README.md

metadata

title: Scriptura
emoji: 🏆
colorFrom: yellow
colorTo: blue
sdk: gradio
sdk_version: 5.32.1
app_file: app.py
pinned: false
license: mit
tag: agent-demo-track

Scriptura

Introduction

Scriptura is a multi-agent AI system designed to assist authors in creating screenplays, storyboards, and soundtracks. Its main goal is to automate and accelerate the stages of analysis, summarization, and enrichment of narrative text, allowing screenwriters to focus on the creative aspects.

The core stack includes:

• DeepSeek (deepseek-ai/DeepSeek-R1) as the base model for all text operations (analysis, summarization, generation) via APIs managed by Nebius AI.
• FLUX (black-forest-labs/FLUX.1-dev) for image generation (storyboards, concept art) integrated into the narrative flow.
• MusicGen (facebook/musicgen-melody) to create short audio tracks or sound effects, useful for prototyping or presenting.
• Optional web search (integrated with DuckDuckGo API) to fetch external resources (original scripts, sound effects, reference materials).

Scriptura supports inputs in various formats:

• Text: TXT, PDF, DOCX (automatically converted to structured plain text)
• Images: JPEG, PNG (for analyzing existing storyboards or screenshots)
• Audio: MP3, WAV (for transcribing dialogue or analyzing uploaded soundtracks)

There are size and duration checks on uploaded files to prevent excessively large inputs.

---

Agent Capabilities

Input File Parsing
: - Formats accepted: TXT, PDF, DOCX, JPEG/PNG, MP3/WAV

• Process: PDF/DOCX → plain text; OCR on images; speech-to-text on audio.
• Why it matters: Provides structured input for all downstream modules.

Overall Plot Summary
: - Model: DeepSeek-R1

• Output: 4–6 sentence summary of main narrative threads (timeframe, tone).
• Mechanics: API calls to DeepSeek with retry logic for improved coherence.

Entity & Theme Extraction
: - Technique: Named Entity Recognition (via DeepSeek)

• Extracts: Characters, locations, key events, recurring themes, narrative tone.
• Output: JSON/CSV + ~5-sentence abstract.

Rights & Licensing Verification
: - Web Search ON: Queries DuckDuckGo API → fetch license info if match.

• Web Search OFF: May recognize very famous works internally (e.g. “Harry Potter”) but not guaranteed.
• If no match & search OFF: No licensing check.

Image Generation (Storyboard & Concept Art)
: - Model: FLUX (black-forest-labs/FLUX.1-dev)

• Trigger: “Generate Image” / storyboard phase.
• Process: DeepSeek crafts cinematic prompt → FLUX returns PNG/JPEG + caption.

Audio Generation (Music & Sound Effects)
: - Model: MusicGen (facebook/musicgen-melody)

• Trigger: “Generate Audio.”
• Process: Send prompt → receive MP3/WAV (standalone audio, no text/images).

In-Depth Analysis of Key Points
: - Extracts:
- Characters (role, gender, description)
- Locations (interior/exterior, period, geography)
- Plot Points (crucial narrative beats via Story Understanding models)

• Extras: Semantic toponym extraction → internal scene maps; detect transitions (“Suddenly,” “Meanwhile”).

Optional Web Search
: - Checkbox toggles DuckDuckGo API lookups.

• If Enabled: search preconfigured sites (free & paid) for scripts, sound effects.
• Output: List of links + short summaries.

---

Agent Flow

flowchart LR
    A[Start Agent] --> B[Load Input (text, image, audio)]
    B --> C[Preprocessing: PDF/DOCX → text, OCR, audio transcription]
    C --> D[Generate Plot Summary (DeepSeek)]
    D --> E[Extract Entities & Themes (DeepSeek)]
    E --> F {Web Search Enabled?}
    F -->|Yes| G[Web Search via DuckDuckGo API]
    F -->|No| H[Continue Offline Analysis]
    H --> I[Rights & Licensing Check]
    I --> J[Deep Analysis: characters, locations, plot points]
    J --> K {Image Generation Requested?}
    K -->|Yes| L[API Call to FLUX for storyboard/concept art]
    K -->|No| M[Skip Image Generation]
    M --> N {Audio Generation Requested?}
    N -->|Yes| O[API Call to MusicGen for audio tracks]
    N -->|No| P[Skip Audio Generation]
    L & O --> Q[Final Output: text, JSON/CSV, images, audio]

---

Deployment & Access and the Code Overview

---

Use Cases

Independent Writer
: - Upload a screenplay and quickly get a summary, a list of characters, and locations.

• Create visual storyboards of key narrative moments via FLUX (PNG/JPEG outputs).
• Generate brief soundtracks or sound effects to accompany script presentations (MP3/WAV).

Film Production Company
: - Import multiple screenplays (PDF, DOCX) and automatically receive reports on characters, locations, and potential copyright issues.

• Use the web search feature to find reference scripts or specific sound effects from free/paid sources.
• Develop visual storyboards and audio prototypes to share with directors, artists, and investors.

Translation and Adaptation Agency
: - Upload foreign-language scripts and obtain a structured text version with extracted entities (JSON/CSV).

• Generate contextual images for cultural adaptation (e.g., images matching the original setting via FLUX).
• Produce reference audio via MusicGen to test culturally appropriate music for the target audience.

Digital Humanities Course
: - Demonstrate how to build a text-mining tool applied to performing arts, combining NLP, image, and audio pipelines.

• Allow students to analyze real scripts, generate abstracts, scene maps, and visual/audio prototypes in a hands-on environment.
• Explore Transformer models (DeepSeek), OCR, speech-to-text, and AI-driven media generation as part of the curriculum.

---

Credits

---

Acknowledgements

---

Contributors:

• Code Implementation made by luke9705 and DDPM;
• Ideas creation and testing conducted by OrianIce and Loren1214.

---

Sources

• Russell, S., & Norvig, P. (2021). Artificial Intelligence: A Modern Approach (3rd ed.). Pearson.
• Cambria, E., & White, B. (2014). Jumping NLP Curves: A Review of Natural Language Processing Research. IEEE Computational Intelligence Magazine, 9(2), 48–57.
• Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., … & Sutskever, I. (2022). Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125.