Hugging Face's logo

Spaces:
DroolingPanda
/
dia-tts-server
Running

App Files Community
dia-tts-server / documentation.md
Michael Hu
initial check in of the dia tts server
ac5de5b
preview code
|
raw
history blame contribute delete
41.7 kB

Dia TTS Server - Technical Documentation

Version: 1.0.0 Date: 2025-04-22

Table of Contents:

  1. Overview
  2. Visual Overview
  3. System Prerequisites
  4. Installation and Setup
  5. Configuration
  6. Running the Server
  7. Usage
  8. Troubleshooting
  9. Project Architecture
  10. License and Disclaimer

1. Overview

The Dia TTS Server provides a backend service and web interface for generating high-fidelity speech, including dialogue with multiple speakers and non-verbal sounds, using the Dia text-to-speech model family (originally from Nari Labs, with support for community conversions like SafeTensors).

This server is built using the FastAPI framework and offers both a RESTful API (including an OpenAI-compatible endpoint) and an interactive web UI powered by Jinja2, Tailwind CSS, and JavaScript. It supports voice cloning via audio prompts and allows configuration of various generation parameters.

Key Features:

  • High-Quality TTS: Leverages the Dia model for realistic speech synthesis.
  • Dialogue Generation: Supports [S1] and [S2] tags for multi-speaker dialogue.
  • Non-Verbal Sounds: Can generate sounds like (laughs), (sighs), etc., when included in the text.
  • Voice Cloning: Allows conditioning the output voice on a provided reference audio file.
  • Flexible Model Loading: Supports loading models from Hugging Face repositories, including both .pth and .safetensors formats (defaults to BF16 SafeTensors for efficiency).
  • API Access: Provides a custom API endpoint (/tts) and an OpenAI-compatible endpoint (/v1/audio/speech).
  • Web Interface: Offers an easy-to-use UI for text input, parameter adjustment, preset loading, reference audio management, and audio playback.
  • Configuration: Server settings, model sources, paths, and default generation parameters are configurable via an .env file.
  • GPU Acceleration: Utilizes NVIDIA GPUs via CUDA for significantly faster inference when available, falling back to CPU otherwise.

2. Visual Overview

2.1 Directory Structure 

dia-tts-server/
│
├── .env                  # Local configuration overrides (user-created)
├── config.py             # Default configuration and management class
├── engine.py             # Core model loading and generation logic
├── models.py             # Pydantic models for API requests
├── requirements.txt      # Python dependencies
├── server.py             # Main FastAPI application, API endpoints, UI routes
├── utils.py              # Utility functions (audio encoding, saving, etc.)
│
├── dia/                  # Core Dia model implementation package
│   ├── __init__.py
│   ├── audio.py          # Audio processing helpers (delay, codebook conversion)
│   ├── config.py         # Pydantic models for Dia model architecture config
│   ├── layers.py         # Custom PyTorch layers for the Dia model
│   └── model.py          # Dia model class wrapper (loading, generation)
│
├── static/               # Static assets (e.g., favicon.ico)
│   └── favicon.ico
│
├── ui/                   # Web User Interface files
│   ├── index.html        # Main HTML template (Jinja2)
│   ├── presets.yaml      # Predefined UI examples
│   ├── script.js         # Frontend JavaScript logic
│   └── style.css         # Frontend CSS styling (Tailwind via CDN/build)
│
├── model_cache/          # Default directory for downloaded model files (configurable)
├── outputs/              # Default directory for saved audio output (configurable)
└── reference_audio/      # Default directory for voice cloning reference files (configurable)

2.2 Component Diagram 

┌───────────────────┐      ┌───────────────────┐      ┌───────────────────┐      ┌───────────────────┐
│ User (Web UI /    │────→ │ FastAPI Server    │────→ │ TTS Engine        │────→ │ Dia Model Wrapper │
│ API Client)       │      │ (server.py)       │      │ (engine.py)       │      │ (dia/model.py)    │
└───────────────────┘      └─────────┬─────────┘      └─────────┬─────────┘      └─────────┬─────────┘
                                     │                          │                          │
                                     │ Uses                     │ Uses                     │ Uses
                                     ▼                          ▼                          ▼
                           ┌───────────────────┐      ┌───────────────────┐      ┌───────────────────┐
                           │ Configuration     │ ←─── │ .env File         │      │ Dia Model Layers  │
                           │ (config.py)       │      └───────────────────┘      │ (dia/layers.py)   │
                           └───────────────────┘                                 └───────────────────┘
                                     │                                                   │ Uses
                                     │ Uses                                                   │
                                     ▼                                                   │
                           ┌───────────────────┐                                         │ Uses
                           │ Utilities         │                                         ▼
                           │ (utils.py)        │                               ┌───────────────────┐
                           └───────────────────┘                               │ PyTorch / CUDA    │
                                     ▲                                         └───────────────────┘
                                     │ Uses                                             │ Uses
                                     │                                                  ▼
┌───────────────────┐      ┌───────────────────┐                           ┌───────────────────┐
│ Web UI Files      │ ←─── │ Jinja2 Templates  │                           │ DAC Model         │
│ (ui/)             │      └───────────────────┘                           │ (descript-audio..)│
└───────────────────┘               ▲                                      └───────────────────┘
                                    │ Renders                                        ▲
                                    │                                                │ Uses
                                    └────────────────────────────────────────────────┘

Diagram Legend:

  • Boxes represent major components or file groups.
  • Arrows (→) indicate primary data flow or control flow.
  • Lines with "Uses" indicate dependencies or function calls.

3. System Prerequisites

Before installing and running the Dia TTS Server, ensure your system meets the following requirements:

  • Operating System:
    • Windows 10/11 (64-bit)
    • Linux (Debian/Ubuntu recommended, other distributions may require adjustments)
  • Python: Python 3.10 or later (Python 3.10.x recommended based on tracebacks). Ensure Python and Pip are added to your system's PATH.
  • Version Control: Git (for cloning the repository).
  • Internet Connection: Required for downloading dependencies and model files.
  • (Optional but Highly Recommended for Performance):
    • NVIDIA GPU: A CUDA-compatible NVIDIA GPU (Maxwell architecture or newer). Check compatibility here. Sufficient VRAM is needed (BF16 model requires ~5-6GB, full precision ~10GB).
    • NVIDIA Drivers: Latest appropriate drivers for your GPU and OS.
    • CUDA Toolkit: Version compatible with the chosen PyTorch build (e.g., 11.8, 12.1). See Section 4.4.
  • (Linux System Libraries):
    • libsndfile1: Required by the soundfile Python library for audio I/O. Install using your package manager (e.g., sudo apt install libsndfile1 on Debian/Ubuntu).

4. Installation and Setup

Follow these steps to set up the project environment and install necessary dependencies.

4.1 Cloning the Repository

Open your terminal or command prompt and navigate to the directory where you want to store the project. Then, clone the repository:

git clone https://github.com/devnen/dia-tts-server.git # Replace with the actual repo URL if different
cd dia-tts-server

4.2 Setting up Python Virtual Environment

Using a virtual environment is strongly recommended to isolate project dependencies.

4.2.1 Windows Setup

  1. Open PowerShell or Command Prompt in the project directory (dia-tts-server).
  2. Create the virtual environment:
    python -m venv venv
  3. Activate the virtual environment:
    .\venv\Scripts\activate
    Your terminal prompt should now be prefixed with (venv).

4.2.2 Linux Setup (Debian/Ubuntu Example)

  1. Install prerequisites (if not already present):
    sudo apt update
sudo apt install python3 python3-venv python3-pip libsndfile1 -y
  2. Open your terminal in the project directory (dia-tts-server).
  3. Create the virtual environment:
    python3 -m venv venv
  4. Activate the virtual environment:
    source venv/bin/activate
    Your terminal prompt should now be prefixed with (venv).

4.3 Installing Dependencies

With your virtual environment activated ((venv) prefix visible), install the required Python packages:

# Upgrade pip first (optional but good practice)
pip install --upgrade pip

# Install all dependencies from requirements.txt
pip install -r requirements.txt

Note: This command installs the CPU-only version of PyTorch by default. If you have a compatible NVIDIA GPU and want acceleration, proceed to Section 4.4 before running the server.

4.4 NVIDIA Driver and CUDA Setup (Required for GPU Acceleration)

Follow these steps only if you have a compatible NVIDIA GPU and want faster inference.

4.4.1 Step 1: Check/Install NVIDIA Drivers

  1. Check Existing Driver: Open Command Prompt (Windows) or Terminal (Linux) and run:
    nvidia-smi
  2. Interpret Output:
    • If the command runs successfully, note the Driver Version and the CUDA Version listed in the top right corner. This CUDA version is the maximum supported by your current driver.
    • If the command fails ("not recognized"), you need to install or update your NVIDIA drivers.
  3. Install/Update Drivers: Go to the NVIDIA Driver Downloads page. Select your GPU model and OS, then download and install the latest recommended driver (Game Ready or Studio). Reboot your computer after installation. Run nvidia-smi again to confirm it works.

4.4.2 Step 2: Install PyTorch with CUDA Support

  1. Go to PyTorch Website: Visit https://pytorch.org/get-started/locally/.
  2. Configure: Select:
    • PyTorch Build: Stable
    • Your OS: Windows or Linux
    • Package: Pip
    • Language: Python
    • Compute Platform: Choose the CUDA version equal to or lower than the version reported by nvidia-smi. For example, if nvidia-smi shows CUDA Version: 12.4, select CUDA 12.1. If it shows 11.8, select CUDA 11.8. Do not select a version higher than your driver supports. (CUDA 12.1 or 11.8 are common stable choices).
  3. Copy Command: Copy the generated installation command. It will look similar to:
    # Example for CUDA 12.1 (Windows/Linux):
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
# Example for CUDA 11.8 (Windows/Linux):
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
    (Use pip instead of pip3 if that's your command)
  4. Install in Activated venv:
    • Ensure your (venv) is active.
    • Uninstall CPU PyTorch first:
      pip uninstall torch torchvision torchaudio -y
    • Paste and run the copied command from the PyTorch website.

4.4.3 Step 3: Verify PyTorch CUDA Installation

  1. With the (venv) still active, start a Python interpreter:
    python
  2. Run the following Python code:
    import torch
print(f"PyTorch version: {torch.__version__}")
cuda_available = torch.cuda.is_available()
print(f"CUDA available: {cuda_available}")
if cuda_available:
    print(f"CUDA version used by PyTorch: {torch.version.cuda}")
    print(f"Device count: {torch.cuda.device_count()}")
    print(f"Current device index: {torch.cuda.current_device()}")
    print(f"Device name: {torch.cuda.get_device_name(torch.cuda.current_device())}")
else:
    print("CUDA not available to PyTorch. Ensure drivers and CUDA-enabled PyTorch are installed correctly.")
exit()
  3. If CUDA available: shows True, the setup was successful. If False, review driver installation and the PyTorch installation command.

5. Configuration

The server's behavior, including model selection, paths, and default generation parameters, is controlled via configuration settings.

5.1 Configuration Files (.env and config.py)

  • config.py: Defines the default values for all configuration parameters in the DEFAULT_CONFIG dictionary. It also contains the ConfigManager class and getter functions used by the application.
  • .env File: This file, located in the project root directory (dia-tts-server/.env), allows you to override the default values. Create this file if it doesn't exist. Settings are defined as KEY=VALUE pairs, one per line. The server reads this file on startup using python-dotenv.

Priority: Values set in the .env file take precedence over the defaults in config.py. Environment variables set directly in your system also override .env file values (though using .env is generally recommended for project-specific settings).

5.2 Configuration Parameters

The following parameters can be set in your .env file:

Parameter Name (in .env) Default Value (config.py) Description Example .env Value
Server Settings
HOST 0.0.0.0 The network interface address the server listens on. 0.0.0.0 makes it accessible on your local network. 127.0.0.1 (localhost only)
PORT 8003 The port number the server listens on. 8080
Model Source Settings
DIA_MODEL_REPO_ID ttj/dia-1.6b-safetensors The Hugging Face repository ID containing the model files. nari-labs/Dia-1.6B
DIA_MODEL_CONFIG_FILENAME config.json The filename of the model's configuration JSON within the repository. config.json
DIA_MODEL_WEIGHTS_FILENAME dia-v0_1_bf16.safetensors The filename of the model weights file (.safetensors or .pth) within the repository to load. dia-v0_1.safetensors or dia-v0_1.pth
Path Settings
DIA_MODEL_CACHE_PATH ./model_cache Local directory to store downloaded model files. Relative paths are based on the project root. /path/to/shared/cache
REFERENCE_AUDIO_PATH ./reference_audio Local directory to store reference audio files (.wav, .mp3) used for voice cloning. ./voices
OUTPUT_PATH ./outputs Local directory where generated audio files from the Web UI are saved. ./generated_speech
Default Generation Parameters (These set the initial UI values and can be saved via the UI)
GEN_DEFAULT_SPEED_FACTOR 0.90 Default playback speed factor applied after generation (UI slider initial value). 1.0
GEN_DEFAULT_CFG_SCALE 3.0 Default Classifier-Free Guidance scale (UI slider initial value). 2.5
GEN_DEFAULT_TEMPERATURE 1.3 Default sampling temperature (UI slider initial value). 1.2
GEN_DEFAULT_TOP_P 0.95 Default nucleus sampling probability (UI slider initial value). 0.9
GEN_DEFAULT_CFG_FILTER_TOP_K 35 Default Top-K value for CFG filtering (UI slider initial value). 40

Example .env File (Using Original Nari Labs Model):

# .env
# Example configuration to use the original Nari Labs model

HOST=0.0.0.0
PORT=8003

DIA_MODEL_REPO_ID=nari-labs/Dia-1.6B
DIA_MODEL_CONFIG_FILENAME=config.json
DIA_MODEL_WEIGHTS_FILENAME=dia-v0_1.pth

# Keep other paths as default or specify custom ones
# DIA_MODEL_CACHE_PATH=./model_cache
# REFERENCE_AUDIO_PATH=./reference_audio
# OUTPUT_PATH=./outputs

# Keep default generation parameters or override them
# GEN_DEFAULT_SPEED_FACTOR=0.90
# GEN_DEFAULT_CFG_SCALE=3.0
# GEN_DEFAULT_TEMPERATURE=1.3
# GEN_DEFAULT_TOP_P=0.95
# GEN_DEFAULT_CFG_FILTER_TOP_K=35

Important: You must restart the server after making changes to the .env file for them to take effect.

6. Running the Server

  1. Activate Virtual Environment: Ensure your virtual environment is activated ((venv) prefix).

    • Windows: .\venv\Scripts\activate
    • Linux: source venv/bin/activate

  2. Navigate to Project Root: Make sure your terminal is in the dia-tts-server directory.

  3. Run the Server:

    python server.py

  4. Server Output: You should see log messages indicating the server is starting, including:

    • The configuration being used (repo ID, filenames, paths).
    • The device being used (CPU or CUDA).
    • Model loading progress (downloading if necessary).
    • Confirmation that the server is running (e.g., Uvicorn running on http://0.0.0.0:8003).
    • URLs for accessing the Web UI and API Docs.

  5. Accessing the Server:

    • Web UI: Open your web browser and go to http://localhost:PORT (e.g., http://localhost:8003 if using the default port). If running on a different machine or VM, replace localhost with the server's IP address.
    • API Docs: Access the interactive API documentation (Swagger UI) at http://localhost:PORT/docs.

  6. Stopping the Server: Press CTRL+C in the terminal where the server is running.

Auto-Reload: The server is configured to run with reload=True. This means Uvicorn will automatically restart the server if it detects changes in .py, .html, .css, .js, .env, or .yaml files within the project or ui directory. This is useful for development but should generally be disabled in production.

7. Usage

The Dia TTS Server can be used via its Web UI or its API endpoints.

7.1 Web User Interface (Web UI)

Access the UI by navigating to the server's base URL (e.g., http://localhost:8003).

7.1.1 Main Generation Form

  • Text to speak: Enter the text you want to synthesize.
    • Use [S1] and [S2] tags to indicate speaker turns for dialogue.
    • Include non-verbal cues like (laughs), (sighs), (clears throat) directly in the text where desired.
    • For voice cloning, prepend the exact transcript of the selected reference audio before the text you want generated (e.g., [S1] Reference transcript text. [S1] This is the new text to generate in the cloned voice.).
  • Voice Mode: Select the desired generation mode:
    • Single / Dialogue (Use [S1]/[S2]): Use this for single-speaker text (you can use [S1] or omit tags if the model handles it) or multi-speaker dialogue (using [S1] and [S2]).
    • Voice Clone (from Reference): Enables voice cloning based on a selected audio file. Requires selecting a file below and prepending its transcript to the text input.
  • Generate Speech Button: Submits the text and settings to the server to start generation.

7.1.2 Presets

  • Located below the Voice Mode selection.
  • Clicking a preset button (e.g., "Standard Dialogue", "Expressive Narration") will automatically populate the "Text to speak" area and the "Generation Parameters" sliders with predefined values, demonstrating different use cases.

7.1.3 Voice Cloning

  • This section appears only when "Voice Clone" mode is selected.
  • Reference Audio File Dropdown: Lists available .wav and .mp3 files found in the configured REFERENCE_AUDIO_PATH. Select the file whose voice you want to clone. Remember to prepend its transcript to the main text input.
  • Load Button: Click this to open your system's file browser. You can select one or more .wav or .mp3 files to upload. The selected files will be copied to the server's REFERENCE_AUDIO_PATH, and the dropdown list will refresh automatically. The first newly uploaded file will be selected in the dropdown.

7.1.4 Generation Parameters

  • Expand this section to fine-tune the generation process. These values correspond to the parameters used by the underlying Dia model.
  • Sliders: Adjust Speed Factor, CFG Scale, Temperature, Top P, and CFG Filter Top K. The current value is displayed next to the label.
  • Save Generation Defaults Button: Saves the current values of these sliders to the .env file (as GEN_DEFAULT_... keys). These saved values will become the default settings loaded into the UI the next time the server starts.

7.1.5 Server Configuration (UI)

  • Expand this section to view and modify server-level settings stored in the .env file.
  • Fields: Edit Model Repo ID, Config/Weights Filenames, Cache/Reference/Output Paths, Host, and Port.
  • Save Server Configuration Button: Saves the values currently shown in these fields to the .env file. A server restart is required for most of these changes (especially model source or paths) to take effect.
  • Restart Server Button: (Appears after saving) Attempts to trigger a server restart. This works best if the server was started with reload=True or is managed by a process manager like systemd or Supervisor.

7.1.6 Generated Audio Player

  • Appears below the main form after a successful generation.
  • Waveform: Visual representation of the generated audio.
  • Play/Pause Button: Controls audio playback.
  • Download WAV Button: Downloads the generated audio as a .wav file.
  • Info: Displays the voice mode used, generation time, and audio duration.

7.1.7 Theme Toggle

  • Located in the top-right navigation bar.
  • Click the Sun/Moon icon to switch between Light and Dark themes. Your preference is saved in your browser's localStorage.

7.2 API Endpoints

Access the interactive API documentation via the /docs path (e.g., http://localhost:8003/docs).

7.2.1 POST /v1/audio/speech (OpenAI Compatible)

  • Purpose: Provides an endpoint compatible with the basic OpenAI TTS API for easier integration with existing tools.
  • Request Body: (application/json) - Uses the OpenAITTSRequest model.
    Field Type Required Description Example
    model string No Ignored by this server (always uses Dia). Included for compatibility. Defaults to dia-1.6b. "dia-1.6b"
    input string Yes The text to synthesize. Use [S1]/[S2] tags for dialogue. For cloning, prepend reference transcript. "Hello [S1] world."
    voice string No Maps to Dia modes. Use "S1", "S2", "dialogue", or the filename of a reference audio (e.g., "my_ref.wav") for cloning. Defaults to S1. "dialogue" or "ref.mp3"
    response_format "opus" | "wav" No Desired audio output format. Defaults to opus. "wav"
    speed float No Playback speed factor (0.5-2.0). Applied after generation. Defaults to 1.0. 0.9
  • Response:
    • Success (200 OK): StreamingResponse containing the binary audio data (audio/opus or audio/wav).
    • Error: Standard FastAPI JSON error response (e.g., 400, 404, 500).

7.2.2 POST /tts (Custom Parameters)

  • Purpose: Allows generation using all specific Dia generation parameters.
  • Request Body: (application/json) - Uses the CustomTTSRequest model.
    Field Type Required Description Default
    text string Yes The text to synthesize. Use [S1]/[S2] tags. Prepend transcript for cloning.
    voice_mode "dialogue" | "clone" No Generation mode. Note: single_s1/single_s2 are handled via dialogue mode with appropriate tags in the text. dialogue
    clone_reference_filename string | null No Filename of reference audio in REFERENCE_AUDIO_PATH. Required if voice_mode is clone. null
    output_format "opus" | "wav" No Desired audio output format. opus
    max_tokens integer | null No Maximum audio tokens to generate. null uses the model's default. null
    cfg_scale float No Classifier-Free Guidance scale. 3.0
    temperature float No Sampling temperature. 1.3
    top_p float No Nucleus sampling probability. 0.95
    speed_factor float No Playback speed factor (0.5-2.0). Applied after generation. 0.90
    cfg_filter_top_k integer No Top-K value for CFG filtering. 35
  • Response:
    • Success (200 OK): StreamingResponse containing the binary audio data (audio/opus or audio/wav).
    • Error: Standard FastAPI JSON error response (e.g., 400, 404, 500).

7.2.3 Configuration & Helper Endpoints

  • GET /get_config: Returns the current server configuration as JSON.
  • POST /save_config: Saves server configuration settings provided in the JSON request body to the .env file. Requires server restart.
  • POST /save_generation_defaults: Saves default generation parameters provided in the JSON request body to the .env file. Affects UI defaults on next load.
  • POST /restart_server: Attempts to trigger a server restart (reliability depends on execution environment).
  • POST /upload_reference: Uploads one or more audio files (.wav, .mp3) as multipart/form-data to the reference audio directory. Returns JSON with status and updated file list.
  • GET /health: Basic health check endpoint. Returns {"status": "healthy", "model_loaded": true/false}.

8. Troubleshooting

  • Error: CUDA available: False or Slow Performance:
    • Verify NVIDIA drivers are installed correctly (nvidia-smi command).
    • Ensure you installed the correct PyTorch version with CUDA support matching your driver (See Section 4.4). Reinstall PyTorch using the command from the official website if unsure.
    • Check if another process is using all GPU VRAM.
  • Error: ImportError: No module named 'dac' (or safetensors, yaml, etc.):
    • Make sure your virtual environment is activated.
    • Run pip install -r requirements.txt again to install missing dependencies.
    • Specifically for dac, ensure you installed descript-audio-codec and not a different package named dac. Run pip uninstall dac -y && pip install descript-audio-codec.
  • Error: libsndfile library not found (or similar soundfile error, mainly on Linux):
    • Install the system library: sudo apt update && sudo apt install libsndfile1 (Debian/Ubuntu) or the equivalent for your distribution.
  • Error: Model Download Fails (e.g., HTTPError, ConnectionError):
    • Check your internet connection.
    • Verify the DIA_MODEL_REPO_ID, DIA_MODEL_CONFIG_FILENAME, and DIA_MODEL_WEIGHTS_FILENAME in your .env file (or defaults in config.py) are correct and accessible on Hugging Face Hub.
    • Check Hugging Face Hub status if multiple downloads fail.
    • Ensure the cache directory (DIA_MODEL_CACHE_PATH) is writable.
  • Error: RuntimeError: Failed to load DAC model...:
    • This usually indicates an issue with the descript-audio-codec installation or version incompatibility. Ensure it's installed correctly (see ImportError above).
    • Check logs for specific AttributeError messages (like missing utils or download) which might indicate version mismatches between the Dia code's expectation and the installed library. The current code expects dac.utils.download().
  • Error: FileNotFoundError during generation (Reference Audio):
    • Ensure the filename selected/provided for voice cloning exists in the configured REFERENCE_AUDIO_PATH.
    • Check that the path in config.py or .env is correct and the server has permission to read from it.
  • Error: Cannot Save Output/Reference Files (PermissionError, etc.):
    • Ensure the directories specified by OUTPUT_PATH and REFERENCE_AUDIO_PATH exist and the server process has write permissions to them.
  • Web UI Issues (Buttons don't work, styles missing):
    • Clear your browser cache.
    • Check the browser's developer console (usually F12) for JavaScript errors.
    • Ensure ui/script.js and ui/style.css are being loaded correctly (check network tab in developer tools).
  • Generation Cancel Button Doesn't Stop Process:
    • This is expected ("Fake Cancel"). The button currently only prevents the UI from processing the result when it eventually arrives. True cancellation is complex and not implemented. Clicking "Generate" again will cancel the previous UI request's result processing before starting the new one.

9. Project Architecture

  • server.py: The main entry point using FastAPI. Defines API routes, serves the Web UI using Jinja2, handles requests, and orchestrates calls to the engine.
  • engine.py: Responsible for loading the Dia model (including downloading files via huggingface_hub), managing the model instance, preparing inputs for the model's generate method based on user requests (handling voice modes), and calling the model's generation function. Also handles post-processing like speed adjustment.
  • config.py: Manages all configuration settings using default values and overrides from a .env file. Provides getter functions for easy access to settings.
  • dia/ package: Contains the core implementation of the Dia model itself.
    • model.py: Defines the Dia class, which wraps the underlying PyTorch model (DiaModel). It handles loading weights (.pth or .safetensors), loading the required DAC model, preparing inputs specifically for the DiaModel forward pass (including CFG logic), and running the autoregressive generation loop.
    • config.py (within dia/): Defines Pydantic models representing the structure and hyperparameters of the Dia model architecture (encoder, decoder, data parameters). This is loaded from the config.json file associated with the model weights.
    • layers.py: Contains custom PyTorch nn.Module implementations used within the DiaModel (e.g., Attention blocks, MLP blocks, RoPE).
    • audio.py: Includes helper functions for audio processing specific to the model's tokenization and delay patterns (e.g., audio_to_codebook, codebook_to_audio, apply_audio_delay).
  • ui/ directory: Contains all files related to the Web UI.
    • index.html: The main Jinja2 template.
    • script.js: Frontend JavaScript for interactivity, API calls, theme switching, etc.
    • presets.yaml: Definitions for the UI preset examples.
  • utils.py: General utility functions, such as audio encoding (encode_audio) and saving (save_audio_to_file) using the soundfile library.
  • Dependencies: Relies heavily on FastAPI, Uvicorn, PyTorch, torchaudio, huggingface_hub, safetensors, descript-audio-codec, soundfile, PyYAML, python-dotenv, pydantic, and Jinja2.

10. License and Disclaimer

  • License: This project is licensed under the MIT License.

  • Disclaimer: This project offers a high-fidelity speech generation model intended solely for research and educational use. The following uses are strictly forbidden:

    • Identity Misuse: Do not produce audio resembling real individuals without permission.
    • Deceptive Content: Do not use this model to generate misleading content (e.g. fake news)
    • Illegal or Malicious Use: Do not use this model for activities that are illegal or intended to cause harm.

    By using this model, you agree to uphold relevant legal standards and ethical responsibilities. The creators are not responsible for any misuse and firmly oppose any unethical usage of this technology.