Hugging Face's logo

Spaces:
Luigi
/
ZeroGPU-LLM-Inference
Running on Zero

App Files Community
ZeroGPU-LLM-Inference / README.md
Luigi's picture
Luigi
Update README.md
371669a verified
preview code
|
raw
history blame contribute delete
4.74 kB
metadata
title: ZeroGPU-LLM-Inference
emoji: 🧠
colorFrom: pink
colorTo: purple
sdk: gradio
sdk_version: 5.29.0
app_file: app.py
pinned: false
license: apache-2.0
short_description: Streaming LLM chat with web search and debug

This Gradio app provides token-streaming, chat-style inference on a wide variety of Transformer models—leveraging ZeroGPU for free GPU acceleration on HF Spaces.

Key features:

  • Real-time DuckDuckGo web search (background thread, configurable timeout) with results injected into the system prompt.
  • Prompt preview panel for debugging and prompt-engineering insights—see exactly what’s sent to the model.
  • Thought vs. Answer streaming: any <think>…</think> blocks emitted by the model are shown as separate “💭 Thought.”
  • Cancel button to immediately stop generation.
  • Dynamic system prompt: automatically inserts today’s date when you toggle web search.
  • Extensive model selection: over 30 LLMs (from Phi-4 mini to Qwen3-14B, SmolLM2, Taiwan-ELM, Mistral, Meta-Llama, MiMo, Gemma, DeepSeek-R1, etc.).
  • Memory-safe design: loads one model at a time, clears cache after each generation.
  • Customizable generation parameters: max tokens, temperature, top-k, top-p, repetition penalty.
  • Web-search settings: max results, max chars per result, search timeout.
  • Requirements pinned to ensure reproducible deployment.

🔄 Supported Models

Use the dropdown to select any of these:

Name Repo ID
Taiwan-ELM-1_1B-Instruct liswei/Taiwan-ELM-1_1B-Instruct
Taiwan-ELM-270M-Instruct liswei/Taiwan-ELM-270M-Instruct
Qwen3-0.6B Qwen/Qwen3-0.6B
Qwen3-1.7B Qwen/Qwen3-1.7B
Qwen3-4B Qwen/Qwen3-4B
Qwen3-8B Qwen/Qwen3-8B
Qwen3-14B Qwen/Qwen3-14B
Gemma-3-4B-IT unsloth/gemma-3-4b-it
SmolLM2-135M-Instruct-TaiwanChat Luigi/SmolLM2-135M-Instruct-TaiwanChat
SmolLM2-135M-Instruct HuggingFaceTB/SmolLM2-135M-Instruct
SmolLM2-360M-Instruct-TaiwanChat Luigi/SmolLM2-360M-Instruct-TaiwanChat
Llama-3.2-Taiwan-3B-Instruct lianghsun/Llama-3.2-Taiwan-3B-Instruct
MiniCPM3-4B openbmb/MiniCPM3-4B
Qwen2.5-3B-Instruct Qwen/Qwen2.5-3B-Instruct
Qwen2.5-7B-Instruct Qwen/Qwen2.5-7B-Instruct
Phi-4-mini-Reasoning microsoft/Phi-4-mini-reasoning
Phi-4-mini-Instruct microsoft/Phi-4-mini-instruct
Meta-Llama-3.1-8B-Instruct MaziyarPanahi/Meta-Llama-3.1-8B-Instruct
DeepSeek-R1-Distill-Llama-8B unsloth/DeepSeek-R1-Distill-Llama-8B
Mistral-7B-Instruct-v0.3 MaziyarPanahi/Mistral-7B-Instruct-v0.3
Qwen2.5-Coder-7B-Instruct Qwen/Qwen2.5-Coder-7B-Instruct
Qwen2.5-Omni-3B Qwen/Qwen2.5-Omni-3B
MiMo-7B-RL XiaomiMiMo/MiMo-7B-RL

(…and more can easily be added in MODELS in app.py.)

⚙️ Generation & Search Parameters

  • Max Tokens: 64–16384

  • Temperature: 0.1–2.0

  • Top-K: 1–100

  • Top-P: 0.1–1.0

  • Repetition Penalty: 1.0–2.0

  • Enable Web Search: on/off

  • Max Results: integer

  • Max Chars/Result: integer

  • Search Timeout (s): 0.0–30.0

🚀 How It Works

  1. User message enters chat history.
  2. If search is enabled, a background DuckDuckGo thread fetches snippets.
  3. After up to Search Timeout seconds, snippets merge into the system prompt.
  4. The selected model pipeline is loaded (bf16→f16→f32 fallback) on ZeroGPU.
  5. Prompt is formatted—any <think>…</think> blocks will be streamed as separate “💭 Thought.”
  6. Tokens stream to the Chatbot UI. Press Cancel to stop mid-generation.