Spaces:
Running
on
Zero
Running
on
Zero
File size: 4,738 Bytes
ef1afaf 371669a cd26609 ef1afaf a703203 371669a ef1afaf 076c1f2 ef1afaf 076c1f2 371669a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 |
---
title: ZeroGPU-LLM-Inference
emoji: 🧠
colorFrom: pink
colorTo: purple
sdk: gradio
sdk_version: 5.29.0
app_file: app.py
pinned: false
license: apache-2.0
short_description: Streaming LLM chat with web search and debug
---
This Gradio app provides **token-streaming, chat-style inference** on a wide variety of Transformer models—leveraging ZeroGPU for free GPU acceleration on HF Spaces.
Key features:
- **Real-time DuckDuckGo web search** (background thread, configurable timeout) with results injected into the system prompt.
- **Prompt preview panel** for debugging and prompt-engineering insights—see exactly what’s sent to the model.
- **Thought vs. Answer streaming**: any `<think>…</think>` blocks emitted by the model are shown as separate “💭 Thought.”
- **Cancel button** to immediately stop generation.
- **Dynamic system prompt**: automatically inserts today’s date when you toggle web search.
- **Extensive model selection**: over 30 LLMs (from Phi-4 mini to Qwen3-14B, SmolLM2, Taiwan-ELM, Mistral, Meta-Llama, MiMo, Gemma, DeepSeek-R1, etc.).
- **Memory-safe design**: loads one model at a time, clears cache after each generation.
- **Customizable generation parameters**: max tokens, temperature, top-k, top-p, repetition penalty.
- **Web-search settings**: max results, max chars per result, search timeout.
- **Requirements pinned** to ensure reproducible deployment.
## 🔄 Supported Models
Use the dropdown to select any of these:
| Name | Repo ID |
| ------------------------------------- | -------------------------------------------------- |
| Taiwan-ELM-1_1B-Instruct | liswei/Taiwan-ELM-1_1B-Instruct |
| Taiwan-ELM-270M-Instruct | liswei/Taiwan-ELM-270M-Instruct |
| Qwen3-0.6B | Qwen/Qwen3-0.6B |
| Qwen3-1.7B | Qwen/Qwen3-1.7B |
| Qwen3-4B | Qwen/Qwen3-4B |
| Qwen3-8B | Qwen/Qwen3-8B |
| Qwen3-14B | Qwen/Qwen3-14B |
| Gemma-3-4B-IT | unsloth/gemma-3-4b-it |
| SmolLM2-135M-Instruct-TaiwanChat | Luigi/SmolLM2-135M-Instruct-TaiwanChat |
| SmolLM2-135M-Instruct | HuggingFaceTB/SmolLM2-135M-Instruct |
| SmolLM2-360M-Instruct-TaiwanChat | Luigi/SmolLM2-360M-Instruct-TaiwanChat |
| Llama-3.2-Taiwan-3B-Instruct | lianghsun/Llama-3.2-Taiwan-3B-Instruct |
| MiniCPM3-4B | openbmb/MiniCPM3-4B |
| Qwen2.5-3B-Instruct | Qwen/Qwen2.5-3B-Instruct |
| Qwen2.5-7B-Instruct | Qwen/Qwen2.5-7B-Instruct |
| Phi-4-mini-Reasoning | microsoft/Phi-4-mini-reasoning |
| Phi-4-mini-Instruct | microsoft/Phi-4-mini-instruct |
| Meta-Llama-3.1-8B-Instruct | MaziyarPanahi/Meta-Llama-3.1-8B-Instruct |
| DeepSeek-R1-Distill-Llama-8B | unsloth/DeepSeek-R1-Distill-Llama-8B |
| Mistral-7B-Instruct-v0.3 | MaziyarPanahi/Mistral-7B-Instruct-v0.3 |
| Qwen2.5-Coder-7B-Instruct | Qwen/Qwen2.5-Coder-7B-Instruct |
| Qwen2.5-Omni-3B | Qwen/Qwen2.5-Omni-3B |
| MiMo-7B-RL | XiaomiMiMo/MiMo-7B-RL |
*(…and more can easily be added in `MODELS` in `app.py`.)*
## ⚙️ Generation & Search Parameters
- **Max Tokens**: 64–16384
- **Temperature**: 0.1–2.0
- **Top-K**: 1–100
- **Top-P**: 0.1–1.0
- **Repetition Penalty**: 1.0–2.0
- **Enable Web Search**: on/off
- **Max Results**: integer
- **Max Chars/Result**: integer
- **Search Timeout (s)**: 0.0–30.0
## 🚀 How It Works
1. **User message** enters chat history.
2. If search is enabled, a background DuckDuckGo thread fetches snippets.
3. After up to *Search Timeout* seconds, snippets merge into the system prompt.
4. The selected model pipeline is loaded (bf16→f16→f32 fallback) on ZeroGPU.
5. Prompt is formatted—any `<think>…</think>` blocks will be streamed as separate “💭 Thought.”
6. Tokens stream to the Chatbot UI. Press **Cancel** to stop mid-generation. |