Instructions to use mayankbhaskarr/qwen3-4b-coder-ft-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use mayankbhaskarr/qwen3-4b-coder-ft-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0
# Run inference directly in the terminal:
llama cli -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0
# Run inference directly in the terminal:
llama cli -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0
# Run inference directly in the terminal:
./build/bin/llama cli -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Use Docker

docker model run hf.co/mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Llama new
LM Studio
Jan

vLLM

How to use mayankbhaskarr/qwen3-4b-coder-ft-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mayankbhaskarr/qwen3-4b-coder-ft-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mayankbhaskarr/qwen3-4b-coder-ft-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Ollama
How to use mayankbhaskarr/qwen3-4b-coder-ft-gguf with Ollama:
```
ollama run hf.co/mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0
```
Unsloth Desktop

How to use mayankbhaskarr/qwen3-4b-coder-ft-gguf with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use mayankbhaskarr/qwen3-4b-coder-ft-gguf with Docker Model Runner:
```
docker model run hf.co/mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0
```

Lemonade

How to use mayankbhaskarr/qwen3-4b-coder-ft-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Run and chat with the model

lemonade run user.qwen3-4b-coder-ft-gguf-Q8_0

List all available models

lemonade list

Hermes Agent

How to use mayankbhaskarr/qwen3-4b-coder-ft-gguf with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mayankbhaskarr/qwen3-4b-coder-ft-gguf with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mayankbhaskarr/qwen3-4b-coder-ft-gguf:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

qwen3-4b-coder-ft — GGUF (Q8_0)

Q8_0 GGUF quantization of mayankbhaskarr/qwen3-4b-coder-ft, a QLoRA fine-tune of Qwen/Qwen3-4B-Instruct-2507 (trained with Unsloth + Hugging Face TRL) targeting coding and code-instruction tasks. The repo ships the full 38-item live runtime benchmark of this model: executed Python coding tasks with held-out tests, instruction-following rubric checks, and standardized llama-bench throughput profiling.

Model lineage

Stage	Repo	Format
Base	Qwen/Qwen3-4B-Instruct-2507	BF16
Fine-tune	mayankbhaskarr/qwen3-4b-coder-ft	BF16 safetensors (~8.0 GB)
This repo	mayankbhaskarr/qwen3-4b-coder-ft-gguf	Q8_0 GGUF (4.28 GB)

Q8_0 is a documented near-lossless format: k-quants reference measurements report only +0.0004% relative perplexity vs BF16 (llama.cpp PR #1684), so this file is a faithful representation of the fine-tuned weights at half the disk footprint.

Provided formats

Format	Size	BPW	Relative ppl (ref.)	Availability
BF16 safetensors	8.04 GB	16.00	reference	parent repo
Q8_0	4.28 GB	8.50	+0.0004%	this repo (`qwen3-4b-coder-ft-Q8_0.gguf`)
Q5_K_M	2.89 GB	5.70	+0.0142%	evaluation derivative (reproduce with one command below)
Q4_K_M	2.50 GB	4.95	+0.0535%	evaluation derivative (reproduce with one command below)

Integrity: the uploaded Q8_0 file was hash-verified across independent downloads — SHA-256 ffeb04e87ab636150002c6a22a67a5d01c19d462117bbb98445316ea3a2f9410.

Benchmark evaluation (October 22, 2026)

Method

Suite: 38 items — 12 easy + 12 medium + 6 hard executable Python coding tasks (generated code is run against held-out unit tests), plus 8 instruction-following items scored with objective rubric checks (exact-format compliance, JSON schema conformance, executed refactors, MCQ, IPv4 regex).
Engine: llama.cpp v0.7.0 (CPU build), llama-server with --jinja (repo chat template), greedy decoding (--temp 0.0), context 1280, 2 threads.
Hardware: CPU-only host, 2× Intel Xeon vCPU, 4 GB RAM (~3.7 GB usable), no GPU.
Why Q5_K_M / Q4_K_M? The BF16 (~8.0 GB) and Q8_0 (4.28 GB) artifacts exceed the host's usable RAM, so the suite ran on Q5_K_M and Q4_K_M derivatives quantized from the uploaded Q8_0 with llama-quantize --allow-requantize (dequant-f32 path — a documented near-lossless composite). Since Q8_0 itself is near-lossless vs BF16, these results represent the fine-tuned weights faithfully, with a slight conservative bias.

Task accuracy (38/38 items executed per variant)

Tier	Q5_K_M	Q4_K_M
Easy coding (12)	12/12 (100%)	12/12 (100%)
Medium coding (12)	12/12 (100%)	12/12 (100%)
Hard coding (6)	3/6 (50.0%)	2/6 (33.3%)
Instruction following (8)	6/8 (75.0%)	7/8 (87.5%)
Overall (38)	33/38 (86.8%)	33/38 (86.8%)

Throughput (`llama-bench`, pp512 / tg128, 3 reps)

Config	Q5_K_M	Q4_K_M
Prompt eval, t=1	28.88 tok/s	28.30 tok/s
Prompt eval, t=2	40.54 tok/s	41.93 tok/s
Generation, t=1	2.58 tok/s	3.32 tok/s
Generation, t=2	3.33 tok/s	4.58 tok/s

Server cold-load to first healthy response: 8.55 s (Q5_K_M) / 6.3 s (Q4_K_M); peak resident memory ~3.8 GB including the 1,280-token KV cache.

Quantization agreement

The two quantizations produced identical pass/fail outcomes on 36 of 38 items (94.7%). The two disagreements flip in both directions — H04 (JSON flattener): Q5 pass / Q4 fail; I07 (fib refactor): Q5 fail (empty/EOS single-token response under greedy decoding) / Q4 pass — i.e., greedy-decoding tie-breaking noise near the pass threshold, not a systematic quality gap. Combined with the accuracy table above, Q4_K_M is the recommended deployment quant for constrained hardware: 37.5% faster generation, 0.39 GB smaller, faster loading, identical overall accuracy on this suite.

Failure analysis

All hard-tier failures (LRU cache, spec-precise tokenizer, sliding-window rate limiter) are boundary-condition assertion errors inside otherwise correct scaffolding — the same three items fail on both variants.
The only instruction-tier failure is I04 (code review, two-part rubric): both variants produce the correct fixed loop, but only partially name the bug class (Q5 0/2, Q4 1/2).

Footprint and feasibility

Usage — llama.cpp

# Serve straight from the Hub (chat template via --jinja)
llama-server -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf -c 1280 -t 4 --jinja

# Or from a local file
llama-server -m qwen3-4b-coder-ft-Q8_0.gguf -c 4096 -t 4 --jinja

# One command to reproduce the evaluation derivatives from this Q8_0
llama-quantize --allow-requantize qwen3-4b-coder-ft-Q8_0.gguf out-Q5_K_M.gguf Q5_K_M
llama-quantize --allow-requantize qwen3-4b-coder-ft-Q8_0.gguf out-Q4_K_M.gguf Q4_K_M

Note: Qwen3-4B-Instruct-2507 is a non-thinking instruct model — no enable_thinking handling is required; keep --temp 0 for deterministic code generation, or --temp 0.7 --top-p 0.8 per the base model's recommended sampling.

Raw data & reproduce

Per-item outputs (prompts, generated text, extracted code, test verdicts, timings): benchmark/results/
Harness: benchmark/harness/bench_suite.py (suite definition) and benchmark/harness/run_bench.py (llama-server runner with per-item checkpointing)

git clone --depth 1 --branch v0.7.0 https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=OFF -DLLAMA_CURL=OFF && cmake --build build -j --target llama-server llama-bench
./build/bin/llama-bench -m <model>.gguf -p 512 -n 128 -r 3 -t 1,2

Citation

@misc{bhaskar2026qwen3coderft,
  title  = {Comparative Benchmark Evaluation of a QLoRA Fine-Tuned Qwen3-4B Coder Model and its GGUF Quantization},
  author = {Bhaskar, Mayank},
  year   = {2026},
  month  = {October},
  url    = {https://huggingface.co/mayankbhaskarr}
}

References

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs (arXiv:2305.14314)
Yang, A., et al. (2025). Qwen3 Technical Report (arXiv:2505.09388)
Kawrakow, I. (2022). K-quantization methods (llama.cpp PR #1684)
Kurt, U. (2026). Which quantization should I use? A unified evaluation of llama.cpp quantization methods (arXiv:2601.14277)

Downloads last month: 278

GGUF

Model size

4B params

Architecture

qwen3

Hardware compatibility

8-bit

Model tree for mayankbhaskarr/qwen3-4b-coder-ft-gguf

Base model

Qwen/Qwen3-4B-Instruct-2507

Quantized

unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit

Finetuned

mayankbhaskarr/qwen3-4b-coder-ft

Quantized

(1)

this model

Papers for mayankbhaskarr/qwen3-4b-coder-ft-gguf

Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct

Paper • 2601.14277 • Published Jan 11

Qwen3 Technical Report

Paper • 2505.09388 • Published May 14, 2025 • 348

QLoRA: Efficient Finetuning of Quantized LLMs

Paper • 2305.14314 • Published May 23, 2023 • 65

qwen3-4b-coder-ft — GGUF (Q8_0)

Model lineage

Provided formats

Benchmark evaluation (October 22, 2026)

Method

Task accuracy (38/38 items executed per variant)

Throughput (llama-bench, pp512 / tg128, 3 reps)

Quantization agreement

Failure analysis

Footprint and feasibility

Usage — llama.cpp

Raw data & reproduce

Citation

References

Model tree for mayankbhaskarr/qwen3-4b-coder-ft-gguf

Papers for mayankbhaskarr/qwen3-4b-coder-ft-gguf

Throughput (`llama-bench`, pp512 / tg128, 3 reps)