How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf sunil-pathak/gemma-4-E2B-it-Q5_K_M:Q5_K_M
# Run inference directly in the terminal:
llama cli -hf sunil-pathak/gemma-4-E2B-it-Q5_K_M:Q5_K_M
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf sunil-pathak/gemma-4-E2B-it-Q5_K_M:Q5_K_M
# Run inference directly in the terminal:
llama cli -hf sunil-pathak/gemma-4-E2B-it-Q5_K_M:Q5_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf sunil-pathak/gemma-4-E2B-it-Q5_K_M:Q5_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf sunil-pathak/gemma-4-E2B-it-Q5_K_M:Q5_K_M
Use Docker
docker model run hf.co/sunil-pathak/gemma-4-E2B-it-Q5_K_M:Q5_K_M
Quick Links

gemma-4-E2B-it โ€” GGUF (Q5_K_M)


๐Ÿ“Š Performance Metrics

  • Size: 3.38 GB
  • Speed: 5.51 tokens/sec
  • Format: GGUF (llama.cpp optimized)
  • Quantization: Q5_K_M

๐Ÿ”ท Model Overview

This repository contains a GGUF quantized version of:

  • Base Model: gemma-4-E2B-it
  • Format: GGUF (optimized for llama.cpp inference)
  • Precision: Q5_K_M
  • Purpose: Efficient local inference on CPU/GPU

GGUF format provides:

  • Fast loading via memory mapping
  • Single-file model distribution
  • Cross-platform compatibility
  • Efficient inference with llama.cpp

๐Ÿ“ฆ Files

File Description
gemma-4-E2B-it-Q5_K_M.gguf Quantized GGUF model file

โš™๏ธ Technical Details

Parameter Value
Architecture gemma-4-E2B-it
Format GGUF
Precision Q5_K_M
Runtime llama.cpp
Use Case Local inference / deployment

โšก Why GGUF?

GGUF is designed for efficient inference:

  • Optimized for llama.cpp
  • Supports CPU and GPU inference
  • Single-file deployment
  • Memory-mapped loading for speed
  • Ideal for edge / local environments

โš ๏ธ License & Usage

This is a converted derivative model.

  • You must comply with the original model license of gemma-4-E2B-it
  • This is not an official release
  • No additional rights are granted
  • Original ownership remains with the base model creator

๐Ÿš€ Quick Start (llama.cpp)

./llama-cli -m gemma-4-E2B-it-Q5_K_M.gguf -p "Explain AI simply"
Downloads last month
7
GGUF
Model size
5B params
Architecture
gemma4
Hardware compatibility
Log In to add your hardware

5-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support