Instructions to use mlx-community/gemma-4-12B-it-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlx-community/gemma-4-12B-it-bf16 with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("mlx-community/gemma-4-12B-it-bf16")
config = load_config("mlx-community/gemma-4-12B-it-bf16")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use mlx-community/gemma-4-12B-it-bf16 with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/gemma-4-12B-it-bf16"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mlx-community/gemma-4-12B-it-bf16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent

How to use mlx-community/gemma-4-12B-it-bf16 with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/gemma-4-12B-it-bf16"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mlx-community/gemma-4-12B-it-bf16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mlx-community/gemma-4-12B-it-bf16 with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/gemma-4-12B-it-bf16"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mlx-community/gemma-4-12B-it-bf16" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Gemma 4 12B MLX Model Fails to Load: gemma4_unified Not Supported

by rishi454 - opened Jun 4

Discussion

rishi454

Jun 4

The model downloads successfully (~12.7 GB), but fails during model initialization with:
ModuleNotFoundError: No module named 'mlx_lm.models.gemma4_unified'

ValueError: Model type gemma4_unified not supported.

Command used :
python3 -m mlx_lm server
--model "mlx-community/gemma-4-12B-it-8bit"

The same error occurs with both mlx-community/gemma-4-12B-it-bf16 and mlx-community/gemma-4-12B-it-8bit

rishi454 changed discussion status to closed Jun 8

jokernifty

MLX Community org Jun 8

https://huggingface.co/jokernifty/gemma-4-12B-it-mlx-4bit-multimodal

usermma

Jun 8

just upgrade your version...

madyarbr

Jun 24

@usermma what version, exactly? pip3 install mlx-lm? thanks.

usermma

Jun 28

@usermma what version, exactly? pip3 install mlx-lm? thanks.

did it work?
pip3 install mlx-lm -U

madyarbr

Jun 28

$ pip3 list | grep mlx
mlx                0.31.2
mlx-audio          0.4.4
mlx-lm             0.31.3
mlx-metal          0.31.2
mlx-optiq          0.2.7
mlx-vlm            0.6.3

$ pip3 --version
pip 26.1.2 from /Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/pip (python 3.13)

unfortunately it does not work... is it working for you, and if so, could you detail how you have installed it?
thanks!

usermma

Jun 29

use this:
to upgrade all of them:

Very-FIRST Upgrade pip:
pip3 install -U pip

and afterwards do a little-execution of this:
pip3 install -U mlx mlx-lm mlx-metal mlx-optiq mlx-vlm mlx-audio

madyarbr

Jun 29

thanks! I did, and pip said there was nothing to do, and the 2nd one updated optiq (0.2.7 -> 0.2.9); so now I have:

mlx                0.31.2
mlx-audio          0.4.4
mlx-lm             0.31.3
mlx-metal          0.31.2
mlx-optiq          0.2.9
mlx-vlm            0.6.3

do these versions match yours?

usermma

Jun 29

i hope you stop trying to run gemma4-12B , its slow and not very great at coding...
thats only in case if you managed to run it...

Actually No , i don't have a any of apple products currently..

consider try a different model or cheap Nvidia GPUs or DGX small computers...

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment