GLM-5.2-MXFP4xFP8_BLOCK

Model Overview

• Model Architecture: GlmMoeDsaForCausalLM
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Weight quantization: Mixed (MXFP4 / FP8_BLOCK)
  • Activation quantization: FP8
• Version: 1.0
• Model Developers: RedHatAI

This model is a quantized version of zai-org/GLM-5.2.

Model Optimizations

This model was obtained by applying mixed-precision quantization to zai-org/GLM-5.2, quantizing the MoE expert linears to 4-bit MXFP4 and the remaining quantized linears (attention, dense, and shared-expert layers) to 8-bit block-scaled FP8 with dynamic FP8 activation scaling, ready for inference with vLLM.

This optimization reduces the number of bits per parameter from 16 to an average of ~4.4 bits, shrinking the on-disk model size from ~1.47 TB (BF16) to ~411 GB and reducing disk and GPU memory requirements by approximately 72%.

Only the linear operators within transformer blocks are quantized using LLM Compressor.

Note: this quantized checkpoint is experimentally supported in vLLM.

Deployment

Use with vLLM

vllm serve RedHatAI/GLM-5.2-MXFP4xFP8_BLOCK \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 5}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice

Creation

This model was created by applying LLM Compressor with the mixed-precision scheme described above (RTN, no calibration data), exported in compressed-tensors format.