ToolSure CONSTRAINED (Qwen2.5 1.5B Instruct)

Small models break function calls: malformed JSON, invented function names, missing or mistyped arguments, and calls when no tool fits. Retrieval cannot fix that, because the failure is in how the model writes the call. ToolSure trains Qwen/Qwen2.5-1.5B-Instruct to call tools reliably. This model is one of six in the ToolSure collection: five training methods on the same data, plus constrained decoding, all on one table.

Method

The same weights as the best trained ToolSure model, served with grammar constrained decoding. Whenever the model opens a tool call, decoding is forced to one of the offered function names and to arguments that match that function's JSON schema, so calls always parse. The model can still answer in plain text when no tool fits.

Training data: AmareshHebbar/ToolSure_Calls_Qwen2.5_1.5B. Same weights as the source model listed below.

Why finetune instead of prompting

The table includes the base model prompted normally, the base model with a strict system prompt, and the base model with grammar constrained decoding. Constrained decoding guarantees valid JSON and real function names, but it cannot choose the right function, fill the right values, or decide when not to call. That gap is what training closes.

Results

All rows use greedy decoding and the same checker. BFCL columns are AST accuracy on the Berkeley Function Calling Leaderboard non live categories, scored with a reimplementation of the BFCL AST checker (function name, required parameters, no unexpected parameters, values against the possible answers; parallel calls matched in any order). Irrelevance is the share of prompts where the model correctly made no call. xLAM held out is exact call accuracy on 500 xLAM examples never used in training. JSON parse rate counts outputs that attempted a call. Latency is batch size 1 on vLLM; cost assumes $1.0/hour GPU at the measured batched throughput.

ModelBFCL SimpleBFCL MultipleBFCL ParallelBFCL Parallel multipleBFCL AST meanIrrelevancexLAM held outJSON parse rateHallucinated function rateMMLULatency p50 (s)Cost per 1k queries (USD)
Qwen2.5 1.5B Instruct (prompted)91.586.580.079.084.264.684.299.20.70.210.0054
Qwen2.5 1.5B Instruct (strict system prompt)91.588.080.076.584.029.283.899.20.90.210.0051
Qwen2.5 1.5B Instruct (constrained decoding)92.087.080.579.084.663.783.299.60.01.210.0546
Qwen2.5-7B-Instruct (prompted, reference)94.593.589.085.590.667.586.899.80.50.730.0155
ToolSure SFT91.890.582.079.585.929.686.299.31.00.200.0044
ToolSure ORPO91.584.584.579.084.960.484.099.20.50.210.0045
ToolSure GRPO92.285.082.082.585.443.884.099.30.80.210.0044
ToolSure CONSTRAINED91.886.083.579.585.260.482.499.90.01.190.0621

scores

reliability

Training hyperparameters

sourceorpo
source_repoAmareshHebbar/ToolSure_Qwen2.5_1.5B_ORPO
source_overall0.7643055555555556

Usage

from huggingface_hub import hf_hub_download
import importlib.util
from vllm import LLM

path = hf_hub_download("AmareshHebbar/ToolSure_Qwen2.5_1.5B_CONSTRAINED", "toolsure_constrained.py")
spec = importlib.util.spec_from_file_location("toolsure_constrained", path)
tc = importlib.util.module_from_spec(spec)
spec.loader.exec_module(tc)

llm = LLM("AmareshHebbar/ToolSure_Qwen2.5_1.5B_CONSTRAINED")
tools = [{"name": "get_weather", "description": "Current weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}]
print(tc.generate(llm, [{"role": "user", "content": "What is the weather in Paris?"}], tools))

toolsure_constrained.py builds an xgrammar structural tag from the offered tools: free text is allowed, but once the model writes <tool_call> the rest of the call is forced to a real function name and schema valid arguments.

Files

The repository root holds bf16 safetensors. adapter/ holds the LoRA adapter where one was trained, training/ holds logs and curves, and eval/ holds raw results and sample generations.

Downloads last month
111
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AmareshHebbar/ToolSure_Qwen2.5_1.5B_CONSTRAINED

Finetuned
(2)
this model
Quantizations
2 models

Dataset used to train AmareshHebbar/ToolSure_Qwen2.5_1.5B_CONSTRAINED

Collection including AmareshHebbar/ToolSure_Qwen2.5_1.5B_CONSTRAINED