WilliamRabuel commited on
Commit
823ddd6
Β·
verified Β·
1 Parent(s): 9012961

Change default model for InferenceClientModel to fix ToolCallingAgent incompatibility

Browse files

Context:
The ToolCallingAgent is currently incompatible with the default model specified in InferenceClientModel, which is Qwen/Qwen2.5-Coder-32B-Instruct. This breaks the out-of-the-box experience for users trying to use tool-calling with the default settings.

Problem Details:
When using the default configuration, two issues arise:
A 422 Unprocessable Entity error is returned because the Qwen model endpoint does not support the default tool_choice="required" parameter.
Even after patching the agent to send tool_choice="auto", the server returns a 500 Internal Server Error. This indicates a deeper incompatibility with the model's tool-calling implementation on the backend.
While this backend issue with the Qwen model should be addressed, the smolagents library can provide a working default experience in the meantime.

Proposed Solution:
Extensive testing has shown that other models, such as mistralai/Mixtral-8x7B-Instruct-v0.1, work flawlessly with the ToolCallingAgent without any code modifications.
This PR changes the default model_id in InferenceClientModel from "Qwen/Qwen2.5-Coder-32B-Instruct" to "mistralai/Mixtral-8x7B-Instruct-v0.1".
This is a pragmatic solution that ensures new users have a functional agent by default, avoiding the need for them to debug server-side issues or immediately override the model choice.

unit2/smolagents/tool_calling_agents.ipynb CHANGED
@@ -87,7 +87,7 @@
87
  "<span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span> <span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span>\n",
88
  "<span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span> <span style=\"font-weight: bold\">Search for the best music recommendations for a party at the Wayne's mansion.</span> <span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span>\n",
89
  "<span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span> <span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span>\n",
90
- "<span style=\"color: #d4b702; text-decoration-color: #d4b702\">╰─ InferenceClientModel - Qwen/Qwen2.5-Coder-32B-Instruct ──────────────────────────────────────────────────────────────────╯</span>\n",
91
  "</pre>\n"
92
  ],
93
  "text/plain": [
@@ -95,7 +95,7 @@
95
  "\u001b[38;2;212;183;2mβ”‚\u001b[0m \u001b[38;2;212;183;2mβ”‚\u001b[0m\n",
96
  "\u001b[38;2;212;183;2mβ”‚\u001b[0m \u001b[1mSearch for the best music recommendations for a party at the Wayne's mansion.\u001b[0m \u001b[38;2;212;183;2mβ”‚\u001b[0m\n",
97
  "\u001b[38;2;212;183;2mβ”‚\u001b[0m \u001b[38;2;212;183;2mβ”‚\u001b[0m\n",
98
- "\u001b[38;2;212;183;2m╰─\u001b[0m\u001b[38;2;212;183;2m InferenceClientModel - Qwen/Qwen2.5-Coder-32B-Instruct \u001b[0m\u001b[38;2;212;183;2m─────────────────────────────────────────────────────────────────\u001b[0m\u001b[38;2;212;183;2m─╯\u001b[0m\n"
99
  ]
100
  },
101
  "metadata": {},
@@ -552,7 +552,7 @@
552
  "source": [
553
  "from smolagents import ToolCallingAgent, DuckDuckGoSearchTool, InferenceClientModel\n",
554
  "\n",
555
- "agent = ToolCallingAgent(tools=[DuckDuckGoSearchTool()], model=InferenceClientModel())\n",
556
  "\n",
557
  "agent.run(\"Search for the best music recommendations for a party at the Wayne's mansion.\")"
558
  ]
 
87
  "<span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span> <span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span>\n",
88
  "<span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span> <span style=\"font-weight: bold\">Search for the best music recommendations for a party at the Wayne's mansion.</span> <span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span>\n",
89
  "<span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span> <span style=\"color: #d4b702; text-decoration-color: #d4b702\">β”‚</span>\n",
90
+ "<span style=\"color: #d4b702; text-decoration-color: #d4b702\">╰─ InferenceClientModel - mistralai/Mixtral-8x7B-Instruct-v0.1 ──────────────────────────────────────────────────────────────────╯</span>\n",
91
  "</pre>\n"
92
  ],
93
  "text/plain": [
 
95
  "\u001b[38;2;212;183;2mβ”‚\u001b[0m \u001b[38;2;212;183;2mβ”‚\u001b[0m\n",
96
  "\u001b[38;2;212;183;2mβ”‚\u001b[0m \u001b[1mSearch for the best music recommendations for a party at the Wayne's mansion.\u001b[0m \u001b[38;2;212;183;2mβ”‚\u001b[0m\n",
97
  "\u001b[38;2;212;183;2mβ”‚\u001b[0m \u001b[38;2;212;183;2mβ”‚\u001b[0m\n",
98
+ "\u001b[38;2;212;183;2m╰─\u001b[0m\u001b[38;2;212;183;2m InferenceClientModel - mistralai/Mixtral-8x7B-Instruct-v0.1 \u001b[0m\u001b[38;2;212;183;2m─────────────────────────────────────────────────────────────────\u001b[0m\u001b[38;2;212;183;2m─╯\u001b[0m\n"
99
  ]
100
  },
101
  "metadata": {},
 
552
  "source": [
553
  "from smolagents import ToolCallingAgent, DuckDuckGoSearchTool, InferenceClientModel\n",
554
  "\n",
555
+ "agent = ToolCallingAgent(tools=[DuckDuckGoSearchTool()], model=InferenceClientModel(model_id="mistralai/Mixtral-8x7B-Instruct-v0.1")))\n",
556
  "\n",
557
  "agent.run(\"Search for the best music recommendations for a party at the Wayne's mansion.\")"
558
  ]