Sergio Paniego PRO
AI & ML interests
Recent Activity
Organizations
I did that exercise for RL in post-training: from RLHF and PPO, to verifiable rewards, to the GRPO family of variants, to agents acting in environments. Everything is backed by what the labs themselves say in their public reports (DeepSeek, Qwen, Kimi, GLM-5, Nemotron, Mistral and more), in their own words
This is the companion piece to Class 3 of our Training Agents series with @burtenshaw . The class explains how GRPO works, with three hands-on experiments. The article shows where the same ideas appear at frontier scale
https://huggingface.co/blog/sergiopaniego/agentic-rl-2026
How frontier models train on outcomes in 2026
Really like that every rollout gets its own sandbox โ isolation is the part most agent-training setups skimp on.
We run coding agents in production harnesses, and the failure mode we see most isn't wrong code โ it's non-terminating turns: hand an agent an open-ended objective and it can think in a loop for hours (one of ours burned its full daily inference cap doing exactly that this morning; it's the failure class we built ThumbGate around).
Does the reward setup here penalize rollouts that never emit a final answer, or do you hard-cap steps in the env? Curious because we ended up putting runaway detection at the harness layer, outside the model โ prompt-level bounds kept getting reasoned around.
thanks for the question @IgorGanapolsky ! this is actually just a small experiment but yes, we added a hard timeout at the harness level. additionally the reward only pays off if it actually solves, plus a small penalty for runaway tool loops
Baseten on Hugging Face Inference Providers ๐ฅ
- +6
you can take a real coding agent (OpenCode), let it run its own tool loop against real coding problems, and train it with RL on the exact tokens it produced
and every rollout runs in its own remote HF sandbox, so rollouts scale out beyond one machine
the loop:
- OpenCode owns its tool loop inside an OpenEnv sandbox
- an in-sandbox proxy records the real token ids + logprobs, per turn
- a hidden-test verifier scores the result, and that is the reward
- TRL trains with AsyncGRPO, weights sync back to vLLM over NCCL
blog + runnable example: https://huggingface.co/blog/sergiopaniego/trl-openenv-harness-training
Training a coding agent using the OpenCode harness in remote HF sandboxes with TRL and OpenEnv
and the @liquidai blog comes with some nice details about the training procedure, so let's analyze it.
basically, a full agent training pipeline but compressed into 2.6B
base model โ SFT โ specialized teachers per domain (SFT + RLVR) โ on-policy distillation back into one student โ agentic RL
the two most interesting stages
โ MOPD: the student generates, each prompt routes to its domain teacher for token-level feedback. teachers branch from the same SFT checkpoint, so their signal stays close to the student's distribution
โ agentic RL: multi-turn GRPO inside real harnesses (OpenClaw, Hermes Agent), one sandbox per rollout, a proxy captures token-level trajectories while the harness stays a black box
this makes a 2.6B that beats much larger models on instruction following and tool use
SFT, distillation, RL, RL envs: exactly what we're covering in our Training Agents livestream series (next one coming soon!)
โ model: LiquidAI/LFM2.5-2.6B
โ blog: https://www.liquid.ai/blog/lfm2-5-2-6b
โ live series: https://www.youtube.com/playlist?list=PLo2EIpI_JMQvQZm-kVlz4wY1vWF0LBcf5