Observability for One, Observability for All: Extending Trackio Across the ML Lifecycle
The Observability Gap in Modern ML Frameworks
The growth of machine learning frameworks has been nothing short of remarkable. They enable practitioners to perform complex tasks such as orchestrating ML pipelines, fine-tuning models on custom datasets and equipping models with specialized capabilities such as tool calling. While conventional tracking libraries help bridge this gap, many impose usage limits and can become costly to operate at scale, especially as projects grow in complexity. As a result, experiment data often remains siloed within individual frameworks, making it difficult to maintain a coherent view of a project as it progresses through the ML lifecycle.
Figure 1. Trackio's Design Architecture and Coupling with HuggingFace Hub
This is precisely the gap that Trackio aims to address. At its core, Trackio is built around two simple notions. First, it is local-first by design. Experiments and metrics remain on your machine by default, enabling fast iteration without imposing project limits. Secondly, it is designed to integrate naturally within the broader Hugging Face concepts such as Datasets, Spaces and Buckets while exposing a deliberately small API centered around logging runs.
This minimal interface makes Trackio straightforward to integrate into higher-level frameworks, allowing experiments to transition seamlessly from local development to collaborative sharing while keeping observability as part of a unified workflow. In the following sections, we'll explore how Trackio addresses these challenges by providing a framework-agnostic approach for experiment observability.
Wiring Tracking in MLOps Pipelines
We can define MLOps as the process of creating machine learning systems with reproducibility and deployment built in from the start. ZenML is an open-source framework which helps to deploy and evaluate your ML systems abstracted behind stacks.
A stack represents a named, versioned collection of components that define where a pipeline runs and where its metadata is stored. It can contain everything an ML system needs to execute, including an orchestrator, an experiment tracker, and other infrastructure components. Because experiment tracking is modeled as a stack component, ZenML provides a natural extension point for integrating tracking backends.
Figure 2. Trackio as a First-Class Experiment Tracker in ZenML
Trackio can be defined as an first-class Experiment Tracker flavor. The integration implements a TrackioExperimentTracker that hooks into ZenML's lifecycle, calling trackio.init() on entry and trackio.finish() on exit. Pipeline code remains focused on it's core capabilities while tracking is handled independently by the stack configuration. This approach aligns with ZenML's philosophy of separating pipeline logic from operational concerns, allowing the same pipeline to be executed locally, or across different environments.
With this integration, the trial data can be viewed as a Dataset for structured logs, a Space for the live dashboard and a Bucket for artifacts.It is worth noting that the hf_dataset backend has since been deprecated, with hf_bucket being the preferred backend. Runs, parameters and metrics remain connected throughout the pipeline, making experiments reproducible and easier to inspect over time.
Applying Instrumentation for Model Fine-Tuning
Model fine-tuning is the process of continuing training on a smaller, task-specific dataset to steer a model's behavior. The training loop itself can contain loss computation, weight updates, evaluation, checkpointing, and parameter-efficient adaptation methods such as LoRA. LlamaFactory is an open-source framework for fine-tuning large language models.
It abstracts much of the training configuration by allowing users to declare the model, dataset, training strategy, and reporting backends through a single YAML file. The framework then orchestrates the underlying training pipeline while exposing a consistent interface across different models and fine-tuning methods. Also, it often involves comparing multiple experiments that differ only in small changes to the training configuration, such as the learning rate, LoRA rank, or dataset version. Capturing these runs in a consistent format makes it easier to understand why one configuration outperformed another.
Figure 3. Trackio Across the LlamaFactory Fine-Tuning Pipeline
We can now understand where observability fits into this abstraction. LlamaFactory routes training metrics through the reporting interface inherited from Transformers via the report_to training argument. The integration extends this interface by accepting trackio as a reporting backend alongside wandb or tensorboard. With this integration, a complete record of each fine-tuning run in a HF Bucket that can be revisited, compared, and shared is provided by Trackio.
Watching the Model Learn Through Rewards and Steps in RL
Reinforcement learning trains a model by having it take actions and learn from the feedback those actions receive. ROLL is a distributed reinforcement learning framework that scales this process across multiple workers using Ray and a modular architecture. It separates reward computation, policy optimization and evaluation into independent components, allowing each stage of the training pipeline to scale while exposing a consistent interface for distributed reinforcement learning workloads.
This distributed environment introduces additional observability challenges. Rather than tracking a single training loop, reinforcement learning produces metrics and artifacts across multiple workers and stages of optimization. Reward distributions, preference scores and evaluation metrics all contribute to understanding whether a policy is actually improving over time.
Figure 4. Trackio Monitoring Across ROLL's Distributed RL Pipeline
The integration hooks into ROLL's native monitoring layer, allowing Trackio to capture logs and artifacts throughout each stage of the training pipeline. Evaluation results and policy checkpoints remain associated with each run, making large-scale reinforcement learning experiments easier to inspect and reproduce. Beyond the ROLL integration itself, the work was extended upstream by another contributor to AReaL by introducing a similar monitoring backend for Trackio.
Tracking Every Trial in Hyperparameter Optimization
Hyperparameter optimization (HPO) can be defined as systematically searching for the best parameters, such as learning rate or batch size. Unlike model weights, hyperparameters are set before training begins and define the conditions under which learning happens. Optuna is an automatic hyperparameter optimization framework that can perform studies with a vast variety of HPO samplers and pruners.
A study is the top-level optimization task in Optuna and consists of multiple trials. Each trial represents a single evaluation of a sampled hyperparameter configuration, recording the sampled values, intermediate metrics, and the final objective value. Together, studies and trials describe how the optimization progresses over time, making it important to observe not only the best-performing configuration but also how the search converged.
Figure 5. Trackio Callback Lifecycle Within an Optuna Study
The initial integration, which was contributed to optuna-integration, focused on the core features of the Trackio public API rather than feature completeness. However, as Optuna evolved, the complete set of integrations were deprecated due to concerns around supply chain security and a smaller maintenance scope. Instead, contributors were encouraged to port integrations to optuna-hub, a registry of third-party packages designed for Optuna.
The upgraded integration now features a couple of additional features such as support for syncing a local project to a HF Space and Bucket as set by the user. Once synchronized, users can inspect the trials that almost won, compare the cost of each trial and identify regions of the hyperparameter space that were never explored.
Conclusion
The modern ML Lifecycle rarely consist of a single framework. Instead, each framework has its own purpose. A model may begin as a reproducible pipeline with ZenML, undergo hyperparameter optimization with Optuna, be fine-tuned for a downstream task using LlamaFactory and be refined through reinforcement learning with a framework like ROLL.
While each stage introduces different configurations and abstractions, the need to understand what happened during each phase remains constant. By building upon each framework's extension points, Trackio provides a consistent way to capture experiment metadata without disrupting existing monitoring workflows. Furthermore, the approach reflects Trackio's broader design philosophy: A lightweight, local-first experiment tracker that integrates naturally with the Hugging Face ecosystem while adapting to the constraints introduced by modern ML frameworks.
The integrations presented throughout this article demonstrate that observability does not need to be reinvented for every framework. Together, they represent the culmination of roughly six months of engineering work across multiple frameworks, each requiring a different integration strategy while preserving a common observability model. Ultimately, this work aims to contribute to a more connected and interoperable ecosystem for machine learning experimentation.
