---
language: dna
license: apache-2.0
tags:
- dna-language-model
- causal-lm
- htt
- polyq
- ceno
library_name: transformers
pipeline_tag: text-generation
base_model: CladeTeam/CENO-base-1b-preview
---

# CENO Rice CDS 1-Epoch Finetune

CENO model finetuned for one epoch on Oryza sativa CDS sequences.

CENO is derived from NVIDIA's Nemotron-H (Apache-2.0). The custom Transformers
remote code in this repository (`configuration_ceno.py`, `modeling_ceno.py`) is a
rename of the upstream Nemotron-H implementation.

This repository includes custom Transformers remote code for `CENOForCausalLM`
and `CENOCharLevelTokenizer`. Load with `trust_remote_code=True`.

## Files

- `model.safetensors`: model weights
- `config.json`: model config with `auto_map`
- `generation_config.json`: generation config
- `configuration_ceno.py`, `modeling_ceno.py`: custom model code
- `ceno_tokenizer.py`, `tokenizer_config.json`, `special_tokens_map.json`, `vocab.json`: tokenizer files
- `training_metrics.json`: finetuning metrics

## Loading

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "CladeTeam/CENO-rice-cds"
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
```

The model code depends on PyTorch and the Mamba/Triton stack used by Nemotron-H.
The bundled config sets `use_mamba_kernels=false`, using the pure-PyTorch Mamba
fallback so no `mamba-ssm`/`causal-conv1d` install is required.

## Training

Finetuned for 1 epoch on rice CDS with learning_rate=5e-5, effective_batch_size=64, bf16, max_length=8192.

## Training Metrics

```json
{
  "species": "rice",
  "train_loss": 10.05208391170438,
  "eval_loss": 1.21553373336792,
  "learning_rate": 5e-05,
  "epochs": 1,
  "epoch_losses": [
    {
      "epoch": 0.9987473903966597,
      "eval_loss": 1.21553373336792
    },
    {
      "epoch": 0.9987473903966597,
      "eval_loss": 1.21553373336792
    }
  ],
  "n_gpu": 8,
  "effective_batch_size": 64
}
```

## Intended Use

These models are released to reproduce HTT/polyQ sequence scoring experiments.
The average log-likelihood scores reflect sequence-model likelihood, not biological
fitness or pathogenicity.

## License

This model and its bundled code are released under the Apache License 2.0,
inheriting the license of the upstream Nemotron-H model code (Copyright 2024 AI21
Labs Ltd. and the HuggingFace Inc. team; Copyright (c) 2025 NVIDIA CORPORATION).
Modifications for CENO by CladeTeam.