File size: 3,377 Bytes
c6a4981
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a68887d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
---
language: dna
license: apache-2.0
tags:
- dna-language-model
- causal-lm
- htt
- polyq
- ceno
library_name: transformers
pipeline_tag: text-generation
base_model: CladeTeam/CENO-base-1b-preview
---

# CENO Rice CDS 1-Epoch Finetune

CENO model finetuned for one epoch on Oryza sativa CDS sequences.

CENO is derived from NVIDIA's Nemotron-H (Apache-2.0). The custom Transformers
remote code in this repository (`configuration_ceno.py`, `modeling_ceno.py`) is a
rename of the upstream Nemotron-H implementation.

This repository includes custom Transformers remote code for `CENOForCausalLM`
and `CENOCharLevelTokenizer`. Load with `trust_remote_code=True`.

## Files

- `model.safetensors`: model weights
- `config.json`: model config with `auto_map`
- `generation_config.json`: generation config
- `configuration_ceno.py`, `modeling_ceno.py`: custom model code
- `ceno_tokenizer.py`, `tokenizer_config.json`, `special_tokens_map.json`, `vocab.json`: tokenizer files
- `training_metrics.json`: finetuning metrics

## Loading

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "CladeTeam/CENO-rice-cds"
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
```

The model code depends on PyTorch and the Mamba/Triton stack used by Nemotron-H.
The bundled config sets `use_mamba_kernels=false`, using the pure-PyTorch Mamba
fallback so no `mamba-ssm`/`causal-conv1d` install is required.

## Training

Finetuned for 1 epoch on rice CDS with learning_rate=5e-5, effective_batch_size=64, bf16, max_length=8192.

## Training Metrics

```json
{
  "species": "rice",
  "train_loss": 10.05208391170438,
  "eval_loss": 1.21553373336792,
  "learning_rate": 5e-05,
  "epochs": 1,
  "epoch_losses": [
    {
      "epoch": 0.9987473903966597,
      "eval_loss": 1.21553373336792
    },
    {
      "epoch": 0.9987473903966597,
      "eval_loss": 1.21553373336792
    }
  ],
  "n_gpu": 8,
  "effective_batch_size": 64
}
```

## Intended Use

These models are released to reproduce HTT/polyQ sequence scoring experiments.
The average log-likelihood scores reflect sequence-model likelihood, not biological
fitness or pathogenicity.

## License

This model and its bundled code are released under the Apache License 2.0,
inheriting the license of the upstream Nemotron-H model code (Copyright 2024 AI21
Labs Ltd. and the HuggingFace Inc. team; Copyright (c) 2025 NVIDIA CORPORATION).
Modifications for CENO by CladeTeam.


<!-- CENO_INFERENCE_CURRENT -->
## Current inference implementation

Model weights and token IDs are unchanged. Current source and usage instructions
are maintained at [CladeTeam/CENO](https://github.com/CladeTeam/CENO).
Use the updated model revision instead of historical diagnostic archives.

The current loader, tokenizer and cache fixes are included. This checkpoint keeps
its existing inference contract; a base-model CP profile is not inferred from its
name or substituted for unknown post-training provenance.

The tested HF stack is PyTorch 2.7.0a0 (NVIDIA 25.03), Transformers 4.48.3,
mamba-ssm 2.2.4, causal-conv1d 1.2.2.post1 and FlashAttention 2.7.3. HF and vLLM
use separate environments. Numerical correctness is not a guarantee of
repetition-free or biologically functional generation.
<!-- /CENO_INFERENCE_CURRENT -->