Cortex Baseline: SHINE β Qwen3-4B-Thinking-2507, rank 30
We trained this SHINE baseline for the Cortex experiments using the SHINE training code. SHINE maps an input context to LoRA parameters through a hypernetwork, allowing a language model to use context-dependent parameter updates during inference.
This variant uses Qwen3-4B-Thinking-2507, with a generated LoRA rank of 30 and a MetaLoRA rank of 128.
For usage and evaluation, please refer to the Cortex GitHub repository.
Released variants
| Repository | Backbone | LoRA rank | MetaLoRA rank |
|---|---|---|---|
| Cortex-Baseline-SHINE-Qwen2.5-14B-r10 | Qwen2.5-14B-Instruct | 10 | 128 |
| Cortex-Baseline-SHINE-Qwen2.5-14B-r30 | Qwen2.5-14B-Instruct | 30 | 128 |
| Cortex-Baseline-SHINE-Qwen3-4B-Thinking-2507-r10 | Qwen3-4B-Thinking-2507 | 10 | 128 |
| Cortex-Baseline-SHINE-Qwen3-4B-Thinking-2507-r30 | Qwen3-4B-Thinking-2507 | 30 | 128 |
r10 and r30 denote the generated LoRA rank. All four variants use MetaLoRA rank 128.
Training configuration
We use the SHINE implementation and its meta_train_parallel.py training entry point. The training configurations are summarized below.
Shared settings
| Parameter | Value |
|---|---|
| Hypernetwork | Transformer, 4 layers |
| Generated LoRA rank | 10 or 30, depending on the variant |
| MetaLoRA rank | 128 |
| Optimizer | AdamW |
| Learning-rate schedule | Linear decay with warmup |
| Weight decay | 0.01; 0 for bias and normalization parameter groups |
| Gradient clipping norm | 1.0 |
| GPUs | 8 |
| Per-device training batch size | 1 |
| Gradient accumulation steps | 4 |
| Effective batch size | 32 |
| Seed | 42 |
Training stages
The training pipeline consists of group pretraining, IFT-PWC, and IFT-C1QA. The data-source identifiers below follow the SHINE code.
| Stage | Data-source identifier | Learning rate | Warmup steps | Epochs |
|---|---|---|---|---|
| Group pretraining | grouptransmla |
5e-5 | 200 | 1 |
| IFT-PWC | ift-pwc |
3e-5 | 400 | 2 |
| IFT-C1QA | ift-c1qa |
1e-5 | 400 | 1 |
Training length limits for Qwen3-4B-Thinking-2507-r30:
| Stage | context_max_length |
conversation_max_length |
|---|---|---|
| Group pretraining | 1141 | 1150 |
| IFT-PWC | 1150 | 1150 |
| IFT-C1QA | 2380 | 100 |
Citation
If you use these checkpoints in your research, please cite Cortex: Consolidating Context into Parametric Memory.
@misc{zhang2026cortex,
title = {Cortex: Consolidating Context into Parametric Memory},
author = {Jintian Zhang and Feng He and Xinnian Liang and Ziwen Xu and Haoming Xu and Haojing Huang and Shuangzhi Wu and Ningyu Zhang},
year = {2026},
eprint = {2610.20399},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2610.20399}
}
Model tree for zjunlp/Cortex-Baseline-SHINE-Qwen3-4B-Thinking-2507-r30
Base model
Qwen/Qwen3-4B-Thinking-2507