Cortex Baseline: SHINE β€” Qwen3-4B-Thinking-2507, rank 30

We trained this SHINE baseline for the Cortex experiments using the SHINE training code. SHINE maps an input context to LoRA parameters through a hypernetwork, allowing a language model to use context-dependent parameter updates during inference.

This variant uses Qwen3-4B-Thinking-2507, with a generated LoRA rank of 30 and a MetaLoRA rank of 128.

For usage and evaluation, please refer to the Cortex GitHub repository.

Released variants

Repository Backbone LoRA rank MetaLoRA rank
Cortex-Baseline-SHINE-Qwen2.5-14B-r10 Qwen2.5-14B-Instruct 10 128
Cortex-Baseline-SHINE-Qwen2.5-14B-r30 Qwen2.5-14B-Instruct 30 128
Cortex-Baseline-SHINE-Qwen3-4B-Thinking-2507-r10 Qwen3-4B-Thinking-2507 10 128
Cortex-Baseline-SHINE-Qwen3-4B-Thinking-2507-r30 Qwen3-4B-Thinking-2507 30 128

r10 and r30 denote the generated LoRA rank. All four variants use MetaLoRA rank 128.

Training configuration

We use the SHINE implementation and its meta_train_parallel.py training entry point. The training configurations are summarized below.

Shared settings

Parameter Value
Hypernetwork Transformer, 4 layers
Generated LoRA rank 10 or 30, depending on the variant
MetaLoRA rank 128
Optimizer AdamW
Learning-rate schedule Linear decay with warmup
Weight decay 0.01; 0 for bias and normalization parameter groups
Gradient clipping norm 1.0
GPUs 8
Per-device training batch size 1
Gradient accumulation steps 4
Effective batch size 32
Seed 42

Training stages

The training pipeline consists of group pretraining, IFT-PWC, and IFT-C1QA. The data-source identifiers below follow the SHINE code.

Stage Data-source identifier Learning rate Warmup steps Epochs
Group pretraining grouptransmla 5e-5 200 1
IFT-PWC ift-pwc 3e-5 400 2
IFT-C1QA ift-c1qa 1e-5 400 1

Training length limits for Qwen3-4B-Thinking-2507-r30:

Stage context_max_length conversation_max_length
Group pretraining 1141 1150
IFT-PWC 1150 1150
IFT-C1QA 2380 100

Citation

If you use these checkpoints in your research, please cite Cortex: Consolidating Context into Parametric Memory.

@misc{zhang2026cortex,
  title         = {Cortex: Consolidating Context into Parametric Memory},
  author        = {Jintian Zhang and Feng He and Xinnian Liang and Ziwen Xu and Haoming Xu and Haojing Huang and Shuangzhi Wu and Ningyu Zhang},
  year          = {2026},
  eprint        = {2610.20399},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.20399}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for zjunlp/Cortex-Baseline-SHINE-Qwen3-4B-Thinking-2507-r30

Adapter
(78)
this model

Collection including zjunlp/Cortex-Baseline-SHINE-Qwen3-4B-Thinking-2507-r30

Paper for zjunlp/Cortex-Baseline-SHINE-Qwen3-4B-Thinking-2507-r30