You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

PIT-4B 2023 YaRN-16K β€” instruction-tuned checkpoint

Private research checkpoint, completed October 20, 2026, using historically dated training sources. This repository contains both the exact final 2023 continued-pretraining backbone and its final instruction-tuning LoRA adapter. It is not a merged model: loading the root backbone without adapter/ does not enable the instruction-tuned weights.

Contents and lineage

  • Root model/tokenizer files: exact BF16 export of the 2023 final CPT checkpoint checkpoint-00101725, counter 101,725, with 9,999,941,632 new CPT input tokens.
  • Original model: Diamegs/PIT-4B-202312, pinned revision c605346d44eaaefcba96772045d04f7af944c5f9. The CPT backbone here is not those original unchanged weights and is not a 2024-model descendant.
  • adapter/: final instruction-tuned LoRA weights and original adapter configuration, unchanged from the completed run sft-2023-finqa-dolly-20261020T1602Z.
  • model_code/pit_hf/: the exact local architecture implementation used for training. Inspect this Python code before importing it. This export has model_type=pit_hf and no automatic remote-code mapping.
  • provenance/: export lineage, original completion markers, SFT completion and final-checkpoint provenance, independent completion receipt, and package inventory. No optimizer state, raw datasets, credentials or W&B cache is included.

The pit_base_provenance field retained in config.json describes the original upstream base conversion. For the actual trained CPT weights, use provenance/pit_temporal_export_provenance.json; for the adapter, use provenance/sft-final-checkpoint-provenance.json and provenance/sft-completed.json. Original provenance/COMPLETE files are retained unchanged; they describe their original artifact directories, not a new resume-ready optimizer checkpoint in this repository.

Training summary and limits

The intended historical cutoff is December 31, 2023. Continued-pretraining used the specified November 2023 Wikipedia / CC-MAIN-2023-50 general FineWeb recipe. SFT used a screened common historical core: original FinQA arithmetic programs and published answers first, then original Dolly instruction responses. No modern teacher-generated reasoning targets were introduced. This publication does not establish or retroactively waive any original CPT scientific/data-acceptance limitation.

SFT stage Training examples Updates Supervised tokens
FinQA source workings and answer 2,054 257 56,761
Dolly original answer 12,801 1,601 1,011,306
Total 14,855 1,858 1,068,067

Frozen BF16 backbone; fresh FP32 LoRA, rank 32 / alpha 64 / dropout 0; one epoch per stage; fresh AdamW at the stage boundary. The final partial batches were retained. No CPT optimizer was restored. The source workings stage teaches published arithmetic programs, not equivalence to an OLMo Think model or faithful internal reasoning.

Architecture context is 16,384 tokens with fixed YaRN factor 8, original vocabulary 50,304 and tied embeddings. SFT examples were capped at 8,192 without target truncation; this is not evidence of 16K instruction-task performance. No new special tokens or chat template were added. Historical-source checks do not establish a universal knowledge-cutoff guarantee. No held-out performance or financial-advice suitability claim is made here. Upstream model, code and dataset terms remain applicable; this private package does not grant additional rights.

Run: https://wandb.ai/joao-sedoc/dbot/runs/is4efulw

Load the exact backbone plus adapter

Authenticate with an account that can read this private repository. Pin REVISION to the complete upload's commit SHA rather than a moving branch. The following is a loading example, not a newly executed Hub inference/evaluation result. The original adapter configuration retains a server-local base path for provenance; explicitly loading the packaged backbone first avoids relying on that path. Plain AutoPeftModel.from_pretrained(repo_id) is not the loading contract.

Run the example in a fresh Python process, so an already imported different pit_hf package cannot take precedence. Use the training-compatible package versions recorded in provenance/software-versions.json. On the training server, importing PEFT's installed optional acceleration stack required a visible CUDA driver; CPU-only loading of that environment is not certified here.

from pathlib import Path
import sys
import torch
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer
from peft import PeftModel

REPO_ID = "codesj/PIT-4B-2023-16k-instruct-20261020"
REVISION = "REPLACE_WITH_UPLOADED_COMMIT_SHA"
snapshot = Path(snapshot_download(REPO_ID, revision=REVISION))
sys.path.insert(0, str(snapshot / "model_code"))
from pit_hf import PITForCausalLM

tokenizer = AutoTokenizer.from_pretrained(snapshot, local_files_only=True)
backbone = PITForCausalLM.from_pretrained(
    snapshot, dtype=torch.bfloat16, local_files_only=True
)
model = PeftModel.from_pretrained(
    backbone, snapshot / "adapter", is_trainable=False,
    autocast_adapter_dtype=True, local_files_only=True
)
model = model.to("cuda").eval()  # Device-only move; keep saved FP32 LoRA precision.

Do not cast the complete PEFT model to BF16: the saved adapter is FP32. Use BF16 autocast for the forward pass as in the original training runtime. Downloading a snapshot and importing its Python code is an explicit trust decision, even with trust_remote_code not used.

Prompt format

General instruction examples used literal text, not a chat template:

Instruction:
{instruction}
Context:
{optional context}
Response:

Omit the Context: section when no context is supplied. FinQA prompts instead used:

Use only the supplied financial-report context. Return the published arithmetic program and published answer.
Context:
{financial-report context}
Question:
{question}
Response:

FinQA targets were ordinary text <think>\n{published program}\n</think>\n{published answer}; Dolly targets were the original answer only. The <think> markers are not special tokens, and the final answer follows the closing marker. No financial-report context, question or target rows are distributed in this model package.

Downloads last month
-
Safetensors
Model size
4B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support