SizheZ03/DRIVES_LLM_rescorer_linear

DRIVES LLM Rescorer Linear

This repository contains a linear-head LLM rescorer for strict next-location candidate reranking in the DRIVES next-location prediction pipeline.

The model is not a normal generative chat model and should not be used with the standard text-generation pipeline. It uses Qwen/Qwen2.5-7B-Instruct as a contextual text encoder, applies a LoRA adapter, pools the final-token hidden state, and feeds that vector into a trained scalar linear score head. Candidate destinations are ranked by descending scalar score.

candidate_text_i = shared_trip_context + "\n\n" + Candidate focus(candidate_i)
h_i = final-token hidden state from Qwen + LoRA
score_i = linear_head(h_i)
ranking = argsort_i(-score_i)

What this model does

Given a fixed candidate set for one trip, this model assigns one scalar score to each candidate destination. It does not retrieve candidates by itself.

Typical use:

1. Build a candidate set using the DRIVES Markov/table + learned candidate generator.
2. Build the same compact scorer prompt used during training.
3. Add a candidate-specific Candidate focus block for each candidate.
4. Score all candidate texts with the LoRA-adapted Qwen model plus linear head.
5. Sort candidates by descending score.

Required files

The model directory should contain the LoRA adapter, tokenizer files, and the separate score-head artifacts:

adapter_config.json
adapter_model.safetensors
score_head.pt
scorer_config.json
tokenizer.json / tokenizer.model / tokenizer_config.json / special_tokens_map.json

Important: score_head.pt is not loaded automatically by AutoModelForCausalLM or PeftModel. You must load it with the DRIVES scorer wrapper or equivalent code.

Installation

Clone the DRIVES repository because the scorer wrapper is implemented there:

git clone https://github.com/novaz03/DRIVES.git
cd DRIVES

Install the Python dependencies used by the scorer:

pip install torch transformers peft huggingface_hub pandas numpy pyarrow

On the WashU cluster, use the project environment if available:

PYTHON=/storage3/fs1/nlin/Active/sizhe/conda/envs/stage/bin/python

Loading the model from Hugging Face

from pathlib import Path
from huggingface_hub import snapshot_download

model_dir = Path(snapshot_download("SizheZ03/DRIVES_LLM_rescorer_linear"))

Then load it with the DRIVES scorer wrapper:

import sys
from pathlib import Path

DRIVES_ROOT = Path("/path/to/DRIVES")
sys.path.insert(0, str(DRIVES_ROOT / "scripts" / "next_location"))

from llm_scorer import load_pairwise_scorer, build_scoring_text

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
SCORER_DIR = model_dir

tokenizer, scorer, config = load_pairwise_scorer(
    base_model_name=BASE_MODEL,
    scorer_dir=SCORER_DIR,
)

The loader performs four operations:

1. Load the tokenizer from the scorer directory.
2. Load Qwen/Qwen2.5-7B-Instruct.
3. Attach the LoRA adapter from this repository.
4. Load score_head.pt according to scorer_config.json.

Minimal scoring example

from llm_scorer import build_scoring_text

context_prompt = """
You are scoring candidate destinations for a next-location prediction task.
Use the current trip context, participant history summary, retrieved examples,
transition priors, time/gap context, parking/POI metadata, and distance features.
Return no text; the model wrapper reads only the scalar score.
""".strip()

candidates = [
    {
        "candidate_id": "cell_A",
        "extras": {
            "candidate_label": "C1",
            "distance_from_start_km": 0.42,
            "candidate_is_parking": False,
            "candidate_total_poi": 12,
            "candidate_num_poi_nearby": 5,
        },
    },
    {
        "candidate_id": "cell_B",
        "extras": {
            "candidate_label": "C2",
            "distance_from_start_km": 3.10,
            "candidate_is_parking": True,
            "candidate_total_poi": 2,
            "candidate_num_poi_nearby": 1,
        },
    },
]

texts = [
    build_scoring_text(context_prompt, c["candidate_id"], c["extras"])
    for c in candidates
]

scores = scorer.score_texts(
    tokenizer,
    texts,
    max_length=3072,
    batch_size=4,
)

ranking = sorted(
    zip(candidates, scores),
    key=lambda item: item[1],
    reverse=True,
)

for candidate, score in ranking:
    print(candidate["candidate_id"], score)

Proper DRIVES evaluation command

For the full DRIVES next-location pipeline, use scripts/next_location/llm_candidate_reranker.py in scorer mode. The following example evaluates the validation split under the strict cap-32 candidate setting:

cd /path/to/DRIVES

PYTHON=/storage3/fs1/nlin/Active/sizhe/conda/envs/stage/bin/python

$PYTHON -u scripts/next_location/llm_candidate_reranker.py \
  --provider local \
  --model Qwen/Qwen2.5-7B-Instruct \
  --score-model-dir /path/to/SizheZ03/DRIVES_LLM_rescorer_linear \
  --split val \
  --input data/next_location/processed/next_location_supervised_250m_test10pct.parquet \
  --out-dir data/next_location/baselines/hf_linear_rescorer_val \
  --candidate-generator-dir data/next_location/candidate_retrieval_models/xgb_lambdarank_v1 \
  --candidate-generator-top-k 32 \
  --max-total-candidates 32 \
  --max-eval-negatives-per-row 31 \
  --scorer-context-mode compact \
  --history-source train_only \
  --history-scope same_day_from_timestamp \
  --same-day-history-k 8 \
  --participant-summary-top-n 5 \
  --max-length 3072 \
  --score-batch-size 4

For the test split, change:

--split test
--out-dir data/next_location/baselines/hf_linear_rescorer_test

Training/evaluation setup used for the reported model

This model was trained as a compact listwise scorer over the Markov + XGB cap-32 candidate distribution.

Item	Value
Base model	Qwen/Qwen2.5-7B-Instruct
Fine-tuning method	LoRA
Score head	Linear scalar head
Hidden size	3584
Objective	Listwise candidate ranking
Candidate distribution	Markov/table candidates union XGB top-32, capped at 32
Max sequence length	3072
Context mode	Compact scorer context
Candidate order during training	Randomized
Evaluation rule	Strict: truth is not appended to held-out candidate sets

Reported strict held-out performance

The reported evaluation used:

Dataset: data/next_location/processed/next_location_supervised_250m_test10pct.parquet
Validation rows: 4,788
Test rows: 4,788
Candidate cap: 32
History source: train_only
History scope: same_day_from_timestamp:trip_start_time

Split	Candidate coverage	MRR	Hit@1	Hit@5	Hit@10
Validation	0.7909	0.4531	0.3791	0.5215	0.6023
Test	0.7556	0.4197	0.3409	0.5038	0.5733

Because evaluation is strict, rows whose true destination is absent from the candidate set contribute zero to MRR and Hit@k.

Input format details

The scorer expects one text per candidate. Each text should have:

1. A shared context prompt containing the current trip, train-only participant history, retrieved examples, transition priors, temporal context, and candidate-list metadata.
2. A candidate-specific block produced by build_candidate_focus_text, which has the form:

Candidate focus:
{"candidate_id":"...", ...candidate extras...}

Do not ask the model to generate JSON or natural language. The wrapper directly reads the scalar score from the score head.

Common mistakes

Mistake 1: using pipeline("text-generation")

This model is not intended for free-form generation. The score head is external to the normal generation head, so pipeline("text-generation") will not use the learned reranking head.

Mistake 2: loading only the LoRA adapter

Loading the LoRA adapter without score_head.pt gives you the adapted language model, but not the candidate scoring function. Always load the score head.

Mistake 3: evaluating with a different candidate distribution

The scorer was trained for the Markov + XGB cap-32 candidate distribution. Metrics are not directly comparable if you change the candidate generator, cap, history source, or strict evaluation rule.

Mistake 4: appending truth during held-out evaluation

Do not append the true destination to validation/test candidate sets. The reported metrics are strict all-row metrics.

Citation / project reference

This model is part of the DRIVES next-location prediction experiments:

https://github.com/novaz03/DRIVES

Relevant project files:

scripts/next_location/llm_scorer.py
scripts/next_location/llm_candidate_reranker.py
docs/details/linear_head_llm_rescorer.md

Limitations

• Domain-specific to DRIVES-style next-location candidate reranking.
• Requires an externally constructed candidate set.
• Requires the DRIVES scoring wrapper or equivalent code to use score_head.pt.
• Metrics are bounded by candidate coverage.
• Intended for research use, not as a general-purpose location recommendation model.

License

Set the Hugging Face license field to match the license policy of the DRIVES repository, the Qwen base model, and any dataset constraints that apply to the released adapter and score head.