Pixiano45/freebot_qwen0.5

FreeBot Qwen2.5-0.5B (Mark II): Paraphraser + Summarizer

Live site: https://freequillbot.netlify.app

Author: Vatsal K. (alias VatsalDaPro)

The models behind FreeBot, a free, self-hosted paraphrasing and summarizing tool built as a personal project (replicating the core behavior of tools like QuillBot using small open models instead of a commercial API). This repo now holds Mark II and supersedes the original upload.

Part	What it is	Where
Paraphraser	LoRA fine-tune of Qwen2.5-0.5B-Instruct, 4 modes (Natural / Formal / Simple / Shorter), now structure-preserving	adapter_model.safetensors + adapter_config.json (LoRA), qwen2.5-0.5b-finetuned-720-q8_0.gguf (merged, Q8_0)
Summarizer	The untrained base Qwen2.5-0.5B-Instruct, Q8_0, driven by a length-scaled prompt	summarization/qwen2.5-0.5b-instruct-q8_0.gguf
Mark I (superseded)	The original adapter and GGUF	mark_i_legacy/

What changed in Mark II

Real traffic exposed a weakness the Mark I fine-tune never showed in testing: plain prose paraphrased fine, but structured documents (numbered headings, bullet lists) were flattened into one undifferentiated paragraph. Prompt-level patches only covered the exact reported pattern, so the fix was retraining with 720 structure-preserving examples (180 unique documents x 4 modes) added to the original 5,200, for 5,920 total. See the companion dataset: Pixiano45/freebot (gated), folder mark_ii/.

Diversity mattered more than repetition: a small pool repeated many times taught the model those documents, not the pattern; more unique documents seen once generalized.

Mark II training

• Base model: unsloth/Qwen2.5-0.5B-Instruct
• Method: LoRA via Unsloth, rank 16, alpha 16, dropout 0, targeting q_proj/k_proj/v_proj/o_proj/gate_proj/up_proj/down_proj (unchanged from Mark I)
• Data: 5,920 examples (Mark I's 5,200 + 720 structure examples), mode-conditioned via a prompt prefix
• Schedule: 3 epochs, 1,110 steps; final training loss 0.7346

Mark II results

Structural preservation = headings, list items, and paragraph breaks in the input survive in the output.

Metric	Mark I model	240-example retrain	Mark II (this repo)
Structural, held-out (12 documents never trained on x 4 modes)	10.4% (5/48)	66.7% (32/48)	77.1% (37/48)
Structural, adversarial (documents from the training pool)	6.2% (45/720)	97.1% (233/240)*	95.4% (687/720)
Overlap ratio on eval_set.json (lower = more restructuring)	0.430	0.407	0.418
Semantic similarity on eval_set.json	0.833	0.827	0.830
Entailment on eval_set.json	0.600	0.592	0.571

* Measured against its own 240-document pool, not the 720-document pool. Adversarial rows reflect training fit; the held-out row is the generalization measure. Plain-text quality did not regress.

Usage

Paraphraser, prompts

The model expects the mode instruction as a plain-language prefix in a user turn, formatted with the chat template (it was trained only on chat-formatted turns). The prompts used in production:

STRUCTURE_NOTE = " Keep the original structure: preserve line breaks, numbered/bulleted lists, and headings exactly as given -- paraphrase the text within each part, not the structure itself."
NO_NEW_PARAGRAPHS = " Do not add, remove, or split paragraphs -- the output must have exactly the same number of paragraph breaks (blank lines) as the input. If the input is a single paragraph, the output must also be a single paragraph."

MODE_PROMPTS = {
    "natural": "Paraphrase the following text so it sounds natural, preserving the exact meaning." + STRUCTURE_NOTE + NO_NEW_PARAGRAPHS,
    "formal":  "Paraphrase the following text in a more formal/professional tone, preserving the exact meaning." + STRUCTURE_NOTE + NO_NEW_PARAGRAPHS,
    "simple":  "Paraphrase the following text using simpler words and shorter sentences, preserving the exact meaning." + STRUCTURE_NOTE + NO_NEW_PARAGRAPHS,
    "shorter": "Paraphrase the following text to be more concise, preserving the exact meaning.",
}
# user turn = MODE_PROMPTS[mode] + "\n" + text

Shorter mode deliberately has no structure instruction: "condense this" reliably overrides "preserve structure", so it still tends to flatten headings and lists.

LoRA adapter (transformers/Unsloth)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Pixiano45/freebot_qwen0.5", max_seq_length=2048, load_in_4bit=False,
)
FastLanguageModel.for_inference(model)
chat = tokenizer.apply_chat_template(
    [{"role": "user", "content": MODE_PROMPTS["natural"] + "\nThe mayor announced a new recycling program for downtown businesses."}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer([chat], return_tensors="pt", add_special_tokens=False).to("cuda")
out = model.generate(**inputs, max_new_tokens=400, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

GGUF with llama-cpp-python: read this first

llama-cpp-python's create_completion() / create_chat_completion() never feed the prompt into the repetition-penalty sampler (upstream llama.cpp's own CLI does), and its penalty window defaults to only the last 64 tokens. For a paraphraser this silently turns the repetition penalty into a no-op against copying the input, and output collapses toward near-verbatim copies. This is how the Mark I GGUF measured ~0.60 overlap (barely better than zero-shot) when served normally, despite the model itself being fine. Use the whole context as the window and prime the sampler with the prompt tokens yourself:

from llama_cpp import Llama

N_CTX = 2048
llm = Llama(model_path="qwen2.5-0.5b-finetuned-720-q8_0.gguf", n_ctx=N_CTX,
            n_gpu_layers=-1, verbose=False, last_n_tokens_size=N_CTX)

def generate(llm, prompt, max_new_tokens=1024, repeat_penalty=1.1):
    text = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
    prompt_tokens = llm.tokenize(text.encode("utf-8"), add_bos=True, special=True)
    llm.reset()
    llm.eval(prompt_tokens)
    sampler = llm._init_sampler(temp=0.0, repeat_penalty=repeat_penalty)
    for t in prompt_tokens:          # the fix: prime the penalty with the prompt
        sampler.accept(t)
    eos, out = llm.token_eos(), []
    for _ in range(max_new_tokens):
        tok = sampler.sample(llm._ctx, -1)
        sampler.accept(tok)
        if tok == eos:
            break
        out.append(tok)
        llm.eval([tok])
    return llm.detokenize(out).decode("utf-8", errors="ignore").strip()

With this, the served GGUF reproduces the unmerged adapter's behavior (~0.37-0.38 overall overlap on eval_set.json) at roughly 0.2 s per request on an RTX 5060 Ti.

Summarization

The Summarize feature uses the base Qwen2.5-0.5B-Instruct, not fine-tuned: summarization/qwen2.5-0.5b-instruct-q8_0.gguf. These are unmodified Qwen weights (Apache-2.0, see the original model), included here so the whole site stack is in one place.

Why the base model: the paraphrase fine-tune refuses to compress. On one test document it returned 871 characters for a 769-character input, because paraphrasing is what it was trained to do. The base model genuinely cuts redundant content.

Output size scales with input length, via the prompt:

def summary_prompt(text: str) -> str:
    n = len(text)
    if n < 350:    ask = "a 2-3 sentence summary"
    elif n < 800:  ask = "a summary as bullet points (at most 7 bullets)"
    elif n < 1400: ask = "a summary as bullet points (at most 10 bullets)"
    else:          ask = "a summary as bullet points (at most 15 bullets)"
    return f"Write {ask} covering only the key points of the following text. Do not include minor details."
# user turn = summary_prompt(text) + "\n" + text   (same generate() as above)

Known limitation, numbers. On numbers-heavy input the base model hallucinates: one test flipped a profit margin the wrong direction, another invented a year that was not in the source. This finding is qualitative (no scored benchmark was saved), so treat it as a warning, not a measured rate. The FreeBot site shows a warning whenever the input contains more than five numbers; do the equivalent if you build on this.

Limitations

• Fact drift: across every version tested, paraphrase outputs invent or drop a small fact (a rounded number, a swapped medical detail) roughly one time in seven or eight. Double-check names, numbers, and dates.
• Weakest case: Shorter mode on dense mixed lists.
• Evaluation is on a small held-out structure set (12 documents), so the 77.1% figure carries wide uncertainty.

Mark I (superseded)

The original upload is preserved under mark_i_legacy/ (adapter and GGUF) for reproducibility and is superseded by Mark II. Its documented numbers (overlap 0.305 / 0.409 / 0.378 / 0.454 for Natural / Formal / Simple / Shorter, vs. ~0.60-0.79 zero-shot) were measured on the unmerged adapter and reproduce. The Mark I GGUF is affected by the sampler issue above when served with default llama-cpp-python settings. Mark I had the structure-flattening weakness described above.

License / usage

Access to this model requires agreeing to the repository's usage terms before use. Contact vatsaldapro@gmail.com for permitted-use details before any reuse, redistribution, or derivative work. The unmodified Qwen2.5-0.5B-Instruct weights in summarization/ remain under their original Apache-2.0 license.

---

Author: Vatsal K. (alias VatsalDaPro)