Qwen2.5-3B-Paper-Quality-Filter

Model Description

Based on 'Qwen2.5-3B-Instruct', this model aim to serve as a 'Data Quality Gatekeeper'. It reduce this task to a binary classification problem. It precisely identifies and filter out invalid text chunks, thereby ensure the high purity of the text corpus. Best act in RAG system.

Intended Use

Input: A text chunk parsed out from an academic PDF Output: 1/0

• 1: Valid academic body text
• 0: Headers, footers, auther informations, references, and similar elements are not considered part of the main body of the paper

Evaluation Metrics

The model underwent rigorous blind testing on independent test set that preserved the authentic distribution ratio of real-world data.

Project	Value
Accurancy	84.29%
Precision	96.32%
Recall	80.51%
F1-score	87.71%
True Positive	157
False Positive	6
True Negtive	79
False Negtive	38

Low FP count indicate that the model is rigorous when approving a data. meeting the requirements for high-quality data ingestion in RAG system.

Training Details

• PDF source: scraped and parse raw PDF academic papers from ChinaXiv.
• Data construction: "Dual-Teacher" cross-validation approach using "DeepSeek v4 Pro" and "ChatGLM 5.1 Pro" for data labeling.
  • The two teacher labeled over 3500+ segments... They Drain the balance in my account without me notice. QwQ
  • Ambiguous samples exhibiting cognition conflicts were filter out to distill a gold standard dataset of over 2000 high purity entries.
• Fine-tuning framework: Utilized 'Unsolth' for efficient Supervised Fine-Tuning(SFT). Preserving the native 'ChatML' format.

How to Use

Download model and load it via your favorite local apps. Set your Prompt Template to 'ChatML' format. Then run this model.

client = OpenAI(
  base_url=<Your URL Here>,
  api_key=<Your API key here>
)

SYSTEM_PROMPT = "As an expert reviewer of thesis excerpts, you are required to evaluate the provide text fragment to determine whether it constitutes legitimate body content. An excerpt is deemed illegitimate if elements such as headers, footers, extensive figures or table, equations, reference, acknowledgments, non-academic materials, titles, or auther informations collectively exceed 30% of the total content. Please respond with a single character only:'0' if the provided excerpt does not constitute legitimate thesis content, and '1' if it does"
SYSTEM_PROMPT_ORIGIN = "作为一个论文片段评审专家，你需要针对给出的论文片段，检查其是否是合法的正文论文片段，不合法的正文片段为“页眉页脚、大量图表、公式、参考文献、致谢、非学术论文片段、标题、作者等信息”超过总量的30%。请只回复一个字符，0代表用户提供的论文片段不是合法的论文内容，1代表用户提供的论文片段是合法的论文内容。"

response = client.chat.completions.create(
  model="qwen2.5-3b-instruct.Q8_0.gguf",
  messages=[
    {"role": "system", "content": SYSTEM_PROMPT}
    {"role": "user", "content": <Your content here>}
  ]
)

print(response.choices[0].message.content.strip())

For example, if you use LM Studio, please set these in the siderbar.

Project	Value
System Prompt	作为一个论文片段评审专家，你需要针对给出的论文片段，检查其是否是合法的正文论文片段，不合法的正文片段为“页眉页脚、大量图表、公式、参考文献、致谢、非学术论文片段、标题、作者等信息”超过总量的30%。请只回复一个字符，0代表用户提供的论文片段不是合法的论文内容，1代表用户提供的论文片段是合法的论文内容。
Temperature	0
Prompt Template	ChatML

After your load the model, you could set 'base_url' to http://<your_url>:1234/v1 to access this model for default settings.