README.md

metadata

title: README
emoji: 🏃
colorFrom: pink
colorTo: red
sdk: static
pinned: false

Knowledge Fusion of Large Language Models

| 📑 FuseLLM Paper @ICLR2024 | 🤗 HuggingFace Repo | 🐱 GitHub Repo |

FuseAI

FuseAI is an open-source research community focused on model fusion topics.

The community members currently applying model fusion on Foundation and Chat LLMs, with future plans to fuse Agent/MoE LLMs.

Welcome to join us!

News

FuseLLM [Surpassing Llama-2-7B]

• Jan 22, 2024: 🔥 We release FuseLLM-7B, which is the fusion of three open-source foundation LLMs with distinct architectures, including Llama-2-7B, OpenLLaMA-7B, and MPT-7B.

Model	BBH	ARC-easy	ARC-challenge	BoolQ	HellaSwag	OpenBookQA
OpenLLaMA-7B	33.87	69.70	41.38	72.29	74.53	41.00
MPT-7B	33.38	70.12	42.15	74.74	76.25	42.40
Llama-2-7B	39.70	74.58	46.33	77.71	76.00	44.20
Llama-2-CLM-7B	40.44	74.54	46.50	76.88	76.57	44.80
🤗 FuseLLM-7B	41.75	75.04	47.44	78.13	76.78	45.40

Model	MultiPL-E	TrivialQA	DROP	LAMBADA	IWSLT2017	SciBench
OpenLLaMA-7B	18.11	39.96	22.31	70.31	5.51	0.68
MPT-7B	17.26	28.89	23.54	70.08	5.49	0.88
Llama-2-7B	14.63	52.46	27.25	73.28	6.48	0.14
Llama-2-CLM-7B	14.83	53.14	28.51	73.45	6.91	0.94
🤗 FuseLLM-7B	15.56	54.49	28.97	73.72	6.75	1.65

Citation

Please cite the following paper if you reference our model, code, data, or paper related to FuseLLM.

@inproceedings{wan2024knowledge,
  title={Knowledge Fusion of Large Language Models},
  author={Fanqi Wan and Xinting Huang and Deng Cai and Xiaojun Quan and Wei Bi and Shuming Shi},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024},
  url={https://openreview.net/pdf?id=jiDsk12qcz}
}