Update docs.md
Browse files
docs.md
CHANGED
@@ -22,12 +22,11 @@
|
|
22 |
|
23 |
</div>
|
24 |
|
25 |
-
|
26 |
<h2>π Background</h2>
|
27 |
<p>Recent advances in <strong>Large Language Models (LLMs)</strong> have demonstrated transformative potential in <strong>healthcare</strong>, yet concerns remain around their reliability and clinical validity across diverse clinical tasks, specialties, and languages. To support timely and trustworthy evaluation, building upon our <a href="https://ai.nejm.org/doi/full/10.1056/AIra2400012">systematic review</a> of global clinical text resources, we introduce <a href="https://arxiv.org/abs/2504.19467">BRIDGE</a>, <strong>a multilingual benchmark that comprises 87 real-world clinical text tasks spanning nine languages and more than one million samples</strong>. Furthermore, we construct this leaderboard of LLM in clinical text understanding by systematically evaluating <strong>52 state-of-the-art LLMs</strong> (by 2024/04/29).</p>
|
28 |
|
29 |
<div style="text-align: center;">
|
30 |
-
<img src="
|
31 |
</div>
|
32 |
|
33 |
<h2>π BRIDGE Leaderboard</h2>
|
@@ -40,7 +39,7 @@
|
|
40 |
<p>In addition, BRIDGE offers multiple <strong>model filters</strong> and <strong>task filters</strong> to enable users to explore LLM performance across <strong>different clinical contexts</strong>, empowering researchers and clinicians to make informed decisions and track model advancements over time.</p>
|
41 |
|
42 |
<div style="text-align: center;">
|
43 |
-
<img src="
|
44 |
</div>
|
45 |
|
46 |
|
@@ -64,13 +63,13 @@ More Details can be found in our <a href="https://arxiv.org/abs/2504.19467">BRID
|
|
64 |
<h2>π οΈ How to Evaluate Your Model on BRIDGE ?</h2>
|
65 |
<h4>π Dataset Access</h4>
|
66 |
<p>All fully open-access datasets in BRIDGE are available in <a href="https://huggingface.co/datasets/YLab-Open/BRIDGE-Open">BRIDGE-Open</a>. To ensure fairness of this leaderboard, we publicly release the following data for each task:
|
67 |
-
Five completed samples
|
68 |
|
69 |
<p>Due to privacy and security considerations of clinical data, regulated-access datasets can not be directly published. However, all detailed task descriptions and their corresponding data sources are available in our <a href="https://arxiv.org/abs/2504.19467">BRIDGE paper</a>.
|
70 |
Importantly, all 87 datasets have been verified to be either fully open-access or publicly accessible via reasonable request.</p>
|
71 |
|
72 |
<h4>π₯ Result Submission and Evaluation</h4>
|
73 |
-
If you would like to submit your model results to BRIDGE and demonstrate its performance, please follow these steps
|
74 |
<ul>
|
75 |
<li><strong>Run local inference:</strong> Download the <a href="https://huggingface.co/datasets/YLab-Open/BRIDGE-Open">BRIDGE-Open</a> dataset and perform inference locally. Save the generated output of each sample in its "pred" field for each dataset file.</li>
|
76 |
<li><strong>Prepare your submission:</strong> Send your results to us (see the following Contact Information). Please clearly describe your model details and experimental settings, including the inference strategy used (zero-shot, few-shot, or CoT), and note any additional relevant configurations if applicable.</li>
|
@@ -92,7 +91,7 @@ If you have clinical text datasets that you would like to share for broader expl
|
|
92 |
</ul>
|
93 |
|
94 |
<h2>π¬ Contact Information</h2>
|
95 |
-
If you have any questions about BRIDGE or the leaderboard, feel free to reach out
|
96 |
<ul>
|
97 |
<li><strong>Leaderboard Managers</strong>: Jiageng Wu ([email protected]), Kevin Xie ([email protected])</li>
|
98 |
<li><strong>Benchmark Managers</strong>: Bowen Gu ([email protected]), Jiageng Wu ([email protected])</li>
|
@@ -100,8 +99,8 @@ If you have any questions about BRIDGE or the leaderboard, feel free to reach ou
|
|
100 |
</ul>
|
101 |
|
102 |
<h2>π Citation</h2>
|
103 |
-
<
|
104 |
-
|
105 |
title={BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text},
|
106 |
author={Wu, Jiageng and Gu, Bowen and Zhou, Ren and Xie, Kevin and Snyder, Doug and Jiang, Yixing and Carducci, Valentina and Wyss, Richard and Desai, Rishi J and Alsentzer, Emily and Celi, Leo Anthony and Rodman, Adam and Schneeweiss, Sebastian and Chen, Jonathan H. and Romero-Brufau, Santiago and Lin, Kueiyu Joshua and Yang, Jie},
|
107 |
year={2025},
|
|
|
22 |
|
23 |
</div>
|
24 |
|
|
|
25 |
<h2>π Background</h2>
|
26 |
<p>Recent advances in <strong>Large Language Models (LLMs)</strong> have demonstrated transformative potential in <strong>healthcare</strong>, yet concerns remain around their reliability and clinical validity across diverse clinical tasks, specialties, and languages. To support timely and trustworthy evaluation, building upon our <a href="https://ai.nejm.org/doi/full/10.1056/AIra2400012">systematic review</a> of global clinical text resources, we introduce <a href="https://arxiv.org/abs/2504.19467">BRIDGE</a>, <strong>a multilingual benchmark that comprises 87 real-world clinical text tasks spanning nine languages and more than one million samples</strong>. Furthermore, we construct this leaderboard of LLM in clinical text understanding by systematically evaluating <strong>52 state-of-the-art LLMs</strong> (by 2024/04/29).</p>
|
27 |
|
28 |
<div style="text-align: center;">
|
29 |
+
<img src="./dataset.png" style="width: 100%;" alt="BRIDGE benchmark graphic">
|
30 |
</div>
|
31 |
|
32 |
<h2>π BRIDGE Leaderboard</h2>
|
|
|
39 |
<p>In addition, BRIDGE offers multiple <strong>model filters</strong> and <strong>task filters</strong> to enable users to explore LLM performance across <strong>different clinical contexts</strong>, empowering researchers and clinicians to make informed decisions and track model advancements over time.</p>
|
40 |
|
41 |
<div style="text-align: center;">
|
42 |
+
<img src="./model.png" style="width: 100%;" alt="BRIDGE benchmark graphic">
|
43 |
</div>
|
44 |
|
45 |
|
|
|
63 |
<h2>π οΈ How to Evaluate Your Model on BRIDGE ?</h2>
|
64 |
<h4>π Dataset Access</h4>
|
65 |
<p>All fully open-access datasets in BRIDGE are available in <a href="https://huggingface.co/datasets/YLab-Open/BRIDGE-Open">BRIDGE-Open</a>. To ensure fairness of this leaderboard, we publicly release the following data for each task:
|
66 |
+
Five completed samples serve as few-shot examples, and all testing samples with instruction and input information.</p>
|
67 |
|
68 |
<p>Due to privacy and security considerations of clinical data, regulated-access datasets can not be directly published. However, all detailed task descriptions and their corresponding data sources are available in our <a href="https://arxiv.org/abs/2504.19467">BRIDGE paper</a>.
|
69 |
Importantly, all 87 datasets have been verified to be either fully open-access or publicly accessible via reasonable request.</p>
|
70 |
|
71 |
<h4>π₯ Result Submission and Evaluation</h4>
|
72 |
+
<p>If you would like to submit your model results to BRIDGE and demonstrate its performance, please follow these steps:</p>
|
73 |
<ul>
|
74 |
<li><strong>Run local inference:</strong> Download the <a href="https://huggingface.co/datasets/YLab-Open/BRIDGE-Open">BRIDGE-Open</a> dataset and perform inference locally. Save the generated output of each sample in its "pred" field for each dataset file.</li>
|
75 |
<li><strong>Prepare your submission:</strong> Send your results to us (see the following Contact Information). Please clearly describe your model details and experimental settings, including the inference strategy used (zero-shot, few-shot, or CoT), and note any additional relevant configurations if applicable.</li>
|
|
|
91 |
</ul>
|
92 |
|
93 |
<h2>π¬ Contact Information</h2>
|
94 |
+
<p>If you have any questions about BRIDGE or the leaderboard, feel free to reach out!</p>
|
95 |
<ul>
|
96 |
<li><strong>Leaderboard Managers</strong>: Jiageng Wu ([email protected]), Kevin Xie ([email protected])</li>
|
97 |
<li><strong>Benchmark Managers</strong>: Bowen Gu ([email protected]), Jiageng Wu ([email protected])</li>
|
|
|
99 |
</ul>
|
100 |
|
101 |
<h2>π Citation</h2>
|
102 |
+
<p>If you find this leaderboard useful for your research and applications, please cite the following papers:</p>
|
103 |
+
<pre><code>@article{BRIDGE-benchmark,
|
104 |
title={BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text},
|
105 |
author={Wu, Jiageng and Gu, Bowen and Zhou, Ren and Xie, Kevin and Snyder, Doug and Jiang, Yixing and Carducci, Valentina and Wyss, Richard and Desai, Rishi J and Alsentzer, Emily and Celi, Leo Anthony and Rodman, Adam and Schneeweiss, Sebastian and Chen, Jonathan H. and Romero-Brufau, Santiago and Lin, Kueiyu Joshua and Yang, Jie},
|
106 |
year={2025},
|