Spaces:
Running
Running
| title: Open GAMMA - 'GAMJA' V2 | |
| emoji: ๐ฅ | |
| colorFrom: blue | |
| colorTo: yellow | |
| sdk: gradio | |
| sdk_version: 5.34.2 | |
| app_file: app.py | |
| pinned: false | |
| short_description: AI generates PPT with diagrams and images from given topics | |
| models: | |
| - VIDraft/Gemma-3-R1984-27B | |
| - VIDraft/Gemma-3-R1984-12B | |
| - VIDraft/Gemma-3-R1984-4B | |
| FACTS Grounding Leaderboard - Medical AI Evaluation | |
| ๐ฅ Overview | |
| FACTS Grounding is an AI reliability evaluation system developed by Google DeepMind that verifies whether AI responses are grounded solely in provided documents. This evaluation is particularly crucial in healthcare, where inaccurate information can be life-threatening. | |
| ๐ฏ Key Features | |
| Evaluation Methodology | |
| Long Medical Document Input (~32,000 tokens โ 40 A4 pages) | |
| AI Response Generation Based on Documents | |
| Dual-Criteria Assessment | |
| โ Quality Check: Does the AI accurately understand the question? | |
| โ Grounding Check: Are all responses based on the provided documents? | |
| Medical-Focused Version | |
| 236 medical cases selected from 860 total problems | |
| Strict evaluation criteria reflecting healthcare field requirements | |
| ๐ Current Leaderboard Rankings (June 5, 2025) | |
| Overall Score TOP 5(https://huggingface.co/spaces/MaziyarPanahi/FACTS-Leaderboard) | |
| 1. deepseek-ai/DeepSeek-R1-Distill-Qwen-14B | |
| 2. VIDraft/Gemma-3-R1984-27B | |
| 3. meta-llama/Llama-3.3-70B-Instruct | |
| 4. Qwen/Qwen3-30B-A3B | |
| 5. Qwen/Qwen3-4B | |
| ๐ก Why FACTS Matters for Medical AI | |
| 1. Patient Safety Assurance | |
| General AI Error: "The capital of France is London" โ Simple mistake | |
| Medical AI Error: "This medication is safe for pregnant women" โ Life-threatening! | |
| 2. Building Healthcare Provider Trust | |
| Ensures all responses are grounded in medical literature | |
| Transparent scoring enables AI system reliability verification | |
| 3. Regulatory Compliance & Standardization | |
| Emerging as a global medical AI standard | |
| Potential reference for FDA, CE, and other regulatory approvals | |
| ๐ Advancing Global Medical AI | |
| Core Values of FACTS | |
| Accuracy: Provides only evidence-based medical responses | |
| Transparency: All evaluation processes and data are public | |
| Accessibility: Global participation in evaluation | |
| Practicality: Reflects real-world healthcare scenarios | |
| Medical-Specific Features | |
| Understanding complex medical terminology and drug interactions | |
| Recognition of diverse symptom descriptions | |
| Verification of clinical guideline and protocol adherence | |
| Consideration of medical ethics and patient privacy | |
| ๐ The Future of Medical AI | |
| FACTS Grounding is establishing itself as the 'quality certification system' for medical AI. | |
| Expected Impact | |
| Global Standardization: Unified criteria for worldwide medical AI evaluation | |
| Quality Improvement: Continuous model enhancement through benchmarking | |
| Accelerated Clinical Adoption: Rapid deployment of validated AI systems | |
| Enhanced Patient Care: More accurate and safer healthcare services | |
| ๐ค How to Participate | |
| The leaderboard operates with complete transparency, allowing anyone to submit and evaluate their models. | |
| Download the FACTS Grounding dataset | |
| Train and optimize your model | |
| Submit evaluation results | |
| Check rankings on the leaderboard | |
| ๐ Utilizing Evaluation Results | |
| Healthcare Institutions | |
| Objective performance metrics for AI adoption decisions | |
| Selection criteria among multiple AI systems | |
| AI Developers | |
| Model performance benchmarking and improvement direction | |
| Marketing and reliability verification materials | |
| Regulatory Bodies | |
| Reference material for AI medical device approval processes | |
| Supplementary indicator for safety assessments | |
| ๐ Key Takeaways | |
| FACTS Grounding = Global standard for verifying AI medical accuracy | |
| Medical-Specific Evaluation = Testing based on 236 real healthcare scenarios | |
| Transparent Operation = Open system accessible to all participants | |
| Real-World Impact = Promoting safer and more reliable medical AI development | |
| "What cannot be measured cannot be improved" - FACTS Grounding objectively measures medical AI reliability, contributing to global healthcare AI advancement. | |
| We look forward to more research teams and developers joining this challenge to create better AI for human health! ๐ |