Amarthya7 commited on
Commit
43850e2
Β·
verified Β·
1 Parent(s): 2655fdb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +109 -109
README.md CHANGED
@@ -1,109 +1,109 @@
1
- ---
2
- title: MediSync - Multi-Modal Medical Analysis System
3
- emoji: 🩺
4
- colorFrom: blue
5
- colorTo: purple
6
- sdk: gradio
7
- sdk_version: "3.41.0"
8
- app_file: app.py
9
- pinned: false
10
- ---
11
-
12
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
13
-
14
- # MediSync: Multi-Modal Medical Analysis System
15
-
16
- MediSync is an AI-powered healthcare solution that combines X-ray image analysis with patient report text processing to provide comprehensive medical insights.
17
-
18
- Introduction
19
- MediSync is a multi-modal AI system that combines X-ray image analysis with medical report text processing to provide comprehensive medical insights. By leveraging state-of-the-art deep learning models for both vision and language understanding, MediSync can:
20
-
21
- Analyze chest X-ray images to detect abnormalities
22
- Extract key clinical information from medical reports
23
- Fuse insights from both modalities for enhanced diagnosis support
24
- Provide comprehensive visualization of analysis results
25
- This AI system demonstrates the power of multi-modal fusion in the healthcare domain, where integrating information from multiple sources can lead to more robust and accurate analyses.
26
-
27
- System Architecture
28
- MediSync follows a modular architecture with three main components:
29
-
30
- Image Analysis Module: Processes X-ray images using pre-trained vision models
31
- Text Analysis Module: Analyzes medical reports using NLP models
32
- Multimodal Fusion Module: Combines insights from both modalities
33
- The system uses the following high-level workflow:
34
-
35
- β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
36
- β”‚ X-ray Image β”‚
37
- β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
38
- β”‚
39
- β–Ό
40
- β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
41
- β”‚ Preprocessing │───▢│ Image Analysis │───▢│ β”‚
42
- β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
43
- β”‚ Multimodal β”‚
44
- β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Fusion │───▢ Results
45
- β”‚ Medical Report │───▢│ Text Analysis │───▢│ β”‚
46
- β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
47
- β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
48
-
49
-
50
- ## Features
51
-
52
- - **X-ray Image Analysis**: Detects abnormalities in chest X-rays using pre-trained vision models from Hugging Face.
53
- - **Medical Report Processing**: Extracts key information from patient reports using NLP models.
54
- - **Multi-modal Integration**: Combines insights from both image and text data for more accurate diagnosis suggestions.
55
- - **User-friendly Interface**: Simple web interface for uploading images and reports.
56
-
57
- ## Project Structure
58
-
59
- ```
60
- mediSync/
61
- β”œβ”€β”€ app.py # Main application with Gradio interface
62
- β”œβ”€β”€ models/
63
- β”‚ β”œβ”€β”€ image_analyzer.py # X-ray image analysis module
64
- β”‚ β”œβ”€β”€ text_analyzer.py # Medical report text analysis module
65
- β”‚ └── multimodal_fusion.py # Fusion of image and text insights
66
- β”œβ”€β”€ utils/
67
- β”‚ β”œβ”€β”€ preprocessing.py # Data preprocessing utilities
68
- β”‚ └── visualization.py # Result visualization utilities
69
- β”œβ”€β”€ data/
70
- β”‚ └── sample/ # Sample data for testing
71
- └── tests/ # Unit tests
72
- ```
73
-
74
- ## Setup Instructions
75
-
76
- 1. Clone this repository:
77
- ```bash
78
- git clone [repository-url]
79
- cd MediSync
80
- ```
81
-
82
- 2. Install dependencies:
83
- ```bash
84
- pip install -r requirements.txt
85
- ```
86
-
87
- 3. Run the application:
88
- ```bash
89
- python app.py
90
- ```
91
-
92
- 4. Access the web interface at `http://localhost:7860`
93
-
94
- ## Models Used
95
-
96
- - **X-ray Analysis**: facebook/deit-base-patch16-224-medical-cxr
97
- - **Medical Text Analysis**: medicalai/ClinicalBERT
98
- - **Additional Support Models**: Medical question answering and entity recognition models
99
-
100
- ## Use Cases
101
-
102
- - Preliminary screening of chest X-rays
103
- - Cross-validation of radiologist reports
104
- - Educational tool for medical students
105
- - Research tool for studying correlation between visual findings and written reports
106
-
107
- ## Note
108
-
109
- This system is designed as a support tool and should not replace professional medical diagnosis. Always consult with healthcare professionals for medical decisions.
 
1
+ ---
2
+ title: MediSync - Multi-Modal Medical Analysis System
3
+ emoji: 🩺
4
+ colorFrom: blue
5
+ colorTo: purple
6
+ sdk: gradio
7
+ sdk_version: 5.20.1
8
+ app_file: app.py
9
+ pinned: false
10
+ ---
11
+
12
+ Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
13
+
14
+ # MediSync: Multi-Modal Medical Analysis System
15
+
16
+ MediSync is an AI-powered healthcare solution that combines X-ray image analysis with patient report text processing to provide comprehensive medical insights.
17
+
18
+ Introduction
19
+ MediSync is a multi-modal AI system that combines X-ray image analysis with medical report text processing to provide comprehensive medical insights. By leveraging state-of-the-art deep learning models for both vision and language understanding, MediSync can:
20
+
21
+ Analyze chest X-ray images to detect abnormalities
22
+ Extract key clinical information from medical reports
23
+ Fuse insights from both modalities for enhanced diagnosis support
24
+ Provide comprehensive visualization of analysis results
25
+ This AI system demonstrates the power of multi-modal fusion in the healthcare domain, where integrating information from multiple sources can lead to more robust and accurate analyses.
26
+
27
+ System Architecture
28
+ MediSync follows a modular architecture with three main components:
29
+
30
+ Image Analysis Module: Processes X-ray images using pre-trained vision models
31
+ Text Analysis Module: Analyzes medical reports using NLP models
32
+ Multimodal Fusion Module: Combines insights from both modalities
33
+ The system uses the following high-level workflow:
34
+
35
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
36
+ β”‚ X-ray Image β”‚
37
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
38
+ β”‚
39
+ β–Ό
40
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
41
+ β”‚ Preprocessing │───▢│ Image Analysis │───▢│ β”‚
42
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
43
+ β”‚ Multimodal β”‚
44
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Fusion │───▢ Results
45
+ β”‚ Medical Report │───▢│ Text Analysis │───▢│ β”‚
46
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
47
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
48
+
49
+
50
+ ## Features
51
+
52
+ - **X-ray Image Analysis**: Detects abnormalities in chest X-rays using pre-trained vision models from Hugging Face.
53
+ - **Medical Report Processing**: Extracts key information from patient reports using NLP models.
54
+ - **Multi-modal Integration**: Combines insights from both image and text data for more accurate diagnosis suggestions.
55
+ - **User-friendly Interface**: Simple web interface for uploading images and reports.
56
+
57
+ ## Project Structure
58
+
59
+ ```
60
+ mediSync/
61
+ β”œβ”€β”€ app.py # Main application with Gradio interface
62
+ β”œβ”€β”€ models/
63
+ β”‚ β”œβ”€β”€ image_analyzer.py # X-ray image analysis module
64
+ β”‚ β”œβ”€β”€ text_analyzer.py # Medical report text analysis module
65
+ β”‚ └── multimodal_fusion.py # Fusion of image and text insights
66
+ β”œβ”€β”€ utils/
67
+ β”‚ β”œβ”€β”€ preprocessing.py # Data preprocessing utilities
68
+ β”‚ └── visualization.py # Result visualization utilities
69
+ β”œβ”€β”€ data/
70
+ β”‚ └── sample/ # Sample data for testing
71
+ └── tests/ # Unit tests
72
+ ```
73
+
74
+ ## Setup Instructions
75
+
76
+ 1. Clone this repository:
77
+ ```bash
78
+ git clone [repository-url]
79
+ cd MediSync
80
+ ```
81
+
82
+ 2. Install dependencies:
83
+ ```bash
84
+ pip install -r requirements.txt
85
+ ```
86
+
87
+ 3. Run the application:
88
+ ```bash
89
+ python app.py
90
+ ```
91
+
92
+ 4. Access the web interface at `http://localhost:7860`
93
+
94
+ ## Models Used
95
+
96
+ - **X-ray Analysis**: facebook/deit-base-patch16-224-medical-cxr
97
+ - **Medical Text Analysis**: medicalai/ClinicalBERT
98
+ - **Additional Support Models**: Medical question answering and entity recognition models
99
+
100
+ ## Use Cases
101
+
102
+ - Preliminary screening of chest X-rays
103
+ - Cross-validation of radiologist reports
104
+ - Educational tool for medical students
105
+ - Research tool for studying correlation between visual findings and written reports
106
+
107
+ ## Note
108
+
109
+ This system is designed as a support tool and should not replace professional medical diagnosis. Always consult with healthcare professionals for medical decisions.