dataset/scripts/commit_message_20250726_special_terms.txt

refactor: migrate special terms to JSON configuration

BREAKING CHANGE: Move hardcoded special terms mapping to external JSON files

1. Create New Configuration Files:
- Add special_terms_emergency.json
  - Organize emergency terms by categories (cardiac, respiratory, etc.)
  - Include all existing mappings with standardized structure
- Add special_terms_treatment.json
  - Organize treatment terms by categories (imaging, medications, etc.)
  - Maintain all existing term variants

2. Update Processing Scripts:
- Modify 01_filter_emergency_opt.py:
  - Load terms from JSON configuration
  - Add term standardization
  - Implement deduplication
  - Preserve category information
- Modify 02_filter_treatment_opt.py:
  - Similar updates for treatment terms
  - Maintain consistent processing logic

3. New Features:
- Term standardization: Convert variants to standard form
- Deduplication: Remove repeated terms while preserving order
- Category-aware: Support for term categorization
- Improved maintainability: Configuration separated from code

4. Technical Details:
- Use pathlib for file path handling
- JSON structure supports hierarchical organization
- Maintain backward compatibility
- Add type hints for better code clarity

Testing:
- Verify JSON format
- Confirm all mappings migrated correctly
- Check term standardization
- Validate deduplication logic