.github/instructions/copilot-instructions.md

Copilot Instructions for OpenSight-Deepfake-Detection-Models-Playground (2025)

Project Overview

OpenSight is a modular, agent-driven toolkit for deepfake detection and digital forensics. It leverages an ensemble of models, advanced forensic tools, and smart agents for explainable, extensible, and robust detection. The system is optimized for integration with vision LLMs and multimodal AI agents, and supports logging to Hugging Face datasets.

Key Technologies

• Gradio: Main UI and API server (app.py).
• Hugging Face Hub: Model and dataset management, logging, and deployment.
• Git LFS: Required for storing binary files (e.g., PNGs) in the repo. See .gitattributes for tracked types.
• Agents: Smart agents for ensemble monitoring, weight optimization, system health, context intelligence, and anomaly detection (agents/).
• Forensic Tools: Modular forensic techniques in forensics/ (ELA, gradient, minmax, bitplane, wavelet, exif, etc.).
• Model Management: Models are registered and managed in utils/registry.py, loaded via ONNX/Hugging Face/Gradio API.
• Utilities: Logging, augmentation, health checks, and more in utils/.

Data Flow & Prediction Pipeline

1. Image Preprocessing: Normalize to PIL RGB, optional augmentation (rotate, noise, sharpen).
2. Agent Initialization: Monitoring, optimization, and context agents setup.
3. Model Inference: Each model predicts independently; results tracked by agents.
4. Consensus: Model weights dynamically adjusted based on context and agent feedback.
5. Forensic Analysis: Multiple forensic tools run in parallel; outputs analyzed for anomalies.
6. Logging: All results (images, predictions, agent data) are logged to Hugging Face datasets (hf_logger.py).

Developer Workflow

• Run the App:

  python app.py
  

• Dependencies: See requirements.txt (gradio, PIL, numpy, torch, huggingface_hub, etc.).
• Binary Files: All PNGs and other binaries must be tracked with Git LFS. If you see a push error, run:

  git lfs track "*.png"
  git add .gitattributes
  git add <yourfile.png>
  git commit -m "Track PNG files with Git LFS"
  git push origin main
  

  If the file is already in history, use:

  git lfs migrate import --include="*.png"
  git push origin main
  

• Extending Forensics/Agents: Add new tools in forensics/, new agents in agents/, and register in the main app.
• Testing: Unit tests for agents/models are planned (see roadmap in README.md).

Project Patterns & Conventions

• Forensic Tool Naming: Use tool_* or descriptive names (e.g., tool_ela, tool_waveletnoise).
• Agent Classes: Use *Agent suffix (e.g., EnsembleMonitorAgent).
• API Exposure: Functions are exposed for LLM/MCP calls with clear parameter/return docs (see README.md).
• Consensus Logic: Model weights are contextually adjusted; see agent code for dynamic weighting.
• Logging: Inference data is logged to Hugging Face datasets for analysis.

Integration & Extension

• Add Models: Update ensemble logic and register in agent system.
• Add Forensic Tools: Implement in forensics/, expose via main app, and document parameters/returns.
• LLM/Multimodal Integration: Hybrid input strategies (e.g., ELA+RGB, metadata+image) are encouraged; see README.md for details.

References

• Forensic Techniques: See forensics/ for implementation details.
• Agent Logic: See agents/ for smart agent patterns.
• API & Usage: README.md documents all callable functions, parameters, and return types.
• Roadmap: Ongoing and planned features are tracked in README.md.

---

Tip: Always check agent logic and consensus weighting when extending or debugging, as these are central to system behavior. For binary file push errors, ensure Git LFS is set up and files are tracked correctly.