Spaces:
Sleeping
Sleeping
| title: LeRobot Arena - AI Inference Server | |
| emoji: 🤖 | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 7860 | |
| suggested_hardware: t4-small | |
| suggested_storage: medium | |
| short_description: Real-time ACT model inference server for robot control | |
| tags: | |
| - robotics | |
| - ai | |
| - inference | |
| - control | |
| - act-model | |
| - transformer | |
| - real-time | |
| - gradio | |
| - fastapi | |
| - computer-vision | |
| pinned: false | |
| fullWidth: true | |
| # Inference Server | |
| 🤖 **Real-time ACT Model Inference Server for Robot Control** | |
| This server provides ACT (Action Chunking Transformer) model inference for robotics applications using the transport server communication system. It includes a user-friendly Gradio web interface for easy setup and management. | |
| ## ✨ Features | |
| - **Real-time AI Inference**: Run ACT models for robot control at 20Hz control frequency | |
| - **Multi-Camera Support**: Handle multiple camera streams with different names | |
| - **Web Interface**: User-friendly Gradio UI for setup and monitoring | |
| - **Session Management**: Create, start, stop, and monitor inference sessions | |
| - **Automatic Timeout**: Sessions automatically cleanup after 10 minutes of inactivity | |
| - **Debug Tools**: Built-in debugging and monitoring endpoints | |
| - **Flexible Configuration**: Support for custom model paths, camera configurations | |
| - **No External Dependencies**: Direct Python execution without subprocess calls | |
| ## 🚀 Quick Start | |
| ### Prerequisites | |
| - Python 3.12+ | |
| - UV package manager (recommended) | |
| - Trained ACT model | |
| - Transport server running | |
| ### 1. Installation | |
| ```bash | |
| cd backend/ai-server | |
| # Install dependencies using uv (recommended) | |
| uv sync | |
| # Or using pip | |
| pip install -e . | |
| ``` | |
| ### 2. Launch the Application | |
| #### **🚀 Simple Integrated Mode (Recommended)** | |
| ```bash | |
| # Everything runs in one process - no subprocess issues! | |
| python launch_simple.py | |
| # Or using the CLI | |
| python -m inference_server.cli --simple | |
| ``` | |
| This will: | |
| - Run everything on `http://localhost:7860` | |
| - Direct session management (no HTTP API calls) | |
| - No external subprocess dependencies | |
| - Most robust and simple deployment! | |
| #### **🔧 Development Mode (Separate Processes)** | |
| ```bash | |
| # Traditional approach with separate server and UI | |
| python -m inference_server.cli | |
| ``` | |
| This will: | |
| - Start the AI server on `http://localhost:8001` | |
| - Launch the Gradio UI on `http://localhost:7860` | |
| - Better for development and debugging | |
| ### 3. Using the Web Interface | |
| 1. **Check Server Status**: The interface will automatically check if the AI server is running | |
| 2. **Configure Your Robot**: Enter your model path and camera setup | |
| 3. **Create & Start Session**: Click the button to set up AI control | |
| 4. **Monitor Performance**: Use the status panel to monitor inference | |
| ## 🎯 Workflow Guide | |
| ### Step 1: AI Server | |
| - The server status will be displayed at the top | |
| - Click "Start Server" if it's not already running | |
| - Use "Check Status" to verify connectivity | |
| ### Step 2: Set Up Robot AI | |
| - **Session Name**: Give your session a unique name (e.g., "my-robot-01") | |
| - **AI Model Path**: Path to your trained ACT model (e.g., "./checkpoints/act_so101_beyond") | |
| - **Camera Names**: Comma-separated list of camera names (e.g., "front,wrist,overhead") | |
| - Click "Create & Start AI Control" to begin | |
| ### Step 3: Control Session | |
| - The session ID will be auto-filled after creation | |
| - Use Start/Stop buttons to control inference | |
| - Click "Status" to see detailed performance metrics | |
| ## 🛠️ Advanced Usage | |
| ### CLI Options | |
| ```bash | |
| # Simple integrated mode (recommended) | |
| python -m inference_server.cli --simple | |
| # Development mode (separate processes) | |
| python -m inference_server.cli | |
| # Launch only the server | |
| python -m inference_server.cli --server-only | |
| # Launch only the UI (server must be running separately) | |
| python -m inference_server.cli --ui-only | |
| # Custom ports | |
| python -m inference_server.cli --server-port 8002 --ui-port 7861 | |
| # Enable public sharing | |
| python -m inference_server.cli --share | |
| # For deployment (recommended) | |
| python -m inference_server.cli --simple --host 0.0.0.0 --share | |
| ``` | |
| ### API Endpoints | |
| The server provides a REST API for programmatic access: | |
| - `GET /health` - Server health check | |
| - `POST /sessions` - Create new session | |
| - `GET /sessions` - List all sessions | |
| - `GET /sessions/{id}` - Get session details | |
| - `POST /sessions/{id}/start` - Start inference | |
| - `POST /sessions/{id}/stop` - Stop inference | |
| - `POST /sessions/{id}/restart` - Restart inference | |
| - `DELETE /sessions/{id}` - Delete session | |
| #### Debug Endpoints | |
| - `GET /debug/system` - System information (CPU, memory, GPU) | |
| - `GET /debug/sessions/{id}/queue` - Action queue details | |
| - `POST /debug/sessions/{id}/reset` - Reset session state | |
| ### Configuration | |
| #### Joint Value Convention | |
| - All joint inputs/outputs use **NORMALIZED VALUES** | |
| - Most joints: -100 to +100 (RANGE_M100_100) | |
| - Gripper: 0 to 100 (RANGE_0_100) | |
| - This matches the training data format exactly | |
| #### Camera Support | |
| - Supports arbitrary number of camera streams | |
| - Each camera has a unique name (e.g., "front", "wrist", "overhead") | |
| - All camera streams are synchronized for inference | |
| - Images expected in RGB format, uint8 [0-255] | |
| ## 📊 Monitoring | |
| ### Session Status Indicators | |
| - 🟢 **Running**: Inference active and processing | |
| - 🟡 **Ready**: Session created but inference not started | |
| - 🔴 **Stopped**: Inference stopped | |
| - 🟠 **Initializing**: Session being set up | |
| ### Smart Session Control | |
| The UI now provides intelligent feedback: | |
| - ℹ️ **Already Running**: When trying to start a running session | |
| - ℹ️ **Already Stopped**: When trying to stop a stopped session | |
| - 💡 **Smart Suggestions**: Context-aware tips based on current status | |
| ### Performance Metrics | |
| - **Inferences**: Total number of model inferences performed | |
| - **Commands Sent**: Joint commands sent to robot | |
| - **Queue Length**: Actions waiting in the queue | |
| - **Errors**: Number of errors encountered | |
| - **Data Flow**: Images and joint states received | |
| ## 🐳 Docker Usage | |
| ### Build the Image | |
| ```bash | |
| cd services/inference-server | |
| docker build -t inference-server . | |
| ``` | |
| ### Run the Container | |
| ```bash | |
| # Basic usage | |
| docker run -p 7860:7860 inference-server | |
| # With environment variables | |
| docker run -p 7860:7860 \ | |
| -e DEFAULT_ARENA_SERVER_URL=http://your-server.com \ | |
| -e DEFAULT_MODEL_PATH=./checkpoints/your-model \ | |
| inference-server | |
| # With GPU support | |
| docker run --gpus all -p 7860:7860 inference-server | |
| ``` | |
| ## 🔧 Troubleshooting | |
| ### Common Issues | |
| 1. **Server Won't Start** | |
| - Check if port 8001 is available | |
| - Verify model path exists and is accessible | |
| - Check dependencies are installed correctly | |
| 2. **Session Creation Fails** | |
| - Verify model path is correct | |
| - Check Arena server is running on specified URL | |
| - Ensure camera names match your robot configuration | |
| 3. **Poor Performance** | |
| - Monitor system resources in the debug panel | |
| - Check if GPU is being used for inference | |
| - Verify control/inference frequency settings | |
| 4. **Connection Issues** | |
| - Verify Arena server URL is correct | |
| - Check network connectivity | |
| - Ensure workspace/room IDs are valid | |
| ### Debug Mode | |
| Enable debug mode for detailed logging: | |
| ```bash | |
| uv run python -m lerobot_arena_ai_server.cli --debug | |
| ``` | |
| ### System Requirements | |
| - **CPU**: Multi-core recommended for 30Hz control | |
| - **Memory**: 8GB+ RAM recommended | |
| - **GPU**: CUDA-compatible GPU for fast inference (optional but recommended) | |
| - **Network**: Stable connection to Arena server | |
| ## 📚 Architecture | |
| ### Integrated Mode (Recommended) | |
| ``` | |
| ┌─────────────────────────────────────┐ ┌─────────────────┐ | |
| │ Single Application │ │ LeRobot Arena │ | |
| │ ┌─────────────┐ ┌─────────────┐ │◄──►│ (Port 8000) │ | |
| │ │ Gradio UI │ │ AI Server │ │ └─────────────────┘ | |
| │ │ (/) │ │ (/api/*) │ │ │ | |
| │ └─────────────┘ └─────────────┘ │ │ | |
| │ (Port 7860) │ Robot/Cameras | |
| └─────────────────────────────────────┘ | |
| │ | |
| Web Browser | |
| ``` | |
| ### Development Mode | |
| ``` | |
| ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ | |
| │ Gradio UI │ │ AI Server │ │ LeRobot Arena │ | |
| │ (Port 7860) │◄──►│ (Port 8001) │◄──►│ (Port 8000) │ | |
| └─────────────────┘ └─────────────────┘ └─────────────────┘ | |
| │ │ │ | |
| │ │ │ | |
| Web Browser ACT Model Robot/Cameras | |
| Inference | |
| ``` | |
| ### Data Flow | |
| 1. **Camera Data**: Robot cameras → Arena → AI Server | |
| 2. **Joint State**: Robot joints → Arena → AI Server | |
| 3. **AI Inference**: Images + Joint State → ACT Model → Actions | |
| 4. **Control Commands**: Actions → Arena → Robot | |
| ### Session Lifecycle | |
| 1. **Create**: Set up rooms in Arena, load ACT model | |
| 2. **Start**: Begin inference loop (3Hz) and control loop (30Hz) | |
| 3. **Running**: Process camera/joint data, generate actions | |
| 4. **Stop**: Pause inference, maintain connections | |
| 5. **Delete**: Clean up resources, disconnect from Arena | |
| ## 🤝 Contributing | |
| 1. Follow the existing code style | |
| 2. Add tests for new features | |
| 3. Update documentation | |
| 4. Submit pull requests | |
| ## 📄 License | |
| This project follows the same license as the parent LeRobot Arena project. | |
| --- | |
| For more information, see the [LeRobot Arena documentation](../../README.md). | |