feat: move teleoperators out of teleoperate into their own classes (keyboard / direct)

docs/conventions.md

@@ -20,15 +20,19 @@
 - **Direct Library Usage**: End users call library functions directly (e.g., `calibrate()`, `teleoperate()`) - avoid unnecessary abstraction layers
 - **Comments**: Write about the functionality, not what you did. We only need to know what the code is doing to make it more easy to understand, not a history of the changes
 - **No Reference Comments**: Never write comments like "same pattern as calibrate.ts", "matches Node.js", "copied from X", etc. Comments should explain what the code does, not where it came from or what it's similar to
-- **No Change Explanation Comments**: NEVER add comments explaining what you changed or why you removed something. Examples of forbidden comments:
+- **ABSOLUTELY FORBIDDEN: No Change Explanation Comments**: NEVER EVER add comments explaining what you changed, what's new, what's updated, or why you removed something. This is a standard library - there is no "new API", no "old way", no "updated approach". Just code that does what it does. Examples of STRICTLY FORBIDDEN comments:
 
+  - `// Create teleoperation process using new API`
+  - `// Updated API to match Node.js`
+  - `// New extensible architecture`
+  - `// Breaking change from old API`
   - `// React import not needed with modern JSX transform`
   - `// Removed unused import`
   - `// Note: Connection manager registration removed for build compatibility`
   - `// Card components not needed in this file`
   - `// SerialPortRequestOptions unused in current implementation`
 
-  Just make the change cleanly without explaining it in comments.
+  **VIOLATION OF THIS RULE IS NOT TOLERATED.** Just make the change cleanly without explaining it in comments. If you catch yourself writing "new", "old", "updated", "changed", "removed", "added" in a comment - DELETE IT.
 
 ## Project Goals
 

docs/planning/005_extensible_teleoperate.md

@@ -0,0 +1,606 @@
+# User Story 005: Extensible Teleoperation Architecture
+
+## Story
+
+**As a** robotics developer building teleoperation systems with various input devices  
+**I want** to use different teleoperators (keyboard, leader arms, joysticks, VR controllers) to control my robot arms  
+**So that** I can choose the most appropriate control method for my application without being locked into keyboard-only control
+
+## Background
+
+The current Web teleoperation implementation has keyboard controls hardcoded into the `WebTeleoperationController`, making it impossible to use other input devices like leader arms, joysticks, or future control methods. Meanwhile, the Node.js implementation already has the correct extensible architecture with pluggable teleoperators.
+
+This architectural inconsistency creates several problems:
+
+- **Limited Extensibility**: Web users cannot use leader arms or other advanced teleoperators
+- **API Mismatch**: Web API `teleoperate(robotConnection, options)` differs from Node.js API `teleoperate(config: TeleoperateConfig)`
+- **Hardcoded Assumptions**: Keyboard logic is baked into the core teleoperation controller
+- **Future Limitations**: Adding new input devices requires core architecture changes
+
+The Python lerobot and Node.js lerobot.js both follow a proper separation where:
+
+- **Robots** handle motor communication and hardware control
+- **Teleoperators** handle input device reading and command generation
+- **Teleoperation orchestrator** connects teleoperators to robots
+
+We need to refactor the Web implementation to match this proven architecture, enabling seamless extension to leader arms, joysticks, VR controllers, and other future input devices.
+
+## Acceptance Criteria
+
+### Core Functionality
+
+- [ ] **Pluggable Teleoperators**: Support multiple teleoperator types (keyboard, leader arms, etc.)
+- [ ] **API Alignment**: Web API matches Node.js: `teleoperate(config: TeleoperateConfig)`
+- [ ] **Keyboard Teleoperator**: Extract existing keyboard logic into dedicated teleoperator class
+- [ ] **Teleoperator Abstraction**: Base interface that all teleoperators implement
+- [ ] **Type Safety**: Each teleoperator type has its own configuration interface
+- [ ] **State Management**: Maintain current TeleoperationState approach with teleoperator-specific extensions
+
+### User Experience
+
+- [ ] **Breaking Change**: Clean API break - no backward compatibility with old hardcoded approach
+- [ ] **Consistent Interface**: Same teleoperation process object regardless of teleoperator type
+- [ ] **Future-Ready**: Easy addition of new teleoperator types without core changes
+- [ ] **Error Handling**: Clear error messages for unsupported or misconfigured teleoperators
+
+### Technical Requirements
+
+- [ ] **Architecture Separation**: Clear separation between robot control and teleoperator input
+- [ ] **Web Implementation**: Focus on Web platform to match Node.js architecture
+- [ ] **TypeScript**: Fully typed with union types for teleoperator configurations
+- [ ] **No Code Duplication**: Reuse existing motor communication and robot control logic
+- [ ] **Configuration-Driven**: Teleoperator behavior determined by config, not hardcoded logic
+
+## Expected User Flow
+
+### Keyboard Teleoperation (Current Functionality Preserved)
+
+```typescript
+import { teleoperate } from "lerobot/web/teleoperate";
+
+// New API - explicitly specify teleoperator
+const teleoperationProcess = await teleoperate({
+  robot: {
+    type: "so100_follower",
+    port: selectedPort,
+    // ... existing robot config
+  },
+  teleop: {
+    type: "keyboard",
+    stepSize: 25,
+    updateRate: 60,
+  },
+  calibrationData: loadedCalibrationData,
+  onStateUpdate: (state) => {
+    // Update UI with current state
+    console.log("Robot positions:", state.motorConfigs);
+    console.log("Active keys:", state.keyStates);
+  },
+});
+
+// Same process interface as before
+teleoperationProcess.start();
+teleoperationProcess.updateKeyState("ArrowUp", true);
+teleoperationProcess.stop();
+```
+
+### Leader Arm Teleoperation (Future)
+
+```typescript
+// Future leader arm teleoperator
+const teleoperationProcess = await teleoperate({
+  robot: {
+    type: "so100_follower",
+    port: followerPort,
+  },
+  teleop: {
+    type: "so100_leader",
+    port: leaderPort,
+    calibrationData: leaderCalibration,
+    positionSmoothing: true,
+  },
+  calibrationData: followerCalibration,
+  onStateUpdate: (state) => {
+    console.log("Follower positions:", state.motorConfigs);
+    console.log("Leader positions:", state.leaderPositions);
+  },
+});
+
+teleoperationProcess.start(); // Reads from leader, writes to follower
+```
+
+### Direct Motor Control
+
+```typescript
+import { teleoperate, DirectTeleoperator } from "@lerobot/web";
+
+// Direct motor control for programmatic use (sliders, API calls)
+const teleoperationProcess = await teleoperate({
+  robot: {
+    type: "so100_follower",
+    port: robotPort,
+  },
+  teleop: {
+    type: "direct",
+  },
+  calibrationData: calibrationData,
+  onStateUpdate: (state) => {
+    console.log("Robot positions:", state.motorConfigs);
+  },
+});
+
+// Control motors programmatically
+teleoperationProcess.start();
+const directTeleoperator =
+  teleoperationProcess.teleoperator as DirectTeleoperator;
+await directTeleoperator.moveMotor("shoulder_pan", 2500);
+await directTeleoperator.setMotorPositions({
+  shoulder_lift: 1800,
+  gripper: 3000,
+});
+```
+
+### Joystick Teleoperation (Future)
+
+```typescript
+// Future joystick teleoperator
+const teleoperationProcess = await teleoperate({
+  robot: {
+    type: "so100_follower",
+    port: robotPort,
+  },
+  teleop: {
+    type: "gamepad",
+    controllerIndex: 0,
+    axisMapping: {
+      leftStick: "shoulder_pan",
+      rightStick: "shoulder_lift",
+      triggers: "gripper",
+    },
+  },
+  calibrationData: calibrationData,
+});
+```
+
+### Error Handling
+
+```typescript
+// Unsupported teleoperator type
+try {
+  await teleoperate({
+    robot: { type: "so100_follower", port: "COM4" },
+    teleop: { type: "unsupported_device" },
+  });
+} catch (error) {
+  console.error("Error: Unsupported teleoperator type: unsupported_device");
+  console.error("Supported types: keyboard, so100_leader");
+}
+```
+
+## Implementation Details
+
+### File Structure
+
+```
+packages/web/src/
+├── teleoperate.ts                    # Updated main API (breaking change)
+├── teleoperators/
+│   ├── base-teleoperator.ts          # Base teleoperator interface
+│   ├── keyboard-teleoperator.ts      # Extracted keyboard logic
+│   ├── index.ts                      # Barrel exports
+│   └── [future]
+│       ├── leader-arm-teleoperator.ts
+│       ├── gamepad-teleoperator.ts
+│       └── vr-teleoperator.ts
+└── types/
+    └── teleoperation.ts              # Updated with teleoperator config types
+```
+
+### Key Dependencies
+
+#### No New Dependencies
+
+- **Existing**: Reuse all current Web dependencies (Web Serial API, motor communication utils)
+- **Architecture Only**: This is purely an architectural refactor - no new external dependencies
+
+### Core Functions to Implement
+
+#### Updated API (Breaking Change)
+
+```typescript
+// teleoperate.ts - New API matching Node.js
+interface TeleoperateConfig {
+  robot: RobotConnection;
+  teleop: TeleoperatorConfig;
+  calibrationData?: { [motorName: string]: any };
+  onStateUpdate?: (state: TeleoperationState) => void;
+}
+
+// Union type for all teleoperator configurations
+type TeleoperatorConfig =
+  | KeyboardTeleoperatorConfig
+  | LeaderArmTeleoperatorConfig
+  | GamepadTeleoperatorConfig;
+
+// Main function - breaking change from old API
+async function teleoperate(
+  config: TeleoperateConfig
+): Promise<TeleoperationProcess>;
+```
+
+#### Teleoperator Configuration Types
+
+```typescript
+// Base interface all teleoperators implement
+interface BaseTeleoperatorConfig {
+  type: string;
+}
+
+// Keyboard teleoperator configuration
+interface KeyboardTeleoperatorConfig extends BaseTeleoperatorConfig {
+  type: "keyboard";
+  stepSize?: number; // Default: 25
+  updateRate?: number; // Default: 60 (FPS)
+  keyTimeout?: number; // Default: 10000ms
+}
+
+// Future: Leader arm teleoperator configuration
+interface LeaderArmTeleoperatorConfig extends BaseTeleoperatorConfig {
+  type: "so100_leader";
+  port: string;
+  calibrationData?: any;
+  positionSmoothing?: boolean;
+  scaleFactor?: number;
+}
+
+// Future: Gamepad teleoperator configuration
+interface GamepadTeleoperatorConfig extends BaseTeleoperatorConfig {
+  type: "gamepad";
+  controllerIndex?: number;
+  axisMapping?: { [axis: string]: string };
+  deadzone?: number;
+}
+```
+
+#### Base Teleoperator Interface
+
+```typescript
+// base-teleoperator.ts
+interface WebTeleoperator {
+  // Lifecycle management
+  initialize(robotConnection: RobotConnection): Promise<void>;
+  start(): void;
+  stop(): void;
+  disconnect(): Promise<void>;
+
+  // State management
+  getState(): TeleoperatorSpecificState;
+
+  // Robot interaction
+  onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void;
+}
+
+// Base class with common functionality
+abstract class BaseWebTeleoperator implements WebTeleoperator {
+  protected port: MotorCommunicationPort;
+  protected motorConfigs: MotorConfig[] = [];
+  protected isActive: boolean = false;
+  protected onStateUpdate?: (state: TeleoperationState) => void;
+
+  constructor(
+    port: MotorCommunicationPort,
+    motorConfigs: MotorConfig[],
+    onStateUpdate?: (state: TeleoperationState) => void
+  ) {
+    this.port = port;
+    this.motorConfigs = motorConfigs;
+    this.onStateUpdate = onStateUpdate;
+  }
+
+  abstract initialize(): Promise<void>;
+  abstract start(): void;
+  abstract stop(): void;
+  abstract getState(): TeleoperatorSpecificState;
+
+  async disconnect(): Promise<void> {
+    this.stop();
+  }
+
+  onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void {
+    this.motorConfigs = motorConfigs;
+  }
+}
+```
+
+#### Keyboard Teleoperator (Extracted Logic)
+
+```typescript
+// keyboard-teleoperator.ts - Extract from current WebTeleoperationController
+class KeyboardTeleoperator extends BaseWebTeleoperator {
+  private keyboardControls: { [key: string]: KeyboardControl } = {};
+  private updateInterval: NodeJS.Timeout | null = null;
+  private keyStates: {
+    [key: string]: { pressed: boolean; timestamp: number };
+  } = {};
+
+  // Configuration from KeyboardTeleoperatorConfig
+  private readonly stepSize: number;
+  private readonly updateRate: number;
+  private readonly keyTimeout: number;
+
+  constructor(
+    config: KeyboardTeleoperatorConfig,
+    port: MotorCommunicationPort,
+    motorConfigs: MotorConfig[],
+    keyboardControls: { [key: string]: KeyboardControl },
+    onStateUpdate?: (state: TeleoperationState) => void
+  ) {
+    super(port, motorConfigs, onStateUpdate);
+    this.keyboardControls = keyboardControls;
+
+    // Extract configuration
+    this.stepSize = config.stepSize ?? 25;
+    this.updateRate = config.updateRate ?? 60;
+    this.keyTimeout = config.keyTimeout ?? 10000;
+  }
+
+  async initialize(): Promise<void> {
+    // Move existing initialization logic here
+    for (const config of this.motorConfigs) {
+      const position = await readMotorPosition(this.port, config.id);
+      if (position !== null) {
+        config.currentPosition = position;
+      }
+    }
+  }
+
+  start(): void {
+    // Move existing start logic here
+    if (this.isActive) return;
+
+    this.isActive = true;
+    this.updateInterval = setInterval(() => {
+      this.updateMotorPositions();
+    }, 1000 / this.updateRate);
+  }
+
+  stop(): void {
+    // Move existing stop logic here
+    if (!this.isActive) return;
+
+    this.isActive = false;
+    if (this.updateInterval) {
+      clearInterval(this.updateInterval);
+      this.updateInterval = null;
+    }
+    this.keyStates = {};
+
+    if (this.onStateUpdate) {
+      this.onStateUpdate(this.buildTeleoperationState());
+    }
+  }
+
+  updateKeyState(key: string, pressed: boolean): void {
+    this.keyStates[key] = { pressed, timestamp: Date.now() };
+  }
+
+  getState(): KeyboardTeleoperatorState {
+    return {
+      keyStates: { ...this.keyStates },
+    };
+  }
+
+  private buildTeleoperationState(): TeleoperationState {
+    return {
+      isActive: this.isActive,
+      motorConfigs: [...this.motorConfigs],
+      lastUpdate: Date.now(),
+      keyStates: { ...this.keyStates },
+    };
+  }
+
+  private updateMotorPositions(): void {
+    // Move existing updateMotorPositions logic here
+    // ... (existing keyboard processing logic)
+  }
+}
+```
+
+#### Teleoperator Factory
+
+```typescript
+// teleoperate.ts - Factory pattern
+async function createTeleoperator(
+  config: TeleoperateConfig,
+  port: MotorCommunicationPort,
+  motorConfigs: MotorConfig[],
+  robotHardwareConfig: RobotHardwareConfig
+): Promise<WebTeleoperator> {
+  switch (config.teleop.type) {
+    case "keyboard":
+      return new KeyboardTeleoperator(
+        config.teleop,
+        port,
+        motorConfigs,
+        robotHardwareConfig.keyboardControls,
+        config.onStateUpdate
+      );
+
+    case "so100_leader":
+      // Future implementation
+      throw new Error("Leader arm teleoperator not yet implemented");
+
+    case "gamepad":
+      // Future implementation
+      throw new Error("Gamepad teleoperator not yet implemented");
+
+    default:
+      throw new Error(
+        `Unsupported teleoperator type: ${(config.teleop as any).type}`
+      );
+  }
+}
+```
+
+#### Updated Main Teleoperate Function
+
+```typescript
+// teleoperate.ts - Updated main function (breaking change)
+export async function teleoperate(
+  config: TeleoperateConfig
+): Promise<TeleoperationProcess> {
+  // Validate required fields
+  if (!config.robot.robotType) {
+    throw new Error(
+      "Robot type is required for teleoperation. Please configure the robot first."
+    );
+  }
+
+  // Create web serial port wrapper (same as before)
+  const port = new WebSerialPortWrapper(config.robot.port);
+  await port.initialize();
+
+  // Get robot-specific configuration (same as before)
+  let robotHardwareConfig: RobotHardwareConfig;
+  if (config.robot.robotType.startsWith("so100")) {
+    robotHardwareConfig = createSO100Config(config.robot.robotType);
+  } else {
+    throw new Error(`Unsupported robot type: ${config.robot.robotType}`);
+  }
+
+  // Create motor configs (same as before)
+  const defaultMotorConfigs =
+    createMotorConfigsFromRobotConfig(robotHardwareConfig);
+  const motorConfigs = config.calibrationData
+    ? applyCalibrationToMotorConfigs(
+        defaultMotorConfigs,
+        config.calibrationData
+      )
+    : defaultMotorConfigs;
+
+  // Create teleoperator using factory pattern (NEW)
+  const teleoperator = await createTeleoperator(
+    config,
+    port,
+    motorConfigs,
+    robotHardwareConfig
+  );
+  await teleoperator.initialize();
+
+  // Return process object (same interface as before)
+  return {
+    start: () => {
+      teleoperator.start();
+      // State update loop (same as before)
+      if (config.onStateUpdate) {
+        const updateLoop = () => {
+          if (teleoperator.getState()) {
+            config.onStateUpdate!(
+              buildTeleoperationStateFromTeleoperator(teleoperator)
+            );
+            setTimeout(updateLoop, 100);
+          }
+        };
+        updateLoop();
+      }
+    },
+    stop: () => teleoperator.stop(),
+    updateKeyState: (key: string, pressed: boolean) => {
+      // Delegate to teleoperator if it supports keyboard input
+      if (teleoperator instanceof KeyboardTeleoperator) {
+        teleoperator.updateKeyState(key, pressed);
+      }
+    },
+    getState: () => buildTeleoperationStateFromTeleoperator(teleoperator),
+    moveMotor: async (motorName: string, position: number) => {
+      // Direct motor control through teleoperator
+      if (teleoperator instanceof DirectTeleoperator) {
+        return teleoperator.moveMotor(motorName, position);
+      }
+      throw new Error(
+        `Motor control not supported for ${config.teleop.type} teleoperator`
+      );
+    },
+    setMotorPositions: async (positions: { [motorName: string]: number }) => {
+      // Direct motor control through teleoperator
+      if (teleoperator instanceof DirectTeleoperator) {
+        return teleoperator.setMotorPositions(positions);
+      }
+      throw new Error(
+        `Motor control not supported for ${config.teleop.type} teleoperator`
+      );
+    },
+    disconnect: () => teleoperator.disconnect(),
+  };
+}
+```
+
+### Technical Considerations
+
+#### State Management Strategy
+
+Maintain current `TeleoperationState` structure but extend with teleoperator-specific state:
+
+```typescript
+interface TeleoperationState {
+  isActive: boolean;
+  motorConfigs: MotorConfig[];
+  lastUpdate: number;
+
+  // Teleoperator-specific state (optional fields for different types)
+  keyStates?: { [key: string]: { pressed: boolean; timestamp: number } }; // keyboard
+  leaderPositions?: { [motor: string]: number }; // leader arm
+  gamepadState?: { axes: number[]; buttons: boolean[] }; // gamepad
+}
+```
+
+#### Migration Strategy
+
+**Breaking Change Approach:**
+
+1. **Remove old API** - No backward compatibility
+2. **Update examples** - All demo applications must be updated to use new API
+3. **Clear documentation** - Document the API change and migration path
+4. **Type safety** - TypeScript will catch all usages of old API
+
+#### Future Extensibility
+
+The architecture supports easy addition of new teleoperators:
+
+```typescript
+// Future: Add VR controller
+interface VRTeleoperatorConfig extends BaseTeleoperatorConfig {
+  type: "vr_controller";
+  handedness: "left" | "right";
+  trackingSpace: "local" | "world";
+}
+
+class VRTeleoperator extends BaseWebTeleoperator {
+  // VR-specific implementation
+}
+
+// Add to factory in teleoperate.ts
+case "vr_controller":
+  return new VRTeleoperator(config.teleop, port, motorConfigs, config.onStateUpdate);
+```
+
+#### Performance Considerations
+
+- **Same Performance**: No performance impact - just architectural refactoring
+- **Memory Usage**: Slightly lower memory usage due to cleaner separation
+- **Extensibility**: No overhead for unused teleoperator types
+
+## Definition of Done
+
+- [ ] **API Breaking Change**: Web API updated to `teleoperate(config: TeleoperateConfig)` matching Node.js
+- [ ] **Keyboard Teleoperator**: Existing keyboard functionality extracted into `KeyboardTeleoperator` class
+- [ ] **Base Teleoperator**: `BaseWebTeleoperator` abstract class with common functionality
+- [ ] **Teleoperator Factory**: Factory pattern for creating appropriate teleoperator instances
+- [ ] **Type Safety**: Full TypeScript coverage with union types for teleoperator configurations
+- [ ] **State Management**: Current `TeleoperationState` approach preserved with teleoperator extensions
+- [ ] **Process Interface**: `TeleoperationProcess` interface remains the same for existing UI code
+- [ ] **Error Handling**: Clear error messages for unsupported teleoperator types
+- [ ] **No Regression**: Keyboard teleoperation functionality identical to current implementation
+- [ ] **Future Ready**: Architecture supports easy addition of leader arms, joysticks, VR controllers
+- [ ] **Code Quality**: No code duplication, clean separation of concerns
+- [ ] **Documentation**: Updated examples and documentation for new API

examples/robot-control-web/components/TeleoperationPanel.tsx

@@ -7,6 +7,7 @@ import {
   teleoperate,
   type TeleoperationProcess,
   type TeleoperationState,
+  type TeleoperateConfig,
 } from "@lerobot/web";
 import { getUnifiedRobotData } from "../lib/unified-storage";
 import type { RobotConnection } from "@lerobot/web";
@@ -31,9 +32,11 @@ export function TeleoperationPanel({
   const [error, setError] = useState<string | null>(null);
   const [, setIsInitialized] = useState(false);
 
-  const teleoperationProcessRef = useRef<TeleoperationProcess | null>(null);
+  // Separate refs for keyboard and direct teleoperators
+  const keyboardProcessRef = useRef<TeleoperationProcess | null>(null);
+  const directProcessRef = useRef<TeleoperationProcess | null>(null);
 
-  // Initialize teleoperation process
+  // Initialize both teleoperation processes
   useEffect(() => {
     const initializeTeleoperation = async () => {
       if (!robot || !robot.robotType) {
@@ -52,18 +55,36 @@ export function TeleoperationPanel({
           }
         }
 
-        // Create teleoperation process using clean library API
-        const process = await teleoperate(robot, {
+        // Create keyboard teleoperation process
+        const keyboardConfig: TeleoperateConfig = {
+          robot: robot,
+          teleop: {
+            type: "keyboard",
+          },
           calibrationData,
           onStateUpdate: (state: TeleoperationState) => {
             setTeleoperationState(state);
           },
-        });
+        };
+        const keyboardProcess = await teleoperate(keyboardConfig);
+
+        // Create direct teleoperation process
+        const directConfig: TeleoperateConfig = {
+          robot: robot,
+          teleop: {
+            type: "direct",
+          },
+          calibrationData,
+        };
+        const directProcess = await teleoperate(directConfig);
 
-        teleoperationProcessRef.current = process;
-        setTeleoperationState(process.getState());
+        keyboardProcessRef.current = keyboardProcess;
+        directProcessRef.current = directProcess;
+        setTeleoperationState(keyboardProcess.getState());
         setIsInitialized(true);
         setError(null);
+
+        console.log("✅ Initialized both keyboard and direct teleoperators");
       } catch (error) {
         const errorMessage =
           error instanceof Error
@@ -78,9 +99,13 @@ export function TeleoperationPanel({
 
     return () => {
       // Cleanup on unmount
-      if (teleoperationProcessRef.current) {
-        teleoperationProcessRef.current.disconnect();
-        teleoperationProcessRef.current = null;
+      if (keyboardProcessRef.current) {
+        keyboardProcessRef.current.disconnect();
+        keyboardProcessRef.current = null;
+      }
+      if (directProcessRef.current) {
+        directProcessRef.current.disconnect();
+        directProcessRef.current = null;
       }
     };
   }, [robot]);
@@ -88,24 +113,22 @@ export function TeleoperationPanel({
   // Keyboard event handlers
   const handleKeyDown = useCallback(
     (event: KeyboardEvent) => {
-      if (!teleoperationState.isActive || !teleoperationProcessRef.current)
-        return;
+      if (!teleoperationState.isActive || !keyboardProcessRef.current) return;
 
       const key = event.key;
       event.preventDefault();
-      teleoperationProcessRef.current.updateKeyState(key, true);
+      keyboardProcessRef.current.updateKeyState(key, true);
     },
     [teleoperationState.isActive]
   );
 
   const handleKeyUp = useCallback(
     (event: KeyboardEvent) => {
-      if (!teleoperationState.isActive || !teleoperationProcessRef.current)
-        return;
+      if (!teleoperationState.isActive || !keyboardProcessRef.current) return;
 
       const key = event.key;
       event.preventDefault();
-      teleoperationProcessRef.current.updateKeyState(key, false);
+      keyboardProcessRef.current.updateKeyState(key, false);
     },
     [teleoperationState.isActive]
   );
@@ -124,14 +147,15 @@ export function TeleoperationPanel({
   }, [teleoperationState.isActive, handleKeyDown, handleKeyUp]);
 
   const handleStart = () => {
-    if (!teleoperationProcessRef.current) {
+    if (!keyboardProcessRef.current || !directProcessRef.current) {
       setError("Teleoperation not initialized");
       return;
     }
 
     try {
-      teleoperationProcessRef.current.start();
-      console.log("🎮 Teleoperation started");
+      keyboardProcessRef.current.start();
+      directProcessRef.current.start();
+      console.log("🎮 Both keyboard and direct teleoperation started");
     } catch (error) {
       const errorMessage =
         error instanceof Error
@@ -142,36 +166,46 @@ export function TeleoperationPanel({
   };
 
   const handleStop = () => {
-    if (!teleoperationProcessRef.current) return;
-
-    teleoperationProcessRef.current.stop();
-    console.log("🛑 Teleoperation stopped");
+    if (keyboardProcessRef.current) {
+      keyboardProcessRef.current.stop();
+    }
+    if (directProcessRef.current) {
+      directProcessRef.current.stop();
+    }
+    console.log("🛑 Both keyboard and direct teleoperation stopped");
   };
 
   const handleClose = () => {
-    if (teleoperationProcessRef.current) {
-      teleoperationProcessRef.current.stop();
+    if (keyboardProcessRef.current) {
+      keyboardProcessRef.current.stop();
+    }
+    if (directProcessRef.current) {
+      directProcessRef.current.stop();
     }
     onClose();
   };
 
   const simulateKeyPress = (key: string) => {
-    if (!teleoperationProcessRef.current) return;
-    teleoperationProcessRef.current.updateKeyState(key, true);
+    if (!keyboardProcessRef.current) return;
+    keyboardProcessRef.current.updateKeyState(key, true);
   };
 
   const simulateKeyRelease = (key: string) => {
-    if (!teleoperationProcessRef.current) return;
-    teleoperationProcessRef.current.updateKeyState(key, false);
+    if (!keyboardProcessRef.current) return;
+    keyboardProcessRef.current.updateKeyState(key, false);
   };
 
+  // Unified motor control: Both sliders AND keyboard use the same teleoperator
+  // This ensures the UI always shows the correct motor positions
   const moveMotorToPosition = async (motorIndex: number, position: number) => {
-    if (!teleoperationProcessRef.current) return;
+    if (!keyboardProcessRef.current) return;
 
     try {
       const motorName = teleoperationState.motorConfigs[motorIndex]?.name;
       if (motorName) {
-        await teleoperationProcessRef.current.moveMotor(motorName, position);
+        const keyboardTeleoperator = keyboardProcessRef.current
+          .teleoperator as any;
+        await keyboardTeleoperator.moveMotor(motorName, position);
       }
     } catch (error) {
       console.warn(
@@ -189,7 +223,7 @@ export function TeleoperationPanel({
   // Virtual keyboard component
   const VirtualKeyboard = () => {
     const isKeyPressed = (key: string) => {
-      return keyStates[key]?.pressed || false;
+      return keyStates?.[key]?.pressed || false;
     };
 
     const KeyButton = ({
@@ -208,42 +242,19 @@ export function TeleoperationPanel({
           keyCode as keyof typeof SO100_KEYBOARD_CONTROLS
         ];
       const pressed = isKeyPressed(keyCode);
-      const intervalRef = useRef<NodeJS.Timeout | null>(null);
 
-      const startContinuousPress = () => {
-        if (!isActive || !teleoperationProcessRef.current) return;
-
-        // Initial press
+      const handleMouseDown = (e: React.MouseEvent) => {
+        e.preventDefault();
+        if (!isActive) return;
         simulateKeyPress(keyCode);
-
-        // Set up continuous updates to maintain key state
-        // Update every 50ms to stay well within the 10 second timeout
-        intervalRef.current = setInterval(() => {
-          if (teleoperationProcessRef.current) {
-            simulateKeyPress(keyCode);
-          }
-        }, 50);
       };
 
-      const stopContinuousPress = () => {
-        if (intervalRef.current) {
-          clearInterval(intervalRef.current);
-          intervalRef.current = null;
-        }
-        if (isActive) {
-          simulateKeyRelease(keyCode);
-        }
+      const handleMouseUp = (e: React.MouseEvent) => {
+        e.preventDefault();
+        if (!isActive) return;
+        simulateKeyRelease(keyCode);
       };
 
-      // Cleanup interval on unmount
-      useEffect(() => {
-        return () => {
-          if (intervalRef.current) {
-            clearInterval(intervalRef.current);
-          }
-        };
-      }, []);
-
       return (
         <Button
           variant={pressed ? "default" : "outline"}
@@ -259,18 +270,9 @@ export function TeleoperationPanel({
             ${!isActive ? "opacity-50 cursor-not-allowed" : ""}
           `}
           disabled={!isActive}
-          onMouseDown={(e) => {
-            e.preventDefault();
-            startContinuousPress();
-          }}
-          onMouseUp={(e) => {
-            e.preventDefault();
-            stopContinuousPress();
-          }}
-          onMouseLeave={(e) => {
-            e.preventDefault();
-            stopContinuousPress();
-          }}
+          onMouseDown={handleMouseDown}
+          onMouseUp={handleMouseUp}
+          onMouseLeave={handleMouseUp}
           title={control?.description || keyCode}
         >
           {children}
@@ -413,8 +415,9 @@ export function TeleoperationPanel({
                 <span className="text-sm text-gray-600">Active Keys</span>
                 <Badge variant="outline">
                   {
-                    Object.values(keyStates).filter((state) => state.pressed)
-                      .length
+                    Object.values(keyStates || {}).filter(
+                      (state) => state.pressed
+                    ).length
                   }
                 </Badge>
               </div>

package.json

@@ -42,7 +42,7 @@
     "install-global": "pnpm run build && npm link"
   },
   "dependencies": {
-    "@lerobot/web": "^0.1.1",
+    "@lerobot/web": "workspace:*",
     "@radix-ui/react-dialog": "^1.1.14",
     "@radix-ui/react-progress": "^1.1.7",
     "log-update": "^6.1.0",

packages/web/src/calibrate.ts

@@ -144,7 +144,7 @@ export async function calibrate(
   const port = new WebSerialPortWrapper(robotConnection.port);
   await port.initialize();
 
-  // Get robot-specific configuration (extensible - add new robot types here)
+  // Get robot-specific configuration
   let config: RobotHardwareConfig;
   if (robotConnection.robotType.startsWith("so100")) {
     config = createSO100Config(robotConnection.robotType);

packages/web/src/index.ts

@@ -35,6 +35,9 @@ export type {
   MotorConfig,
   TeleoperationState,
   TeleoperationProcess,
+  TeleoperateConfig,
+  TeleoperatorConfig,
+  DirectTeleoperatorConfig,
 } from "./types/teleoperation.js";
 
 export type {
@@ -48,3 +51,4 @@ export {
   createSO100Config,
   SO100_KEYBOARD_CONTROLS,
 } from "./robots/so100_config.js";
+export { KEYBOARD_TELEOPERATOR_DEFAULTS } from "./teleoperators/index.js";

packages/web/src/teleoperate.ts

@@ -3,10 +3,7 @@
  */
 
 import { createSO100Config } from "./robots/so100_config.js";
-import type {
-  RobotHardwareConfig,
-  KeyboardControl,
-} from "./types/robot-config.js";
+import type { RobotHardwareConfig } from "./types/robot-config.js";
 import type { RobotConnection } from "./types/robot-connection.js";
 import { WebSerialPortWrapper } from "./utils/serial-port-wrapper.js";
 import {
@@ -18,18 +15,29 @@ import type {
   MotorConfig,
   TeleoperationState,
   TeleoperationProcess,
+  TeleoperateConfig,
+  TeleoperatorConfig,
+  DirectTeleoperatorConfig,
 } from "./types/teleoperation.js";
+import {
+  KeyboardTeleoperator,
+  DirectTeleoperator,
+  type WebTeleoperator,
+} from "./teleoperators/index.js";
 
 // Re-export types for external use
 export type {
   MotorConfig,
   TeleoperationState,
   TeleoperationProcess,
+  TeleoperateConfig,
+  TeleoperatorConfig,
+  DirectTeleoperatorConfig,
 } from "./types/teleoperation.js";
 
 /**
  * Create motor configurations from robot hardware config
- * Pure function - converts robot specs to motor configs with defaults
+ * Pure function - converts robot specs to motor configs
  */
 function createMotorConfigsFromRobotConfig(
   robotConfig: RobotHardwareConfig
@@ -75,283 +83,145 @@ export function applyCalibrationToMotorConfigs(
 }
 
 /**
- * Web teleoperation controller
- * Now uses shared utilities instead of custom port handling
+ * Create appropriate teleoperator based on configuration
  */
-export class WebTeleoperationController {
-  private port: MotorCommunicationPort;
-  private motorConfigs: MotorConfig[] = [];
-  private keyboardControls: { [key: string]: KeyboardControl } = {};
-  private isActive: boolean = false;
-  private updateInterval: NodeJS.Timeout | null = null;
-  private keyStates: {
-    [key: string]: { pressed: boolean; timestamp: number };
-  } = {};
-  private onStateUpdate?: (state: TeleoperationState) => void;
-
-  // Movement parameters
-  private readonly STEP_SIZE = 8;
-  private readonly UPDATE_RATE = 60; // 60 FPS
-  private readonly KEY_TIMEOUT = 10000; // ms - very long timeout (10 seconds) for virtual buttons
-
-  constructor(
-    port: MotorCommunicationPort,
-    motorConfigs: MotorConfig[],
-    keyboardControls: { [key: string]: KeyboardControl },
-    onStateUpdate?: (state: TeleoperationState) => void
-  ) {
-    this.port = port;
-    this.motorConfigs = motorConfigs;
-    this.keyboardControls = keyboardControls;
-    this.onStateUpdate = onStateUpdate;
-  }
-
-  async initialize(): Promise<void> {
-    // Read current motor positions
-    for (const config of this.motorConfigs) {
-      const position = await readMotorPosition(this.port, config.id);
-      if (position !== null) {
-        config.currentPosition = position;
-      }
-    }
-  }
-
-  getMotorConfigs(): MotorConfig[] {
-    return [...this.motorConfigs];
-  }
-
-  getState(): TeleoperationState {
-    return {
-      isActive: this.isActive,
-      motorConfigs: [...this.motorConfigs],
-      lastUpdate: Date.now(),
-      keyStates: { ...this.keyStates },
-    };
-  }
-
-  updateKeyState(key: string, pressed: boolean): void {
-    this.keyStates[key] = {
-      pressed,
-      timestamp: Date.now(),
-    };
-  }
-
-  start(): void {
-    if (this.isActive) return;
-
-    this.isActive = true;
-    this.updateInterval = setInterval(() => {
-      this.updateMotorPositions();
-    }, 1000 / this.UPDATE_RATE);
-
-    console.log("🎮 Web teleoperation started");
-  }
-
-  stop(): void {
-    if (!this.isActive) return;
-
-    this.isActive = false;
-
-    if (this.updateInterval) {
-      clearInterval(this.updateInterval);
-      this.updateInterval = null;
-    }
-
-    // Clear all key states
-    this.keyStates = {};
-
-    console.log("⏹️ Web teleoperation stopped");
-
-    // Notify UI of state change
-    if (this.onStateUpdate) {
-      this.onStateUpdate(this.getState());
-    }
-  }
-
-  async disconnect(): Promise<void> {
-    this.stop();
-    // No need to manually disconnect - port wrapper handles this
-  }
-
-  private updateMotorPositions(): void {
-    const now = Date.now();
-
-    // Clear timed-out keys
-    Object.keys(this.keyStates).forEach((key) => {
-      if (now - this.keyStates[key].timestamp > this.KEY_TIMEOUT) {
-        delete this.keyStates[key];
-      }
-    });
-
-    // Process active keys
-    const activeKeys = Object.keys(this.keyStates).filter(
-      (key) =>
-        this.keyStates[key].pressed &&
-        now - this.keyStates[key].timestamp <= this.KEY_TIMEOUT
-    );
-
-    // Emergency stop check
-    if (activeKeys.includes("Escape")) {
-      this.stop();
-      return;
-    }
-
-    // Calculate target positions based on active keys
-    const targetPositions: { [motorName: string]: number } = {};
-
-    for (const key of activeKeys) {
-      const control = this.keyboardControls[key];
-      if (!control || control.motor === "emergency_stop") continue;
-
-      const motorConfig = this.motorConfigs.find(
-        (m) => m.name === control.motor
+async function createTeleoperator(
+  config: TeleoperateConfig,
+  port: MotorCommunicationPort,
+  motorConfigs: MotorConfig[],
+  robotHardwareConfig: RobotHardwareConfig
+): Promise<WebTeleoperator> {
+  switch (config.teleop.type) {
+    case "keyboard":
+      return new KeyboardTeleoperator(
+        config.teleop,
+        port,
+        motorConfigs,
+        robotHardwareConfig.keyboardControls,
+        config.onStateUpdate
       );
-      if (!motorConfig) continue;
-
-      // Calculate new position
-      const currentTarget =
-        targetPositions[motorConfig.name] ?? motorConfig.currentPosition;
-      const newPosition = currentTarget + control.direction * this.STEP_SIZE;
 
-      // Apply limits
-      targetPositions[motorConfig.name] = Math.max(
-        motorConfig.minPosition,
-        Math.min(motorConfig.maxPosition, newPosition)
+    case "direct":
+      return new DirectTeleoperator(
+        config.teleop,
+        port,
+        motorConfigs,
+        config.onStateUpdate
       );
-    }
 
-    // Send motor commands
-    Object.entries(targetPositions).forEach(([motorName, targetPosition]) => {
-      const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
-      if (motorConfig && targetPosition !== motorConfig.currentPosition) {
-        writeMotorPosition(
-          this.port,
-          motorConfig.id,
-          Math.round(targetPosition)
-        )
-          .then(() => {
-            motorConfig.currentPosition = targetPosition;
-          })
-          .catch((error) => {
-            console.warn(
-              `Failed to write motor ${motorConfig.id} position:`,
-              error
-            );
-          });
-      }
-    });
-  }
+    case "so100_leader":
+      throw new Error("Leader arm teleoperator not yet implemented");
 
-  // Programmatic control methods
-  async moveMotor(motorName: string, targetPosition: number): Promise<boolean> {
-    const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
-    if (!motorConfig) return false;
-
-    const clampedPosition = Math.max(
-      motorConfig.minPosition,
-      Math.min(motorConfig.maxPosition, targetPosition)
-    );
+    case "gamepad":
+      throw new Error("Gamepad teleoperator not yet implemented");
 
-    try {
-      await writeMotorPosition(
-        this.port,
-        motorConfig.id,
-        Math.round(clampedPosition)
+    default:
+      throw new Error(
+        `Unsupported teleoperator type: ${(config.teleop as any).type}`
       );
-      motorConfig.currentPosition = clampedPosition;
-      return true;
-    } catch (error) {
-      console.warn(`Failed to move motor ${motorName}:`, error);
-      return false;
-    }
   }
+}
 
-  async setMotorPositions(positions: {
-    [motorName: string]: number;
-  }): Promise<boolean> {
-    const results = await Promise.all(
-      Object.entries(positions).map(([motorName, position]) =>
-        this.moveMotor(motorName, position)
-      )
-    );
+/**
+ * Build TeleoperationState from teleoperator and motor configs
+ */
+function buildTeleoperationStateFromTeleoperator(
+  teleoperator: WebTeleoperator
+): TeleoperationState {
+  const teleoperatorState = teleoperator.getState();
+  const isActive = (teleoperator as any).isActive;
 
-    return results.every((result) => result);
-  }
+  return {
+    isActive: isActive || false,
+    motorConfigs: [...teleoperator.motorConfigs], // Get fresh motor configs from teleoperator
+    lastUpdate: Date.now(),
+    ...teleoperatorState,
+  };
 }
 
 /**
- * Main teleoperate function - simple API
- * Handles robot types internally, creates appropriate motor configurations
+ * Main teleoperate function
  */
 export async function teleoperate(
-  robotConnection: RobotConnection,
-  options?: {
-    calibrationData?: { [motorName: string]: any };
-    onStateUpdate?: (state: TeleoperationState) => void;
-  }
+  config: TeleoperateConfig
 ): Promise<TeleoperationProcess> {
+  const teleoperator = await createTeleoperatorProcess(config);
+  const motorConfigs = teleoperator.motorConfigs;
+
+  return {
+    start: () => {
+      teleoperator.start();
+      // CRITICAL: State update loop for UI synchronization
+      // This ensures sliders and UI reflect actual motor positions when moved via keyboard
+      if (config.onStateUpdate) {
+        const updateLoop = () => {
+          const state = buildTeleoperationStateFromTeleoperator(teleoperator);
+          if (state.isActive) {
+            config.onStateUpdate!(state);
+            setTimeout(updateLoop, 100); // 10fps state updates - keeps sliders in sync
+          }
+        };
+        updateLoop();
+      }
+    },
+    stop: () => teleoperator.stop(),
+    updateKeyState: (key: string, pressed: boolean) => {
+      // Delegate to teleoperator if it supports keyboard input
+      if (teleoperator instanceof KeyboardTeleoperator) {
+        teleoperator.updateKeyState(key, pressed);
+      }
+    },
+    getState: () => buildTeleoperationStateFromTeleoperator(teleoperator),
+    teleoperator,
+    disconnect: () => teleoperator.disconnect(),
+  };
+}
+
+/**
+ * Create teleoperator instance (shared logic)
+ */
+async function createTeleoperatorProcess(
+  config: TeleoperateConfig
+): Promise<WebTeleoperator> {
   // Validate required fields
-  if (!robotConnection.robotType) {
+  if (!config.robot.robotType) {
     throw new Error(
       "Robot type is required for teleoperation. Please configure the robot first."
     );
   }
 
   // Create web serial port wrapper
-  const port = new WebSerialPortWrapper(robotConnection.port);
+  const port = new WebSerialPortWrapper(config.robot.port);
   await port.initialize();
 
   // Get robot-specific configuration
-  let config: RobotHardwareConfig;
-  if (robotConnection.robotType.startsWith("so100")) {
-    config = createSO100Config(robotConnection.robotType);
+  let robotHardwareConfig: RobotHardwareConfig;
+  if (config.robot.robotType.startsWith("so100")) {
+    robotHardwareConfig = createSO100Config(config.robot.robotType);
   } else {
-    throw new Error(`Unsupported robot type: ${robotConnection.robotType}`);
+    throw new Error(`Unsupported robot type: ${config.robot.robotType}`);
   }
 
-  // Create motor configs from robot hardware specs (single call, no duplication)
-  const defaultMotorConfigs = createMotorConfigsFromRobotConfig(config);
+  // Create motor configs from robot hardware specs
+  const defaultMotorConfigs =
+    createMotorConfigsFromRobotConfig(robotHardwareConfig);
 
   // Apply calibration data if provided
-  const motorConfigs = options?.calibrationData
+  const motorConfigs = config.calibrationData
     ? applyCalibrationToMotorConfigs(
         defaultMotorConfigs,
-        options.calibrationData
+        config.calibrationData
       )
     : defaultMotorConfigs;
 
-  // Create and initialize controller using shared utilities
-  const controller = new WebTeleoperationController(
+  // Create teleoperator
+  const teleoperator = await createTeleoperator(
+    config,
     port,
     motorConfigs,
-    config.keyboardControls,
-    options?.onStateUpdate
+    robotHardwareConfig
   );
-  await controller.initialize();
+  await teleoperator.initialize();
 
-  // Wrap controller in process object
-  return {
-    start: () => {
-      controller.start();
-      // Optional state update callback
-      if (options?.onStateUpdate) {
-        const updateLoop = () => {
-          if (controller.getState().isActive) {
-            options.onStateUpdate!(controller.getState());
-            setTimeout(updateLoop, 100); // 10fps state updates
-          }
-        };
-        updateLoop();
-      }
-    },
-    stop: () => controller.stop(),
-    updateKeyState: (key: string, pressed: boolean) =>
-      controller.updateKeyState(key, pressed),
-    getState: () => controller.getState(),
-    moveMotor: (motorName: string, position: number) =>
-      controller.moveMotor(motorName, position),
-    setMotorPositions: (positions: { [motorName: string]: number }) =>
-      controller.setMotorPositions(positions),
-    disconnect: () => controller.disconnect(),
-  };
+  return teleoperator;
 }

packages/web/src/teleoperators/base-teleoperator.ts

@@ -0,0 +1,60 @@
+/**
+ * Base teleoperator interface and abstract class for Web platform
+ * Defines the contract that all teleoperators must implement
+ */
+
+import type { MotorConfig } from "../types/teleoperation.js";
+import type { MotorCommunicationPort } from "../utils/motor-communication.js";
+
+/**
+ * Base interface that all Web teleoperators must implement
+ */
+export interface WebTeleoperator {
+  initialize(): Promise<void>;
+  start(): void;
+  stop(): void;
+  disconnect(): Promise<void>;
+  getState(): TeleoperatorSpecificState;
+  onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void;
+  motorConfigs: MotorConfig[];
+}
+
+/**
+ * Teleoperator-specific state (union type for different teleoperator types)
+ */
+export type TeleoperatorSpecificState = {
+  keyStates?: { [key: string]: { pressed: boolean; timestamp: number } }; // keyboard
+  leaderPositions?: { [motor: string]: number }; // leader arm
+  gamepadState?: { axes: number[]; buttons: boolean[] }; // gamepad
+};
+
+/**
+ * Base abstract class with common functionality for all teleoperators
+ */
+export abstract class BaseWebTeleoperator implements WebTeleoperator {
+  protected port: MotorCommunicationPort;
+  public motorConfigs: MotorConfig[] = [];
+  protected isActive: boolean = false;
+
+  constructor(port: MotorCommunicationPort, motorConfigs: MotorConfig[]) {
+    this.port = port;
+    this.motorConfigs = motorConfigs;
+  }
+
+  abstract initialize(): Promise<void>;
+  abstract start(): void;
+  abstract stop(): void;
+  abstract getState(): TeleoperatorSpecificState;
+
+  async disconnect(): Promise<void> {
+    this.stop();
+  }
+
+  onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void {
+    this.motorConfigs = motorConfigs;
+  }
+
+  get isActiveTeleoperator(): boolean {
+    return this.isActive;
+  }
+}

packages/web/src/teleoperators/direct-teleoperator.ts

@@ -0,0 +1,115 @@
+/**
+ * Direct teleoperator for Web platform
+ * Handles programmatic motor control (sliders, API calls, etc.)
+ */
+
+import {
+  BaseWebTeleoperator,
+  type TeleoperatorSpecificState,
+} from "./base-teleoperator.js";
+import type {
+  DirectTeleoperatorConfig,
+  MotorConfig,
+  TeleoperationState,
+} from "../types/teleoperation.js";
+import type { MotorCommunicationPort } from "../utils/motor-communication.js";
+import {
+  readMotorPosition,
+  writeMotorPosition,
+} from "../utils/motor-communication.js";
+
+export class DirectTeleoperator extends BaseWebTeleoperator {
+  private onStateUpdate?: (state: TeleoperationState) => void;
+
+  constructor(
+    config: DirectTeleoperatorConfig,
+    port: MotorCommunicationPort,
+    motorConfigs: MotorConfig[],
+    onStateUpdate?: (state: TeleoperationState) => void
+  ) {
+    super(port, motorConfigs);
+    this.onStateUpdate = onStateUpdate;
+  }
+
+  async initialize(): Promise<void> {
+    // Read current motor positions
+    for (const config of this.motorConfigs) {
+      const position = await readMotorPosition(this.port, config.id);
+      if (position !== null) {
+        config.currentPosition = position;
+      }
+    }
+  }
+
+  start(): void {
+    this.isActive = true;
+  }
+
+  stop(): void {
+    this.isActive = false;
+
+    // Notify UI of state change
+    if (this.onStateUpdate) {
+      this.onStateUpdate(this.buildTeleoperationState());
+    }
+  }
+
+  getState(): TeleoperatorSpecificState {
+    return {};
+  }
+
+  /**
+   * Move motor to exact position
+   */
+  async moveMotor(motorName: string, targetPosition: number): Promise<boolean> {
+    const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
+    if (!motorConfig) return false;
+
+    const clampedPosition = Math.max(
+      motorConfig.minPosition,
+      Math.min(motorConfig.maxPosition, targetPosition)
+    );
+
+    try {
+      await writeMotorPosition(
+        this.port,
+        motorConfig.id,
+        Math.round(clampedPosition)
+      );
+      motorConfig.currentPosition = clampedPosition;
+
+      // Notify UI of position change
+      if (this.onStateUpdate) {
+        this.onStateUpdate(this.buildTeleoperationState());
+      }
+
+      return true;
+    } catch (error) {
+      console.warn(`Failed to move motor ${motorName}:`, error);
+      return false;
+    }
+  }
+
+  /**
+   * Set multiple motor positions at once
+   */
+  async setMotorPositions(positions: {
+    [motorName: string]: number;
+  }): Promise<boolean> {
+    const results = await Promise.all(
+      Object.entries(positions).map(([motorName, position]) =>
+        this.moveMotor(motorName, position)
+      )
+    );
+
+    return results.every((result) => result);
+  }
+
+  private buildTeleoperationState(): TeleoperationState {
+    return {
+      isActive: this.isActive,
+      motorConfigs: [...this.motorConfigs],
+      lastUpdate: Date.now(),
+    };
+  }
+}

packages/web/src/teleoperators/index.ts

@@ -0,0 +1,14 @@
+/**
+ * Teleoperators barrel exports
+ */
+
+export {
+  BaseWebTeleoperator,
+  type WebTeleoperator,
+  type TeleoperatorSpecificState,
+} from "./base-teleoperator.js";
+export {
+  KeyboardTeleoperator,
+  KEYBOARD_TELEOPERATOR_DEFAULTS,
+} from "./keyboard-teleoperator.js";
+export { DirectTeleoperator } from "./direct-teleoperator.js";

packages/web/src/teleoperators/keyboard-teleoperator.ts

@@ -0,0 +1,243 @@
+/**
+ * Keyboard teleoperator for Web platform
+ */
+
+import {
+  BaseWebTeleoperator,
+  type TeleoperatorSpecificState,
+} from "./base-teleoperator.js";
+import type { KeyboardControl } from "../types/robot-config.js";
+import type {
+  KeyboardTeleoperatorConfig,
+  MotorConfig,
+  TeleoperationState,
+} from "../types/teleoperation.js";
+import type { MotorCommunicationPort } from "../utils/motor-communication.js";
+import {
+  readMotorPosition,
+  writeMotorPosition,
+} from "../utils/motor-communication.js";
+
+/**
+ * Default configuration values for keyboard teleoperator
+ */
+export const KEYBOARD_TELEOPERATOR_DEFAULTS = {
+  stepSize: 8, // Position units per keypress (smooth responsive control)
+  updateRate: 60, // Control loop FPS (60 Hz for smooth updates)
+  keyTimeout: 10000, // Key state timeout in ms (10 seconds for virtual buttons)
+} as const;
+
+export class KeyboardTeleoperator extends BaseWebTeleoperator {
+  private keyboardControls: { [key: string]: KeyboardControl } = {};
+  private updateInterval: NodeJS.Timeout | null = null;
+  private keyStates: {
+    [key: string]: { pressed: boolean; timestamp: number };
+  } = {};
+  private onStateUpdate?: (state: TeleoperationState) => void;
+
+  // Configuration values
+  private readonly stepSize: number;
+  private readonly updateRate: number;
+  private readonly keyTimeout: number;
+
+  constructor(
+    config: KeyboardTeleoperatorConfig,
+    port: MotorCommunicationPort,
+    motorConfigs: MotorConfig[],
+    keyboardControls: { [key: string]: KeyboardControl },
+    onStateUpdate?: (state: TeleoperationState) => void
+  ) {
+    super(port, motorConfigs);
+    this.keyboardControls = keyboardControls;
+    this.onStateUpdate = onStateUpdate;
+
+    // Set configuration values
+    this.stepSize = config.stepSize ?? KEYBOARD_TELEOPERATOR_DEFAULTS.stepSize;
+    this.updateRate =
+      config.updateRate ?? KEYBOARD_TELEOPERATOR_DEFAULTS.updateRate;
+    this.keyTimeout =
+      config.keyTimeout ?? KEYBOARD_TELEOPERATOR_DEFAULTS.keyTimeout;
+  }
+
+  async initialize(): Promise<void> {
+    // Read current motor positions
+    for (const config of this.motorConfigs) {
+      const position = await readMotorPosition(this.port, config.id);
+      if (position !== null) {
+        config.currentPosition = position;
+      }
+    }
+  }
+
+  start(): void {
+    if (this.isActive) return;
+
+    this.isActive = true;
+    this.updateInterval = setInterval(() => {
+      this.updateMotorPositions();
+    }, 1000 / this.updateRate);
+  }
+
+  stop(): void {
+    if (!this.isActive) return;
+
+    this.isActive = false;
+
+    if (this.updateInterval) {
+      clearInterval(this.updateInterval);
+      this.updateInterval = null;
+    }
+
+    // Clear all key states
+    this.keyStates = {};
+
+    // Notify UI of state change
+    if (this.onStateUpdate) {
+      this.onStateUpdate(this.buildTeleoperationState());
+    }
+  }
+
+  getState(): TeleoperatorSpecificState {
+    return {
+      keyStates: { ...this.keyStates },
+    };
+  }
+
+  updateKeyState(key: string, pressed: boolean): void {
+    this.keyStates[key] = {
+      pressed,
+      timestamp: Date.now(),
+    };
+  }
+
+  /**
+   * Move motor to exact position (for sliders and direct control)
+   * This ensures sliders update the same motor configs that the UI displays
+   */
+  async moveMotor(motorName: string, targetPosition: number): Promise<boolean> {
+    const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
+    if (!motorConfig) return false;
+
+    const clampedPosition = Math.max(
+      motorConfig.minPosition,
+      Math.min(motorConfig.maxPosition, targetPosition)
+    );
+
+    try {
+      await writeMotorPosition(
+        this.port,
+        motorConfig.id,
+        Math.round(clampedPosition)
+      );
+      motorConfig.currentPosition = clampedPosition;
+
+      // Notify UI of position change for immediate slider update
+      if (this.onStateUpdate) {
+        this.onStateUpdate(this.buildTeleoperationState());
+      }
+
+      return true;
+    } catch (error) {
+      console.warn(`Failed to move motor ${motorName}:`, error);
+      return false;
+    }
+  }
+
+  private buildTeleoperationState(): TeleoperationState {
+    return {
+      isActive: this.isActive,
+      motorConfigs: [...this.motorConfigs],
+      lastUpdate: Date.now(),
+      keyStates: { ...this.keyStates },
+    };
+  }
+
+  /**
+   * IMPORTANT: This method implements the WORKING keyboard control logic.
+   *
+   * ⚠️  DO NOT MODIFY THIS LOGIC! ⚠️
+   *
+   * This simple approach works perfectly:
+   * - If key is pressed → apply movement every update cycle
+   * - stepSize: 8 units per cycle at 60Hz = smooth, responsive control
+   * - Single taps work naturally (brief key press = few cycles = small movement)
+   * - Held keys work naturally (continuous press = continuous movement)
+   *
+   * Previous "improvements" that BROKE this:
+   * ❌ Trying to detect "first press" vs "held" - breaks everything
+   * ❌ Adding delays/thresholds - makes it clunky
+   * ❌ Event-driven immediate movement - causes multiple applications
+   * ❌ Higher stepSize values - too large jumps
+   *
+   * Keep it simple - this works!
+   */
+  private updateMotorPositions(): void {
+    const now = Date.now();
+
+    // Clear timed-out keys
+    Object.keys(this.keyStates).forEach((key) => {
+      if (now - this.keyStates[key].timestamp > this.keyTimeout) {
+        delete this.keyStates[key];
+      }
+    });
+
+    // Process active keys
+    const activeKeys = Object.keys(this.keyStates).filter(
+      (key) =>
+        this.keyStates[key].pressed &&
+        now - this.keyStates[key].timestamp <= this.keyTimeout
+    );
+
+    // Emergency stop check
+    if (activeKeys.includes("Escape")) {
+      this.stop();
+      return;
+    }
+
+    // Calculate target positions based on active keys
+    // SIMPLE RULE: If key is pressed → apply movement (works perfectly!)
+    const targetPositions: { [motorName: string]: number } = {};
+
+    for (const key of activeKeys) {
+      const control = this.keyboardControls[key];
+      if (!control || control.motor === "emergency_stop") continue;
+
+      const motorConfig = this.motorConfigs.find(
+        (m) => m.name === control.motor
+      );
+      if (!motorConfig) continue;
+
+      // Calculate new position
+      const currentTarget =
+        targetPositions[motorConfig.name] ?? motorConfig.currentPosition;
+      const newPosition = currentTarget + control.direction * this.stepSize;
+
+      // Apply limits
+      targetPositions[motorConfig.name] = Math.max(
+        motorConfig.minPosition,
+        Math.min(motorConfig.maxPosition, newPosition)
+      );
+    }
+
+    // Send motor commands and update positions
+    Object.entries(targetPositions).forEach(([motorName, targetPosition]) => {
+      const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
+      if (motorConfig && targetPosition !== motorConfig.currentPosition) {
+        writeMotorPosition(
+          this.port,
+          motorConfig.id,
+          Math.round(targetPosition)
+        )
+          .then(() => {
+            motorConfig.currentPosition = targetPosition;
+          })
+          .catch((error) => {
+            console.warn(
+              `Failed to write motor ${motorConfig.id} position:`,
+              error
+            );
+          });
+      }
+    });
+  }
+}

packages/web/src/types/teleoperation.ts

@@ -2,6 +2,9 @@
  * Teleoperation-related types for web implementation
  */
 
+import type { RobotConnection } from "./robot-connection.js";
+import type { WebTeleoperator } from "../teleoperators/index.js";
+
 /**
  * Motor position and limits for teleoperation
  */
@@ -20,7 +23,11 @@ export interface TeleoperationState {
   isActive: boolean;
   motorConfigs: MotorConfig[];
   lastUpdate: number;
-  keyStates: { [key: string]: { pressed: boolean; timestamp: number } };
+
+  // Teleoperator-specific state (optional fields for different types)
+  keyStates?: { [key: string]: { pressed: boolean; timestamp: number } }; // keyboard
+  leaderPositions?: { [motor: string]: number }; // leader arm
+  gamepadState?: { axes: number[]; buttons: boolean[] }; // gamepad
 }
 
 /**
@@ -31,9 +38,70 @@ export interface TeleoperationProcess {
   stop(): void;
   updateKeyState(key: string, pressed: boolean): void;
   getState(): TeleoperationState;
-  moveMotor(motorName: string, position: number): Promise<boolean>;
-  setMotorPositions(positions: {
-    [motorName: string]: number;
-  }): Promise<boolean>;
+  teleoperator: WebTeleoperator;
   disconnect(): Promise<void>;
 }
+
+/**
+ * Base interface for all teleoperator configurations
+ */
+export interface BaseTeleoperatorConfig {
+  type: string;
+}
+
+/**
+ * Keyboard teleoperator configuration
+ */
+export interface KeyboardTeleoperatorConfig extends BaseTeleoperatorConfig {
+  type: "keyboard";
+  stepSize?: number; // Default: KEYBOARD_TELEOPERATOR_DEFAULTS.stepSize
+  updateRate?: number; // Default: KEYBOARD_TELEOPERATOR_DEFAULTS.updateRate
+  keyTimeout?: number; // Default: KEYBOARD_TELEOPERATOR_DEFAULTS.keyTimeout
+}
+
+/**
+ * Leader arm teleoperator configuration (future)
+ */
+export interface LeaderArmTeleoperatorConfig extends BaseTeleoperatorConfig {
+  type: "so100_leader";
+  port: string;
+  calibrationData?: any;
+  positionSmoothing?: boolean;
+  scaleFactor?: number;
+}
+
+/**
+ * Direct teleoperator configuration
+ */
+export interface DirectTeleoperatorConfig extends BaseTeleoperatorConfig {
+  type: "direct";
+}
+
+/**
+ * Gamepad teleoperator configuration (future)
+ */
+export interface GamepadTeleoperatorConfig extends BaseTeleoperatorConfig {
+  type: "gamepad";
+  controllerIndex?: number;
+  axisMapping?: { [axis: string]: string };
+  deadzone?: number;
+}
+
+/**
+ * Union type for all teleoperator configurations
+ */
+export type TeleoperatorConfig =
+  | KeyboardTeleoperatorConfig
+  | LeaderArmTeleoperatorConfig
+  | DirectTeleoperatorConfig
+  | GamepadTeleoperatorConfig;
+
+/**
+ * Main teleoperation configuration
+ */
+export interface TeleoperateConfig {
+  robot: RobotConnection;
+  teleop: TeleoperatorConfig;
+  calibrationData?: { [motorName: string]: any };
+  onStateUpdate?: (state: TeleoperationState) => void;
+}

pnpm-lock.yaml

@@ -9,8 +9,8 @@ importers:
   .:
     dependencies:
       '@lerobot/web':
-        specifier: ^0.1.1
-        version: 0.1.1([email protected])
+        specifier: workspace:*
+        version: link:packages/web
       '@radix-ui/react-dialog':
         specifier: ^1.1.14
         version: 1.1.14(@types/[email protected](@types/[email protected]))(@types/[email protected])([email protected]([email protected]))([email protected])
@@ -402,11 +402,6 @@ packages:
   '@jridgewell/[email protected]':
     resolution: {integrity: sha512-vNk6aEwybGtawWmy/PzwnGDOjCkLWSD2wqvjGGAgOAwCGWySYXfYoxt00IJkTF+8Lb57DwOb3Aa0o9CApepiYQ==}
 
-  '@lerobot/[email protected]':
-    resolution: {integrity: sha512-8xLGBTIQQetJzqauM9OtiSUIaSLphCH2qAqiGVzszmJk7pAcucqouezcIGRHiDOVRFDfpAYerRyfoeFLdoKqDQ==}
-    peerDependencies:
-      typescript: '>=4.5.0'
-
   '@manypkg/[email protected]':
     resolution: {integrity: sha512-mki5uBvhHzO8kYYix/WRy2WX8S3B5wdVSc9D6KcU5lQNglP2yt58/VfLuAK49glRXChosY8ap2oJ1qgma3GUVA==}
 
@@ -2085,10 +2080,6 @@ snapshots:
       '@jridgewell/resolve-uri': 3.1.2
       '@jridgewell/sourcemap-codec': 1.5.0
 
-  '@lerobot/[email protected]([email protected])':
-    dependencies:
-      typescript: 5.8.3
-
   '@manypkg/[email protected]':
     dependencies:
       '@babel/runtime': 7.27.6