Aditya Shankar NERDDISCO commited on
Commit
4384839
·
unverified ·
1 Parent(s): 78599e3

feat: added dataset recording; hf uploader, s3 uploader; runpod trainer (#6)

Browse files

* Added recording options
- Added dataset recorder in record.ts
- Added dataset record show example in recorder.tsx
- Added huggingface upload options
- Added zip download options
- Added S3 upload options

* fixed huggingface upload

* fixed video recording

* feat: added react-router-dom to be able to navigate

* feat: show the recorder all the time; make camera config nicer

* - Added threejs display changes
- fixed storage to work episode wise

* feat(recorder): persist camera; better configuration

* docs: how to refactor the pr to make record fit into the lib

* fixed teleoperator, added frame wise tracking in ui

* fixed multi teleoperator recording, added statistical measurement

* refactored code to move recording into record.ts

* updated README to reflect recorder and export functions

* fixed merge errors

* removed 3d rendering

---------

Co-authored-by: Tim Pietrusky <[email protected]>

docs/planning/007_record.md ADDED
@@ -0,0 +1,483 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # User Story 007: Clean Record API Implementation
2
+
3
+ ## Story
4
+
5
+ **As a** robotics developer building teleoperation recording systems
6
+ **I want** to record robot motor positions and control data using a clean `record()` function API
7
+ **So that** I can capture teleoperation sessions for training AI models, analysis, and replay without dealing with complex class-based APIs or mixed concerns
8
+
9
+ ## Background
10
+
11
+ A community contributor has provided a recording implementation in this PR branch, which includes a comprehensive `LeRobotDatasetRecorder` class with video recording, data export, and LeRobot dataset format support. However, the current implementation violates several of our core conventions and doesn't match the clean API patterns established by `calibrate()`, `teleoperate()`, and `findPort()`.
12
+
13
+ ### Current Implementation Problems
14
+
15
+ The existing `LeRobotDatasetRecorder` implementation has several architectural issues:
16
+
17
+ - **Missing Standard Library Pattern**: Uses class instantiation instead of simple function call like our other APIs
18
+ - **Library vs Demo Separation Violation**: Mixes hardware recording (library concern) with video streams, export formats, and UI (demo concerns)
19
+ - **Teleoperator Integration Issues**: Recording logic deeply embedded in `BaseWebTeleoperator` with complex state management
20
+ - **Complex Constructor Anti-Pattern**: Requires pre-configured teleoperators and video streams, violating our "direct library usage" principle
21
+ - **Export API Complexity**: ZIP, HuggingFace, and S3 upload belong in demo code, not standard library
22
+ - **No Clean Process API**: Doesn't follow our consistent `start()/stop()/result` pattern
23
+ - **Redundant Event System**: Uses `dispatchMotorPositionChanged` events that aren't consumed and duplicate callback functionality
24
+ - **Artificial Polling**: 100ms polling in teleoperate instead of immediate callbacks when motors change
25
+
26
+ ### Convention Alignment Needed
27
+
28
+ Our established patterns from `calibrate()`, `teleoperate()`, and `findPort()` follow these principles:
29
+
30
+ - **Simple Function API**: `const process = await record(config)`
31
+ - **Clean Process Objects**: Consistent `start()`, `stop()`, `getState()`, `result` interface
32
+ - **Hardware-Only Library**: Standard library handles only robotics hardware, not UI/storage/export
33
+ - **Demo Handles UI**: Examples handle video, export formats, browser storage, file downloads
34
+ - **Immediate Callbacks**: Real-time updates via callbacks, not polling or unused events
35
+ - **Direct Usage**: End users call library functions directly without complex setup
36
+
37
+ ## Acceptance Criteria
38
+
39
+ ### Core Functionality
40
+
41
+ - [ ] **Standard Library API**: Clean `record(config)` function matching our established patterns
42
+ - [ ] **Process Object Interface**: Consistent `RecordProcess` with `start()`, `stop()`, `getState()`, `result` methods
43
+ - [ ] **Hardware-Only Recording**: Library captures only robot motor positions and teleoperation data
44
+ - [ ] **Real-Time Callbacks**: Immediate `onDataUpdate` and `onStateUpdate` callbacks, no polling
45
+ - [ ] **Device-Agnostic**: Works with any robot type through configuration, not hardcoded values
46
+ - [ ] **Clean Teleoperator Integration**: Recording subscribes to teleoperation changes without embedding in teleoperator classes
47
+
48
+ ### User Experience
49
+
50
+ - [ ] **Simple Integration**: Easy to add recording to existing teleoperation workflows
51
+ - [ ] **Consistent API**: Same patterns as `calibrate()` and `teleoperate()` for familiar developer experience
52
+ - [ ] **Immediate Feedback**: Real-time recording state and data updates for responsive UI
53
+ - [ ] **Error Handling**: Clear error messages for recording failures or invalid configurations
54
+ - [ ] **Resource Management**: Proper cleanup of recording resources on stop/disconnect
55
+
56
+ ### Technical Requirements
57
+
58
+ - [ ] **Library/Demo Separation**: Move video, export, and storage logic to examples/demo layer
59
+ - [ ] **Remove Event System**: Eliminate unused `dispatchMotorPositionChanged` events, use callbacks only
60
+ - [ ] **Extract from Teleoperators**: Remove recording state and logic from `BaseWebTeleoperator`
61
+ - [ ] **TypeScript**: Fully typed with proper interfaces for recording configuration and data
62
+ - [ ] **No Code Duplication**: Reuse existing teleoperation and motor communication infrastructure
63
+ - [ ] **Performance**: Immediate callbacks when data changes, no unnecessary polling
64
+
65
+ ## Expected User Flow
66
+
67
+ ### Basic Robot Recording
68
+
69
+ ```typescript
70
+ import { record } from "@lerobot/web";
71
+
72
+ // Clean API matching our conventions
73
+ const recordProcess = await record({
74
+ robot: connectedRobot,
75
+ options: {
76
+ fps: 30,
77
+ onDataUpdate: (data) => {
78
+ // Real-time recording data for UI feedback
79
+ console.log(`Recorded ${data.frameCount} frames`);
80
+ updateRecordingUI(data);
81
+ },
82
+ onStateUpdate: (state) => {
83
+ // Recording state changes
84
+ console.log(`Recording: ${state.isActive}`);
85
+ updateRecordingStatus(state);
86
+ },
87
+ },
88
+ });
89
+
90
+ // Consistent process interface
91
+ recordProcess.start();
92
+
93
+ // Recording runs automatically while teleoperation is active
94
+ setTimeout(() => {
95
+ recordProcess.stop();
96
+ }, 30000);
97
+
98
+ // Get pure robot recording data
99
+ const robotData = await recordProcess.result;
100
+ console.log("Episodes:", robotData.episodes);
101
+ console.log("Metadata:", robotData.metadata);
102
+ ```
103
+
104
+ ### Recording with Teleoperation
105
+
106
+ ```typescript
107
+ import { teleoperate, record } from "@lerobot/web";
108
+
109
+ // Start teleoperation
110
+ const teleoperationProcess = await teleoperate({
111
+ robot: connectedRobot,
112
+ teleop: { type: "keyboard" },
113
+ calibrationData: calibrationData,
114
+ onStateUpdate: (state) => {
115
+ updateTeleoperationUI(state);
116
+ },
117
+ });
118
+
119
+ // Add recording to existing teleoperation
120
+ const recordProcess = await record({
121
+ robot: connectedRobot,
122
+ options: {
123
+ onDataUpdate: (data) => {
124
+ console.log(`Recording frame ${data.frameCount}`);
125
+ },
126
+ },
127
+ });
128
+
129
+ // Both run independently
130
+ teleoperationProcess.start();
131
+ recordProcess.start();
132
+
133
+ // Control independently
134
+ setTimeout(() => {
135
+ recordProcess.stop(); // Stop recording, keep teleoperation
136
+ }, 60000);
137
+
138
+ setTimeout(() => {
139
+ teleoperationProcess.stop(); // Stop teleoperation
140
+ }, 120000);
141
+ ```
142
+
143
+ ### Demo-Layer Dataset Export
144
+
145
+ ```typescript
146
+ // In examples/demo - NOT in standard library
147
+ import { record } from "@lerobot/web";
148
+ import { DatasetExporter } from "./dataset-exporter"; // Demo code
149
+
150
+ const recordProcess = await record({ robot, options });
151
+ recordProcess.start();
152
+
153
+ // ... recording session ...
154
+
155
+ recordProcess.stop();
156
+ const robotData = await recordProcess.result;
157
+
158
+ // Demo handles complex export logic
159
+ const exporter = new DatasetExporter({
160
+ robotData,
161
+ videoStreams: cameraStreams, // Demo manages video
162
+ taskDescription: "Pick and place task",
163
+ });
164
+
165
+ // Export options handled by demo
166
+ await exporter.downloadZip();
167
+ await exporter.uploadToHuggingFace({ apiKey, repoName });
168
+ await exporter.uploadToS3({ credentials });
169
+ ```
170
+
171
+ ### Component Integration
172
+
173
+ ```typescript
174
+ // React component - direct library usage like calibration
175
+ const [recordingState, setRecordingState] = useState<RecordingState>();
176
+ const [recordingData, setRecordingData] = useState<RecordingData>();
177
+ const recordProcessRef = useRef<RecordProcess | null>(null);
178
+
179
+ useEffect(() => {
180
+ const initRecording = async () => {
181
+ const process = await record({
182
+ robot,
183
+ options: {
184
+ onStateUpdate: setRecordingState,
185
+ onDataUpdate: setRecordingData,
186
+ },
187
+ });
188
+ recordProcessRef.current = process;
189
+ };
190
+ initRecording();
191
+ }, [robot]);
192
+
193
+ const handleStartRecording = () => {
194
+ recordProcessRef.current?.start();
195
+ };
196
+
197
+ const handleStopRecording = async () => {
198
+ recordProcessRef.current?.stop();
199
+ const data = await recordProcessRef.current?.result;
200
+ // Handle recorded data
201
+ };
202
+ ```
203
+
204
+ ## Implementation Details
205
+
206
+ ### File Structure Refactoring
207
+
208
+ ```
209
+ packages/web/src/
210
+ ├── record.ts # NEW: Clean record() function
211
+ ├── types/
212
+ │ └── recording.ts # NEW: Recording-specific types
213
+ ├── utils/
214
+ │ └── recording-manager.ts # NEW: Internal recording logic
215
+ ├── teleoperators/
216
+ │ └── base-teleoperator.ts # UPDATED: Remove recording logic
217
+ └── [MOVED TO EXAMPLES]
218
+ ├── LeRobotDatasetRecorder.ts # Complex export logic
219
+ ├── dataset-exporter.ts # Video + export functionality
220
+ └── upload-handlers.ts # HuggingFace, S3 upload logic
221
+ ```
222
+
223
+ ### Key Dependencies
224
+
225
+ #### No New Dependencies for Standard Library
226
+
227
+ - **Existing**: Reuse all current dependencies (motor communication, teleoperation integration)
228
+ - **Architecture Only**: Pure refactoring to clean up existing functionality
229
+
230
+ #### Demo Dependencies (Moved)
231
+
232
+ - **Video/Export**: `parquet-wasm`, `apache-arrow`, `jszip` - moved to examples
233
+ - **Upload**: `@huggingface/hub`, AWS SDK - moved to examples
234
+
235
+ ### Core Functions to Implement
236
+
237
+ #### Clean Record API
238
+
239
+ ```typescript
240
+ // record.ts - New clean API
241
+ interface RecordConfig {
242
+ robot: RobotConnection;
243
+ options?: {
244
+ fps?: number; // Default: 30
245
+ onDataUpdate?: (data: RecordingData) => void;
246
+ onStateUpdate?: (state: RecordingState) => void;
247
+ };
248
+ }
249
+
250
+ interface RecordProcess {
251
+ start(): void;
252
+ stop(): void;
253
+ getState(): RecordingState;
254
+ result: Promise<RobotRecordingData>;
255
+ }
256
+
257
+ interface RecordingState {
258
+ isActive: boolean;
259
+ frameCount: number;
260
+ episodeCount: number;
261
+ duration: number; // milliseconds
262
+ lastUpdate: number;
263
+ }
264
+
265
+ interface RecordingData {
266
+ frameCount: number;
267
+ currentEpisode: number;
268
+ recentFrames: MotorPositionFrame[]; // Last few frames for UI
269
+ }
270
+
271
+ interface RobotRecordingData {
272
+ episodes: MotorPositionFrame[][]; // Pure motor data only
273
+ metadata: {
274
+ fps: number;
275
+ robotType: string;
276
+ startTime: number;
277
+ endTime: number;
278
+ totalFrames: number;
279
+ totalEpisodes: number;
280
+ };
281
+ }
282
+
283
+ // Main function - matches our conventions
284
+ export async function record(config: RecordConfig): Promise<RecordProcess>;
285
+ ```
286
+
287
+ #### Recording Manager (Internal)
288
+
289
+ ```typescript
290
+ // utils/recording-manager.ts - Internal implementation
291
+ class RecordingManager {
292
+ private robot: RobotConnection;
293
+ private isActive: boolean = false;
294
+ private episodes: MotorPositionFrame[][] = [];
295
+ private currentEpisode: MotorPositionFrame[] = [];
296
+ private startTime: number = 0;
297
+ private frameCount: number = 0;
298
+
299
+ constructor(
300
+ robot: RobotConnection,
301
+ private options: RecordOptions,
302
+ private onDataUpdate?: (data: RecordingData) => void,
303
+ private onStateUpdate?: (state: RecordingState) => void
304
+ ) {
305
+ this.robot = robot;
306
+ }
307
+
308
+ start(): void {
309
+ if (this.isActive) return;
310
+
311
+ this.isActive = true;
312
+ this.startTime = Date.now();
313
+
314
+ // Subscribe to teleoperation changes (NO events, just callbacks)
315
+ this.subscribeToRobotChanges();
316
+
317
+ this.notifyStateUpdate();
318
+ }
319
+
320
+ stop(): void {
321
+ if (!this.isActive) return;
322
+
323
+ this.isActive = false;
324
+ this.finishCurrentEpisode();
325
+ this.unsubscribeFromRobotChanges();
326
+
327
+ this.notifyStateUpdate();
328
+ }
329
+
330
+ private subscribeToRobotChanges(): void {
331
+ // Listen to existing teleoperation callbacks - no new events needed
332
+ // This integrates with the existing onStateUpdate mechanism
333
+ }
334
+
335
+ private recordFrame(motorConfigs: MotorConfig[]): void {
336
+ const frame: MotorPositionFrame = {
337
+ timestamp: Date.now() - this.startTime,
338
+ motorPositions: motorConfigs.map((config) => ({
339
+ id: config.id,
340
+ name: config.name,
341
+ position: config.currentPosition,
342
+ })),
343
+ frameIndex: this.frameCount++,
344
+ };
345
+
346
+ this.currentEpisode.push(frame);
347
+
348
+ if (this.onDataUpdate) {
349
+ this.onDataUpdate({
350
+ frameCount: this.frameCount,
351
+ currentEpisode: this.episodes.length,
352
+ recentFrames: this.currentEpisode.slice(-10), // Last 10 frames
353
+ });
354
+ }
355
+ }
356
+
357
+ getState(): RecordingState {
358
+ return {
359
+ isActive: this.isActive,
360
+ frameCount: this.frameCount,
361
+ episodeCount: this.episodes.length,
362
+ duration: this.isActive ? Date.now() - this.startTime : 0,
363
+ lastUpdate: Date.now(),
364
+ };
365
+ }
366
+
367
+ async getResult(): Promise<RobotRecordingData> {
368
+ return {
369
+ episodes: [...this.episodes],
370
+ metadata: {
371
+ fps: this.options.fps || 30,
372
+ robotType: this.robot.robotType || "unknown",
373
+ startTime: this.startTime,
374
+ endTime: Date.now(),
375
+ totalFrames: this.frameCount,
376
+ totalEpisodes: this.episodes.length,
377
+ },
378
+ };
379
+ }
380
+ }
381
+ ```
382
+
383
+ #### Updated Teleoperate Integration
384
+
385
+ ```typescript
386
+ // teleoperate.ts - Remove 100ms polling, add immediate callbacks
387
+ export async function teleoperate(
388
+ config: TeleoperateConfig
389
+ ): Promise<TeleoperationProcess> {
390
+ const teleoperator = await createTeleoperatorProcess(config);
391
+
392
+ return {
393
+ start: () => {
394
+ teleoperator.start();
395
+
396
+ // NO MORE 100ms polling! Use immediate callbacks
397
+ if (config.onStateUpdate) {
398
+ teleoperator.setStateUpdateCallback(config.onStateUpdate);
399
+ }
400
+ },
401
+ // ... rest of interface
402
+ };
403
+ }
404
+ ```
405
+
406
+ #### Clean Teleoperator Base
407
+
408
+ ```typescript
409
+ // teleoperators/base-teleoperator.ts - Remove recording logic
410
+ export abstract class BaseWebTeleoperator extends WebTeleoperator {
411
+ protected port: MotorCommunicationPort;
412
+ public motorConfigs: MotorConfig[] = [];
413
+ protected isActive: boolean = false;
414
+
415
+ // REMOVED: All recording-related properties
416
+ // REMOVED: dispatchMotorPositionChanged events
417
+ // REMOVED: recordedMotorPositions, episodeIndex, etc.
418
+
419
+ private stateUpdateCallback?: (state: TeleoperationState) => void;
420
+
421
+ setStateUpdateCallback(callback: (state: TeleoperationState) => void): void {
422
+ this.stateUpdateCallback = callback;
423
+ }
424
+
425
+ protected motorPositionsChanged(): void {
426
+ // Call immediately when motors change - no events, no 100ms delay
427
+ if (this.stateUpdateCallback) {
428
+ const state = this.buildCurrentState();
429
+ this.stateUpdateCallback(state);
430
+ }
431
+ }
432
+
433
+ // Clean implementation without recording concerns
434
+ }
435
+ ```
436
+
437
+ ### Technical Considerations
438
+
439
+ #### Migration Strategy
440
+
441
+ **Preserve Existing Functionality:**
442
+
443
+ 1. **Move Complex Logic**: `LeRobotDatasetRecorder` moves to `examples/` as demo code
444
+ 2. **Extract Clean Core**: Create new `record()` function for standard library
445
+ 3. **Update Examples**: Cyberpunk demo uses new API with demo-layer export functionality
446
+ 4. **Remove Event System**: Clean up unused `dispatchMotorPositionChanged` events
447
+ 5. **Fix Polling**: Replace 100ms polling with immediate callbacks
448
+
449
+ #### Performance Improvements
450
+
451
+ - **Remove Polling**: Eliminate artificial 100ms delays in favor of immediate callbacks
452
+ - **Event-Driven**: Only fire callbacks when robot state actually changes
453
+ - **Memory Efficiency**: No unused event listeners or redundant data structures
454
+ - **Responsive UI**: Immediate feedback for recording status and data updates
455
+
456
+ #### Future Extensibility
457
+
458
+ The clean architecture supports advanced recording features as demo enhancements:
459
+
460
+ ```typescript
461
+ // Future: Advanced demo features (NOT in standard library)
462
+ class AdvancedDatasetExporter extends DatasetExporter {
463
+ // Video synchronization, multi-camera support
464
+ // Cloud storage, data preprocessing
465
+ // Visualization, playback, analysis tools
466
+ }
467
+ ```
468
+
469
+ ## Definition of Done
470
+
471
+ - [ ] **Clean Record API**: `record(config)` function implemented matching our established patterns
472
+ - [ ] **Process Interface**: `RecordProcess` with consistent `start()`, `stop()`, `getState()`, `result` methods
473
+ - [ ] **Hardware-Only Library**: Standard library captures only robot motor data, no video/export complexity
474
+ - [ ] **Demo Separation**: Video recording, export formats, and UI logic moved to examples layer
475
+ - [ ] **Remove Events**: `dispatchMotorPositionChanged` events eliminated, callbacks used exclusively
476
+ - [ ] **Fix Polling**: 100ms artificial polling replaced with immediate callbacks when motors change
477
+ - [ ] **Clean Teleoperators**: Recording logic extracted from `BaseWebTeleoperator` and teleoperator classes
478
+ - [ ] **TypeScript Coverage**: Full type safety with proper interfaces for all recording functionality
479
+ - [ ] **Performance**: Immediate, event-driven updates with no unnecessary polling or unused listeners
480
+ - [ ] **Integration**: Easy integration with existing teleoperation workflows using familiar patterns
481
+ - [ ] **Example Updates**: Cyberpunk demo updated to use new clean API with demo-layer export features
482
+ - [ ] **No Regression**: All existing recording functionality preserved through demo layer
483
+ - [ ] **Documentation**: Clear examples showing standard library vs demo separation
examples/cyberpunk-standalone/package.json CHANGED
@@ -10,7 +10,10 @@
10
  "preview": "vite preview"
11
  },
12
  "dependencies": {
 
 
13
  "@hookform/resolvers": "^3.9.1",
 
14
  "@lerobot/web": "file:../../packages/web",
15
  "@radix-ui/react-accordion": "1.2.2",
16
  "@radix-ui/react-alert-dialog": "1.1.4",
@@ -39,7 +42,11 @@
39
  "@radix-ui/react-toggle": "1.1.1",
40
  "@radix-ui/react-toggle-group": "1.1.1",
41
  "@radix-ui/react-tooltip": "1.1.6",
 
 
 
42
  "@types/react-syntax-highlighter": "^15.5.13",
 
43
  "autoprefixer": "^10.4.20",
44
  "class-variance-authority": "^0.7.1",
45
  "clsx": "^2.1.1",
@@ -49,16 +56,19 @@
49
  "input-otp": "1.4.1",
50
  "lucide-react": "^0.454.0",
51
  "next-themes": "^0.4.6",
 
52
  "react": "^19",
53
  "react-day-picker": "8.10.1",
54
  "react-dom": "^19",
55
  "react-hook-form": "^7.54.1",
56
  "react-resizable-panels": "^2.1.7",
 
57
  "react-syntax-highlighter": "^15.6.1",
58
  "recharts": "2.15.0",
59
  "sonner": "^1.7.1",
60
  "tailwind-merge": "^2.5.5",
61
  "tailwindcss-animate": "^1.0.7",
 
62
  "vaul": "^0.9.6",
63
  "zod": "^3.24.1"
64
  },
 
10
  "preview": "vite preview"
11
  },
12
  "dependencies": {
13
+ "@aws-sdk/client-s3": "^3.850.0",
14
+ "@aws-sdk/lib-storage": "^3.850.0",
15
  "@hookform/resolvers": "^3.9.1",
16
+ "@huggingface/hub": "^2.4.0",
17
  "@lerobot/web": "file:../../packages/web",
18
  "@radix-ui/react-accordion": "1.2.2",
19
  "@radix-ui/react-alert-dialog": "1.1.4",
 
42
  "@radix-ui/react-toggle": "1.1.1",
43
  "@radix-ui/react-toggle-group": "1.1.1",
44
  "@radix-ui/react-tooltip": "1.1.6",
45
+ "@react-three/cannon": "^6.6.0",
46
+ "@react-three/drei": "^10.6.0",
47
+ "@react-three/fiber": "^9.2.0",
48
  "@types/react-syntax-highlighter": "^15.5.13",
49
+ "apache-arrow": "^21.0.0",
50
  "autoprefixer": "^10.4.20",
51
  "class-variance-authority": "^0.7.1",
52
  "clsx": "^2.1.1",
 
56
  "input-otp": "1.4.1",
57
  "lucide-react": "^0.454.0",
58
  "next-themes": "^0.4.6",
59
+ "parquet-wasm": "^0.6.1",
60
  "react": "^19",
61
  "react-day-picker": "8.10.1",
62
  "react-dom": "^19",
63
  "react-hook-form": "^7.54.1",
64
  "react-resizable-panels": "^2.1.7",
65
+ "react-router-dom": "^7.7.1",
66
  "react-syntax-highlighter": "^15.6.1",
67
  "recharts": "2.15.0",
68
  "sonner": "^1.7.1",
69
  "tailwind-merge": "^2.5.5",
70
  "tailwindcss-animate": "^1.0.7",
71
+ "three": "^0.178.0",
72
  "vaul": "^0.9.6",
73
  "zod": "^3.24.1"
74
  },
examples/cyberpunk-standalone/pnpm-lock.yaml CHANGED
The diff for this file is too large to render. See raw diff
 
examples/cyberpunk-standalone/src/App.tsx CHANGED
@@ -1,4 +1,11 @@
1
  import { useState, useEffect, useRef } from "react";
 
 
 
 
 
 
 
2
  import { ChevronLeft } from "lucide-react";
3
  import { Header } from "@/components/header";
4
  import { Footer } from "@/components/footer";
@@ -27,19 +34,15 @@ import {
27
  } from "@/lib/unified-storage";
28
 
29
  function App() {
30
- const [view, setView] = useState<
31
- "dashboard" | "calibrating" | "teleoperating"
32
- >("dashboard");
33
  const [robots, setRobots] = useState<RobotConnection[]>([]);
34
- const [selectedRobot, setSelectedRobot] = useState<RobotConnection | null>(
35
- null
36
- );
37
  const [editingRobot, setEditingRobot] = useState<RobotConnection | null>(
38
  null
39
  );
40
  const [isConnecting, setIsConnecting] = useState(false);
41
  const hardwareSectionRef = useRef<HTMLDivElement>(null);
42
  const { toast } = useToast();
 
 
43
 
44
  // Check browser support
45
  const isSupported = isWebSerialSupported();
@@ -231,8 +234,7 @@ function App() {
231
  return;
232
  }
233
 
234
- setSelectedRobot(robot);
235
- setView("calibrating");
236
  };
237
 
238
  const handleTeleoperate = (robot: RobotConnection) => {
@@ -245,122 +247,57 @@ function App() {
245
  return;
246
  }
247
 
248
- setSelectedRobot(robot);
249
- setView("teleoperating");
250
  };
251
 
252
- const handleCloseSubView = () => {
253
- setSelectedRobot(null);
254
- setView("dashboard");
255
  };
256
 
257
  const scrollToHardware = () => {
258
  hardwareSectionRef.current?.scrollIntoView({ behavior: "smooth" });
259
  };
260
 
261
- const renderView = () => {
262
- switch (view) {
263
- case "calibrating":
264
- return selectedRobot && <CalibrationView robot={selectedRobot} />;
265
- case "teleoperating":
266
- return selectedRobot && <TeleoperationView robot={selectedRobot} />;
267
- case "dashboard":
268
- default:
269
- return (
270
- <div className="space-y-20">
271
- <DeviceDashboard
272
- robots={robots}
273
- onCalibrate={handleCalibrate}
274
- onTeleoperate={handleTeleoperate}
275
- onRemove={handleRemoveRobot}
276
- onEdit={setEditingRobot}
277
- onFindNew={handleFindNewRobots}
278
- isConnecting={isConnecting}
279
- onScrollToHardware={scrollToHardware}
280
- />
281
- <div>
282
- <div className="mb-6">
283
- <h2 className="text-3xl font-bold font-mono tracking-wider mb-2 uppercase">
284
- install
285
- </h2>
286
- <p className="text-sm text-muted-foreground font-mono">
287
- Choose your preferred development environment
288
- </p>
289
- </div>
290
- <SetupCards />
291
- </div>
292
- <DocsSection />
293
- <RoadmapSection />
294
- <div ref={hardwareSectionRef}>
295
- <HardwareSupportSection />
296
- </div>
297
- </div>
298
- );
299
- }
300
- };
301
-
302
- const PageHeader = () => {
303
- return (
304
- <div className="flex items-center justify-between mb-12">
305
- <div className="flex items-center gap-4">
306
- <div>
307
- {view === "calibrating" && selectedRobot ? (
308
- <h1 className="font-mono text-4xl font-bold tracking-wider">
309
- <span className="text-muted-foreground uppercase">
310
- calibrate:
311
- </span>{" "}
312
- <span
313
- className="text-primary text-glitch uppercase"
314
- data-text={selectedRobot.robotId}
315
- >
316
- {selectedRobot.robotId?.toUpperCase()}
317
- </span>
318
- </h1>
319
- ) : view === "teleoperating" && selectedRobot ? (
320
- <h1 className="font-mono text-4xl font-bold tracking-wider">
321
- <span className="text-muted-foreground uppercase">
322
- teleoperate:
323
- </span>{" "}
324
- <span
325
- className="text-primary text-glitch uppercase"
326
- data-text={selectedRobot.robotId}
327
- >
328
- {selectedRobot.robotId?.toUpperCase()}
329
- </span>
330
- </h1>
331
- ) : (
332
- <h1
333
- className="font-mono text-4xl font-bold text-primary tracking-wider text-glitch uppercase"
334
- data-text="dashboard"
335
- >
336
- DASHBOARD
337
- </h1>
338
- )}
339
- <div className="h-6 flex items-center">
340
- {view !== "dashboard" ? (
341
- <button
342
- onClick={handleCloseSubView}
343
- className="flex items-center gap-2 text-sm text-muted-foreground font-mono hover:text-primary transition-colors"
344
- >
345
- <ChevronLeft className="w-4 h-4" />
346
- <span className="uppercase">back to dashboard</span>
347
- </button>
348
- ) : (
349
- <p className="text-sm text-muted-foreground font-mono">{""} </p>
350
- )}
351
- </div>
352
- </div>
353
- </div>
354
- </div>
355
- );
356
- };
357
-
358
  return (
359
  <div className="flex flex-col min-h-screen font-sans bg-gray-200 dark:bg-background">
360
  <Header />
361
  <main className="flex-grow container mx-auto py-12 px-4 md:px-6">
362
- <PageHeader />
363
- {renderView()}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
364
  <EditRobotDialog
365
  robot={editingRobot}
366
  isOpen={!!editingRobot}
@@ -374,4 +311,203 @@ function App() {
374
  );
375
  }
376
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
377
  export default App;
 
1
  import { useState, useEffect, useRef } from "react";
2
+ import {
3
+ Routes,
4
+ Route,
5
+ useNavigate,
6
+ useParams,
7
+ useLocation,
8
+ } from "react-router-dom";
9
  import { ChevronLeft } from "lucide-react";
10
  import { Header } from "@/components/header";
11
  import { Footer } from "@/components/footer";
 
34
  } from "@/lib/unified-storage";
35
 
36
  function App() {
 
 
 
37
  const [robots, setRobots] = useState<RobotConnection[]>([]);
 
 
 
38
  const [editingRobot, setEditingRobot] = useState<RobotConnection | null>(
39
  null
40
  );
41
  const [isConnecting, setIsConnecting] = useState(false);
42
  const hardwareSectionRef = useRef<HTMLDivElement>(null);
43
  const { toast } = useToast();
44
+ const navigate = useNavigate();
45
+ const location = useLocation();
46
 
47
  // Check browser support
48
  const isSupported = isWebSerialSupported();
 
234
  return;
235
  }
236
 
237
+ navigate(`/device/${robot.serialNumber}/calibrate`);
 
238
  };
239
 
240
  const handleTeleoperate = (robot: RobotConnection) => {
 
247
  return;
248
  }
249
 
250
+ navigate(`/device/${robot.serialNumber}/control`);
 
251
  };
252
 
253
+ const handleBackToDashboard = () => {
254
+ navigate("/");
 
255
  };
256
 
257
  const scrollToHardware = () => {
258
  hardwareSectionRef.current?.scrollIntoView({ behavior: "smooth" });
259
  };
260
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
261
  return (
262
  <div className="flex flex-col min-h-screen font-sans bg-gray-200 dark:bg-background">
263
  <Header />
264
  <main className="flex-grow container mx-auto py-12 px-4 md:px-6">
265
+ <Routes>
266
+ <Route
267
+ path="/"
268
+ element={
269
+ <DashboardPage
270
+ robots={robots}
271
+ onCalibrate={handleCalibrate}
272
+ onTeleoperate={handleTeleoperate}
273
+ onRemove={handleRemoveRobot}
274
+ onEdit={setEditingRobot}
275
+ onFindNew={handleFindNewRobots}
276
+ isConnecting={isConnecting}
277
+ onScrollToHardware={scrollToHardware}
278
+ hardwareSectionRef={hardwareSectionRef}
279
+ />
280
+ }
281
+ />
282
+ <Route
283
+ path="/device/:serialNumber/calibrate"
284
+ element={
285
+ <CalibratePage
286
+ robots={robots}
287
+ onBackToDashboard={handleBackToDashboard}
288
+ />
289
+ }
290
+ />
291
+ <Route
292
+ path="/device/:serialNumber/control"
293
+ element={
294
+ <ControlPage
295
+ robots={robots}
296
+ onBackToDashboard={handleBackToDashboard}
297
+ />
298
+ }
299
+ />
300
+ </Routes>
301
  <EditRobotDialog
302
  robot={editingRobot}
303
  isOpen={!!editingRobot}
 
311
  );
312
  }
313
 
314
+ // Dashboard Page Component
315
+ function DashboardPage({
316
+ robots,
317
+ onCalibrate,
318
+ onTeleoperate,
319
+ onRemove,
320
+ onEdit,
321
+ onFindNew,
322
+ isConnecting,
323
+ onScrollToHardware,
324
+ hardwareSectionRef,
325
+ }: {
326
+ robots: RobotConnection[];
327
+ onCalibrate: (robot: RobotConnection) => void;
328
+ onTeleoperate: (robot: RobotConnection) => void;
329
+ onRemove: (robot: RobotConnection) => void;
330
+ onEdit: (robot: RobotConnection | null) => void;
331
+ onFindNew: () => void;
332
+ isConnecting: boolean;
333
+ onScrollToHardware: () => void;
334
+ hardwareSectionRef: React.RefObject<HTMLDivElement>;
335
+ }) {
336
+ return (
337
+ <div>
338
+ <PageHeader />
339
+ <div className="space-y-20">
340
+ <DeviceDashboard
341
+ robots={robots}
342
+ onCalibrate={onCalibrate}
343
+ onTeleoperate={onTeleoperate}
344
+ onRemove={onRemove}
345
+ onEdit={onEdit}
346
+ onFindNew={onFindNew}
347
+ isConnecting={isConnecting}
348
+ onScrollToHardware={onScrollToHardware}
349
+ />
350
+ <div>
351
+ <div className="mb-6">
352
+ <h2 className="text-3xl font-bold font-mono tracking-wider mb-2 uppercase">
353
+ install
354
+ </h2>
355
+ <p className="text-sm text-muted-foreground font-mono">
356
+ Choose your preferred development environment
357
+ </p>
358
+ </div>
359
+ <SetupCards />
360
+ </div>
361
+ <DocsSection />
362
+ <RoadmapSection />
363
+ <div ref={hardwareSectionRef}>
364
+ <HardwareSupportSection />
365
+ </div>
366
+ </div>
367
+ </div>
368
+ );
369
+ }
370
+
371
+ // Calibrate Page Component
372
+ function CalibratePage({
373
+ robots,
374
+ onBackToDashboard,
375
+ }: {
376
+ robots: RobotConnection[];
377
+ onBackToDashboard: () => void;
378
+ }) {
379
+ const { serialNumber } = useParams<{ serialNumber: string }>();
380
+ const selectedRobot = robots.find(
381
+ (robot) => robot.serialNumber === serialNumber
382
+ );
383
+
384
+ if (!selectedRobot) {
385
+ return (
386
+ <div>
387
+ <PageHeader onBackToDashboard={onBackToDashboard} />
388
+ <div className="text-center py-20">
389
+ <p className="text-muted-foreground">
390
+ Device not found or not connected.
391
+ </p>
392
+ </div>
393
+ </div>
394
+ );
395
+ }
396
+
397
+ return (
398
+ <div>
399
+ <PageHeader
400
+ onBackToDashboard={onBackToDashboard}
401
+ selectedRobot={selectedRobot}
402
+ />
403
+ <CalibrationView robot={selectedRobot} />
404
+ </div>
405
+ );
406
+ }
407
+
408
+ // Control Page Component
409
+ function ControlPage({
410
+ robots,
411
+ onBackToDashboard,
412
+ }: {
413
+ robots: RobotConnection[];
414
+ onBackToDashboard: () => void;
415
+ }) {
416
+ const { serialNumber } = useParams<{ serialNumber: string }>();
417
+ const selectedRobot = robots.find(
418
+ (robot) => robot.serialNumber === serialNumber
419
+ );
420
+
421
+ if (!selectedRobot) {
422
+ return (
423
+ <div>
424
+ <PageHeader onBackToDashboard={onBackToDashboard} />
425
+ <div className="text-center py-20">
426
+ <p className="text-muted-foreground">
427
+ Device not found or not connected.
428
+ </p>
429
+ </div>
430
+ </div>
431
+ );
432
+ }
433
+
434
+ return (
435
+ <div>
436
+ <PageHeader
437
+ onBackToDashboard={onBackToDashboard}
438
+ selectedRobot={selectedRobot}
439
+ />
440
+ <TeleoperationView robot={selectedRobot} />
441
+ </div>
442
+ );
443
+ }
444
+
445
+ // Page Header Component
446
+ function PageHeader({
447
+ onBackToDashboard,
448
+ selectedRobot,
449
+ }: {
450
+ onBackToDashboard?: () => void;
451
+ selectedRobot?: RobotConnection | null;
452
+ }) {
453
+ const location = useLocation();
454
+ const isDashboard = location.pathname === "/";
455
+ const isCalibrating = location.pathname.includes("/calibrate");
456
+ const isTeleoperating = location.pathname.includes("/control");
457
+
458
+ return (
459
+ <div className="flex items-center justify-between mb-12">
460
+ <div className="flex items-center gap-4">
461
+ <div>
462
+ {isCalibrating && selectedRobot ? (
463
+ <h1 className="font-mono text-4xl font-bold tracking-wider">
464
+ <span className="text-muted-foreground uppercase">
465
+ calibrate:
466
+ </span>{" "}
467
+ <span
468
+ className="text-primary text-glitch uppercase"
469
+ data-text={selectedRobot.robotId}
470
+ >
471
+ {selectedRobot.robotId?.toUpperCase()}
472
+ </span>
473
+ </h1>
474
+ ) : isTeleoperating && selectedRobot ? (
475
+ <h1 className="font-mono text-4xl font-bold tracking-wider">
476
+ <span className="text-muted-foreground uppercase">
477
+ teleoperate:
478
+ </span>{" "}
479
+ <span
480
+ className="text-primary text-glitch uppercase"
481
+ data-text={selectedRobot.robotId}
482
+ >
483
+ {selectedRobot.robotId?.toUpperCase()}
484
+ </span>
485
+ </h1>
486
+ ) : (
487
+ <h1
488
+ className="font-mono text-4xl font-bold text-primary tracking-wider text-glitch uppercase"
489
+ data-text="dashboard"
490
+ >
491
+ DASHBOARD
492
+ </h1>
493
+ )}
494
+ <div className="h-6 flex items-center">
495
+ {!isDashboard && onBackToDashboard ? (
496
+ <button
497
+ onClick={onBackToDashboard}
498
+ className="flex items-center gap-2 text-sm text-muted-foreground font-mono hover:text-primary transition-colors"
499
+ >
500
+ <ChevronLeft className="w-4 h-4" />
501
+ <span className="uppercase">back to dashboard</span>
502
+ </button>
503
+ ) : (
504
+ <p className="text-sm text-muted-foreground font-mono">{""} </p>
505
+ )}
506
+ </div>
507
+ </div>
508
+ </div>
509
+ </div>
510
+ );
511
+ }
512
+
513
  export default App;
examples/cyberpunk-standalone/src/components/recorder.tsx ADDED
@@ -0,0 +1,1318 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ "use client";
2
+
3
+ import { useState, useEffect, useRef, useCallback, useMemo } from "react";
4
+ import { Button } from "@/components/ui/button";
5
+ import { Card } from "@/components/ui/card";
6
+ import {
7
+ Table,
8
+ TableBody,
9
+ TableCaption,
10
+ TableCell,
11
+ TableHead,
12
+ TableHeader,
13
+ TableRow,
14
+ } from "@/components/ui/table";
15
+ import { Input } from "@/components/ui/input";
16
+ import { Badge } from "@/components/ui/badge";
17
+ import {
18
+ Select,
19
+ SelectContent,
20
+ SelectItem,
21
+ SelectTrigger,
22
+ SelectValue,
23
+ } from "@/components/ui/select";
24
+ import { useToast } from "@/hooks/use-toast";
25
+ import {
26
+ Disc as Record,
27
+ Download,
28
+ Upload,
29
+ PlusCircle,
30
+ Square,
31
+ Camera,
32
+ Trash2,
33
+ Settings,
34
+ RefreshCw,
35
+ X,
36
+ Edit2,
37
+ Check,
38
+ } from "lucide-react";
39
+ import { LeRobotDatasetRecorder, LeRobotDatasetRow, NonIndexedLeRobotDatasetRow, LeRobotEpisode } from "@lerobot/web";
40
+ import { TeleoperatorEpisodesView } from "./teleoperator-episodes-view";
41
+
42
+ interface RecorderProps {
43
+ teleoperators: any[];
44
+ robot: any; // eslint-disable-line @typescript-eslint/no-explicit-any
45
+ onNeedsTeleoperation: () => Promise<boolean>;
46
+ videoStreams?: { [key: string]: MediaStream };
47
+ }
48
+
49
+
50
+
51
+ interface RecorderSettings {
52
+ huggingfaceApiKey: string;
53
+ cameraConfigs: {
54
+ [cameraName: string]: {
55
+ deviceId: string;
56
+ deviceLabel: string;
57
+ };
58
+ };
59
+ }
60
+
61
+ // Storage functions for recorder settings
62
+ const RECORDER_SETTINGS_KEY = "lerobot-recorder-settings";
63
+
64
+ function getRecorderSettings(): RecorderSettings {
65
+ try {
66
+ const stored = localStorage.getItem(RECORDER_SETTINGS_KEY);
67
+ if (stored) {
68
+ return JSON.parse(stored);
69
+ }
70
+ } catch (error) {
71
+ console.warn("Failed to load recorder settings:", error);
72
+ }
73
+ return {
74
+ huggingfaceApiKey: "",
75
+ cameraConfigs: {},
76
+ };
77
+ }
78
+
79
+ function saveRecorderSettings(settings: RecorderSettings): void {
80
+ try {
81
+ localStorage.setItem(RECORDER_SETTINGS_KEY, JSON.stringify(settings));
82
+ } catch (error) {
83
+ console.warn("Failed to save recorder settings:", error);
84
+ }
85
+ }
86
+
87
+ export function Recorder({
88
+ teleoperators,
89
+ robot,
90
+ onNeedsTeleoperation,
91
+ }: RecorderProps) {
92
+ const [isRecording, setIsRecording] = useState(false);
93
+ const [currentEpisode, setCurrentEpisode] = useState(0);
94
+ // Use huggingfaceApiKey from recorderSettings instead of separate state
95
+ const [cameraName, setCameraName] = useState("");
96
+ const [additionalCameras, setAdditionalCameras] = useState<{
97
+ [key: string]: MediaStream;
98
+ }>({});
99
+ const [availableCameras, setAvailableCameras] = useState<MediaDeviceInfo[]>(
100
+ []
101
+ );
102
+ const [selectedCameraId, setSelectedCameraId] = useState<string>("");
103
+ const [previewStream, setPreviewStream] = useState<MediaStream | null>(null);
104
+ const [isLoadingCameras, setIsLoadingCameras] = useState(false);
105
+ const [cameraPermissionState, setCameraPermissionState] = useState<
106
+ "unknown" | "granted" | "denied"
107
+ >("unknown");
108
+ const [showCameraConfig, setShowCameraConfig] = useState(false);
109
+ const [showConfigure, setShowConfigure] = useState(false);
110
+ const [recorderSettings, setRecorderSettings] = useState<RecorderSettings>(
111
+ () => getRecorderSettings()
112
+ );
113
+ const [hasRecordedFrames, setHasRecordedFrames] = useState(false);
114
+ const [editingCameraName, setEditingCameraName] = useState<string | null>(
115
+ null
116
+ );
117
+ const [editingCameraNewName, setEditingCameraNewName] = useState("");
118
+ const [huggingfaceApiKey, setHuggingfaceApiKey] = useState("")
119
+ const recorderRef = useRef<LeRobotDatasetRecorder | null>(null);
120
+ const videoRef = useRef<HTMLVideoElement | null>(null);
121
+ const { toast } = useToast();
122
+
123
+ // Initialize the recorder when teleoperators are available
124
+ useEffect(() => {
125
+ if (teleoperators.length > 0) {
126
+ recorderRef.current = new LeRobotDatasetRecorder(
127
+ teleoperators,
128
+ additionalCameras,
129
+ 30, // fps
130
+ "Robot teleoperation recording"
131
+ );
132
+ }
133
+ }, [teleoperators, additionalCameras]);
134
+
135
+ const handleStartRecording = async () => {
136
+ // If teleoperators aren't available, initialize teleoperation first
137
+ if (teleoperators.length === 0) {
138
+ toast({
139
+ title: "Initializing...",
140
+ description: `Setting up robot control for ${robot.robotId || "robot"}`,
141
+ });
142
+
143
+ const success = await onNeedsTeleoperation();
144
+ if (!success) {
145
+ toast({
146
+ title: "Recording Error",
147
+ description: "Failed to initialize robot control",
148
+ variant: "destructive",
149
+ });
150
+ return;
151
+ }
152
+
153
+ // Wait a moment for the recorder to initialize with new teleoperators
154
+ await new Promise((resolve) => setTimeout(resolve, 100));
155
+ }
156
+
157
+ if (!recorderRef.current) {
158
+ toast({
159
+ title: "Recording Error",
160
+ description: "Recorder not ready yet. Please try again.",
161
+ variant: "destructive",
162
+ });
163
+ return;
164
+ }
165
+
166
+ try {
167
+ // Set the episode index
168
+ recorderRef.current.setEpisodeIndex(currentEpisode);
169
+ recorderRef.current.setTaskIndex(0); // Default task index
170
+
171
+ // Start recording
172
+ recorderRef.current.startRecording();
173
+ setIsRecording(true);
174
+ setHasRecordedFrames(true);
175
+
176
+ toast({
177
+ title: "Recording Started",
178
+ description: `Episode ${currentEpisode} is now recording`,
179
+ });
180
+ } catch (error) {
181
+ const errorMessage =
182
+ error instanceof Error ? error.message : "Failed to start recording";
183
+ toast({
184
+ title: "Recording Error",
185
+ description: errorMessage,
186
+ variant: "destructive",
187
+ });
188
+ }
189
+ };
190
+
191
+ const handleStopRecording = async () => {
192
+ if (!recorderRef.current || !isRecording) {
193
+ return;
194
+ }
195
+
196
+ try {
197
+ const result = await recorderRef.current.stopRecording();
198
+ setIsRecording(false);
199
+
200
+ toast({
201
+ title: "Recording Stopped",
202
+ description: `Episode ${currentEpisode} completed with ${result.teleoperatorData.length} frames`,
203
+ });
204
+ } catch (error) {
205
+ const errorMessage =
206
+ error instanceof Error ? error.message : "Failed to stop recording";
207
+ toast({
208
+ title: "Recording Error",
209
+ description: errorMessage,
210
+ variant: "destructive",
211
+ });
212
+ }
213
+ };
214
+
215
+ const handleNextEpisode = () => {
216
+ // Make sure we're not recording
217
+ if (isRecording) {
218
+ handleStopRecording();
219
+ }
220
+
221
+ // Increment episode counter
222
+ setCurrentEpisode((prev) => prev + 1);
223
+
224
+ toast({
225
+ title: "New Episode",
226
+ description: `Ready to record episode ${currentEpisode + 1}`,
227
+ });
228
+ };
229
+
230
+ // Reset frames by clearing the recorder data
231
+ const handleResetFrames = useCallback(() => {
232
+ if (isRecording) {
233
+ handleStopRecording();
234
+ }
235
+
236
+ if (recorderRef.current) {
237
+ recorderRef.current.clearRecording();
238
+ setHasRecordedFrames(false);
239
+
240
+ toast({
241
+ title: "Frames Reset",
242
+ description: "All recorded frames have been cleared",
243
+ });
244
+ }
245
+ }, [isRecording, toast]);
246
+
247
+ // Load available cameras
248
+ const loadAvailableCameras = useCallback(
249
+ async (isAutoLoad = false) => {
250
+ if (isLoadingCameras) return;
251
+
252
+ setIsLoadingCameras(true);
253
+
254
+ try {
255
+ // Check if we already have permission
256
+ const permission = await navigator.permissions.query({
257
+ name: "camera" as PermissionName,
258
+ });
259
+ setCameraPermissionState(
260
+ permission.state === "granted"
261
+ ? "granted"
262
+ : permission.state === "denied"
263
+ ? "denied"
264
+ : "unknown"
265
+ );
266
+
267
+ let tempStream: MediaStream | null = null;
268
+
269
+ // Try to enumerate devices first (works if we have permission)
270
+ const devices = await navigator.mediaDevices.enumerateDevices();
271
+ const videoDevices = devices.filter(
272
+ (device) => device.kind === "videoinput"
273
+ );
274
+
275
+ // If devices have labels, we already have permission
276
+ const hasLabels = videoDevices.some((device) => device.label);
277
+ let finalVideoDevices = videoDevices;
278
+
279
+ if (!hasLabels && videoDevices.length > 0) {
280
+ // Need to request permission to get device labels
281
+ tempStream = await navigator.mediaDevices.getUserMedia({
282
+ video: true,
283
+ });
284
+
285
+ // Re-enumerate to get labels
286
+ const devicesWithLabels =
287
+ await navigator.mediaDevices.enumerateDevices();
288
+ const videoDevicesWithLabels = devicesWithLabels.filter(
289
+ (device) => device.kind === "videoinput"
290
+ );
291
+ finalVideoDevices = videoDevicesWithLabels;
292
+ setAvailableCameras(videoDevicesWithLabels);
293
+
294
+ if (!isAutoLoad) {
295
+ console.log(
296
+ `Found ${videoDevicesWithLabels.length} video devices:`,
297
+ videoDevicesWithLabels.map((d) => d.label || d.deviceId)
298
+ );
299
+ }
300
+ } else {
301
+ setAvailableCameras(videoDevices);
302
+ if (!isAutoLoad) {
303
+ console.log(
304
+ `Found ${videoDevices.length} video devices:`,
305
+ videoDevices.map((d) => d.label || d.deviceId)
306
+ );
307
+ }
308
+ }
309
+
310
+ // Auto-select and preview first camera if none selected
311
+ if (finalVideoDevices.length > 0 && !selectedCameraId) {
312
+ const firstCameraId = finalVideoDevices[0].deviceId;
313
+
314
+ // Stop temp stream since we'll create a fresh one with switchCameraPreview
315
+ if (tempStream) {
316
+ tempStream.getTracks().forEach((track) => track.stop());
317
+ }
318
+
319
+ setCameraPermissionState("granted");
320
+
321
+ // Use the same logic as manual camera switching
322
+ await switchCameraPreview(firstCameraId);
323
+ } else if (tempStream) {
324
+ // Stop temp stream if we didn't use it
325
+ tempStream.getTracks().forEach((track) => track.stop());
326
+ }
327
+ } catch (error) {
328
+ setCameraPermissionState("denied");
329
+ if (!isAutoLoad) {
330
+ toast({
331
+ title: "Camera Error",
332
+ description: `Failed to load cameras: ${
333
+ error instanceof Error ? error.message : String(error)
334
+ }`,
335
+ variant: "destructive",
336
+ });
337
+ }
338
+ } finally {
339
+ setIsLoadingCameras(false);
340
+ }
341
+ },
342
+ [selectedCameraId, toast]
343
+ );
344
+
345
+ // Switch camera preview
346
+ const switchCameraPreview = useCallback(
347
+ async (deviceId: string) => {
348
+ try {
349
+ // Stop current preview stream
350
+ if (previewStream) {
351
+ previewStream.getTracks().forEach((track) => track.stop());
352
+ }
353
+
354
+ // Start new stream with selected camera
355
+ const newStream = await navigator.mediaDevices.getUserMedia({
356
+ video: {
357
+ deviceId: { exact: deviceId },
358
+ width: { ideal: 1280 },
359
+ height: { ideal: 720 },
360
+ },
361
+ });
362
+
363
+ setPreviewStream(newStream);
364
+ setSelectedCameraId(deviceId);
365
+ } catch (error) {
366
+ toast({
367
+ title: "Camera Error",
368
+ description: `Failed to switch camera: ${
369
+ error instanceof Error ? error.message : String(error)
370
+ }`,
371
+ variant: "destructive",
372
+ });
373
+ }
374
+ },
375
+ [previewStream, toast]
376
+ );
377
+
378
+ // Add a new camera to the recorder
379
+ const handleAddCamera = useCallback(async () => {
380
+ if (!cameraName.trim()) {
381
+ toast({
382
+ title: "Camera Error",
383
+ description: "Please enter a camera name",
384
+ variant: "destructive",
385
+ });
386
+ return;
387
+ }
388
+
389
+ if (hasRecordedFrames) {
390
+ toast({
391
+ title: "Camera Error",
392
+ description: "Cannot add cameras after recording has started",
393
+ variant: "destructive",
394
+ });
395
+ return;
396
+ }
397
+
398
+ if (!selectedCameraId) {
399
+ toast({
400
+ title: "Camera Error",
401
+ description: "Please select a camera first",
402
+ variant: "destructive",
403
+ });
404
+ return;
405
+ }
406
+
407
+ try {
408
+ // Use the current preview stream (already running with correct camera)
409
+ if (!previewStream) {
410
+ throw new Error("No camera preview available");
411
+ }
412
+
413
+ // Clone the stream for recording (keep preview running)
414
+ const recordingStream = previewStream.clone();
415
+
416
+ // Add the new camera to our state
417
+ setAdditionalCameras((prev) => ({
418
+ ...prev,
419
+ [cameraName]: recordingStream,
420
+ }));
421
+
422
+ // Save camera configuration to persistent storage
423
+ const selectedCamera = availableCameras.find(
424
+ (cam) => cam.deviceId === selectedCameraId
425
+ );
426
+ const newSettings = {
427
+ ...recorderSettings,
428
+ cameraConfigs: {
429
+ ...recorderSettings.cameraConfigs,
430
+ [cameraName]: {
431
+ deviceId: selectedCameraId,
432
+ deviceLabel:
433
+ selectedCamera?.label ||
434
+ `Camera ${selectedCameraId.slice(0, 8)}...`,
435
+ },
436
+ },
437
+ };
438
+ setRecorderSettings(newSettings);
439
+ saveRecorderSettings(newSettings);
440
+
441
+ setCameraName(""); // Clear the input
442
+
443
+ toast({
444
+ title: "Camera Added",
445
+ description: `Camera "${cameraName}" has been added to the recorder`,
446
+ });
447
+ } catch (error) {
448
+ toast({
449
+ title: "Camera Error",
450
+ description: `Failed to access camera: ${
451
+ error instanceof Error ? error.message : String(error)
452
+ }`,
453
+ variant: "destructive",
454
+ });
455
+ }
456
+ }, [
457
+ cameraName,
458
+ hasRecordedFrames,
459
+ selectedCameraId,
460
+ previewStream,
461
+ availableCameras,
462
+ recorderSettings,
463
+ toast,
464
+ ]);
465
+
466
+ // Remove a camera from the recorder
467
+ const handleRemoveCamera = useCallback(
468
+ (name: string) => {
469
+ if (hasRecordedFrames) {
470
+ toast({
471
+ title: "Camera Error",
472
+ description: "Cannot remove cameras after recording has started",
473
+ variant: "destructive",
474
+ });
475
+ return;
476
+ }
477
+
478
+ setAdditionalCameras((prev) => {
479
+ const newCameras = { ...prev };
480
+ if (newCameras[name]) {
481
+ // Stop the stream tracks
482
+ newCameras[name].getTracks().forEach((track) => track.stop());
483
+ delete newCameras[name];
484
+ }
485
+ return newCameras;
486
+ });
487
+
488
+ // Remove camera configuration from persistent storage
489
+ const newSettings = {
490
+ ...recorderSettings,
491
+ cameraConfigs: { ...recorderSettings.cameraConfigs },
492
+ };
493
+ delete newSettings.cameraConfigs[name];
494
+ setRecorderSettings(newSettings);
495
+ saveRecorderSettings(newSettings);
496
+
497
+ toast({
498
+ title: "Camera Removed",
499
+ description: `Camera "${name}" has been removed`,
500
+ });
501
+ },
502
+ [hasRecordedFrames, recorderSettings, toast]
503
+ );
504
+
505
+ // Camera name editing functions
506
+ const handleStartEditingCameraName = (cameraName: string) => {
507
+ setEditingCameraName(cameraName);
508
+ setEditingCameraNewName(cameraName);
509
+ };
510
+
511
+ const handleConfirmCameraNameEdit = (oldName: string) => {
512
+ if (editingCameraNewName.trim() && editingCameraNewName !== oldName) {
513
+ const stream = additionalCameras[oldName];
514
+ if (stream) {
515
+ // Update camera streams
516
+ setAdditionalCameras((prev) => {
517
+ const newCameras = { ...prev };
518
+ delete newCameras[oldName];
519
+ newCameras[editingCameraNewName.trim()] = stream;
520
+ return newCameras;
521
+ });
522
+
523
+ // Update camera configuration in persistent storage
524
+ const oldConfig = recorderSettings.cameraConfigs[oldName];
525
+ if (oldConfig) {
526
+ const newSettings = {
527
+ ...recorderSettings,
528
+ cameraConfigs: { ...recorderSettings.cameraConfigs },
529
+ };
530
+ delete newSettings.cameraConfigs[oldName];
531
+ newSettings.cameraConfigs[editingCameraNewName.trim()] = oldConfig;
532
+ setRecorderSettings(newSettings);
533
+ saveRecorderSettings(newSettings);
534
+ }
535
+ }
536
+ }
537
+ setEditingCameraName(null);
538
+ setEditingCameraNewName("");
539
+ };
540
+
541
+ const handleCancelCameraNameEdit = () => {
542
+ setEditingCameraName(null);
543
+ setEditingCameraNewName("");
544
+ };
545
+
546
+ // Restore cameras from saved configurations
547
+ const restoreSavedCameras = useCallback(async () => {
548
+ const savedConfigs = recorderSettings.cameraConfigs;
549
+ if (!savedConfigs || Object.keys(savedConfigs).length === 0) {
550
+ return;
551
+ }
552
+
553
+ for (const [cameraName, config] of Object.entries(savedConfigs)) {
554
+ try {
555
+ // Check if this camera is still available
556
+ const isDeviceAvailable = availableCameras.some(
557
+ (cam) => cam.deviceId === config.deviceId
558
+ );
559
+
560
+ if (!isDeviceAvailable) {
561
+ console.warn(
562
+ `Saved camera "${cameraName}" (${config.deviceId}) is no longer available`
563
+ );
564
+ continue;
565
+ }
566
+
567
+ // Create stream for this saved camera
568
+ const stream = await navigator.mediaDevices.getUserMedia({
569
+ video: {
570
+ deviceId: { exact: config.deviceId },
571
+ width: { ideal: 1280 },
572
+ height: { ideal: 720 },
573
+ },
574
+ });
575
+
576
+ // Add to additional cameras
577
+ setAdditionalCameras((prev) => ({
578
+ ...prev,
579
+ [cameraName]: stream,
580
+ }));
581
+ } catch (error) {
582
+ console.error(`Failed to restore camera "${cameraName}":`, error);
583
+ // Remove invalid configuration
584
+ const newSettings = {
585
+ ...recorderSettings,
586
+ cameraConfigs: { ...recorderSettings.cameraConfigs },
587
+ };
588
+ delete newSettings.cameraConfigs[cameraName];
589
+ setRecorderSettings(newSettings);
590
+ saveRecorderSettings(newSettings);
591
+ }
592
+ }
593
+ }, [availableCameras, recorderSettings]);
594
+
595
+ // Auto-load cameras on component mount (only once)
596
+ useEffect(() => {
597
+ loadAvailableCameras(true);
598
+ // eslint-disable-next-line react-hooks/exhaustive-deps
599
+ }, []); // Empty dependency array to run only once
600
+
601
+ // Handle video stream assignment - runs when stream changes OR when settings panel opens
602
+ useEffect(() => {
603
+ if (videoRef.current && previewStream) {
604
+ videoRef.current.srcObject = previewStream;
605
+ }
606
+ }, [previewStream, showConfigure]); // Also depend on showConfigure so it re-runs when video element appears
607
+
608
+ // Cleanup preview stream on unmount
609
+ useEffect(() => {
610
+ return () => {
611
+ if (previewStream) {
612
+ previewStream.getTracks().forEach((track) => track.stop());
613
+ }
614
+ };
615
+ }, [previewStream]);
616
+
617
+ // Restore saved cameras when available cameras are loaded
618
+ useEffect(() => {
619
+ if (availableCameras.length > 0 && cameraPermissionState === "granted") {
620
+ restoreSavedCameras();
621
+ }
622
+ }, [availableCameras, cameraPermissionState, restoreSavedCameras]);
623
+
624
+ const handleDownloadZip = async () => {
625
+ if (!recorderRef.current) {
626
+ toast({
627
+ title: "Download Error",
628
+ description: "Recorder not initialized",
629
+ variant: "destructive",
630
+ });
631
+ return;
632
+ }
633
+
634
+ await recorderRef.current.exportForLeRobot("zip-download");
635
+ toast({
636
+ title: "Download Started",
637
+ description: "Your dataset is being downloaded as a ZIP file",
638
+ });
639
+ };
640
+
641
+ const handleUploadToHuggingFace = async () => {
642
+ if (!recorderRef.current) {
643
+ toast({
644
+ title: "Upload Error",
645
+ description: "Recorder not initialized",
646
+ variant: "destructive",
647
+ });
648
+ return;
649
+ }
650
+
651
+ if (!recorderSettings.huggingfaceApiKey) {
652
+ toast({
653
+ title: "Upload Error",
654
+ description: "Please enter your Hugging Face API key in Configure",
655
+ variant: "destructive",
656
+ });
657
+ return;
658
+ }
659
+
660
+ try {
661
+ toast({
662
+ title: "Upload Started",
663
+ description: "Uploading dataset to Hugging Face...",
664
+ });
665
+
666
+ // Generate a unique repository name
667
+ const repoName = `lerobot-recording-${Date.now()}`;
668
+
669
+ const uploader = await recorderRef.current.exportForLeRobot(
670
+ "huggingface",
671
+ {
672
+ repoName,
673
+ accessToken: recorderSettings.huggingfaceApiKey,
674
+ }
675
+ );
676
+
677
+ uploader.addEventListener("progress", (event: Event) => {
678
+ console.log(event);
679
+ });
680
+ } catch (error) {
681
+ const errorMessage =
682
+ error instanceof Error
683
+ ? error.message
684
+ : "Failed to upload to Hugging Face";
685
+ toast({
686
+ title: "Upload Error",
687
+ description: errorMessage,
688
+ variant: "destructive",
689
+ });
690
+ }
691
+ };
692
+
693
+ // Helper function to format duration
694
+ const formatDuration = (seconds: number): string => {
695
+ const mins = Math.floor(seconds / 60);
696
+ const secs = Math.floor(seconds % 60);
697
+ return `${mins.toString().padStart(2, "0")}:${secs
698
+ .toString()
699
+ .padStart(2, "0")}`;
700
+ };
701
+
702
+ return (
703
+ <Card className="border-0 rounded-none mt-6">
704
+ <div className="p-4 border-b border-white/10">
705
+ <div className="flex items-center justify-between">
706
+ <div className="flex items-center gap-4">
707
+ <div className="w-1 h-8 bg-primary"></div>
708
+ <div>
709
+ <h3 className="text-xl font-bold text-foreground font-mono tracking-wider uppercase">
710
+ robot movement recorder
711
+ </h3>
712
+ <p className="text-sm text-muted-foreground font-mono">
713
+ dataset <span className="text-muted-foreground">recording</span>{" "}
714
+ interface
715
+ </p>
716
+ </div>
717
+ </div>
718
+ <div className="flex items-center gap-6">
719
+ <div className="border-l border-white/10 pl-6 flex items-center gap-4">
720
+ <Button
721
+ variant="outline"
722
+ size="lg"
723
+ className="gap-2"
724
+ onClick={() => setShowConfigure(!showConfigure)}
725
+ >
726
+ <Settings className="w-5 h-5" />
727
+ Configure
728
+ </Button>
729
+
730
+ <Button
731
+ variant={isRecording ? "destructive" : "default"}
732
+ size="lg"
733
+ className="gap-2"
734
+ onClick={
735
+ isRecording ? handleStopRecording : handleStartRecording
736
+ }
737
+ >
738
+ {isRecording ? (
739
+ <>
740
+ <Square className="w-5 h-5" />
741
+ Stop Recording
742
+ </>
743
+ ) : (
744
+ <>
745
+ <Record className="w-5 h-5" />
746
+ Start Recording
747
+ </>
748
+ )}
749
+ </Button>
750
+ </div>
751
+ </div>
752
+ </div>
753
+ </div>
754
+
755
+ <div className="p-6 space-y-6">
756
+ {/* Recorder Settings - Toggleable Inline */}
757
+ {showConfigure && (
758
+ <div className="space-y-6">
759
+ {/* Hugging Face Settings */}
760
+ <div className="space-y-3">
761
+ <h3 className="text-lg font-semibold text-foreground">
762
+ Settings
763
+ </h3>
764
+ <div className="grid grid-cols-1 md:grid-cols-2 gap-4">
765
+ <div className="space-y-2">
766
+ <label className="text-sm text-muted-foreground">
767
+ Hugging Face API Key
768
+ </label>
769
+ <Input
770
+ placeholder="Enter your Hugging Face API key"
771
+ value={recorderSettings.huggingfaceApiKey}
772
+ onChange={(e) => {
773
+ const newSettings = {
774
+ ...recorderSettings,
775
+ huggingfaceApiKey: e.target.value,
776
+ };
777
+ setRecorderSettings(newSettings);
778
+ saveRecorderSettings(newSettings);
779
+ }}
780
+ type="password"
781
+ className="bg-black/20 border-white/10"
782
+ />
783
+ <p className="text-xs text-white/50">
784
+ Required to upload datasets to Hugging Face Hub
785
+ </p>
786
+ </div>
787
+ </div>
788
+ </div>
789
+
790
+ {/* Camera Configuration */}
791
+ <div className="space-y-4">
792
+ <h3 className="text-lg font-semibold text-foreground">
793
+ Camera Setup
794
+ </h3>
795
+ <div className="bg-black/40 border border-white/20 p-6">
796
+ <div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
797
+ {/* Left Column: Camera Selection & Adding */}
798
+ <div className="space-y-4">
799
+ {/* Camera Selection and Refresh */}
800
+
801
+ {/* Camera Access Request */}
802
+ {cameraPermissionState === "unknown" && (
803
+ <div className="space-y-3">
804
+ <div className="bg-black/60 border border-white/20 rounded-lg p-4 text-center">
805
+ <Camera className="w-8 h-8 mx-auto mb-2 opacity-50" />
806
+ <p className="text-sm text-white/70 mb-3">
807
+ Camera access needed to configure cameras
808
+ </p>
809
+ <Button
810
+ onClick={() => loadAvailableCameras(false)}
811
+ variant="outline"
812
+ className="gap-2"
813
+ disabled={isLoadingCameras}
814
+ >
815
+ <Camera className="w-4 h-4" />
816
+ {isLoadingCameras
817
+ ? "Loading..."
818
+ : "Request Camera Access"}
819
+ </Button>
820
+ </div>
821
+ </div>
822
+ )}
823
+
824
+ {/* Camera Access Denied */}
825
+ {cameraPermissionState === "denied" && (
826
+ <div className="space-y-3">
827
+ <div className="bg-red-900/20 border border-red-500/20 rounded-lg p-4 text-center">
828
+ <Camera className="w-8 h-8 mx-auto mb-2 opacity-50 text-red-400" />
829
+ <p className="text-sm text-red-300 mb-1">
830
+ Camera access denied
831
+ </p>
832
+ <p className="text-xs text-red-400">
833
+ Please allow camera access in your browser settings
834
+ and refresh
835
+ </p>
836
+ </div>
837
+ </div>
838
+ )}
839
+
840
+ {/* Camera List with Refresh Button */}
841
+ {cameraPermissionState === "granted" &&
842
+ availableCameras.length > 0 && (
843
+ <div className="space-y-2">
844
+ <label className="text-sm text-white/70">
845
+ Select Camera:
846
+ </label>
847
+ <div className="flex items-center gap-2">
848
+ <Select
849
+ value={selectedCameraId}
850
+ onValueChange={switchCameraPreview}
851
+ disabled={hasRecordedFrames}
852
+ >
853
+ <SelectTrigger className="flex-1 bg-black/20 border-white/10">
854
+ <SelectValue placeholder="Choose a camera" />
855
+ </SelectTrigger>
856
+ <SelectContent>
857
+ {availableCameras.map((camera) => (
858
+ <SelectItem
859
+ key={camera.deviceId}
860
+ value={camera.deviceId}
861
+ >
862
+ {camera.label ||
863
+ `Camera ${camera.deviceId.slice(
864
+ 0,
865
+ 8
866
+ )}...`}
867
+ </SelectItem>
868
+ ))}
869
+ </SelectContent>
870
+ </Select>
871
+ <Button
872
+ onClick={() => loadAvailableCameras(false)}
873
+ variant="ghost"
874
+ size="sm"
875
+ className="gap-2 text-white/70 hover:text-white"
876
+ disabled={isLoadingCameras}
877
+ >
878
+ <RefreshCw
879
+ className={`w-4 h-4 ${
880
+ isLoadingCameras ? "animate-spin" : ""
881
+ }`}
882
+ />
883
+ Refresh
884
+ </Button>
885
+ </div>
886
+ </div>
887
+ )}
888
+
889
+ {/* Camera Name Input */}
890
+ {selectedCameraId && (
891
+ <div className="space-y-2">
892
+ <label className="text-sm text-white/70">
893
+ Camera Name:
894
+ </label>
895
+ <Input
896
+ placeholder="e.g., 'Overhead View', 'Side Angle', 'Close-up'"
897
+ value={cameraName}
898
+ onChange={(e) => setCameraName(e.target.value)}
899
+ className="bg-black/20 border-white/10"
900
+ disabled={hasRecordedFrames}
901
+ />
902
+ <p className="text-xs text-white/50">
903
+ Give this camera a descriptive name for your recording
904
+ setup
905
+ </p>
906
+ </div>
907
+ )}
908
+
909
+ {/* Add Camera Button */}
910
+ {selectedCameraId && (
911
+ <div className="flex justify-end">
912
+ <Button
913
+ onClick={handleAddCamera}
914
+ className="gap-2"
915
+ disabled={
916
+ hasRecordedFrames ||
917
+ !cameraName.trim() ||
918
+ !selectedCameraId ||
919
+ !previewStream
920
+ }
921
+ >
922
+ <PlusCircle className="w-4 h-4" />
923
+ Add Camera to Recorder
924
+ </Button>
925
+ </div>
926
+ )}
927
+ </div>
928
+
929
+ {/* Right Column: Camera Preview */}
930
+ <div className="space-y-4">
931
+ <div className="aspect-video bg-black/60 border border-white/20 rounded-lg overflow-hidden">
932
+ {previewStream ? (
933
+ <video
934
+ ref={videoRef}
935
+ autoPlay
936
+ muted
937
+ playsInline
938
+ className="w-full h-full object-cover"
939
+ />
940
+ ) : (
941
+ <div className="w-full h-full flex items-center justify-center text-white/60">
942
+ <div className="text-center">
943
+ <Camera className="w-12 h-12 mx-auto mb-2 opacity-50" />
944
+ {cameraPermissionState === "unknown" ? (
945
+ <p className="text-sm">
946
+ Request camera access to preview
947
+ </p>
948
+ ) : cameraPermissionState === "denied" ? (
949
+ <p className="text-sm">Camera access denied</p>
950
+ ) : availableCameras.length === 0 ? (
951
+ <p className="text-sm">No cameras available</p>
952
+ ) : !selectedCameraId ? (
953
+ <p className="text-sm">
954
+ Select a camera to preview
955
+ </p>
956
+ ) : (
957
+ <p className="text-sm">Loading preview...</p>
958
+ )}
959
+ </div>
960
+ </div>
961
+ )}
962
+ </div>
963
+ </div>
964
+ </div>
965
+ </div>
966
+ </div>
967
+ </div>
968
+ )}
969
+
970
+ {/* Added Camera Previews */}
971
+ {Object.keys(additionalCameras).length > 0 && (
972
+ <div className="space-y-4">
973
+ <h3 className="text-lg font-semibold text-foreground">
974
+ Active Cameras
975
+ </h3>
976
+ <div className="grid grid-cols-2 md:grid-cols-3 lg:grid-cols-4 gap-4">
977
+ {Object.entries(additionalCameras).map(([cameraName, stream]) => (
978
+ <div
979
+ key={cameraName}
980
+ className="bg-black/40 border border-white/20 rounded-lg p-3 space-y-2"
981
+ >
982
+ <div className="aspect-video bg-black/60 border border-white/10 rounded overflow-hidden">
983
+ <video
984
+ autoPlay
985
+ muted
986
+ playsInline
987
+ className="w-full h-full object-cover"
988
+ ref={(video) => {
989
+ if (video && stream) {
990
+ video.srcObject = stream;
991
+ }
992
+ }}
993
+ />
994
+ </div>
995
+ <div className="flex items-center justify-between">
996
+ {editingCameraName === cameraName ? (
997
+ <div className="flex items-center gap-1 flex-1">
998
+ <Input
999
+ value={editingCameraNewName}
1000
+ onChange={(e) =>
1001
+ setEditingCameraNewName(e.target.value)
1002
+ }
1003
+ className="text-xs h-6 bg-black/20 border-white/10"
1004
+ onKeyDown={(e) => {
1005
+ if (e.key === "Enter") {
1006
+ handleConfirmCameraNameEdit(cameraName);
1007
+ } else if (e.key === "Escape") {
1008
+ handleCancelCameraNameEdit();
1009
+ }
1010
+ }}
1011
+ autoFocus
1012
+ />
1013
+ <button
1014
+ onClick={() =>
1015
+ handleConfirmCameraNameEdit(cameraName)
1016
+ }
1017
+ className="text-green-400 hover:text-green-300 p-1"
1018
+ >
1019
+ <Check className="w-3 h-3" />
1020
+ </button>
1021
+ <button
1022
+ onClick={handleCancelCameraNameEdit}
1023
+ className="text-red-400 hover:text-red-300 p-1"
1024
+ >
1025
+ <X className="w-3 h-3" />
1026
+ </button>
1027
+ </div>
1028
+ ) : (
1029
+ <button
1030
+ onClick={() => handleStartEditingCameraName(cameraName)}
1031
+ className="text-sm font-medium text-white/90 truncate hover:text-white cursor-pointer flex items-center gap-1 flex-1"
1032
+ disabled={hasRecordedFrames}
1033
+ >
1034
+ {cameraName}
1035
+ <Edit2 className="w-3 h-3 opacity-50" />
1036
+ </button>
1037
+ )}
1038
+ <button
1039
+ onClick={() => handleRemoveCamera(cameraName)}
1040
+ className="text-red-400 hover:text-red-300 p-1 ml-2"
1041
+ disabled={hasRecordedFrames}
1042
+ title="Remove camera"
1043
+ >
1044
+ <X className="w-4 h-4" />
1045
+ </button>
1046
+ </div>
1047
+ </div>
1048
+ ))}
1049
+ </div>
1050
+ </div>
1051
+ )}
1052
+
1053
+ {/* Episode Management & Dataset Actions */}
1054
+ <div className="flex justify-between items-center">
1055
+ <div className="flex items-center gap-2">
1056
+ <Button
1057
+ variant="outline"
1058
+ className="gap-2"
1059
+ onClick={handleResetFrames}
1060
+ disabled={isRecording || !hasRecordedFrames}
1061
+ >
1062
+ <Trash2 className="w-4 h-4" />
1063
+ Reset Frames
1064
+ </Button>
1065
+ <Button
1066
+ variant="outline"
1067
+ className="gap-2"
1068
+ onClick={handleNextEpisode}
1069
+ disabled={isRecording}
1070
+ >
1071
+ <PlusCircle className="w-4 h-4" />
1072
+ Next Episode
1073
+ </Button>
1074
+ </div>
1075
+
1076
+ <div className="flex items-center gap-2">
1077
+ <Button
1078
+ variant="outline"
1079
+ className="gap-2"
1080
+ onClick={handleDownloadZip}
1081
+ disabled={recorderRef.current?.teleoperatorData.length === 0 || isRecording}
1082
+ >
1083
+ <Download className="w-4 h-4" />
1084
+ Download as ZIP
1085
+ </Button>
1086
+ <Button
1087
+ variant="outline"
1088
+ className="gap-2"
1089
+ onClick={handleUploadToHuggingFace}
1090
+ disabled={
1091
+ recorderRef.current?.teleoperatorData.length === 0 ||
1092
+ isRecording ||
1093
+ !recorderSettings.huggingfaceApiKey
1094
+ }
1095
+ >
1096
+ <Upload className="w-4 h-4" />
1097
+ Upload to Hugging Face
1098
+ </Button>
1099
+ </div>
1100
+ </div>
1101
+
1102
+ <div className="border border-white/10 rounded-md overflow-hidden">
1103
+ <TeleoperatorEpisodesView teleoperatorData={recorderRef.current?.teleoperatorData} />
1104
+ </div>
1105
+
1106
+ {/* Camera Configuration */}
1107
+ <div className="border-t border-white/10 pt-4 mt-4">
1108
+ <div className="flex items-center justify-between mb-4">
1109
+ <h3 className="text-lg font-semibold">Camera Setup</h3>
1110
+ <Button
1111
+ onClick={() => setShowCameraConfig(!showCameraConfig)}
1112
+ variant="outline"
1113
+ size="sm"
1114
+ className="gap-2"
1115
+ >
1116
+ <Camera className="w-4 h-4" />
1117
+ {showCameraConfig ? "Hide Camera Config" : "Configure Cameras"}
1118
+ </Button>
1119
+ </div>
1120
+
1121
+ {showCameraConfig && (
1122
+ <div className="bg-black/40 border border-white/20 rounded-lg p-6 mb-4">
1123
+ <div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
1124
+ {/* Left Column: Camera Preview & Selection */}
1125
+ <div className="space-y-4">
1126
+ <h4 className="text-md font-semibold text-white/90">
1127
+ Camera Preview
1128
+ </h4>
1129
+
1130
+ {/* Camera Preview */}
1131
+ <div className="aspect-video bg-black/60 border border-white/20 rounded-lg overflow-hidden">
1132
+ {previewStream ? (
1133
+ <video
1134
+ ref={videoRef}
1135
+ autoPlay
1136
+ muted
1137
+ playsInline
1138
+ className="w-full h-full object-cover"
1139
+ />
1140
+ ) : (
1141
+ <div className="w-full h-full flex items-center justify-center text-white/60">
1142
+ <div className="text-center">
1143
+ <Camera className="w-12 h-12 mx-auto mb-2 opacity-50" />
1144
+ {isLoadingCameras ? (
1145
+ <p>Loading cameras...</p>
1146
+ ) : cameraPermissionState === "denied" ? (
1147
+ <div>
1148
+ <p>Camera access denied</p>
1149
+ <p className="text-xs mt-1">
1150
+ Please allow camera access and refresh
1151
+ </p>
1152
+ </div>
1153
+ ) : availableCameras.length === 0 ? (
1154
+ <p>Click "Load Cameras" to start</p>
1155
+ ) : (
1156
+ <p>Select a camera to preview</p>
1157
+ )}
1158
+ </div>
1159
+ </div>
1160
+ )}
1161
+ </div>
1162
+
1163
+ {/* Camera Controls */}
1164
+ <div className="space-y-3">
1165
+ <Button
1166
+ onClick={() => loadAvailableCameras(false)}
1167
+ variant="outline"
1168
+ className="w-full gap-2"
1169
+ disabled={hasRecordedFrames || isLoadingCameras}
1170
+ >
1171
+ <Camera className="w-4 h-4" />
1172
+ {isLoadingCameras
1173
+ ? "Loading..."
1174
+ : availableCameras.length > 0
1175
+ ? "Refresh Cameras"
1176
+ : "Load Cameras"}
1177
+ </Button>
1178
+
1179
+ {availableCameras.length > 0 && (
1180
+ <div className="space-y-2">
1181
+ <label className="text-sm text-white/70">
1182
+ Select Camera:
1183
+ </label>
1184
+ <Select
1185
+ value={selectedCameraId}
1186
+ onValueChange={switchCameraPreview}
1187
+ disabled={hasRecordedFrames}
1188
+ >
1189
+ <SelectTrigger className="w-full bg-black/20 border-white/10">
1190
+ <SelectValue placeholder="Choose a camera" />
1191
+ </SelectTrigger>
1192
+ <SelectContent>
1193
+ {availableCameras.map((camera) => (
1194
+ <SelectItem
1195
+ key={camera.deviceId}
1196
+ value={camera.deviceId}
1197
+ >
1198
+ {camera.label ||
1199
+ `Camera ${camera.deviceId.slice(0, 8)}...`}
1200
+ </SelectItem>
1201
+ ))}
1202
+ </SelectContent>
1203
+ </Select>
1204
+ </div>
1205
+ )}
1206
+ </div>
1207
+ </div>
1208
+
1209
+ {/* Right Column: Camera Naming & Adding */}
1210
+ <div className="space-y-4">
1211
+ <h4 className="text-md font-semibold text-white/90">
1212
+ Add to Recorder
1213
+ </h4>
1214
+
1215
+ <div className="space-y-4">
1216
+ <div className="space-y-2">
1217
+ <label className="text-sm text-white/70">
1218
+ Camera Name:
1219
+ </label>
1220
+ <Input
1221
+ placeholder="e.g., 'Overhead View', 'Side Angle', 'Close-up'"
1222
+ value={cameraName}
1223
+ onChange={(e) => setCameraName(e.target.value)}
1224
+ className="bg-black/20 border-white/10"
1225
+ disabled={hasRecordedFrames}
1226
+ />
1227
+ <p className="text-xs text-white/50">
1228
+ Give this camera a descriptive name for your recording
1229
+ setup
1230
+ </p>
1231
+ </div>
1232
+
1233
+ <Button
1234
+ onClick={handleAddCamera}
1235
+ className="w-full gap-2"
1236
+ disabled={
1237
+ hasRecordedFrames ||
1238
+ !cameraName.trim() ||
1239
+ !selectedCameraId ||
1240
+ !previewStream
1241
+ }
1242
+ >
1243
+ <PlusCircle className="w-4 h-4" />
1244
+ Add Camera to Recorder
1245
+ </Button>
1246
+ </div>
1247
+ </div>
1248
+ </div>
1249
+ </div>
1250
+ )}
1251
+
1252
+ {/* Display added cameras */}
1253
+ {Object.keys(additionalCameras).length > 0 && (
1254
+ <div className="mb-4 space-y-2">
1255
+ <p className="text-sm text-muted-foreground">Added Cameras:</p>
1256
+ <div className="flex flex-wrap gap-2">
1257
+ {Object.keys(additionalCameras).map((name) => (
1258
+ <Badge
1259
+ key={name}
1260
+ variant="secondary"
1261
+ className="flex items-center gap-1 py-1 px-2"
1262
+ >
1263
+ {name}
1264
+ <button
1265
+ onClick={() => handleRemoveCamera(name)}
1266
+ className="ml-1 text-muted-foreground hover:text-destructive"
1267
+ disabled={hasRecordedFrames}
1268
+ >
1269
+ ×
1270
+ </button>
1271
+ </Badge>
1272
+ ))}
1273
+ </div>
1274
+ </div>
1275
+ )}
1276
+ </div>
1277
+
1278
+ <div className="flex items-center gap-4">
1279
+ <div className="flex items-center gap-2">
1280
+ <Button
1281
+ variant="outline"
1282
+ className="gap-2"
1283
+ onClick={handleDownloadZip}
1284
+ disabled={recorderRef.current?.teleoperatorData.length === 0 || isRecording}
1285
+ >
1286
+ <Download className="w-4 h-4" />
1287
+ Download as ZIP
1288
+ </Button>
1289
+
1290
+ {/* Reset Frames button moved to top bar */}
1291
+ </div>
1292
+
1293
+ <div className="flex items-center gap-2 flex-1">
1294
+ <Button
1295
+ variant="outline"
1296
+ className="gap-2"
1297
+ onClick={handleUploadToHuggingFace}
1298
+ disabled={
1299
+ recorderRef.current?.teleoperatorData.length === 0 || isRecording || !huggingfaceApiKey
1300
+ }
1301
+ >
1302
+ <Upload className="w-4 h-4" />
1303
+ Upload to HuggingFace
1304
+ </Button>
1305
+
1306
+ <Input
1307
+ placeholder="HuggingFace API Key"
1308
+ value={huggingfaceApiKey}
1309
+ onChange={(e) => setHuggingfaceApiKey(e.target.value)}
1310
+ className="flex-1 bg-black/20 border-white/10"
1311
+ type="password"
1312
+ />
1313
+ </div>
1314
+ </div>
1315
+ </div>
1316
+ </Card>
1317
+ );
1318
+ }
examples/cyberpunk-standalone/src/components/teleoperation-view.tsx CHANGED
@@ -1,6 +1,6 @@
1
  "use client";
2
  import { useState, useEffect, useMemo, useRef, useCallback } from "react";
3
- import { Power, PowerOff, Keyboard } from "lucide-react";
4
  import { Button } from "@/components/ui/button";
5
  import { Card } from "@/components/ui/card";
6
  import { Badge } from "@/components/ui/badge";
@@ -22,6 +22,11 @@ import {
22
  } from "@lerobot/web";
23
  import { getUnifiedRobotData } from "@/lib/unified-storage";
24
  import VirtualKey from "@/components/VirtualKey";
 
 
 
 
 
25
 
26
  interface TeleoperationViewProps {
27
  robot: RobotConnection;
@@ -104,7 +109,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
104
  }, [robot.serialNumber]);
105
 
106
  // Lazy initialization function - only connects when user wants to start
107
- const initializeTeleoperation = async () => {
108
  if (!robot || !robot.robotType) {
109
  return false;
110
  }
@@ -130,15 +135,18 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
130
  type: "direct",
131
  },
132
  calibrationData,
 
 
 
133
  };
134
  const directProcess = await teleoperate(directConfig);
135
 
136
  keyboardProcessRef.current = keyboardProcess;
137
  directProcessRef.current = directProcess;
138
- setTeleopState(keyboardProcess.getState());
139
 
140
  // Initialize local motor positions from hardware state
141
- const initialState = keyboardProcess.getState();
142
  const initialPositions: {
143
  [motorName: string]: { position: number; timestamp: number };
144
  } = {};
@@ -165,7 +173,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
165
  });
166
  return false;
167
  }
168
- };
169
 
170
  // Cleanup on unmount
171
  useEffect(() => {
@@ -247,7 +255,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
247
  if (!success) return;
248
  }
249
 
250
- if (!keyboardProcessRef.current || !directProcessRef.current) {
251
  toast({
252
  title: "Teleoperation Error",
253
  description: "Teleoperation not initialized",
@@ -257,8 +265,8 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
257
  }
258
 
259
  try {
260
- keyboardProcessRef.current.start();
261
- directProcessRef.current.start();
262
  } catch (error) {
263
  const errorMessage =
264
  error instanceof Error
@@ -349,12 +357,20 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
349
  const realMotorConfigs = teleopState?.motorConfigs || [];
350
  const now = Date.now();
351
 
352
- // If we have real motor configs, use them with local position overrides when recent
353
  if (realMotorConfigs.length > 0) {
354
  return realMotorConfigs.map((motor) => {
355
  const localData = localMotorPositions[motor.name];
356
- // Use local position only if it's very recent (within 100ms), otherwise use hardware position
357
- const useLocalPosition = localData && now - localData.timestamp < 100;
 
 
 
 
 
 
 
 
358
  return {
359
  ...motor,
360
  currentPosition: useLocalPosition
@@ -368,7 +384,13 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
368
  return DEFAULT_MOTOR_CONFIGS.map((motor) => {
369
  const calibratedMotor = calibrationData?.[motor.name];
370
  const localData = localMotorPositions[motor.name];
371
- const useLocalPosition = localData && now - localData.timestamp < 100;
 
 
 
 
 
 
372
 
373
  return {
374
  ...motor,
@@ -392,332 +414,358 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
392
  const keyStates = teleopState?.keyStates || {};
393
  const controls = SO100_KEYBOARD_CONTROLS;
394
 
 
 
 
 
 
 
 
 
 
395
  return (
396
- <Card className="border-0 rounded-none">
397
- <div className="p-4 border-b border-white/10">
398
- <div className="flex items-center justify-between">
399
- <div className="flex items-center gap-4">
400
- <div className="w-1 h-8 bg-primary"></div>
401
- <div>
402
- <h3 className="text-xl font-bold text-foreground font-mono tracking-wider uppercase">
403
- robot control
404
- </h3>
405
- <p className="text-sm text-muted-foreground font-mono">
406
- manual{" "}
407
- <span className="text-muted-foreground">teleoperate</span>{" "}
408
- interface
409
- </p>
410
- </div>
411
- </div>
412
- <div className="flex items-center gap-6">
413
- <div className="border-l border-white/10 pl-6 flex items-center gap-4">
414
- {teleopState?.isActive ? (
415
- <Button onClick={handleStop} variant="destructive" size="lg">
416
- <PowerOff className="w-5 h-5 mr-2" /> Stop Control
417
- </Button>
418
- ) : (
419
- <Button
420
- onClick={handleStart}
421
- size="lg"
422
- disabled={!robot.isConnected}
423
- >
424
- <Power className="w-5 h-5 mr-2" /> Control Robot
425
- </Button>
426
- )}
427
- <div className="flex items-center gap-2">
428
- <span className="text-sm font-mono text-muted-foreground uppercase">
429
- status:
430
- </span>
431
- <Badge
432
- variant="outline"
433
- className={cn(
434
- "border-primary/50 bg-primary/20 text-primary font-mono text-xs",
435
- teleopState?.isActive && "animate-pulse-slow"
436
- )}
437
- >
438
- {teleopState?.isActive ? "ACTIVE" : "STOPPED"}
439
- </Badge>
440
  </div>
441
  </div>
442
- </div>
443
- </div>
444
- </div>
445
- <div className="pt-6 p-6 grid md:grid-cols-2 gap-8">
446
- <div>
447
- <h3 className="font-sans font-semibold mb-4 text-xl">
448
- Motor Control
449
- </h3>
450
- <div className="space-y-6">
451
- {motorConfigs.map((motor) => (
452
- <div key={motor.name}>
453
- <label className="text-sm font-mono text-muted-foreground">
454
- {motor.name}
455
- </label>
456
- <div className="flex items-center gap-4">
457
- <Slider
458
- value={[motor.currentPosition]}
459
- min={motor.minPosition}
460
- max={motor.maxPosition}
461
- step={1}
462
- onValueChange={(val) => moveMotor(motor.name, val[0])}
463
- disabled={!teleopState?.isActive}
464
- className={!teleopState?.isActive ? "opacity-50" : ""}
465
- />
466
- <span
467
  className={cn(
468
- "text-lg font-mono w-16 text-right",
469
- teleopState?.isActive
470
- ? "text-accent"
471
- : "text-muted-foreground"
472
  )}
473
  >
474
- {Math.round(motor.currentPosition)}
475
- </span>
476
  </div>
477
  </div>
478
- ))}
479
  </div>
480
  </div>
481
- <div>
482
- <h3 className="font-sans font-semibold mb-4 text-xl">
483
- Keyboard Layout & Status
484
- </h3>
485
- <div className="p-4 bg-black/30 rounded-lg space-y-4">
486
- <div className="flex justify-around items-end">
487
- <div className="flex flex-col items-center gap-2">
488
- <VirtualKey
489
- label=""
490
- subLabel="Lift+"
491
- isPressed={
492
- !!keyStates[controls.shoulder_lift.positive]?.pressed
493
- }
494
- onMouseDown={() =>
495
- simulateKeyPress(controls.shoulder_lift.positive)
496
- }
497
- onMouseUp={() =>
498
- simulateKeyRelease(controls.shoulder_lift.positive)
499
- }
500
- disabled={!teleopState?.isActive}
501
- />
502
- <div className="flex gap-2">
503
- <VirtualKey
504
- label=""
505
- subLabel="Pan-"
506
- isPressed={
507
- !!keyStates[controls.shoulder_pan.negative]?.pressed
508
- }
509
- onMouseDown={() =>
510
- simulateKeyPress(controls.shoulder_pan.negative)
511
- }
512
- onMouseUp={() =>
513
- simulateKeyRelease(controls.shoulder_pan.negative)
514
- }
515
- disabled={!teleopState?.isActive}
516
- />
517
- <VirtualKey
518
- label="↓"
519
- subLabel="Lift-"
520
- isPressed={
521
- !!keyStates[controls.shoulder_lift.negative]?.pressed
522
- }
523
- onMouseDown={() =>
524
- simulateKeyPress(controls.shoulder_lift.negative)
525
- }
526
- onMouseUp={() =>
527
- simulateKeyRelease(controls.shoulder_lift.negative)
528
- }
529
- disabled={!teleopState?.isActive}
530
- />
531
- <VirtualKey
532
- label="→"
533
- subLabel="Pan+"
534
- isPressed={
535
- !!keyStates[controls.shoulder_pan.positive]?.pressed
536
- }
537
- onMouseDown={() =>
538
- simulateKeyPress(controls.shoulder_pan.positive)
539
- }
540
- onMouseUp={() =>
541
- simulateKeyRelease(controls.shoulder_pan.positive)
542
- }
543
- disabled={!teleopState?.isActive}
544
- />
545
  </div>
546
- <span className="font-bold text-sm font-sans">Shoulder</span>
547
- </div>
548
- <div className="flex flex-col items-center gap-2">
549
- <VirtualKey
550
- label="W"
551
- subLabel="Elbow+"
552
- isPressed={!!keyStates[controls.elbow_flex.positive]?.pressed}
553
- onMouseDown={() =>
554
- simulateKeyPress(controls.elbow_flex.positive)
555
- }
556
- onMouseUp={() =>
557
- simulateKeyRelease(controls.elbow_flex.positive)
558
- }
559
- disabled={!teleopState?.isActive}
560
- />
561
- <div className="flex gap-2">
562
- <VirtualKey
563
- label="A"
564
- subLabel="Wrist+"
565
- isPressed={
566
- !!keyStates[controls.wrist_flex.positive]?.pressed
567
- }
568
- onMouseDown={() =>
569
- simulateKeyPress(controls.wrist_flex.positive)
570
- }
571
- onMouseUp={() =>
572
- simulateKeyRelease(controls.wrist_flex.positive)
573
- }
574
- disabled={!teleopState?.isActive}
575
- />
576
- <VirtualKey
577
- label="S"
578
- subLabel="Elbow-"
579
- isPressed={
580
- !!keyStates[controls.elbow_flex.negative]?.pressed
581
- }
582
- onMouseDown={() =>
583
- simulateKeyPress(controls.elbow_flex.negative)
584
- }
585
- onMouseUp={() =>
586
- simulateKeyRelease(controls.elbow_flex.negative)
587
- }
588
- disabled={!teleopState?.isActive}
589
- />
590
  <VirtualKey
591
- label="D"
592
- subLabel="Wrist-"
593
  isPressed={
594
- !!keyStates[controls.wrist_flex.negative]?.pressed
595
  }
596
  onMouseDown={() =>
597
- simulateKeyPress(controls.wrist_flex.negative)
598
  }
599
  onMouseUp={() =>
600
- simulateKeyRelease(controls.wrist_flex.negative)
601
  }
602
  disabled={!teleopState?.isActive}
603
  />
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
604
  </div>
605
- <span className="font-bold text-sm font-sans">Elbow/Wrist</span>
606
- </div>
607
- <div className="flex flex-col items-center gap-2">
608
- <div className="flex gap-2">
609
- <VirtualKey
610
- label="Q"
611
- subLabel="Roll+"
612
- isPressed={
613
- !!keyStates[controls.wrist_roll.positive]?.pressed
614
- }
615
- onMouseDown={() =>
616
- simulateKeyPress(controls.wrist_roll.positive)
617
- }
618
- onMouseUp={() =>
619
- simulateKeyRelease(controls.wrist_roll.positive)
620
- }
621
- disabled={!teleopState?.isActive}
622
- />
623
  <VirtualKey
624
- label="E"
625
- subLabel="Roll-"
626
  isPressed={
627
- !!keyStates[controls.wrist_roll.negative]?.pressed
628
  }
629
  onMouseDown={() =>
630
- simulateKeyPress(controls.wrist_roll.negative)
631
  }
632
  onMouseUp={() =>
633
- simulateKeyRelease(controls.wrist_roll.negative)
634
  }
635
  disabled={!teleopState?.isActive}
636
  />
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
637
  </div>
638
- <div className="flex gap-2">
639
- <VirtualKey
640
- label="O"
641
- subLabel="Grip+"
642
- isPressed={!!keyStates[controls.gripper.positive]?.pressed}
643
- onMouseDown={() =>
644
- simulateKeyPress(controls.gripper.positive)
645
- }
646
- onMouseUp={() =>
647
- simulateKeyRelease(controls.gripper.positive)
648
- }
649
- disabled={!teleopState?.isActive}
650
- />
651
- <VirtualKey
652
- label="C"
653
- subLabel="Grip-"
654
- isPressed={!!keyStates[controls.gripper.negative]?.pressed}
655
- onMouseDown={() =>
656
- simulateKeyPress(controls.gripper.negative)
657
- }
658
- onMouseUp={() =>
659
- simulateKeyRelease(controls.gripper.negative)
660
- }
661
- disabled={!teleopState?.isActive}
662
- />
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
663
  </div>
664
- <span className="font-bold text-sm font-sans">Roll/Grip</span>
665
  </div>
666
- </div>
667
- <div className="pt-4 border-t border-white/10">
668
- <div className="flex justify-between items-center font-mono text-sm">
669
- <div className="flex items-center gap-2 text-muted-foreground">
670
- <Keyboard className="w-4 h-4" />
671
- <span>
672
- Active Keys:{" "}
673
- {Object.values(keyStates).filter((k) => k.pressed).length}
674
- </span>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
675
  </div>
676
- <TooltipProvider>
677
- <Tooltip>
678
- <TooltipTrigger asChild>
679
- <div
680
- className={cn(
681
- "w-10 h-6 border rounded-md flex items-center justify-center font-mono text-xs transition-all",
682
- "select-none user-select-none",
683
- !teleopState?.isActive &&
684
- "opacity-50 cursor-not-allowed",
685
- teleopState?.isActive &&
686
- "cursor-pointer hover:bg-white/5",
687
- keyStates[controls.stop]?.pressed
688
- ? "bg-destructive text-destructive-foreground border-destructive"
689
- : "bg-background"
690
- )}
691
- onMouseDown={(e) => {
692
- e.preventDefault();
693
- if (teleopState?.isActive) {
694
- simulateKeyPress(controls.stop);
695
- }
696
- }}
697
- onMouseUp={(e) => {
698
- e.preventDefault();
699
- if (teleopState?.isActive) {
700
- simulateKeyRelease(controls.stop);
701
- }
702
- }}
703
- onMouseLeave={(e) => {
704
- e.preventDefault();
705
- if (teleopState?.isActive) {
706
- simulateKeyRelease(controls.stop);
707
- }
708
- }}
709
- >
710
- ESC
711
- </div>
712
- </TooltipTrigger>
713
- <TooltipContent>Emergency Stop</TooltipContent>
714
- </Tooltip>
715
- </TooltipProvider>
716
  </div>
717
  </div>
718
  </div>
719
  </div>
720
- </div>
721
- </Card>
 
 
 
 
 
 
 
722
  );
723
  }
 
1
  "use client";
2
  import { useState, useEffect, useMemo, useRef, useCallback } from "react";
3
+ import { Power, PowerOff, Keyboard, Box } from "lucide-react";
4
  import { Button } from "@/components/ui/button";
5
  import { Card } from "@/components/ui/card";
6
  import { Badge } from "@/components/ui/badge";
 
22
  } from "@lerobot/web";
23
  import { getUnifiedRobotData } from "@/lib/unified-storage";
24
  import VirtualKey from "@/components/VirtualKey";
25
+ import { Recorder } from "@/components/recorder";
26
+ import { Canvas } from "@react-three/fiber";
27
+ import { Physics } from "@react-three/cannon";
28
+ import * as THREE from "three";
29
+ import { OrbitControls } from "@react-three/drei";
30
 
31
  interface TeleoperationViewProps {
32
  robot: RobotConnection;
 
109
  }, [robot.serialNumber]);
110
 
111
  // Lazy initialization function - only connects when user wants to start
112
+ const initializeTeleoperation = useCallback(async () => {
113
  if (!robot || !robot.robotType) {
114
  return false;
115
  }
 
135
  type: "direct",
136
  },
137
  calibrationData,
138
+ onStateUpdate: (state: TeleoperationState) => {
139
+ setTeleopState(state);
140
+ },
141
  };
142
  const directProcess = await teleoperate(directConfig);
143
 
144
  keyboardProcessRef.current = keyboardProcess;
145
  directProcessRef.current = directProcess;
146
+ setTeleopState(directProcess.getState());
147
 
148
  // Initialize local motor positions from hardware state
149
+ const initialState = directProcess.getState();
150
  const initialPositions: {
151
  [motorName: string]: { position: number; timestamp: number };
152
  } = {};
 
173
  });
174
  return false;
175
  }
176
+ }, [robot, robot.robotType, calibrationData, toast]);
177
 
178
  // Cleanup on unmount
179
  useEffect(() => {
 
255
  if (!success) return;
256
  }
257
 
258
+ if (!(keyboardProcessRef.current || directProcessRef.current)) {
259
  toast({
260
  title: "Teleoperation Error",
261
  description: "Teleoperation not initialized",
 
265
  }
266
 
267
  try {
268
+ keyboardProcessRef.current?.start();
269
+ directProcessRef.current?.start();
270
  } catch (error) {
271
  const errorMessage =
272
  error instanceof Error
 
357
  const realMotorConfigs = teleopState?.motorConfigs || [];
358
  const now = Date.now();
359
 
360
+ // If we have real motor configs, use them with local position overrides when appropriate
361
  if (realMotorConfigs.length > 0) {
362
  return realMotorConfigs.map((motor) => {
363
  const localData = localMotorPositions[motor.name];
364
+
365
+ // Use local position if it exists and either:
366
+ // 1. It's very recent (within 500ms) OR
367
+ // 2. The hardware position is not yet close to our requested position
368
+ const isRecent = localData && now - localData.timestamp < 500;
369
+ const isHardwareNotCaughtUp =
370
+ localData && Math.abs(motor.currentPosition - localData.position) > 5;
371
+ const useLocalPosition =
372
+ localData && (isRecent || isHardwareNotCaughtUp);
373
+
374
  return {
375
  ...motor,
376
  currentPosition: useLocalPosition
 
384
  return DEFAULT_MOTOR_CONFIGS.map((motor) => {
385
  const calibratedMotor = calibrationData?.[motor.name];
386
  const localData = localMotorPositions[motor.name];
387
+ // Use local position if it exists and either:
388
+ // 1. It's very recent (within 500ms) OR
389
+ // 2. We don't have a hardware position yet that's close to our requested position
390
+ const isRecent = localData && now - localData.timestamp < 500;
391
+ const isHardwareNotCaughtUp =
392
+ localData && Math.abs(motor.currentPosition - localData.position) > 5;
393
+ const useLocalPosition = localData && (isRecent || isHardwareNotCaughtUp);
394
 
395
  return {
396
  ...motor,
 
414
  const keyStates = teleopState?.keyStates || {};
415
  const controls = SO100_KEYBOARD_CONTROLS;
416
 
417
+ // Memoize teleoperators array to prevent unnecessary re-renders of the Recorder component
418
+ const memoizedTeleoperators = useMemo(() => {
419
+ if (!directProcessRef.current) return [];
420
+
421
+ return [
422
+ directProcessRef.current?.teleoperator,
423
+ ].filter(Boolean);
424
+ }, [directProcessRef.current]);
425
+
426
  return (
427
+ <>
428
+ <Card className="border-0 rounded-none">
429
+ <div className="p-4 border-b border-white/10">
430
+ <div className="flex items-center justify-between">
431
+ <div className="flex items-center gap-4">
432
+ <div className="w-1 h-8 bg-primary"></div>
433
+ <div>
434
+ <h3 className="text-xl font-bold text-foreground font-mono tracking-wider uppercase">
435
+ robot control
436
+ </h3>
437
+ <p className="text-sm text-muted-foreground font-mono">
438
+ manual{" "}
439
+ <span className="text-muted-foreground">teleoperate</span>{" "}
440
+ interface
441
+ </p>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
442
  </div>
443
  </div>
444
+ <div className="flex items-center gap-6">
445
+ <div className="border-l border-white/10 pl-6 flex items-center gap-4">
446
+ {teleopState?.isActive ? (
447
+ <Button onClick={handleStop} variant="destructive" size="lg">
448
+ <PowerOff className="w-5 h-5 mr-2" /> Stop Control
449
+ </Button>
450
+ ) : (
451
+ <Button
452
+ onClick={handleStart}
453
+ size="lg"
454
+ disabled={!robot.isConnected}
455
+ >
456
+ <Power className="w-5 h-5 mr-2" /> Control Robot
457
+ </Button>
458
+ )}
459
+ <div className="flex items-center gap-2">
460
+ <span className="text-sm font-mono text-muted-foreground uppercase">
461
+ status:
462
+ </span>
463
+ <Badge
464
+ variant="outline"
 
 
 
 
465
  className={cn(
466
+ "border-primary/50 bg-primary/20 text-primary font-mono text-xs",
467
+ teleopState?.isActive && "animate-pulse-slow"
 
 
468
  )}
469
  >
470
+ {teleopState?.isActive ? "ACTIVE" : "STOPPED"}
471
+ </Badge>
472
  </div>
473
  </div>
474
+ </div>
475
  </div>
476
  </div>
477
+ <div className="pt-6 p-6 grid md:grid-cols-2 gap-8">
478
+ <div>
479
+ <h3 className="font-sans font-semibold mb-4 text-xl">
480
+ Motor Control
481
+ </h3>
482
+ <div className="space-y-6">
483
+ {motorConfigs.map((motor) => (
484
+ <div key={motor.name}>
485
+ <label className="text-sm font-mono text-muted-foreground">
486
+ {motor.name}
487
+ </label>
488
+ <div className="flex items-center gap-4">
489
+ <Slider
490
+ value={[motor.currentPosition]}
491
+ min={motor.minPosition}
492
+ max={motor.maxPosition}
493
+ step={1}
494
+ onValueChange={(val) => moveMotor(motor.name, val[0])}
495
+ disabled={!teleopState?.isActive}
496
+ className={!teleopState?.isActive ? "opacity-50" : ""}
497
+ />
498
+ <span
499
+ className={cn(
500
+ "text-lg font-mono w-16 text-right",
501
+ teleopState?.isActive
502
+ ? "text-accent"
503
+ : "text-muted-foreground"
504
+ )}
505
+ >
506
+ {Math.round(motor.currentPosition)}
507
+ </span>
508
+ </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
509
  </div>
510
+ ))}
511
+ </div>
512
+ </div>
513
+ <div>
514
+ <h3 className="font-sans font-semibold mb-4 text-xl">
515
+ Keyboard Layout & Status
516
+ </h3>
517
+ <div className="p-4 bg-black/30 rounded-lg space-y-4">
518
+ <div className="flex justify-around items-end">
519
+ <div className="flex flex-col items-center gap-2">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
520
  <VirtualKey
521
+ label=""
522
+ subLabel="Lift+"
523
  isPressed={
524
+ !!keyStates[controls.shoulder_lift.positive]?.pressed
525
  }
526
  onMouseDown={() =>
527
+ simulateKeyPress(controls.shoulder_lift.positive)
528
  }
529
  onMouseUp={() =>
530
+ simulateKeyRelease(controls.shoulder_lift.positive)
531
  }
532
  disabled={!teleopState?.isActive}
533
  />
534
+ <div className="flex gap-2">
535
+ <VirtualKey
536
+ label="←"
537
+ subLabel="Pan-"
538
+ isPressed={
539
+ !!keyStates[controls.shoulder_pan.negative]?.pressed
540
+ }
541
+ onMouseDown={() =>
542
+ simulateKeyPress(controls.shoulder_pan.negative)
543
+ }
544
+ onMouseUp={() =>
545
+ simulateKeyRelease(controls.shoulder_pan.negative)
546
+ }
547
+ disabled={!teleopState?.isActive}
548
+ />
549
+ <VirtualKey
550
+ label="↓"
551
+ subLabel="Lift-"
552
+ isPressed={
553
+ !!keyStates[controls.shoulder_lift.negative]?.pressed
554
+ }
555
+ onMouseDown={() =>
556
+ simulateKeyPress(controls.shoulder_lift.negative)
557
+ }
558
+ onMouseUp={() =>
559
+ simulateKeyRelease(controls.shoulder_lift.negative)
560
+ }
561
+ disabled={!teleopState?.isActive}
562
+ />
563
+ <VirtualKey
564
+ label="→"
565
+ subLabel="Pan+"
566
+ isPressed={
567
+ !!keyStates[controls.shoulder_pan.positive]?.pressed
568
+ }
569
+ onMouseDown={() =>
570
+ simulateKeyPress(controls.shoulder_pan.positive)
571
+ }
572
+ onMouseUp={() =>
573
+ simulateKeyRelease(controls.shoulder_pan.positive)
574
+ }
575
+ disabled={!teleopState?.isActive}
576
+ />
577
+ </div>
578
+ <span className="font-bold text-sm font-sans">Shoulder</span>
579
  </div>
580
+ <div className="flex flex-col items-center gap-2">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
581
  <VirtualKey
582
+ label="W"
583
+ subLabel="Elbow+"
584
  isPressed={
585
+ !!keyStates[controls.elbow_flex.positive]?.pressed
586
  }
587
  onMouseDown={() =>
588
+ simulateKeyPress(controls.elbow_flex.positive)
589
  }
590
  onMouseUp={() =>
591
+ simulateKeyRelease(controls.elbow_flex.positive)
592
  }
593
  disabled={!teleopState?.isActive}
594
  />
595
+ <div className="flex gap-2">
596
+ <VirtualKey
597
+ label="A"
598
+ subLabel="Wrist+"
599
+ isPressed={
600
+ !!keyStates[controls.wrist_flex.positive]?.pressed
601
+ }
602
+ onMouseDown={() =>
603
+ simulateKeyPress(controls.wrist_flex.positive)
604
+ }
605
+ onMouseUp={() =>
606
+ simulateKeyRelease(controls.wrist_flex.positive)
607
+ }
608
+ disabled={!teleopState?.isActive}
609
+ />
610
+ <VirtualKey
611
+ label="S"
612
+ subLabel="Elbow-"
613
+ isPressed={
614
+ !!keyStates[controls.elbow_flex.negative]?.pressed
615
+ }
616
+ onMouseDown={() =>
617
+ simulateKeyPress(controls.elbow_flex.negative)
618
+ }
619
+ onMouseUp={() =>
620
+ simulateKeyRelease(controls.elbow_flex.negative)
621
+ }
622
+ disabled={!teleopState?.isActive}
623
+ />
624
+ <VirtualKey
625
+ label="D"
626
+ subLabel="Wrist-"
627
+ isPressed={
628
+ !!keyStates[controls.wrist_flex.negative]?.pressed
629
+ }
630
+ onMouseDown={() =>
631
+ simulateKeyPress(controls.wrist_flex.negative)
632
+ }
633
+ onMouseUp={() =>
634
+ simulateKeyRelease(controls.wrist_flex.negative)
635
+ }
636
+ disabled={!teleopState?.isActive}
637
+ />
638
+ </div>
639
+ <span className="font-bold text-sm font-sans">
640
+ Elbow/Wrist
641
+ </span>
642
  </div>
643
+ <div className="flex flex-col items-center gap-2">
644
+ <div className="flex gap-2">
645
+ <VirtualKey
646
+ label="Q"
647
+ subLabel="Roll+"
648
+ isPressed={
649
+ !!keyStates[controls.wrist_roll.positive]?.pressed
650
+ }
651
+ onMouseDown={() =>
652
+ simulateKeyPress(controls.wrist_roll.positive)
653
+ }
654
+ onMouseUp={() =>
655
+ simulateKeyRelease(controls.wrist_roll.positive)
656
+ }
657
+ disabled={!teleopState?.isActive}
658
+ />
659
+ <VirtualKey
660
+ label="E"
661
+ subLabel="Roll-"
662
+ isPressed={
663
+ !!keyStates[controls.wrist_roll.negative]?.pressed
664
+ }
665
+ onMouseDown={() =>
666
+ simulateKeyPress(controls.wrist_roll.negative)
667
+ }
668
+ onMouseUp={() =>
669
+ simulateKeyRelease(controls.wrist_roll.negative)
670
+ }
671
+ disabled={!teleopState?.isActive}
672
+ />
673
+ </div>
674
+ <div className="flex gap-2">
675
+ <VirtualKey
676
+ label="O"
677
+ subLabel="Grip+"
678
+ isPressed={
679
+ !!keyStates[controls.gripper.positive]?.pressed
680
+ }
681
+ onMouseDown={() =>
682
+ simulateKeyPress(controls.gripper.positive)
683
+ }
684
+ onMouseUp={() =>
685
+ simulateKeyRelease(controls.gripper.positive)
686
+ }
687
+ disabled={!teleopState?.isActive}
688
+ />
689
+ <VirtualKey
690
+ label="C"
691
+ subLabel="Grip-"
692
+ isPressed={
693
+ !!keyStates[controls.gripper.negative]?.pressed
694
+ }
695
+ onMouseDown={() =>
696
+ simulateKeyPress(controls.gripper.negative)
697
+ }
698
+ onMouseUp={() =>
699
+ simulateKeyRelease(controls.gripper.negative)
700
+ }
701
+ disabled={!teleopState?.isActive}
702
+ />
703
+ </div>
704
+ <span className="font-bold text-sm font-sans">Roll/Grip</span>
705
  </div>
 
706
  </div>
707
+ <div className="pt-4 border-t border-white/10">
708
+ <div className="flex justify-between items-center font-mono text-sm">
709
+ <div className="flex items-center gap-2 text-muted-foreground">
710
+ <Keyboard className="w-4 h-4" />
711
+ <span>
712
+ Active Keys:{" "}
713
+ {Object.values(keyStates).filter((k) => k.pressed).length}
714
+ </span>
715
+ </div>
716
+ <TooltipProvider>
717
+ <Tooltip>
718
+ <TooltipTrigger asChild>
719
+ <div
720
+ className={cn(
721
+ "w-10 h-6 border rounded-md flex items-center justify-center font-mono text-xs transition-all",
722
+ "select-none user-select-none",
723
+ !teleopState?.isActive &&
724
+ "opacity-50 cursor-not-allowed",
725
+ teleopState?.isActive &&
726
+ "cursor-pointer hover:bg-white/5",
727
+ keyStates[controls.stop]?.pressed
728
+ ? "bg-destructive text-destructive-foreground border-destructive"
729
+ : "bg-background"
730
+ )}
731
+ onMouseDown={(e) => {
732
+ e.preventDefault();
733
+ if (teleopState?.isActive) {
734
+ simulateKeyPress(controls.stop);
735
+ }
736
+ }}
737
+ onMouseUp={(e) => {
738
+ e.preventDefault();
739
+ if (teleopState?.isActive) {
740
+ simulateKeyRelease(controls.stop);
741
+ }
742
+ }}
743
+ onMouseLeave={(e) => {
744
+ e.preventDefault();
745
+ if (teleopState?.isActive) {
746
+ simulateKeyRelease(controls.stop);
747
+ }
748
+ }}
749
+ >
750
+ ESC
751
+ </div>
752
+ </TooltipTrigger>
753
+ <TooltipContent>Emergency Stop</TooltipContent>
754
+ </Tooltip>
755
+ </TooltipProvider>
756
  </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
757
  </div>
758
  </div>
759
  </div>
760
  </div>
761
+ </Card>
762
+
763
+ {/* Robot Movement Recorder - Always show UI */}
764
+ <Recorder
765
+ teleoperators={memoizedTeleoperators}
766
+ robot={robot}
767
+ onNeedsTeleoperation={initializeTeleoperation}
768
+ />
769
+ </>
770
  );
771
  }
examples/cyberpunk-standalone/src/components/teleoperator-episodes-view.tsx ADDED
@@ -0,0 +1,69 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ "use client";
2
+
3
+ import { useState } from "react";
4
+ import { LeRobotEpisode } from "@lerobot/web";
5
+ import { TeleoperatorFramesView } from "./teleoperator-frames-view";
6
+ import { Button } from "./ui/button";
7
+
8
+ interface TeleoperatorEpisodesViewProps {
9
+ teleoperatorData?: LeRobotEpisode[];
10
+ }
11
+
12
+ export function TeleoperatorEpisodesView({ teleoperatorData }: TeleoperatorEpisodesViewProps) {
13
+ // State to track which episodes are expanded
14
+ const [expandedEpisodes, setExpandedEpisodes] = useState<Record<number, boolean>>({});
15
+
16
+ // Toggle expanded state for an episode
17
+ const toggleEpisode = (index: number) => {
18
+ setExpandedEpisodes(prev => ({
19
+ ...prev,
20
+ [index]: !prev[index]
21
+ }));
22
+ };
23
+
24
+ return (
25
+ <div className="flex flex-col gap-2">
26
+ <div className="text-sm text-center text-muted-foreground mb-2">List of recorded episodes</div>
27
+ <div className="flex flex-col gap-1">
28
+ {/* Header */}
29
+ <div className="flex flex-row font-medium text-sm">
30
+ <div className="flex-1 px-4 py-2">id</div>
31
+ <div className="flex-1 px-4 py-2">time length</div>
32
+ <div className="flex-1 px-4 py-2">frames</div>
33
+ </div>
34
+
35
+ {/* Body */}
36
+ {teleoperatorData && teleoperatorData.length > 0 ? (
37
+ teleoperatorData.map((episode: LeRobotEpisode, i: number) => (
38
+ <div key={i} className="flex flex-col border-t border-gray-700">
39
+ {/* Episode row */}
40
+ <div className="flex flex-row">
41
+ <div className="flex-1 px-4 py-2 font-mono">{i}</div>
42
+ <div className="flex-1 px-4 py-2 font-mono">{episode.timespan}</div>
43
+ <div className="flex-1 px-4 py-2 font-mono">{episode.length}</div>
44
+ <div className="px-4 py-2 cursor-pointer" onClick={() => toggleEpisode(i)}>
45
+ {expandedEpisodes[i] ?
46
+ <Button>hide frames</Button> :
47
+ <Button>show frames</Button>}
48
+ </div>
49
+ </div>
50
+
51
+ {/* Frames (collapsible) */}
52
+ {expandedEpisodes[i] && (
53
+ <TeleoperatorFramesView frames={episode.frames} />
54
+ )}
55
+ </div>
56
+ ))
57
+ ) : (
58
+ <div className="flex flex-row border-t border-gray-700">
59
+ <div
60
+ className="flex-1 px-4 py-4 text-center text-muted-foreground"
61
+ >
62
+ No episodes recorded yet. Click "Start Recording" to begin.
63
+ </div>
64
+ </div>
65
+ )}
66
+ </div>
67
+ </div>
68
+ );
69
+ }
examples/cyberpunk-standalone/src/components/teleoperator-frames-view.tsx ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ "use client";
2
+
3
+ import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
4
+ import { TeleoperatorJointGraph } from "./teleoperator-joint-graph";
5
+
6
+ interface TeleoperatorFramesViewProps {
7
+ frames: NonIndexedLeRobotDatasetRow[];
8
+ }
9
+
10
+ export function TeleoperatorFramesView({ frames }: TeleoperatorFramesViewProps) {
11
+ // Joint names in the order they appear in the arrays
12
+ const jointNames = [
13
+ "shoulder_pan",
14
+ "shoulder_lift",
15
+ "elbow_flex",
16
+ "wrist_flex",
17
+ "wrist_roll",
18
+ "gripper"
19
+ ];
20
+
21
+ // Helper function to format an object as a column of key-value pairs with joint names
22
+ const formatArrayAsColumn = (obj: Record<number, number>): string => {
23
+ return Object.entries(obj)
24
+ .map(([key, value]) => {
25
+ // Convert numeric key to joint name if possible
26
+ const index = parseInt(key);
27
+ const jointName = !isNaN(index) && index < jointNames.length ? jointNames[index] : key;
28
+ return `${jointName}: ${value}`;
29
+ })
30
+ .join('\n');
31
+ };
32
+
33
+ return (
34
+ <div className="ml-8 mr-4 mb-2">
35
+ {/* Joint visualization graph */}
36
+ <TeleoperatorJointGraph frames={frames} />
37
+
38
+ {/* Frames container with horizontal scroll */}
39
+ <div className="bg-gray-800/50 rounded-md overflow-hidden">
40
+ <div className="overflow-x-auto">
41
+ {/* Frames header */}
42
+ <table className="w-full min-w-max table-fixed">
43
+ <thead>
44
+ <tr className="text-xs font-medium bg-gray-800/80 text-gray-300">
45
+ <th className="w-16 px-2 py-1 text-left">Frame</th>
46
+ <th className="w-64 px-2 py-1 text-left">Timestamp</th>
47
+ <th className="w-[300px] px-2 py-1 text-left">Action</th>
48
+ <th className="w-[500px] px-2 py-1 text-left">State</th>
49
+ </tr>
50
+ </thead>
51
+
52
+ {/* Frame rows */}
53
+ <tbody className="max-h-60 overflow-y-auto">
54
+ {frames.map((frame: NonIndexedLeRobotDatasetRow, frameIndex: number) => (
55
+ <tr key={frameIndex} className="text-xs border-t border-gray-700/50">
56
+ <td className="w-16 px-2 py-1 font-mono whitespace-nowrap">{frameIndex}</td>
57
+ <td className="w-64 px-2 py-1 font-mono whitespace-nowrap">
58
+ {frame.timestamp}
59
+ </td>
60
+ <td className="w-[200px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
61
+ {Object.keys(frame.action).length > 0 ?
62
+ formatArrayAsColumn(frame.action) :
63
+ '-'}
64
+ </td>
65
+ <td className="w-[300px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
66
+ {Object.keys(frame["observation.state"]).length > 0 ?
67
+ formatArrayAsColumn(frame["observation.state"]) :
68
+ '-'}
69
+ </td>
70
+ </tr>
71
+ ))}
72
+ </tbody>
73
+ </table>
74
+ </div>
75
+ </div>
76
+ </div>
77
+ );
78
+ }
examples/cyberpunk-standalone/src/components/teleoperator-joint-graph.tsx ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import React from "react";
2
+ import {
3
+ LineChart,
4
+ Line,
5
+ XAxis,
6
+ YAxis,
7
+ CartesianGrid,
8
+ Tooltip,
9
+ Legend,
10
+ ResponsiveContainer
11
+ } from "recharts";
12
+ import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
13
+
14
+ interface TeleoperatorJointGraphProps {
15
+ frames: NonIndexedLeRobotDatasetRow[];
16
+ }
17
+
18
+ export function TeleoperatorJointGraph({ frames }: TeleoperatorJointGraphProps) {
19
+ // Skip rendering if no frames
20
+ if (!frames || frames.length === 0) {
21
+ return null;
22
+ }
23
+
24
+ // Use hardcoded joint names that match the LeRobot dataset format
25
+ const jointNames = [
26
+ "shoulder_pan",
27
+ "shoulder_lift",
28
+ "elbow_flex",
29
+ "wrist_flex",
30
+ "wrist_roll",
31
+ "gripper"
32
+ ];
33
+
34
+ // Generate a color palette for the joints
35
+ const colors = [
36
+ "#8884d8", "#82ca9d", "#ffc658", "#ff8042", "#0088fe", "#00C49F",
37
+ "#FFBB28", "#FF8042", "#a4de6c", "#d0ed57"
38
+ ];
39
+
40
+ // Prepare data for the chart - handling arrays
41
+ const chartData = frames.map((frame, index) => {
42
+ // Create base data point with index
43
+ const dataPoint: any = {
44
+ name: index,
45
+ timestamp: frame.timestamp
46
+ };
47
+
48
+ // Add action values (assuming action is an array)
49
+ if (Array.isArray(frame.action)) {
50
+ // Map each array index to the corresponding joint name
51
+ jointNames.forEach((jointName, i) => {
52
+ if (i < frame.action.length) {
53
+ dataPoint[`action_${jointName}`] = frame.action[i];
54
+ }
55
+ });
56
+ }
57
+
58
+ // Add observation state values (assuming observation.state is an array)
59
+ if (Array.isArray(frame["observation.state"])) {
60
+ // Map each array index to the corresponding joint name
61
+ jointNames.forEach((jointName, i) => {
62
+ if (i < frame["observation.state"].length) {
63
+ dataPoint[`state_${jointName}`] = frame["observation.state"][i];
64
+ }
65
+ });
66
+ }
67
+
68
+ return dataPoint;
69
+ });
70
+
71
+ // Create lines for each joint
72
+ const linesToRender = jointNames.flatMap(jointName => [
73
+ {
74
+ key: `action_${jointName}`,
75
+ dataKey: `action_${jointName}`,
76
+ name: `Action: ${jointName}`,
77
+ isDotted: true
78
+ },
79
+ {
80
+ key: `state_${jointName}`,
81
+ dataKey: `state_${jointName}`,
82
+ name: `State: ${jointName}`,
83
+ isDotted: false
84
+ }
85
+ ]);
86
+
87
+ return (
88
+ <div className="w-full bg-gray-800/50 rounded-md p-4 mb-4">
89
+ <h3 className="text-sm font-medium text-gray-300 mb-2">Joint Positions Over Time</h3>
90
+ <ResponsiveContainer width="100%" height={300}>
91
+ <LineChart
92
+ data={chartData}
93
+ margin={{
94
+ top: 5,
95
+ right: 30,
96
+ left: 20,
97
+ bottom: 5
98
+ }}
99
+ >
100
+ <CartesianGrid strokeDasharray="3 3" stroke="#444" />
101
+ <XAxis
102
+ dataKey="name"
103
+ label={{ value: 'Frame Index', position: 'insideBottomRight', offset: -10 }}
104
+ stroke="#aaa"
105
+ />
106
+ <YAxis stroke="#aaa" />
107
+ <Tooltip
108
+ contentStyle={{ backgroundColor: '#333', borderColor: '#555' }}
109
+ labelStyle={{ color: '#eee' }}
110
+ itemStyle={{ color: '#eee' }}
111
+ />
112
+ <Legend />
113
+
114
+ {/* Render all lines */}
115
+ {linesToRender.map((lineConfig, index) => {
116
+ const jointName = lineConfig.dataKey.replace(/^(action|state)_/, '');
117
+ const jointIndex = jointNames.indexOf(jointName);
118
+ const colorIndex = jointIndex >= 0 ? jointIndex : index % colors.length;
119
+
120
+ return (
121
+ <Line
122
+ key={lineConfig.key}
123
+ type="monotone"
124
+ dataKey={lineConfig.dataKey}
125
+ name={lineConfig.name}
126
+ stroke={colors[colorIndex]}
127
+ strokeDasharray={lineConfig.isDotted ? "5 5" : undefined}
128
+ dot={false}
129
+ activeDot={{ r: 4 }}
130
+ />
131
+ );
132
+ })}
133
+ </LineChart>
134
+ </ResponsiveContainer>
135
+ </div>
136
+ );
137
+ }
examples/cyberpunk-standalone/src/main.tsx CHANGED
@@ -1,5 +1,6 @@
1
  import { StrictMode } from "react";
2
  import { createRoot } from "react-dom/client";
 
3
  import "./global.css";
4
  import App from "./App";
5
  import { ThemeProvider } from "./components/theme-provider";
@@ -11,13 +12,15 @@ setTimeout(() => {
11
 
12
  createRoot(document.getElementById("root")!).render(
13
  <StrictMode>
14
- <ThemeProvider
15
- attribute="class"
16
- defaultTheme="system"
17
- enableSystem
18
- disableTransitionOnChange
19
- >
20
- <App />
21
- </ThemeProvider>
 
 
22
  </StrictMode>
23
  );
 
1
  import { StrictMode } from "react";
2
  import { createRoot } from "react-dom/client";
3
+ import { BrowserRouter } from "react-router-dom";
4
  import "./global.css";
5
  import App from "./App";
6
  import { ThemeProvider } from "./components/theme-provider";
 
12
 
13
  createRoot(document.getElementById("root")!).render(
14
  <StrictMode>
15
+ <BrowserRouter>
16
+ <ThemeProvider
17
+ attribute="class"
18
+ defaultTheme="system"
19
+ enableSystem
20
+ disableTransitionOnChange
21
+ >
22
+ <App />
23
+ </ThemeProvider>
24
+ </BrowserRouter>
25
  </StrictMode>
26
  );
package.json CHANGED
@@ -34,14 +34,22 @@
34
  "install-global": "pnpm run build && npm link"
35
  },
36
  "dependencies": {
37
- "@lerobot/web": "workspace:*",
 
 
 
 
 
38
  "log-update": "^6.1.0",
39
- "serialport": "^12.0.0"
 
 
40
  },
41
  "devDependencies": {
42
  "@changesets/cli": "^2.29.5",
43
- "@types/node": "^22.10.5",
44
- "tsx": "^4.19.2",
 
45
  "typescript": "~5.8.3"
46
  },
47
  "repository": {
 
34
  "install-global": "pnpm run build && npm link"
35
  },
36
  "dependencies": {
37
+ "@aws-sdk/client-s3": "^3.856.0",
38
+ "@aws-sdk/lib-storage": "^3.856.0",
39
+ "@huggingface/hub": "^2.4.0",
40
+ "@react-three/cannon": "^6.6.0",
41
+ "apache-arrow": "^21.0.0",
42
+ "jszip": "^3.10.1",
43
  "log-update": "^6.1.0",
44
+ "parquet-wasm": "^0.6.1",
45
+ "serialport": "^12.0.0",
46
+ "three": "^0.178.0"
47
  },
48
  "devDependencies": {
49
  "@changesets/cli": "^2.29.5",
50
+ "@types/node": "^22.17.0",
51
+ "@types/three": "^0.178.1",
52
+ "tsx": "^4.20.3",
53
  "typescript": "~5.8.3"
54
  },
55
  "repository": {
packages/web/README.md CHANGED
@@ -312,6 +312,75 @@ await releaseMotors(robot, [1, 2, 3]);
312
  - `robot: RobotConnection` - Connected robot
313
  - `motorIds?: number[]` - Specific motor IDs (default: all motors for robot type)
314
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
315
  ## Browser Requirements
316
 
317
  - **chromium 89+** with WebSerial and WebUSB API support
 
312
  - `robot: RobotConnection` - Connected robot
313
  - `motorIds?: number[]` - Specific motor IDs (default: all motors for robot type)
314
 
315
+ ---
316
+
317
+ ## Dataset Recording and Export
318
+
319
+ The LeRobot.js library provides functionality to record teleoperator data and export it in the LeRobot dataset format, compatible with machine learning models.
320
+
321
+ ### `LeRobotDatasetRecorder`
322
+
323
+ Records teleoperator movements and camera streams, then exports them in the LeRobot dataset format.
324
+
325
+ ```typescript
326
+ import { LeRobotDatasetRecorder } from "@lerobot/web";
327
+
328
+ // Create a recorder with teleoperator and video streams
329
+ const recorder = new LeRobotDatasetRecorder(
330
+ [teleoperator], // Array of teleoperators to record, currently only supports 1 teleoperator
331
+ { "main": videoStream }, // Video streams by camera key
332
+ 30, // Target FPS
333
+ "Pick and place task" // Task description
334
+ );
335
+
336
+ // Start recording
337
+ await recorder.startRecording();
338
+
339
+ // ... robot performs task ...
340
+
341
+ // Stop recording and get the data
342
+ const recordingData = await recorder.stopRecording();
343
+
344
+ // Export the dataset in various formats
345
+ // 1. As a downloadable zip file
346
+ await recorder.exportForLeRobot('zip-download');
347
+
348
+ // 2. Upload to Hugging Face
349
+ const hfUploader = await recorder.exportForLeRobot('huggingface', {
350
+ repoName: 'my-robot-dataset',
351
+ accessToken: 'hf_...',
352
+ });
353
+
354
+ // 3. Upload to S3
355
+ const s3Uploader = await recorder.exportForLeRobot('s3', {
356
+ bucketName: 'my-bucket',
357
+ accessKeyId: 'AKIA...',
358
+ secretAccessKey: '...',
359
+ region: 'us-east-1',
360
+ });
361
+ ```
362
+
363
+ #### Key Features
364
+
365
+ - **Multi-source Recording**: Records teleoperator movements and synchronized video
366
+ - **Regular Interpolation**: Generates frames at consistent intervals with `episodes` getter
367
+ - **Multiple Export Formats**: Supports local download, Hugging Face, and S3 upload
368
+ - **LeRobot Dataset Format**: Follows the standard format for compatibility with ML models
369
+
370
+ > **Note:** The dataset statistical data currently generated is incorrect and needs to be updated in a future release.
371
+
372
+ #### Dataset Format
373
+
374
+ The exported dataset follows the LeRobot format with this structure:
375
+
376
+ ```
377
+ /data/chunk-000/file-000.parquet # Teleoperator data
378
+ /videos/observation.images.{camera-key}/chunk-000/file-000.mp4 # Video data
379
+ /metadata.json # Dataset metadata
380
+ /statistics.json # Dataset statistics (currently incorrect)
381
+ /README.md # Dataset documentation
382
+ ```
383
+
384
  ## Browser Requirements
385
 
386
  - **chromium 89+** with WebSerial and WebUSB API support
packages/web/package.json CHANGED
@@ -36,6 +36,7 @@
36
  ],
37
  "scripts": {
38
  "build": "tsc --project tsconfig.build.json",
 
39
  "prepublishOnly": "npm run build",
40
  "dev": "vitest --watch",
41
  "test": "vitest run",
@@ -43,10 +44,10 @@
43
  "test:coverage": "vitest run --coverage"
44
  },
45
  "devDependencies": {
46
- "vitest": "^2.0.0",
47
- "@vitest/ui": "^2.0.0",
48
- "jsdom": "^24.0.0",
49
- "vite": "^6.3.5"
50
  },
51
  "peerDependencies": {
52
  "typescript": ">=4.5.0"
@@ -59,5 +60,13 @@
59
  "author": "Tim Pietrusky",
60
  "publishConfig": {
61
  "access": "public"
 
 
 
 
 
 
 
 
62
  }
63
  }
 
36
  ],
37
  "scripts": {
38
  "build": "tsc --project tsconfig.build.json",
39
+ "watch": "tsc --project tsconfig.build.json --watch",
40
  "prepublishOnly": "npm run build",
41
  "dev": "vitest --watch",
42
  "test": "vitest run",
 
44
  "test:coverage": "vitest run --coverage"
45
  },
46
  "devDependencies": {
47
+ "@vitest/ui": "^2.1.9",
48
+ "jsdom": "^24.1.3",
49
+ "vite": "^6.3.5",
50
+ "vitest": "^2.1.9"
51
  },
52
  "peerDependencies": {
53
  "typescript": ">=4.5.0"
 
60
  "author": "Tim Pietrusky",
61
  "publishConfig": {
62
  "access": "public"
63
+ },
64
+ "dependencies": {
65
+ "@aws-sdk/client-s3": "^3.856.0",
66
+ "@aws-sdk/lib-storage": "^3.856.0",
67
+ "@huggingface/hub": "^2.4.0",
68
+ "apache-arrow": "^21.0.0",
69
+ "jszip": "^3.10.1",
70
+ "parquet-wasm": "^0.6.1"
71
  }
72
  }
packages/web/pnpm-lock.yaml ADDED
The diff for this file is too large to render. See raw diff
 
packages/web/src/calibrate.ts CHANGED
@@ -15,7 +15,6 @@ import {
15
  writeHardwarePositionLimits,
16
  } from "./utils/motor-calibration.js";
17
  import { createSO100Config } from "./robots/so100_config.js";
18
- import type { RobotConnection } from "./types/robot-connection.js";
19
  import type { RobotHardwareConfig } from "./types/robot-config.js";
20
  import type {
21
  CalibrateConfig,
 
15
  writeHardwarePositionLimits,
16
  } from "./utils/motor-calibration.js";
17
  import { createSO100Config } from "./robots/so100_config.js";
 
18
  import type { RobotHardwareConfig } from "./types/robot-config.js";
19
  import type {
20
  CalibrateConfig,
packages/web/src/hf_uploader.ts ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import * as hub from "@huggingface/hub";
2
+ import type { RepoDesignation } from "@huggingface/hub";
3
+ import type { ContentSource } from "@huggingface/hub";
4
+
5
+ type FileArray = Array<URL | File | { path: string; content: ContentSource }>;
6
+ /**
7
+ * Uploads a leRobot dataset to huggingface
8
+ */
9
+
10
+ export class LeRobotHFUploader extends EventTarget {
11
+ private _repoDesignation: RepoDesignation;
12
+ private _uploaded : boolean;
13
+ private _created_repo : boolean;
14
+
15
+ constructor(username: string, repoName: string) {
16
+ super();
17
+ this._repoDesignation = {
18
+ name : `${username}/${repoName}`,
19
+ type : "dataset"
20
+ };
21
+
22
+ this._uploaded = false;
23
+ this._created_repo = false;
24
+ }
25
+
26
+ /**
27
+ * Returns whether the repository has been successfully created
28
+ */
29
+ get createdRepo(): boolean {
30
+ return this._created_repo;
31
+ }
32
+
33
+ get uploaded() : boolean {
34
+ return this._uploaded;
35
+ }
36
+
37
+
38
+ /**
39
+ * Uploads the dataset to huggingface
40
+ *
41
+ * A referenceId is used to be able to track progress of the upload,
42
+ * this provides a progressEvent from huggingface.js (see : https://github.com/huggingface/huggingface.js/blob/main/packages/hub/README.md#usage)
43
+ *
44
+ * both this and huggingface.js have pretty bad documentation, some exploration will be required (sorry!, I have university, and work and I already feel guilty procastinating those to to write this)
45
+ *
46
+ * @param dataset The dataset to upload
47
+ * @param accessToken The access token for huggingface
48
+ * @param referenceId The reference id for the upload, to track it (optional)
49
+ */
50
+ async createRepoAndUploadFiles(files : FileArray, accessToken : string, referenceId : string = "") {
51
+ await hub.createRepo({
52
+ repo: this._repoDesignation,
53
+ accessToken: accessToken,
54
+ license: "mit"
55
+ });
56
+
57
+ this._created_repo = true;
58
+ this.dispatchEvent(new CustomEvent("repoCreated", { detail: this._repoDesignation }));
59
+
60
+ const uploadPromises : Promise<void>[] = [];
61
+ uploadPromises.push(this.uploadFilesWithProgress(files, accessToken, referenceId));
62
+
63
+ await Promise.all(uploadPromises);
64
+ }
65
+
66
+ /**
67
+ * Uploads files to huggingface with progress events
68
+ *
69
+ * @param files The files to upload
70
+ * @param accessToken The access token for huggingface
71
+ * @param referenceId The reference id for the upload, to track it (optional)
72
+ */
73
+ async uploadFilesWithProgress(files : FileArray, accessToken : string, referenceId : string = "") {
74
+ for await (const progressEvent of hub.uploadFilesWithProgress({
75
+ repo: this._repoDesignation,
76
+ accessToken: accessToken,
77
+ files: files,
78
+ })) {
79
+ this.dispatchEvent(new CustomEvent("progress", { detail: {
80
+ progressEvent,
81
+ repoDesignation: this._repoDesignation,
82
+ referenceId
83
+ } }));
84
+ }
85
+ }
86
+ }
packages/web/src/index.ts CHANGED
@@ -58,3 +58,7 @@ export {
58
  SO100_KEYBOARD_CONTROLS,
59
  } from "./robots/so100_config.js";
60
  export { KEYBOARD_TELEOPERATOR_DEFAULTS } from "./teleoperators/index.js";
 
 
 
 
 
58
  SO100_KEYBOARD_CONTROLS,
59
  } from "./robots/so100_config.js";
60
  export { KEYBOARD_TELEOPERATOR_DEFAULTS } from "./teleoperators/index.js";
61
+
62
+ // Record
63
+ export { LeRobotDatasetRecorder } from "./record.js";
64
+ export { LeRobotHFUploader } from "./hf_uploader.js";
packages/web/src/record.ts ADDED
@@ -0,0 +1,1412 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import { WebTeleoperator } from "./teleoperators/base-teleoperator";
2
+ import { MotorConfig } from "./types/teleoperation";
3
+ import * as parquet from 'parquet-wasm';
4
+ import * as arrow from 'apache-arrow';
5
+ import JSZip from 'jszip';
6
+ import getMetadataInfo from './utils/record/metadataInfo';
7
+ import type { VideoInfo } from './utils/record/metadataInfo';
8
+ import getStats from './utils/record/stats';
9
+ import generateREADME from './utils/record/generateREADME';
10
+ import { LeRobotHFUploader } from './hf_uploader';
11
+ import { LeRobotS3Uploader } from './s3_uploader';
12
+
13
+ // declare a type leRobot action that's basically an array of numbers
14
+ interface LeRobotAction {
15
+ [key: number]: number;
16
+ }
17
+
18
+ export class LeRobotEpisode {
19
+ // we assume that the frames are ordered
20
+ public frames : NonIndexedLeRobotDatasetRow[]
21
+
22
+ /**
23
+ * Optional start time of the episode
24
+ * If not set, defaults to the timestamp of the first frame
25
+ */
26
+ private _startTime?: number;
27
+
28
+ /**
29
+ * Optional end time of the episode
30
+ * If not set, defaults to the timestamp of the last frame
31
+ */
32
+ private _endTime?: number;
33
+
34
+ /**
35
+ * Creates a new LeRobotEpisode
36
+ *
37
+ * @param frames Optional array of frames to initialize the episode with
38
+ * @param startTime Optional explicit start time for the episode
39
+ * @param endTime Optional explicit end time for the episode
40
+ */
41
+ constructor(frames?: NonIndexedLeRobotDatasetRow[], startTime?: number, endTime?: number){
42
+ this.frames = frames || [];
43
+ this._startTime = startTime;
44
+ this._endTime = endTime;
45
+ }
46
+
47
+ /**
48
+ * Adds a new frame to the episode
49
+ * Ensures frames are always in chronological order
50
+ *
51
+ * @param frame The frame to add
52
+ * @throws Error if the frame's timestamp is before the last frame's timestamp
53
+ */
54
+ add(frame : NonIndexedLeRobotDatasetRow){
55
+ const lastFrame = this.frames.at(-1);
56
+ if (lastFrame && frame.timestamp < lastFrame.timestamp) {
57
+ throw new Error(`Frame timestamp ${frame.timestamp} is before last frame timestamp ${lastFrame.timestamp}`);
58
+ }
59
+ this.frames.push(frame);
60
+ }
61
+
62
+ /**
63
+ * Gets the start time of the episode
64
+ * If not explicitly set, returns the timestamp of the first frame
65
+ * If no frames exist, throws an error
66
+ */
67
+ get startTime(): number {
68
+ if (this._startTime !== undefined) {
69
+ return this._startTime;
70
+ }
71
+
72
+ const firstFrame = this.frames.at(0);
73
+ if (!firstFrame) {
74
+ throw new Error("Cannot determine start time: no frames in episode");
75
+ }
76
+
77
+ return firstFrame.timestamp;
78
+ }
79
+
80
+ /**
81
+ * Sets an explicit start time for the episode
82
+ */
83
+ set startTime(value: number) {
84
+ this._startTime = value;
85
+ }
86
+
87
+ /**
88
+ * Gets the end time of the episode
89
+ * If not explicitly set, returns the timestamp of the last frame
90
+ * If no frames exist, throws an error
91
+ */
92
+ get endTime(): number {
93
+ if (this._endTime !== undefined) {
94
+ return this._endTime;
95
+ }
96
+
97
+ const lastFrame = this.frames.at(-1);
98
+ if (!lastFrame) {
99
+ throw new Error("Cannot determine end time: no frames in episode");
100
+ }
101
+
102
+ return lastFrame.timestamp;
103
+ }
104
+
105
+ /**
106
+ * Sets an explicit end time for the episode
107
+ */
108
+ set endTime(value: number) {
109
+ this._endTime = value;
110
+ }
111
+
112
+ /**
113
+ * The time difference between the start and end time of the episode, in seconds
114
+ */
115
+ get timespan(){
116
+ const hasNoFrames = this.frames.length === 0
117
+ if(hasNoFrames) return 0
118
+
119
+ return this.endTime - this.startTime;
120
+ }
121
+
122
+ /**
123
+ * The number of frames in the episode
124
+ */
125
+ get length(){
126
+ return this.frames.length;
127
+ }
128
+
129
+ /**
130
+ * Creates a new LeRobotEpisode with frames interpolated at regular intervals
131
+ *
132
+ * @param fps The desired frames per second for the interpolated episode
133
+ * @param startIndex The desired starting index for the episode frames, useful when storing multiple episodes
134
+ * @returns A new LeRobotEpisode with interpolated frames
135
+ */
136
+ getInterpolatedRegularEpisode(fps: number, startIndex: number = 0): LeRobotEpisode {
137
+ if (this.frames.length === 0) {
138
+ return new LeRobotEpisode([], this._startTime, this._endTime);
139
+ }
140
+
141
+ const actualStartTime = this._startTime !== undefined ? this._startTime : this.frames[0].timestamp;
142
+ const actualEndTime = this._endTime !== undefined ? this._endTime : this.frames[this.frames.length - 1].timestamp;
143
+ const timeDifference = actualEndTime - actualStartTime;
144
+
145
+ const numFrames = Math.max(1, Math.floor(timeDifference * fps));
146
+ const interpolatedFrames: NonIndexedLeRobotDatasetRow[] = [];
147
+
148
+ const firstFrame = this.frames[0];
149
+ const lastFrame = this.frames[this.frames.length - 1];
150
+
151
+ for (let i = 0; i < numFrames; i++) {
152
+ const timestamp = actualStartTime + (i / fps);
153
+ let frameToAdd: NonIndexedLeRobotDatasetRow;
154
+
155
+ if (timestamp < firstFrame.timestamp) {
156
+ frameToAdd = { ...firstFrame, timestamp };
157
+ frameToAdd.frame_index = i;
158
+ frameToAdd.index = startIndex + i;
159
+ } else if (timestamp > lastFrame.timestamp) {
160
+ frameToAdd = { ...lastFrame, timestamp };
161
+ frameToAdd.frame_index = i;
162
+ frameToAdd.index = startIndex + i;
163
+ } else {
164
+ let lowerIndex = 0;
165
+ for (let j = 0; j < this.frames.length - 1; j++) {
166
+ if (this.frames[j].timestamp <= timestamp && this.frames[j + 1].timestamp > timestamp) {
167
+ lowerIndex = j;
168
+ break;
169
+ }
170
+ }
171
+
172
+ const lowerFrame = this.frames[lowerIndex];
173
+ const upperFrame = this.frames[lowerIndex + 1];
174
+
175
+ frameToAdd = LeRobotEpisode.interpolateFrames(
176
+ lowerFrame,
177
+ upperFrame,
178
+ timestamp
179
+ );
180
+
181
+ frameToAdd.frame_index = i;
182
+ frameToAdd.episode_index = lowerFrame.episode_index;
183
+ frameToAdd.index = startIndex + i;
184
+ frameToAdd.task_index = lowerFrame.task_index;
185
+ }
186
+
187
+ interpolatedFrames.push(frameToAdd);
188
+ }
189
+
190
+ return new LeRobotEpisode(interpolatedFrames, actualStartTime, actualEndTime);
191
+ }
192
+
193
+ /**
194
+ * Interpolates between two frames to create a new frame at the specified timestamp
195
+ *
196
+ * @param frame1 The first frame
197
+ * @param frame2 The second frame
198
+ * @param targetTimestamp The timestamp at which to interpolate
199
+ * @returns A new interpolated frame
200
+ */
201
+ static interpolateFrames(
202
+ frame1: NonIndexedLeRobotDatasetRow,
203
+ frame2: NonIndexedLeRobotDatasetRow,
204
+ targetTimestamp: number
205
+ ): NonIndexedLeRobotDatasetRow {
206
+ if (targetTimestamp < frame1.timestamp || targetTimestamp > frame2.timestamp) {
207
+ throw new Error("Target timestamp must be between the timestamps of the two frames");
208
+ }
209
+
210
+ const timeRange = frame2.timestamp - frame1.timestamp;
211
+ const interpolationFactor = (targetTimestamp - frame1.timestamp) / timeRange;
212
+
213
+ // Interpolate action array
214
+ const interpolatedAction = LeRobotEpisode.interpolateArrays(
215
+ frame1.action,
216
+ frame2.action,
217
+ interpolationFactor
218
+ );
219
+
220
+ // Interpolate observation.state array
221
+ const interpolatedObservationState = LeRobotEpisode.interpolateArrays(
222
+ frame1["observation.state"],
223
+ frame2["observation.state"],
224
+ interpolationFactor
225
+ );
226
+
227
+ // Create the interpolated frame
228
+ return {
229
+ timestamp: targetTimestamp,
230
+ action: interpolatedAction,
231
+ "observation.state": interpolatedObservationState,
232
+ episode_index: frame1.episode_index,
233
+ task_index: frame1.task_index,
234
+ // Optional properties are not interpolated
235
+ frame_index: frame1.frame_index,
236
+ index: frame1.index
237
+ };
238
+ }
239
+
240
+ /**
241
+ * Helper method to interpolate between two arrays
242
+ *
243
+ * @param array1 First array of values
244
+ * @param array2 Second array of values
245
+ * @param factor Interpolation factor (0-1)
246
+ * @returns Interpolated array
247
+ */
248
+ private static interpolateArrays(array1: any, array2: any, factor: number): any {
249
+ // Handle different types of inputs
250
+ if (Array.isArray(array1) && Array.isArray(array2)) {
251
+ // For arrays, interpolate each element
252
+ return array1.map((value, index) => {
253
+ return value + (array2[index] - value) * factor;
254
+ });
255
+ } else if (typeof array1 === 'object' && typeof array2 === 'object') {
256
+ // For objects, interpolate each property
257
+ const result: any = {};
258
+ for (const key of Object.keys(array1)) {
259
+ if (key in array2) {
260
+ result[key] = array1[key] + (array2[key] - array1[key]) * factor;
261
+ } else {
262
+ result[key] = array1[key];
263
+ }
264
+ }
265
+ return result;
266
+ } else {
267
+ // For primitive values
268
+ return array1 + (array2 - array1) * factor;
269
+ }
270
+ }
271
+ }
272
+
273
+ /**
274
+ * Base interface for LeRobot dataset rows with common fields
275
+ */
276
+ export interface NonIndexedLeRobotDatasetRow {
277
+ timestamp: number;
278
+ action: LeRobotAction;
279
+ "observation.state": LeRobotAction;
280
+
281
+ // properties are optional for back-converstion from normal rows
282
+ episode_index : number,
283
+ task_index : number
284
+ frame_index? : number
285
+ index? : number
286
+ }
287
+
288
+ /**
289
+ * Represents a complete row in the LeRobot dataset format after indexing
290
+ * Used in the final exported dataset
291
+ */
292
+ export interface LeRobotDatasetRow extends NonIndexedLeRobotDatasetRow {
293
+ frame_index: number;
294
+ index: number;
295
+ }
296
+
297
+ /**
298
+ * A mechanism to store and record, the video of all associated cameras
299
+ * as well as the teleoperator data
300
+ *
301
+ * follows the lerobot dataset format https://github.com/huggingface/lerobot/blob/cf86b9300dc83fdad408cfe4787b7b09b55f12cf/README.md#the-lerobotdataset-format
302
+ */
303
+ export class LeRobotDatasetRecorder {
304
+ teleoperators: WebTeleoperator[];
305
+ videoStreams: { [key: string]: MediaStream };
306
+ mediaRecorders: { [key: string]: MediaRecorder };
307
+ videoChunks: { [key: string]: Blob[] };
308
+ videoBlobs: { [key: string]: Blob };
309
+ teleoperatorData: LeRobotEpisode[];
310
+ private _isRecording: boolean;
311
+ private episodeIndex: number;
312
+ private taskIndex: number;
313
+ fps: number;
314
+ taskDescription: string;
315
+
316
+ constructor(
317
+ teleoperators: WebTeleoperator[],
318
+ videoStreams: { [key: string]: MediaStream },
319
+ fps: number,
320
+ taskDescription: string = "Default task description"
321
+ ) {
322
+ this.teleoperators = [];
323
+
324
+ if(teleoperators.length > 1)
325
+ throw Error(`
326
+ Currently, only 1 teleoperator can be recorded at a time!
327
+
328
+ Note : Do not attempt to create 2 different recorders via 2 different teleoperators, this would not work either
329
+ `)
330
+
331
+ this.addTeleoperator(teleoperators[0])
332
+ this.mediaRecorders = {};
333
+ this.videoChunks = {};
334
+ this.videoBlobs = {};
335
+ this.videoStreams = {};
336
+ this.teleoperatorData = [];
337
+ this._isRecording = false;
338
+ this.fps = fps;
339
+ this.taskDescription = taskDescription;
340
+
341
+ for(const [key, stream] of Object.entries(videoStreams)) {
342
+ this.addVideoStream(key, stream);
343
+ }
344
+ }
345
+
346
+ get isRecording() : boolean {
347
+ return this._isRecording;
348
+ }
349
+
350
+ get currentEpisode() : LeRobotEpisode | undefined {
351
+ return this.teleoperatorData.at(-1);
352
+ }
353
+
354
+ /**
355
+ * Adds a new video stream to be recorded
356
+ * @param key The key to identify this video stream
357
+ * @param stream The media stream to record from
358
+ */
359
+ addVideoStream(key: string, stream: MediaStream) {
360
+ console.log("Adding video stream", key);
361
+ if (this._isRecording) {
362
+ throw new Error("Cannot add video streams while recording");
363
+ }
364
+
365
+ // Add to video streams dictionary
366
+ this.videoStreams[key] = stream;
367
+
368
+ // Initialize MediaRecorder for this stream
369
+ this.mediaRecorders[key] = new MediaRecorder(stream, {
370
+ mimeType: 'video/mp4'
371
+ });
372
+
373
+ // add a video chunk array for this stream
374
+ this.videoChunks[key] = [];
375
+ }
376
+
377
+ /**
378
+ * Add a new teleoperator and set up state update callbacks
379
+ * for recording joint position data in the LeRobot dataset format
380
+ *
381
+ * @param teleoperator The teleoperator to add callbacks to
382
+ */
383
+ addTeleoperator(teleoperator : WebTeleoperator) {
384
+ teleoperator.addOnStateUpdateCallback(params => {
385
+ if(this._isRecording){
386
+ if(!this.currentEpisode) throw Error("There is no current episode while recording, something is wrong!, this means that no frames exist on the recorder for some reason")
387
+
388
+ // Create a frame with the current state data
389
+ // Using the normalized configs for consistent data ranges
390
+ const frame : NonIndexedLeRobotDatasetRow = {
391
+ timestamp : params.commandSentTimestamp,
392
+ // For observation state, use the current motor positions
393
+ "observation.state" : this.convertMotorConfigToArray(params.newMotorConfigsNormalized),
394
+ // For action, use the target positions that were commanded
395
+ action : this.convertMotorConfigToArray(params.previousMotorConfigsNormalized),
396
+ episode_index : this.episodeIndex,
397
+ task_index : this.taskIndex
398
+ }
399
+
400
+ // Add the frame to the current episode
401
+ this.currentEpisode.add(frame);
402
+ }
403
+ });
404
+
405
+ this.teleoperators.push(teleoperator)
406
+ }
407
+
408
+ /**
409
+ * Starts recording for all teleoperators and video streams
410
+ */
411
+ startRecording() {
412
+ console.log("Starting recording");
413
+ if (this._isRecording) {
414
+ console.warn('Recording already in progress');
415
+ return;
416
+ }
417
+
418
+ this._isRecording = true;
419
+
420
+ // add a new episode
421
+ this.teleoperatorData.push(new LeRobotEpisode())
422
+
423
+ // Start recording video streams
424
+ Object.entries(this.videoStreams).forEach(([key, stream]) => {
425
+ // Create a media recorder for this stream
426
+ const mediaRecorder = new MediaRecorder(stream, {
427
+ mimeType: 'video/mp4'
428
+ });
429
+
430
+ // Handle data available events
431
+ mediaRecorder.ondataavailable = (event) => {
432
+ console.log("data available for", key);
433
+ if (event.data && event.data.size > 0) {
434
+ this.videoChunks[key].push(event.data);
435
+ }
436
+ };
437
+
438
+ // Save the recorder and start recording
439
+ this.mediaRecorders[key] = mediaRecorder;
440
+ mediaRecorder.start(1000); // Capture in 1-second chunks
441
+
442
+ console.log(`Started recording video stream: ${key}`);
443
+ });
444
+ }
445
+
446
+ setEpisodeIndex(index: number): void {
447
+ this.episodeIndex = index
448
+ }
449
+
450
+ setTaskIndex(index: number): void {
451
+ this.taskIndex = index
452
+ }
453
+
454
+ /**
455
+ * teleoperatorData by default only contains data
456
+ * for the episodes in a non-regularized manner
457
+ *
458
+ * this function returns episodes in a regularized manner, wherein
459
+ * the frames in each are interpolated through so that all frames are spaced
460
+ * equally through each other
461
+ */
462
+ get episodes() : LeRobotEpisode[]{
463
+ const regularizedEpisodes : LeRobotEpisode[] = [];
464
+ let lastFrameIndex = 0;
465
+
466
+ for(let i=0; i<this.teleoperatorData.length; i++){
467
+ let episode = this.teleoperatorData[i];
468
+ const regularizedEpisode = episode.getInterpolatedRegularEpisode(this.fps, lastFrameIndex);
469
+ regularizedEpisodes.push(regularizedEpisode)
470
+
471
+ lastFrameIndex += regularizedEpisode.frames?.at(-1)?.index || 0
472
+ }
473
+
474
+ return regularizedEpisodes
475
+ }
476
+
477
+ /**
478
+ * Stops recording for all teleoperators and video streams
479
+ * @returns An object containing teleoperator data and video blobs
480
+ */
481
+ async stopRecording() {
482
+ if (!this._isRecording) {
483
+ console.warn('No recording in progress');
484
+ return { teleoperatorData: [], videoBlobs: {} };
485
+ }
486
+
487
+ this._isRecording = false;
488
+
489
+ // Stop all media recorders
490
+ const stopPromises = Object.entries(this.mediaRecorders).map(([key, recorder]) => {
491
+ return new Promise<void>((resolve) => {
492
+ // Only do this if the recorder is active
493
+ if (recorder.state === 'inactive') {
494
+ resolve();
495
+ return;
496
+ }
497
+
498
+ // When the recorder stops, create a blob
499
+ recorder.onstop = () => {
500
+ // Combine all chunks into a single blob
501
+ const chunks = this.videoChunks[key] || [];
502
+ const blob = new Blob(chunks, { type: 'video/mp4' });
503
+ this.videoBlobs[key] = blob;
504
+ resolve();
505
+ };
506
+
507
+ // Stop the recorder
508
+ recorder.stop();
509
+ });
510
+ });
511
+
512
+ // Wait for all recorders to stop
513
+ await Promise.all(stopPromises);
514
+ return {
515
+ teleoperatorData: this.episodes,
516
+ videoBlobs: this.videoBlobs
517
+ };
518
+ }
519
+
520
+ /**
521
+ * Clears the teleoperator data and video blobs
522
+ */
523
+ clearRecording() {
524
+ this.teleoperatorData = [];
525
+ this.videoBlobs = {};
526
+ }
527
+
528
+ /**
529
+ * Action is a dictionary of motor positions, timestamp1 and timestamp2 are when the actions occurred
530
+ * reqTimestamp must be between timestamp1 and timestamp2
531
+ *
532
+ * the keys in action1 and action2 must match, this will loop through the dictionary
533
+ * and interpolate each action to the required timestamp
534
+ *
535
+ * @param action1 Motor positions at timestamp1
536
+ * @param action2 Motor positions at timestamp2
537
+ * @param timestamp1 The timestamp of action1
538
+ * @param timestamp2 The timestamp of action2
539
+ * @param reqTimestamp The timestamp at which to interpolate
540
+ * @returns The interpolated action
541
+ */
542
+ _actionInterpolatate(action1 : any, action2 : any, timestamp1 : number, timestamp2 : number, reqTimestamp : number) : any {
543
+ if(reqTimestamp < timestamp1 || reqTimestamp > timestamp2) throw new Error("reqTimestamp must be between timestamp1 and timestamp2");
544
+ if(timestamp2 < timestamp1) throw new Error("timestamp2 must be greater than timestamp1");
545
+
546
+ const numActions = Object.keys(action1).length;
547
+ const interpolatedAction : any = {};
548
+ const timeRange = timestamp2 - timestamp1;
549
+
550
+ for(let i = 0; i < numActions; i++){
551
+ const key = Object.keys(action1)[i];
552
+ interpolatedAction[key] = action1[key] + (action2[key] - action1[key]) * (reqTimestamp - timestamp1) / timeRange;
553
+ }
554
+
555
+ return interpolatedAction;
556
+ }
557
+
558
+ /**
559
+ * Converts an action object to an array of numbers
560
+ * follows the same pattern as https://huggingface.co/datasets/lerobot/svla_so100_pickplace
561
+ * I am not really sure if the array can be in a different order
562
+ * but I am not going to risk it tbh 😛
563
+ *
564
+ * @param action The action object to convert
565
+ * @returns An array of numbers
566
+ */
567
+ convertActionToArray(action : any) : number[] {
568
+ return [
569
+ action["shoulder_pan"],
570
+ action["shoulder_lift"],
571
+ action["elbow_flex"],
572
+ action["wrist_flex"],
573
+ action["wrist_roll"],
574
+ action["gripper"]
575
+ ]
576
+ }
577
+
578
+ /**
579
+ * Converts an array of MotorConfig objects to an action object
580
+ * following the same joint order as convertActionToArray
581
+ *
582
+ * @param motorConfigs Array of MotorConfig objects
583
+ * @returns An action object with joint positions
584
+ */
585
+ convertMotorConfigToArray(motorConfigs: MotorConfig[]) : number[] {
586
+ // Create a map for quick lookup of motor positions by name
587
+ const motorMap: Record<string, number> = {};
588
+ for (const config of motorConfigs) {
589
+ motorMap[config.name] = config.currentPosition;
590
+ }
591
+
592
+ // Define required joint names
593
+ const requiredJoints = [
594
+ "shoulder_pan",
595
+ "shoulder_lift",
596
+ "elbow_flex",
597
+ "wrist_flex",
598
+ "wrist_roll",
599
+ "gripper"
600
+ ];
601
+
602
+ // Check that all required joints are present
603
+ const missingJoints = requiredJoints.filter(joint => motorMap[joint] === undefined);
604
+ if (missingJoints.length > 0) {
605
+ throw new Error(`Missing required joints in motor configs: ${missingJoints.join(", ")}. Available joints: ${Object.keys(motorMap).join(", ")}`);
606
+ }
607
+
608
+ // Return in the same order as convertActionToArray
609
+ return [
610
+ motorMap["shoulder_pan"],
611
+ motorMap["shoulder_lift"],
612
+ motorMap["elbow_flex"],
613
+ motorMap["wrist_flex"],
614
+ motorMap["wrist_roll"],
615
+ motorMap["gripper"]
616
+ ];
617
+ }
618
+
619
+ /**
620
+ * Finds the closest timestamp to the target timestamp
621
+ *
622
+ * the data must have timestamps in ascending order
623
+ * uses binary search to get the closest timestamp
624
+ *
625
+ * @param data The data to search through
626
+ * @param targetTimestamp The target timestamp
627
+ * @returns The closest timestamp in the data's index
628
+ */
629
+ _findClosestTimestampBefore(data : any[], targetTimestamp : number): number{
630
+ let firstIndex = 0;
631
+ let lastIndex = data.length - 1;
632
+
633
+ while (firstIndex <= lastIndex) {
634
+ const middleIndex = Math.floor((firstIndex + lastIndex) / 2);
635
+ const middleTimestamp = data[middleIndex].timestamp;
636
+
637
+ if (middleTimestamp === targetTimestamp) {
638
+ return middleIndex;
639
+ } else if (middleTimestamp < targetTimestamp) {
640
+ firstIndex = middleIndex + 1;
641
+ } else {
642
+ lastIndex = middleIndex - 1;
643
+ }
644
+ }
645
+
646
+ return lastIndex;
647
+ }
648
+
649
+ /**
650
+ * Takes non-regularly spaced lerobot-ish data and interpolates it to a regularly spaced dataset
651
+ * also adds additional
652
+ * - frame_index
653
+ * - episode_index
654
+ * - index columns
655
+ *
656
+ * to match lerobot dataset requirements
657
+ */
658
+ _interpolateAndCompleteLerobotData(fps : number, frameData : NonIndexedLeRobotDatasetRow[], lastFrameIndex : number) : LeRobotDatasetRow[]{
659
+ const interpolatedData : LeRobotDatasetRow[] = [];
660
+ const minTimestamp = frameData[0].timestamp;
661
+ const maxTimestamp = frameData[frameData.length - 1].timestamp;
662
+ const timeDifference = maxTimestamp - minTimestamp;
663
+ const numFrames = Math.floor(timeDifference * fps);
664
+ const firstFrame = frameData[0]
665
+
666
+
667
+ console.log("frames before interpolation", numFrames, frameData[0].timestamp, frameData[frameData.length - 1].timestamp, fps)
668
+
669
+ interpolatedData.push({
670
+ timestamp: firstFrame.timestamp,
671
+ action: this.convertActionToArray(firstFrame.action),
672
+ "observation.state": this.convertActionToArray(firstFrame["observation.state"]),
673
+ episode_index: firstFrame.episode_index,
674
+ task_index: firstFrame.task_index,
675
+ frame_index : 0,
676
+ index: lastFrameIndex
677
+ })
678
+
679
+ // start from 1 as the first frame is pushed already (see above)
680
+ for(let i = 1; i < numFrames; i++){
681
+ const timestamp = (i / fps)
682
+ const closestIndex = this._findClosestTimestampBefore(frameData, timestamp);
683
+ const nextIndex = closestIndex + 1;
684
+ const closestItemData = frameData[closestIndex];
685
+ const nextItemData = frameData[nextIndex];
686
+ const action = this._actionInterpolatate(closestItemData.action, nextItemData.action, closestItemData.timestamp, nextItemData.timestamp, timestamp);
687
+ const observation_state = this._actionInterpolatate(closestItemData["observation.state"], nextItemData["observation.state"], closestItemData.timestamp, nextItemData.timestamp, timestamp);
688
+
689
+ interpolatedData.push({
690
+ timestamp: timestamp,
691
+ action: this.convertActionToArray(action),
692
+ "observation.state": this.convertActionToArray(observation_state),
693
+ episode_index: closestItemData.episode_index,
694
+ task_index: closestItemData.task_index,
695
+ frame_index : i,
696
+ index: lastFrameIndex + i
697
+ });
698
+
699
+ }
700
+
701
+ return interpolatedData;
702
+ }
703
+
704
+ /**
705
+ * converts all the frames of a recording into lerobot dataset frame style
706
+ *
707
+ * NOTE : This does not interpolate the data, you are only working with raw data
708
+ * that is called by the teleop when things are actively "changing"
709
+ * @param episodeRough
710
+ */
711
+ _convertToLeRobotDataFormatFrames(episodeRough : any[]) : NonIndexedLeRobotDatasetRow[]{
712
+ const properFormatFrames : NonIndexedLeRobotDatasetRow[] = [];
713
+
714
+ const firstTimestamp = episodeRough[0].commandSentTimestamp;
715
+ for(let i=0; i<episodeRough.length; i++){
716
+ const frameRough = episodeRough[i]
717
+
718
+ properFormatFrames.push({
719
+ timestamp: frameRough.commandSentTimestamp - firstTimestamp, // so timestamps start from 0, and are in seconds
720
+ action: frameRough.previousMotorConfigsNormalized,
721
+ "observation.state": frameRough.newMotorConfigsNormalized,
722
+ episode_index: frameRough.episodeIndex,
723
+ task_index: frameRough.taskIndex
724
+ });
725
+ }
726
+
727
+ return properFormatFrames
728
+ }
729
+
730
+ /**
731
+ * Converts teleoperator data to a parquet blob
732
+ * @private
733
+ * @returns Array of objects containing parquet file content and path
734
+ */
735
+ private async _exportEpisodesToBlob(episodes: LeRobotEpisode[]): Promise<{content: Blob, path: string}[]> {
736
+ // combine all the frames
737
+ let data : NonIndexedLeRobotDatasetRow[] = [];
738
+ const episodeBlobs : any[] = []
739
+
740
+ for(let i=0; i<episodes.length; i++){
741
+ const episode = episodes[i]
742
+ data = episode.frames
743
+ const { tableFromArrays, vectorFromArray } = arrow;
744
+
745
+ const timestamps = data.map((row: any) => row.timestamp);
746
+ const actions = data.map((row: any) => row.action);
747
+ const observationStates = data.map((row: any) => row["observation.state"]);
748
+ const episodeIndexes = data.map((row: any) => row.episode_index);
749
+ const taskIndexes = data.map((row: any) => row.task_index);
750
+ const frameIndexes = data.map((row: any) => row.frame_index);
751
+ const indexes = data.map((row: any) => row.index);
752
+
753
+ const table = tableFromArrays({
754
+ timestamp: timestamps,
755
+ // @ts-ignore, this works, idk why
756
+ action: vectorFromArray(actions, new arrow.List(new arrow.Field("item", new arrow.Float32()))),
757
+ // @ts-ignore, this works, idk why
758
+ "observation.state": vectorFromArray(observationStates, new arrow.List(new arrow.Field("item", new arrow.Float32()))),
759
+ episode_index: episodeIndexes,
760
+ task_index: taskIndexes,
761
+ frame_index: frameIndexes,
762
+ index: indexes
763
+ });
764
+
765
+ const wasmUrl = "https://cdn.jsdelivr.net/npm/[email protected]/esm/parquet_wasm_bg.wasm";
766
+ const initWasm = parquet.default;
767
+ await initWasm(wasmUrl);
768
+
769
+ const wasmTable = parquet.Table.fromIPCStream(arrow.tableToIPC(table, "stream"));
770
+ const writerProperties = new parquet.WriterPropertiesBuilder()
771
+ .setCompression(parquet.Compression.UNCOMPRESSED)
772
+ .build();
773
+
774
+ const parquetUint8Array = parquet.writeParquet(wasmTable, writerProperties);
775
+ const numpadded = i.toString().padStart(6, "0")
776
+ const content = new Blob([parquetUint8Array])
777
+
778
+ episodeBlobs.push({
779
+ content, path: `data/chunk-000/episode_${numpadded}.parquet`
780
+ })
781
+
782
+ }
783
+
784
+ return episodeBlobs
785
+ }
786
+
787
+ /**
788
+ * Exports the teleoperator data in lerobot format
789
+ * @param format The format to return the data in ('json' or 'blob')
790
+ * @returns Either an array of data objects or a Uint8Array blob depending on format
791
+ */
792
+ exportEpisodes(format: 'json' | 'blob' = 'json') {
793
+ if(this._isRecording) throw new Error("This can only be called after recording has stopped!");
794
+ const data = this.episodes;
795
+
796
+ if (format === 'json') {
797
+ return data;
798
+ } else {
799
+ return this._exportEpisodesToBlob(data);
800
+ }
801
+ }
802
+
803
+ /**
804
+ * Exports the media (video) data as blobs
805
+ * @returns A dictionary of video blobs with the same keys as videoStreams
806
+ */
807
+ async exportMediaData(): Promise<{ [key: string]: Blob }> {
808
+ if(this._isRecording) throw new Error("This can only be called after recording has stopped!");
809
+ return this.videoBlobs;
810
+ }
811
+
812
+ /**
813
+ * Generates metadata for the dataset
814
+ * @returns Metadata object for the LeRobot dataset
815
+ */
816
+ async generateMetadata(data : any[]): Promise<any> {
817
+ // Calculate total episodes, frames, and tasks
818
+ let total_episodes = 0;
819
+ const total_frames = data.length;
820
+ let total_tasks = 0;
821
+
822
+ for (const row of data) {
823
+ total_episodes = Math.max(total_episodes, row.episode_index);
824
+ total_tasks = Math.max(total_tasks, row.task_index);
825
+ }
826
+
827
+ // Create video info objects for each video stream
828
+ const videos_info: VideoInfo[] = Object.keys(this.videoBlobs).map(key => {
829
+ // Default values - in a production environment, you would extract
830
+ // these from the actual video metadata using the key to identify the video source
831
+ console.log(`Generating metadata for video stream: ${key}`);
832
+ return {
833
+ height: 480,
834
+ width: 640,
835
+ channels: 3,
836
+ codec: 'h264',
837
+ pix_fmt: 'yuv420p',
838
+ is_depth_map: false,
839
+ has_audio: false
840
+ };
841
+ });
842
+
843
+ // Calculate approximate file sizes in MB
844
+ const data_files_size_in_mb = Math.round(data.length * 0.001); // Estimate
845
+
846
+ // Calculate video size by summing the sizes of all video blobs and converting to MB
847
+ let video_files_size_in_mb = 0;
848
+ for (const blob of Object.values(this.videoBlobs)) {
849
+ video_files_size_in_mb += blob.size / (1024 * 1024);
850
+ }
851
+ video_files_size_in_mb = Math.round(video_files_size_in_mb);
852
+
853
+ // Generate and return the metadata
854
+ return getMetadataInfo({
855
+ total_episodes,
856
+ total_frames,
857
+ total_tasks,
858
+ chunks_size: 1000, // Default chunk size
859
+ fps: this.fps,
860
+ splits: { "train": `0:${total_episodes}` }, // All episodes in train split
861
+ features: {}, // Additional features can be added here
862
+ videos_info,
863
+ data_files_size_in_mb,
864
+ video_files_size_in_mb
865
+ });
866
+ }
867
+
868
+ /**
869
+ * Generates statistics for the dataset
870
+ * @returns Statistics object for the LeRobot dataset
871
+ */
872
+ async getStatistics(data : any[]): Promise<any> {
873
+
874
+ // Get camera keys from the video blobs
875
+ const cameraKeys = Object.keys(this.videoBlobs);
876
+
877
+ // Generate stats using the data and camera keys
878
+ return getStats(data, cameraKeys);
879
+ }
880
+
881
+ /**
882
+ * Creates a tasks.parquet file containing task description
883
+ * @returns A Uint8Array blob containing the parquet data
884
+ */
885
+ async createTasksParquet(): Promise<Uint8Array> {
886
+ // Create a simple data structure with the task description
887
+ const tasksData = [{
888
+ task_index: 0,
889
+ __index_level_0__: this.taskDescription
890
+ }];
891
+
892
+ // Create Arrow table from the data
893
+ const taskIndexArr = arrow.vectorFromArray(tasksData.map(d => d.task_index), new arrow.Int32());
894
+ const descriptionArr = arrow.vectorFromArray(tasksData.map(d => d.__index_level_0__), new arrow.Utf8());
895
+
896
+ const table = arrow.tableFromArrays({
897
+ // @ts-ignore, this works, idk why
898
+ task_index: taskIndexArr,
899
+ // @ts-ignore, this works, idk why
900
+ __index_level_0__: descriptionArr
901
+ });
902
+
903
+ // Initialize the WASM module
904
+ const wasmUrl = "https://cdn.jsdelivr.net/npm/[email protected]/esm/parquet_wasm_bg.wasm";
905
+ const initWasm = parquet.default;
906
+ await initWasm(wasmUrl);
907
+
908
+ // Convert Arrow table to Parquet WASM table
909
+ const wasmTable = parquet.Table.fromIPCStream(arrow.tableToIPC(table, "stream"));
910
+
911
+ // Set compression properties
912
+ const writerProperties = new parquet.WriterPropertiesBuilder()
913
+ .setCompression(parquet.Compression.UNCOMPRESSED)
914
+ .build();
915
+
916
+ // Write the Parquet file
917
+ return parquet.writeParquet(wasmTable, writerProperties);
918
+ }
919
+
920
+ /**
921
+ * Creates the episodes statistics parquet file
922
+ * @returns A Uint8Array blob containing the parquet data
923
+ */
924
+ async getEpisodeStatistics(data : any[]): Promise<Uint8Array> {
925
+ const { vectorFromArray } = arrow;
926
+ const statistics = await this.getStatistics(data);
927
+
928
+ // Calculate total episodes and frames
929
+ let total_episodes = 0;
930
+
931
+ for(let row of data){
932
+ total_episodes = Math.max(total_episodes, row.episode_index)
933
+ }
934
+
935
+ total_episodes += 1; // +1 since episodes start from 0
936
+
937
+ const episodes: any[] = [];
938
+
939
+ // we'll create one row per episode
940
+ for (let episode_index = 0; episode_index < total_episodes; episode_index++) {
941
+ // Get data for this episode only
942
+ const episodeData = data.filter(row => row.episode_index === episode_index);
943
+
944
+ // Extract timestamps for this episode
945
+ const timestamps = episodeData.map(row => row.timestamp);
946
+ let min_timestamp = Infinity;
947
+ let max_timestamp = -Infinity;
948
+
949
+ for(let timestamp of timestamps){
950
+ min_timestamp = Math.min(min_timestamp, timestamp);
951
+ max_timestamp = Math.max(max_timestamp, timestamp);
952
+ }
953
+
954
+
955
+
956
+ // Camera keys from video blobs
957
+ const cameraKeys = Object.keys(this.videoBlobs);
958
+
959
+ // Create entry for this episode
960
+ const episodeEntry: any = {
961
+ // Basic episode information
962
+ episode_index: episode_index,
963
+ "data/chunk_index": 0,
964
+ "data/file_index": 0,
965
+ dataset_from_index: 0,
966
+ dataset_to_index: episodeData.length - 1,
967
+ length: episodeData.length,
968
+ tasks: [0], // Task index 0, could be extended for multiple tasks
969
+
970
+ // Meta information
971
+ "meta/episodes/chunk_index": 0,
972
+ "meta/episodes/file_index": 0,
973
+ };
974
+
975
+ // Add video information for each camera
976
+ cameraKeys.forEach(key => {
977
+ episodeEntry[`videos/observation.images.${key}/chunk_index`] = 0;
978
+ episodeEntry[`videos/observation.images.${key}/file_index`] = 0;
979
+ episodeEntry[`videos/observation.images.${key}/from_timestamp`] = min_timestamp;
980
+ episodeEntry[`videos/observation.images.${key}/to_timestamp`] = max_timestamp;
981
+ });
982
+
983
+ // Add statistics for each field
984
+ // This is a simplified approach - in a real implementation, you'd calculate
985
+ // these values for each episode individually
986
+
987
+ // Add timestamp statistics
988
+ episodeEntry["stats/timestamp/min"] = [statistics.timestamp.min];
989
+ episodeEntry["stats/timestamp/max"] = [statistics.timestamp.max];
990
+ episodeEntry["stats/timestamp/mean"] = [statistics.timestamp.mean];
991
+ episodeEntry["stats/timestamp/std"] = [statistics.timestamp.std];
992
+ episodeEntry["stats/timestamp/count"] = [statistics.timestamp.count];
993
+
994
+ // Add frame_index statistics
995
+ episodeEntry["stats/frame_index/min"] = [statistics.frame_index.min];
996
+ episodeEntry["stats/frame_index/max"] = [statistics.frame_index.max];
997
+ episodeEntry["stats/frame_index/mean"] = [statistics.frame_index.mean];
998
+ episodeEntry["stats/frame_index/std"] = [statistics.frame_index.std];
999
+ episodeEntry["stats/frame_index/count"] = [statistics.frame_index.count];
1000
+
1001
+ // Add episode_index statistics
1002
+ episodeEntry["stats/episode_index/min"] = [statistics.episode_index.min];
1003
+ episodeEntry["stats/episode_index/max"] = [statistics.episode_index.max];
1004
+ episodeEntry["stats/episode_index/mean"] = [statistics.episode_index.mean];
1005
+ episodeEntry["stats/episode_index/std"] = [statistics.episode_index.std];
1006
+ episodeEntry["stats/episode_index/count"] = [statistics.episode_index.count];
1007
+
1008
+ // Add task_index statistics
1009
+ episodeEntry["stats/task_index/min"] = [statistics.task_index.min];
1010
+ episodeEntry["stats/task_index/max"] = [statistics.task_index.max];
1011
+ episodeEntry["stats/task_index/mean"] = [statistics.task_index.mean];
1012
+ episodeEntry["stats/task_index/std"] = [statistics.task_index.std];
1013
+ episodeEntry["stats/task_index/count"] = [statistics.task_index.count];
1014
+
1015
+ // Add index statistics
1016
+ episodeEntry["stats/index/min"] = [0];
1017
+ episodeEntry["stats/index/max"] = [episodeData.length - 1];
1018
+ episodeEntry["stats/index/mean"] = [episodeData.length / 2];
1019
+ episodeEntry["stats/index/std"] = [episodeData.length / 4]; // Approximate std
1020
+ episodeEntry["stats/index/count"] = [episodeData.length];
1021
+
1022
+ // Add action statistics (placeholder)
1023
+ episodeEntry["stats/action/min"] = [0.0];
1024
+ episodeEntry["stats/action/max"] = [1.0];
1025
+ episodeEntry["stats/action/mean"] = [0.5];
1026
+ episodeEntry["stats/action/std"] = [0.2];
1027
+ episodeEntry["stats/action/count"] = [episodeData.length];
1028
+
1029
+ // Add observation.state statistics (placeholder)
1030
+ episodeEntry["stats/observation.state/min"] = [0.0];
1031
+ episodeEntry["stats/observation.state/max"] = [1.0];
1032
+ episodeEntry["stats/observation.state/mean"] = [0.5];
1033
+ episodeEntry["stats/observation.state/std"] = [0.2];
1034
+ episodeEntry["stats/observation.state/count"] = [episodeData.length];
1035
+
1036
+ // Add observation.images statistics for each camera
1037
+ cameraKeys.forEach(key => {
1038
+ // Get the image statistics from the overall statistics
1039
+ const imageStats = statistics[`observation.images.${key}`] || {
1040
+ min: [[[0.0]], [[0.0]], [[0.0]]],
1041
+ max: [[[255.0]], [[255.0]], [[255.0]]],
1042
+ mean: [[[127.5]], [[127.5]], [[127.5]]],
1043
+ std: [[[50.0]], [[50.0]], [[50.0]]],
1044
+ count: [[[episodeData.length * 3]]]
1045
+ };
1046
+
1047
+ episodeEntry[`stats/observation.images.${key}/min`] = imageStats.min;
1048
+ episodeEntry[`stats/observation.images.${key}/max`] = imageStats.max;
1049
+ episodeEntry[`stats/observation.images.${key}/mean`] = imageStats.mean;
1050
+ episodeEntry[`stats/observation.images.${key}/std`] = imageStats.std;
1051
+ episodeEntry[`stats/observation.images.${key}/count`] = imageStats.count;
1052
+ });
1053
+
1054
+ episodes.push(episodeEntry);
1055
+ }
1056
+
1057
+ // Create vector arrays for each column
1058
+ const columns: any = {};
1059
+
1060
+ // Define column names and default types
1061
+ const columnNames = [
1062
+ "episode_index", "data/chunk_index", "data/file_index", "dataset_from_index", "dataset_to_index",
1063
+ "length", "meta/episodes/chunk_index", "meta/episodes/file_index", "tasks"
1064
+ ];
1065
+
1066
+ // Add camera-specific columns
1067
+ const cameraKeys = Object.keys(this.videoBlobs);
1068
+ cameraKeys.forEach(key => {
1069
+ columnNames.push(
1070
+ `videos/observation.images.${key}/chunk_index`,
1071
+ `videos/observation.images.${key}/file_index`,
1072
+ `videos/observation.images.${key}/from_timestamp`,
1073
+ `videos/observation.images.${key}/to_timestamp`
1074
+ );
1075
+ });
1076
+
1077
+ // Add statistic columns for each field
1078
+ const statFields = ["timestamp", "frame_index", "episode_index", "task_index", "index", "action", "observation.state"];
1079
+ statFields.forEach(field => {
1080
+ columnNames.push(
1081
+ `stats/${field}/min`,
1082
+ `stats/${field}/max`,
1083
+ `stats/${field}/mean`,
1084
+ `stats/${field}/std`,
1085
+ `stats/${field}/count`
1086
+ );
1087
+ });
1088
+
1089
+ // Add image statistic columns for each camera
1090
+ cameraKeys.forEach(key => {
1091
+ columnNames.push(
1092
+ `stats/observation.images.${key}/min`,
1093
+ `stats/observation.images.${key}/max`,
1094
+ `stats/observation.images.${key}/mean`,
1095
+ `stats/observation.images.${key}/std`,
1096
+ `stats/observation.images.${key}/count`
1097
+ );
1098
+ });
1099
+
1100
+ // Create vector arrays for each column
1101
+ columnNames.forEach(columnName => {
1102
+ const values = episodes.map(ep => ep[columnName] || 0);
1103
+
1104
+ // Check if the column is an array type and needs special handling
1105
+ if (columnName.includes('stats/') || columnName === 'tasks') {
1106
+ // Handle different types of array columns based on their naming pattern
1107
+ if (columnName.includes('/count')) {
1108
+ // Bigint arrays for count fields
1109
+ // @ts-ignore
1110
+ columns[columnName] = vectorFromArray(values.map(v => Number(v)), new arrow.List(new arrow.Field("item", new arrow.Int64())));
1111
+ } else if (columnName.includes('/min') || columnName.includes('/max') ||
1112
+ columnName.includes('/mean') || columnName.includes('/std')) {
1113
+ // Double arrays for min, max, mean, std fields
1114
+ if (columnName.includes('observation.images') &&
1115
+ (columnName.includes('/min') || columnName.includes('/max') ||
1116
+ columnName.includes('/mean') || columnName.includes('/std'))) {
1117
+ // These are 3D arrays [[[value]]]
1118
+ // For 3D arrays, we need nested Lists
1119
+ // @ts-ignore
1120
+ columns[columnName] = vectorFromArray(values, new arrow.List(new arrow.Field("item",
1121
+ new arrow.List(new arrow.Field("subitem",
1122
+ new arrow.List(new arrow.Field("value", new arrow.Float64())))))));
1123
+ } else {
1124
+ // These are normal arrays [value]
1125
+ // @ts-ignore
1126
+ columns[columnName] = vectorFromArray(values, new arrow.List(new arrow.Field("item", new arrow.Float64())));
1127
+ }
1128
+ } else {
1129
+ // Default to Float64 List for other array types
1130
+ // @ts-ignore
1131
+ columns[columnName] = vectorFromArray(values, new arrow.List(new arrow.Field("item", new arrow.Float64())));
1132
+ }
1133
+ } else {
1134
+ // For non-array columns, use regular vectorFromArray
1135
+ // @ts-ignore
1136
+ columns[columnName] = vectorFromArray(values);
1137
+ }
1138
+ });
1139
+
1140
+ // Create the table with all columns
1141
+ const table = arrow.tableFromArrays(columns);
1142
+
1143
+ // Initialize the WASM module
1144
+ const wasmUrl = "https://cdn.jsdelivr.net/npm/[email protected]/esm/parquet_wasm_bg.wasm";
1145
+ const initWasm = parquet.default;
1146
+ await initWasm(wasmUrl);
1147
+
1148
+ // Convert Arrow table to Parquet WASM table
1149
+ const wasmTable = parquet.Table.fromIPCStream(arrow.tableToIPC(table, "stream"));
1150
+
1151
+ // Set compression properties
1152
+ const writerProperties = new parquet.WriterPropertiesBuilder()
1153
+ .setCompression(parquet.Compression.UNCOMPRESSED)
1154
+ .build();
1155
+
1156
+ // Write the Parquet file
1157
+ return parquet.writeParquet(wasmTable, writerProperties);
1158
+ }
1159
+
1160
+ generateREADME(metaInfo : string) {
1161
+ return generateREADME(metaInfo);
1162
+ }
1163
+
1164
+ /**
1165
+ * Creates an array of path and blob content objects for the LeRobot dataset
1166
+ *
1167
+ * @returns An array of {path, content} objects representing the dataset files
1168
+ * @private
1169
+ */
1170
+ async _exportForLeRobotBlobs() {
1171
+ const teleoperatorDataJson = await this.exportEpisodes('json') as any[];
1172
+ const parquetEpisodeDataFiles = await this._exportEpisodesToBlob(teleoperatorDataJson)
1173
+ const videoBlobs = await this.exportMediaData();
1174
+ const metadata = await this.generateMetadata(teleoperatorDataJson);
1175
+ const statistics = await this.getStatistics(teleoperatorDataJson);
1176
+ const tasksParquet = await this.createTasksParquet();
1177
+ const episodesParquet = await this.getEpisodeStatistics(teleoperatorDataJson);
1178
+ const readme = this.generateREADME(JSON.stringify(metadata));
1179
+
1180
+ // Create the blob array with proper paths
1181
+ const blobArray = [
1182
+ ...parquetEpisodeDataFiles,
1183
+ {
1184
+ path: "meta/info.json",
1185
+ content: new Blob([JSON.stringify(metadata, null, 2)], { type: "application/json" })
1186
+ },
1187
+ {
1188
+ path: "meta/stats.json",
1189
+ content: new Blob([JSON.stringify(statistics, null, 2)], { type: "application/json" })
1190
+ },
1191
+ {
1192
+ path: "meta/tasks.parquet",
1193
+ content: new Blob([tasksParquet])
1194
+ },
1195
+ {
1196
+ path: "meta/episodes/chunk-000/file-000.parquet",
1197
+ content: new Blob([episodesParquet])
1198
+ },
1199
+ {
1200
+ path: "README.md",
1201
+ content: new Blob([readme], { type: "text/markdown" })
1202
+ }
1203
+ ];
1204
+
1205
+ // Add video blobs with proper paths
1206
+ for (const [key, blob] of Object.entries(videoBlobs)) {
1207
+ blobArray.push({
1208
+ path: `videos/chunk-000/observation.images.${key}/episode_000000.mp4`,
1209
+ content: blob
1210
+ });
1211
+ }
1212
+
1213
+ return blobArray;
1214
+ }
1215
+
1216
+ /**
1217
+ * Creates a ZIP file from the dataset blobs
1218
+ *
1219
+ * @returns A Blob containing the ZIP file
1220
+ * @private
1221
+ */
1222
+ async _exportForLeRobotZip() {
1223
+ const blobArray = await this._exportForLeRobotBlobs();
1224
+ const zip = new JSZip();
1225
+
1226
+ // Add all blobs to the zip with their paths
1227
+ for (const item of blobArray) {
1228
+ // Split the path to handle directories
1229
+ const pathParts = item.path.split('/');
1230
+ const fileName = pathParts.pop() || '';
1231
+ let currentFolder = zip;
1232
+
1233
+ // Create nested folders as needed
1234
+ if (pathParts.length > 0) {
1235
+ for (const part of pathParts) {
1236
+ currentFolder = currentFolder.folder(part) || currentFolder;
1237
+ }
1238
+ }
1239
+
1240
+ // Add file to the current folder
1241
+ currentFolder.file(fileName, item.content);
1242
+ }
1243
+
1244
+ // Generate the zip file
1245
+ return await zip.generateAsync({ type: "blob" });
1246
+ }
1247
+
1248
+ /**
1249
+ * Uploads the LeRobot dataset to Hugging Face
1250
+ *
1251
+ * @param username Hugging Face username
1252
+ * @param repoName Repository name for the dataset
1253
+ * @param accessToken Hugging Face access token
1254
+ * @returns The LeRobotHFUploader instance used for upload
1255
+ */
1256
+ async _exportForLeRobotHuggingface(username: string, repoName: string, accessToken: string) {
1257
+ // Create the blobs array for upload
1258
+ const blobArray = await this._exportForLeRobotBlobs();
1259
+
1260
+ // Create the uploader
1261
+ const uploader = new LeRobotHFUploader(username, repoName);
1262
+
1263
+ // Convert blobs to File objects for HF uploader
1264
+ const files = blobArray.map(item => {
1265
+ return {
1266
+ path: item.path,
1267
+ content: item.content
1268
+ };
1269
+ });
1270
+
1271
+ // Generate a unique reference ID for tracking the upload
1272
+ const referenceId = `lerobot-upload-${Date.now()}`;
1273
+
1274
+ try {
1275
+ // Start the upload process
1276
+ uploader.createRepoAndUploadFiles(files, accessToken, referenceId);
1277
+ console.log(`Successfully uploaded dataset to ${username}/${repoName}`);
1278
+ return uploader;
1279
+ } catch (error) {
1280
+ console.error("Error uploading to Hugging Face:", error);
1281
+ throw error;
1282
+ }
1283
+ }
1284
+
1285
+ /**
1286
+ * Uploads the LeRobot dataset to Amazon S3
1287
+ *
1288
+ * @param bucketName S3 bucket name
1289
+ * @param accessKeyId AWS access key ID
1290
+ * @param secretAccessKey AWS secret access key
1291
+ * @param region AWS region (default: us-east-1)
1292
+ * @param prefix Optional prefix (folder) to upload files to within the bucket
1293
+ * @returns The LeRobotS3Uploader instance used for upload
1294
+ */
1295
+ async _exportForLeRobotS3(bucketName: string, accessKeyId: string, secretAccessKey: string, region: string = "us-east-1", prefix: string = "") {
1296
+ // Create the blobs array for upload
1297
+ const blobArray = await this._exportForLeRobotBlobs();
1298
+
1299
+ // Create the uploader
1300
+ const uploader = new LeRobotS3Uploader(bucketName, region);
1301
+
1302
+ // Convert blobs to File objects for S3 uploader
1303
+ const files = blobArray.map(item => {
1304
+ return {
1305
+ path: item.path,
1306
+ content: item.content
1307
+ };
1308
+ });
1309
+
1310
+ // Generate a unique reference ID for tracking the upload
1311
+ const referenceId = `lerobot-s3-upload-${Date.now()}`;
1312
+
1313
+ try {
1314
+ // Start the upload process
1315
+ uploader.checkBucketAndUploadFiles(files, accessKeyId, secretAccessKey, prefix, referenceId);
1316
+ console.log(`Successfully uploaded dataset to S3 bucket: ${bucketName}`);
1317
+ return uploader;
1318
+ } catch (error) {
1319
+ console.error("Error uploading to S3:", error);
1320
+ throw error;
1321
+ }
1322
+ }
1323
+
1324
+ /**
1325
+ * Exports the LeRobot dataset in various formats
1326
+ *
1327
+ * @param format The export format - 'blobs', 'zip', 'zip-download', 'huggingface', or 's3'
1328
+ * @param options Additional options for specific formats
1329
+ * @param options.username Hugging Face username (if not provided for "huggingface" format, it will use the default username)
1330
+ * @param options.repoName Hugging Face repository name (required for 'huggingface' format)
1331
+ * @param options.accessToken Hugging Face access token (required for 'huggingface' format)
1332
+ * @param options.bucketName S3 bucket name (required for 's3' format)
1333
+ * @param options.accessKeyId AWS access key ID (required for 's3' format)
1334
+ * @param options.secretAccessKey AWS secret access key (required for 's3' format)
1335
+ * @param options.region AWS region (optional for 's3' format, default: us-east-1)
1336
+ * @param options.prefix S3 prefix/folder (optional for 's3' format)
1337
+ * @returns The exported data in the requested format or the uploader instance for 'huggingface'/'s3' formats
1338
+ */
1339
+ async exportForLeRobot(format: 'blobs' | 'zip' | 'zip-download' | 'huggingface' | 's3' = 'zip-download', options?: {
1340
+ username?: string;
1341
+ repoName?: string;
1342
+ accessToken?: string;
1343
+ bucketName?: string;
1344
+ accessKeyId?: string;
1345
+ secretAccessKey?: string;
1346
+ region?: string;
1347
+ prefix?: string;
1348
+ }) {
1349
+ switch (format) {
1350
+ case 'blobs':
1351
+ return this._exportForLeRobotBlobs();
1352
+
1353
+ case 'zip':
1354
+ return this._exportForLeRobotZip();
1355
+
1356
+ case 'huggingface':
1357
+ // Validate required options for Hugging Face upload
1358
+ if (!options || !options.repoName || !options.accessToken) {
1359
+ throw new Error('Hugging Face upload requires repoName, and accessToken options');
1360
+ }
1361
+
1362
+ if (!options.username) {
1363
+ const hub = await import("@huggingface/hub");
1364
+ const {name: username} = await hub.whoAmI({accessToken: options.accessToken});
1365
+ options.username = username;
1366
+ }
1367
+
1368
+ return this._exportForLeRobotHuggingface(
1369
+ options.username,
1370
+ options.repoName,
1371
+ options.accessToken
1372
+ );
1373
+
1374
+ case 's3':
1375
+ // Validate required options for S3 upload
1376
+ if (!options || !options.bucketName || !options.accessKeyId || !options.secretAccessKey) {
1377
+ throw new Error('S3 upload requires bucketName, accessKeyId, and secretAccessKey options');
1378
+ }
1379
+
1380
+ return this._exportForLeRobotS3(
1381
+ options.bucketName,
1382
+ options.accessKeyId,
1383
+ options.secretAccessKey,
1384
+ options.region,
1385
+ options.prefix
1386
+ );
1387
+
1388
+ case 'zip-download':
1389
+ default:
1390
+ // Get the zip blob
1391
+ const zipContent = await this._exportForLeRobotZip();
1392
+
1393
+ // Create a URL for the zip file
1394
+ const url = URL.createObjectURL(zipContent);
1395
+
1396
+ // Create a download link and trigger the download
1397
+ const link = document.createElement('a');
1398
+ link.href = url;
1399
+ link.download = `lerobot_dataset_${new Date().toISOString().replace(/[:.]/g, '-')}.zip`;
1400
+ document.body.appendChild(link);
1401
+ link.click();
1402
+
1403
+ // Clean up
1404
+ setTimeout(() => {
1405
+ document.body.removeChild(link);
1406
+ URL.revokeObjectURL(url);
1407
+ }, 100);
1408
+
1409
+ return zipContent;
1410
+ }
1411
+ }
1412
+ }
packages/web/src/runpodTrainer.ts ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /**
2
+ * Class to train a model using runpod
3
+ */
4
+ class LeRobotRunpodTrainer extends EventTarget {
5
+ api_token: string;
6
+ constructor(api_token: string) {
7
+ super();
8
+ this.api_token = api_token;
9
+ }
10
+
11
+ async _startPod(api_token: string) {
12
+ const podOptions = {
13
+ allowedCudaVersions: ["12.8"],
14
+ cloudType: "SECURE",
15
+ computeType: "GPU",
16
+ containerDiskInGb: 50,
17
+ countryCodes: ["<string>"],
18
+ cpuFlavorIds: ["cpu3c"],
19
+ cpuFlavorPriority: "availability",
20
+ dataCenterIds: ["EU-RO-1","CA-MTL-1"],
21
+ dataCenterPriority: "availability",
22
+ dockerEntrypoint: [],
23
+ dockerStartCmd: [],
24
+ env: {"ENV_VAR":"value"},
25
+ globalNetworking: true,
26
+ gpuCount: 1,
27
+ gpuTypeIds: ["NVIDIA GeForce RTX 4090"],
28
+ gpuTypePriority: "availability",
29
+ imageName: "runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04",
30
+ interruptible: false,
31
+ locked: false,
32
+ minDiskBandwidthMBps: 123,
33
+ minDownloadMbps: 123,
34
+ minRAMPerGPU: 8,
35
+ minUploadMbps: 123,
36
+ minVCPUPerGPU: 2,
37
+ name: "my pod",
38
+ ports: ["8888/http","22/tcp"],
39
+ supportPublicIp: true,
40
+ templateId: null,
41
+ vcpuCount: 2,
42
+ volumeInGb: 20,
43
+ volumeMountPath: "/workspace"
44
+ }
45
+ const options = {
46
+ method: 'POST',
47
+ headers: {Authorization: 'Bearer ' + api_token, 'Content-Type': 'application/json'},
48
+ body: JSON.stringify(podOptions)
49
+ };
50
+
51
+ const response = await fetch('https://rest.runpod.io/v1/pods', options);
52
+ const data = await response.json();
53
+ return data;
54
+ }
55
+
56
+ async start() {
57
+ await this._startPod(this.api_token);
58
+ this.dispatchEvent(new CustomEvent("deployed_training_pod"));
59
+ }
60
+ }
packages/web/src/s3_uploader.ts ADDED
@@ -0,0 +1,172 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Note: To use this module, you'll need to install the AWS SDK:
2
+ // npm install @aws-sdk/client-s3 @aws-sdk/lib-storage
3
+
4
+ import { S3Client, HeadBucketCommand } from "@aws-sdk/client-s3";
5
+ import { Upload } from "@aws-sdk/lib-storage";
6
+
7
+ // Define ContentSource type locally to avoid HuggingFace dependency
8
+ type ContentSource = Blob | ArrayBuffer | Uint8Array | string;
9
+ type FileArray = Array<URL | File | { path: string; content: ContentSource }>;
10
+
11
+ /**
12
+ * Uploads a leRobot dataset to Amazon S3
13
+ */
14
+ export class LeRobotS3Uploader extends EventTarget {
15
+ private _bucketName: string;
16
+ private _region: string;
17
+ private _uploaded: boolean;
18
+ private _bucket_exists: boolean;
19
+ private _s3Client: S3Client | null;
20
+
21
+ constructor(bucketName: string, region: string = "us-east-1") {
22
+ super();
23
+ this._bucketName = bucketName;
24
+ this._region = region;
25
+ this._uploaded = false;
26
+ this._bucket_exists = false;
27
+ this._s3Client = null;
28
+ }
29
+
30
+ /**
31
+ * Returns whether the bucket has been successfully checked/created
32
+ */
33
+ get bucketExists(): boolean {
34
+ return this._bucket_exists;
35
+ }
36
+
37
+ get uploaded(): boolean {
38
+ return this._uploaded;
39
+ }
40
+
41
+ /**
42
+ * Initialize the S3 client with credentials
43
+ *
44
+ * @param accessKeyId AWS access key ID
45
+ * @param secretAccessKey AWS secret access key
46
+ */
47
+ initializeClient(accessKeyId: string, secretAccessKey: string): void {
48
+ this._s3Client = new S3Client({
49
+ region: this._region,
50
+ credentials: {
51
+ accessKeyId,
52
+ secretAccessKey
53
+ }
54
+ });
55
+ }
56
+
57
+ /**
58
+ * Checks if the bucket exists and uploads files to it
59
+ *
60
+ * @param files The files to upload
61
+ * @param accessKeyId AWS access key ID
62
+ * @param secretAccessKey AWS secret access key
63
+ * @param prefix Optional prefix (folder) to upload files to within the bucket
64
+ * @param referenceId The reference id for the upload, to track it (optional)
65
+ */
66
+ async checkBucketAndUploadFiles(
67
+ files: FileArray,
68
+ accessKeyId: string,
69
+ secretAccessKey: string,
70
+ prefix: string = "",
71
+ referenceId: string = ""
72
+ ): Promise<void> {
73
+ // Initialize the client if not already done
74
+ if (!this._s3Client) {
75
+ this.initializeClient(accessKeyId, secretAccessKey);
76
+ }
77
+
78
+ // Check if bucket exists
79
+ try {
80
+ await this._s3Client!.send(new HeadBucketCommand({ Bucket: this._bucketName }));
81
+ this._bucket_exists = true;
82
+ this.dispatchEvent(new CustomEvent("bucketExists", {
83
+ detail: { bucketName: this._bucketName }
84
+ }));
85
+ } catch (error) {
86
+ throw new Error(`Bucket ${this._bucketName} does not exist or you don't have permission to access it`);
87
+ }
88
+
89
+ // Upload files
90
+ const uploadPromises: Promise<void>[] = [];
91
+ for (const file of files) {
92
+ uploadPromises.push(this.uploadFileWithProgress([file], prefix, referenceId));
93
+ }
94
+
95
+ await Promise.all(uploadPromises);
96
+ this._uploaded = true;
97
+ }
98
+
99
+ /**
100
+ * Uploads files to S3 with progress events
101
+ *
102
+ * @param files The files to upload
103
+ * @param prefix Optional prefix (folder) to upload files to within the bucket
104
+ * @param referenceId The reference id for the upload, to track it (optional)
105
+ */
106
+ async uploadFileWithProgress(
107
+ files: FileArray,
108
+ prefix: string = "",
109
+ referenceId: string = ""
110
+ ): Promise<void> {
111
+ if (!this._s3Client) {
112
+ throw new Error("S3 client not initialized. Call initializeClient first.");
113
+ }
114
+
115
+ for (const file of files) {
116
+ let key: string;
117
+ let body: any;
118
+
119
+ if (file instanceof URL) {
120
+ const response = await fetch(file);
121
+ body = await response.blob();
122
+ key = `${prefix}${prefix ? '/' : ''}${file.pathname.split('/').pop()}`;
123
+ } else if (file instanceof File) {
124
+ body = file;
125
+ key = `${prefix}${prefix ? '/' : ''}${file.name}`;
126
+ } else {
127
+ body = file.content;
128
+ key = `${prefix}${prefix ? '/' : ''}${file.path}`;
129
+ }
130
+
131
+ const upload = new Upload({
132
+ client: this._s3Client,
133
+ params: {
134
+ Bucket: this._bucketName,
135
+ Key: key,
136
+ Body: body,
137
+ },
138
+ });
139
+
140
+ // Set up progress tracking
141
+ upload.on('httpUploadProgress', (progress) => {
142
+ this.dispatchEvent(new CustomEvent("progress", {
143
+ detail: {
144
+ progressEvent: progress,
145
+ bucketName: this._bucketName,
146
+ key,
147
+ referenceId
148
+ }
149
+ }));
150
+ });
151
+
152
+ await upload.done();
153
+ }
154
+ }
155
+
156
+ /**
157
+ * Generates a pre-signed URL for downloading a file from S3
158
+ *
159
+ * @param _key The key (path) of the file in the S3 bucket
160
+ * @param _expiresIn The number of seconds until the URL expires (default: 3600 = 1 hour)
161
+ * @returns A pre-signed URL for downloading the file
162
+ */
163
+ async generatePresignedUrl(_key: string, _expiresIn: number = 3600): Promise<string> {
164
+ if (!this._s3Client) {
165
+ throw new Error("S3 client not initialized. Call initializeClient first.");
166
+ }
167
+
168
+ // Note: This requires the @aws-sdk/s3-request-presigner package
169
+ // Implementation would go here
170
+ throw new Error("generatePresignedUrl not implemented. Requires @aws-sdk/s3-request-presigner package.");
171
+ }
172
+ }
packages/web/src/teleoperate.ts CHANGED
@@ -146,7 +146,6 @@ export async function teleoperate(
146
  config: TeleoperateConfig
147
  ): Promise<TeleoperationProcess> {
148
  const teleoperator = await createTeleoperatorProcess(config);
149
- const motorConfigs = teleoperator.motorConfigs;
150
 
151
  return {
152
  start: () => {
@@ -173,7 +172,7 @@ export async function teleoperate(
173
  },
174
  getState: () => buildTeleoperationStateFromTeleoperator(teleoperator),
175
  teleoperator,
176
- disconnect: () => teleoperator.disconnect(),
177
  };
178
  }
179
 
 
146
  config: TeleoperateConfig
147
  ): Promise<TeleoperationProcess> {
148
  const teleoperator = await createTeleoperatorProcess(config);
 
149
 
150
  return {
151
  start: () => {
 
172
  },
173
  getState: () => buildTeleoperationStateFromTeleoperator(teleoperator),
174
  teleoperator,
175
+ disconnect: () => teleoperator.disconnect()
176
  };
177
  }
178
 
packages/web/src/teleoperators/base-teleoperator.ts CHANGED
@@ -6,17 +6,63 @@
6
  import type { MotorConfig } from "../types/teleoperation.js";
7
  import type { MotorCommunicationPort } from "../utils/motor-communication.js";
8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9
  /**
10
  * Base interface that all Web teleoperators must implement
11
  */
12
- export interface WebTeleoperator {
13
- initialize(): Promise<void>;
14
- start(): void;
15
- stop(): void;
16
- disconnect(): Promise<void>;
17
- getState(): TeleoperatorSpecificState;
18
- onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void;
19
- motorConfigs: MotorConfig[];
 
 
 
 
 
 
 
 
20
  }
21
 
22
  /**
@@ -31,14 +77,26 @@ export type TeleoperatorSpecificState = {
31
  /**
32
  * Base abstract class with common functionality for all teleoperators
33
  */
34
- export abstract class BaseWebTeleoperator implements WebTeleoperator {
35
  protected port: MotorCommunicationPort;
36
  public motorConfigs: MotorConfig[] = [];
37
  protected isActive: boolean = false;
 
 
 
 
38
 
39
  constructor(port: MotorCommunicationPort, motorConfigs: MotorConfig[]) {
 
40
  this.port = port;
41
  this.motorConfigs = motorConfigs;
 
 
 
 
 
 
 
42
  }
43
 
44
  abstract initialize(): Promise<void>;
@@ -53,10 +111,130 @@ export abstract class BaseWebTeleoperator implements WebTeleoperator {
53
  }
54
  }
55
 
 
 
 
 
 
 
 
 
 
 
 
56
  onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void {
57
  this.motorConfigs = motorConfigs;
58
  }
59
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
60
  get isActiveTeleoperator(): boolean {
61
  return this.isActive;
62
  }
 
6
  import type { MotorConfig } from "../types/teleoperation.js";
7
  import type { MotorCommunicationPort } from "../utils/motor-communication.js";
8
 
9
+ /**
10
+ * Normalizes a value from one range to another
11
+ * @param value The value to normalize
12
+ * @param minVal The minimum value of the original range
13
+ * @param maxVal The maximum value of the original range
14
+ * @param minNorm The minimum value of the normalized range
15
+ * @param maxNorm The maximum value of the normalized range
16
+ * @returns The normalized value
17
+ */
18
+ function normalizeValue(value: number, minVal: number, maxVal: number, minNorm: number, maxNorm: number): number {
19
+ const range = maxVal - minVal;
20
+ const normRange = maxNorm - minNorm;
21
+ const normalized = (value - minVal) / range;
22
+ return normalized * normRange + minNorm;
23
+ }
24
+
25
+ /**
26
+ * Type definition for state update callback parameters
27
+ */
28
+ interface StateUpdateCallbackParams {
29
+ previousMotorConfigs: MotorConfig[];
30
+ newMotorConfigs: MotorConfig[];
31
+ previousMotorConfigsNormalized: MotorConfig[];
32
+ newMotorConfigsNormalized: MotorConfig[];
33
+ commandSentTimestamp: number;
34
+ positionChangedTimestamp: number;
35
+ }
36
+
37
+ /**
38
+ * Type definition for state update callback function
39
+ */
40
+ type StateUpdateCallback = (params: StateUpdateCallbackParams) => void;
41
+
42
+ /**
43
+ * Type definition for array of state update callbacks
44
+ */
45
+ type StateUpdateCallbackArray = Array<StateUpdateCallback>;
46
+
47
  /**
48
  * Base interface that all Web teleoperators must implement
49
  */
50
+ export abstract class WebTeleoperator {
51
+ protected onStateUpdateCallbacks: StateUpdateCallbackArray = [];
52
+ public motorConfigs: MotorConfig[] = [];
53
+ abstract initialize(): Promise<void>;
54
+ abstract start(): void;
55
+ abstract stop(): void;
56
+
57
+ abstract recordingTaskIndex : number;
58
+ abstract recordingEpisodeIndex : number;
59
+ abstract setEpisodeIndex(index: number): void;
60
+ abstract setTaskIndex(index: number): void;
61
+
62
+ abstract disconnect(): Promise<void>;
63
+ abstract getState(): TeleoperatorSpecificState;
64
+ abstract onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void;
65
+ abstract addOnStateUpdateCallback(fn : StateUpdateCallback): void;
66
  }
67
 
68
  /**
 
77
  /**
78
  * Base abstract class with common functionality for all teleoperators
79
  */
80
+ export abstract class BaseWebTeleoperator extends WebTeleoperator {
81
  protected port: MotorCommunicationPort;
82
  public motorConfigs: MotorConfig[] = [];
83
  protected isActive: boolean = false;
84
+ public isRecording: boolean = false;
85
+ public recordingTaskIndex : number;
86
+ public recordingEpisodeIndex : number;
87
+ public recordedMotorPositionEpisodes : any[];
88
 
89
  constructor(port: MotorCommunicationPort, motorConfigs: MotorConfig[]) {
90
+ super();
91
  this.port = port;
92
  this.motorConfigs = motorConfigs;
93
+
94
+ // store episode positions
95
+ this.recordedMotorPositionEpisodes = []
96
+
97
+ this.isRecording = false;
98
+ this.recordingTaskIndex = 0;
99
+ this.recordingEpisodeIndex = 0;
100
  }
101
 
102
  abstract initialize(): Promise<void>;
 
111
  }
112
  }
113
 
114
+ setEpisodeIndex(index: number): void {
115
+ this.recordingEpisodeIndex = index;
116
+
117
+ // create a new empty array at that position on the array
118
+ this.recordedMotorPositionEpisodes[this.recordingEpisodeIndex] = []
119
+ }
120
+
121
+ setTaskIndex(index: number): void {
122
+ this.recordingTaskIndex = index;
123
+ }
124
+
125
  onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void {
126
  this.motorConfigs = motorConfigs;
127
  }
128
 
129
+ normalizeMotorConfigPosition(motorConfig: MotorConfig){
130
+ let minNormPosition;
131
+ let maxNormPosition;
132
+
133
+ /**
134
+ * This follows the guide at https://github.com/huggingface/lerobot/blob/cf86b9300dc83fdad408cfe4787b7b09b55f12cf/src/lerobot/robots/so100_follower/so100_follower.py#L49
135
+ * Meaning, for everything except gripper, it normalizes the positions to between -100 and 100
136
+ * and for gripper it normalizes between 0 - 100
137
+ */
138
+ if(["shoulder_pan", "shoulder_lift", "elbow_flex", "wrist_flex", "wrist_roll"].includes(motorConfig.name)) {
139
+ minNormPosition = -100;
140
+ maxNormPosition = 100;
141
+ } else {
142
+ minNormPosition = 0;
143
+ maxNormPosition = 100;
144
+ }
145
+
146
+ return normalizeValue(motorConfig.currentPosition, motorConfig.minPosition, motorConfig.maxPosition, minNormPosition, maxNormPosition)
147
+ }
148
+
149
+ /**
150
+ * Normalize an entire list of motor configurations
151
+ *
152
+ * This follows the guide at https://github.com/huggingface/lerobot/blob/cf86b9300dc83fdad408cfe4787b7b09b55f12cf/src/lerobot/robots/so100_follower/so100_follower.py#L49
153
+ * Meaning, for everything except gripper, it normalizes the positions to between -100 and 100
154
+ * and for gripper it normalizes between 0 - 100
155
+ */
156
+ normalizeMotorConfigs(motorConfigs : MotorConfig[]) : MotorConfig[] {
157
+ // Create a deep copy of the motor configs
158
+ const normalizedConfigs = JSON.parse(JSON.stringify(motorConfigs)) as MotorConfig[];
159
+
160
+ // Normalize the current position values
161
+ for(let i = 0; i < normalizedConfigs.length; i++) {
162
+ const config = normalizedConfigs[i];
163
+
164
+ if(config.name === "gripper") {
165
+ config.currentPosition = normalizeValue(motorConfigs[i].currentPosition, motorConfigs[i].minPosition, motorConfigs[i].maxPosition, 0, 100);
166
+ } else {
167
+ config.currentPosition = normalizeValue(motorConfigs[i].currentPosition, motorConfigs[i].minPosition, motorConfigs[i].maxPosition, -100, 100);
168
+ }
169
+
170
+ // Also normalize min/max positions for consistency
171
+ if(config.name === "gripper") {
172
+ config.minPosition = 0;
173
+ config.maxPosition = 100;
174
+ } else {
175
+ config.minPosition = -100;
176
+ config.maxPosition = 100;
177
+ }
178
+ }
179
+
180
+ return normalizedConfigs;
181
+ }
182
+
183
+ /**
184
+ * Dispatches a motor position changed event
185
+ * Gets the motor positions, normalized
186
+ *
187
+ * This follows the guide at https://github.com/huggingface/lerobot/blob/cf86b9300dc83fdad408cfe4787b7b09b55f12cf/src/lerobot/robots/so100_follower/so100_follower.py#L49
188
+ * Meaning, for everything except gripper, it normalizes the positions to between -100 and 100
189
+ * and for gripper it normalizes between 0 - 100
190
+ *
191
+ * @param motorName The name of the motor that changed
192
+ * @param motorConfig The motor configuration
193
+ * @param previousPosition The previous position of the motor
194
+ * @param currentPosition The current position of the motor
195
+ * @param timestamp The timestamp of the event
196
+ */
197
+ dispatchMotorPositionChanged(prevMotorConfigs: MotorConfig[], newMotorConfigs: MotorConfig[], commandSentTimestamp: number, positionChangedTimestamp: number): void {
198
+ // Call all registered state update callbacks
199
+ const callbackParams: StateUpdateCallbackParams = {
200
+ previousMotorConfigs: prevMotorConfigs,
201
+ newMotorConfigs: newMotorConfigs,
202
+ previousMotorConfigsNormalized: this.normalizeMotorConfigs(prevMotorConfigs),
203
+ newMotorConfigsNormalized: this.normalizeMotorConfigs(newMotorConfigs),
204
+ commandSentTimestamp: commandSentTimestamp,
205
+ positionChangedTimestamp: positionChangedTimestamp
206
+ };
207
+
208
+ // call all the onStateUpdateCallbacks
209
+ this.onStateUpdateCallbacks.forEach(callback => callback(callbackParams));
210
+
211
+ // if recording, store the changes
212
+ if(this.isRecording) {
213
+ const data = {
214
+ previousMotorConfigs : prevMotorConfigs,
215
+ newMotorConfigs,
216
+ previousMotorConfigsNormalized : this.normalizeMotorConfigs(prevMotorConfigs),
217
+ newMotorConfigsNormalized : this.normalizeMotorConfigs(newMotorConfigs),
218
+ commandSentTimestamp,
219
+ positionChangedTimestamp,
220
+ episodeIndex: this.recordingEpisodeIndex,
221
+ taskIndex: this.recordingTaskIndex,
222
+ }
223
+
224
+ const episodes = this.recordedMotorPositionEpisodes[this.recordingEpisodeIndex]
225
+ episodes.push(data)
226
+ }
227
+ }
228
+
229
+ /**
230
+ * Adds an onstateupdate callback
231
+ * to return a response for the user with motor config states and timestamps
232
+ * @param fn Callback function that receives state update parameters
233
+ */
234
+ addOnStateUpdateCallback(fn: StateUpdateCallback): void {
235
+ this.onStateUpdateCallbacks.push(fn);
236
+ }
237
+
238
  get isActiveTeleoperator(): boolean {
239
  return this.isActive;
240
  }
packages/web/src/teleoperators/direct-teleoperator.ts CHANGED
@@ -62,6 +62,7 @@ export class DirectTeleoperator extends BaseWebTeleoperator {
62
  * Move motor to exact position
63
  */
64
  async moveMotor(motorName: string, targetPosition: number): Promise<boolean> {
 
65
  const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
66
  if (!motorConfig) return false;
67
 
@@ -69,6 +70,7 @@ export class DirectTeleoperator extends BaseWebTeleoperator {
69
  motorConfig.minPosition,
70
  Math.min(motorConfig.maxPosition, targetPosition)
71
  );
 
72
 
73
  try {
74
  await writeMotorPosition(
@@ -77,12 +79,14 @@ export class DirectTeleoperator extends BaseWebTeleoperator {
77
  Math.round(clampedPosition)
78
  );
79
  motorConfig.currentPosition = clampedPosition;
 
80
 
81
  // Notify UI of position change
82
  if (this.onStateUpdate) {
83
  this.onStateUpdate(this.buildTeleoperationState());
84
  }
85
 
 
86
  return true;
87
  } catch (error) {
88
  console.warn(`Failed to move motor ${motorName}:`, error);
@@ -96,11 +100,16 @@ export class DirectTeleoperator extends BaseWebTeleoperator {
96
  async setMotorPositions(positions: {
97
  [motorName: string]: number;
98
  }): Promise<boolean> {
 
 
99
  const results = await Promise.all(
100
  Object.entries(positions).map(([motorName, position]) =>
101
  this.moveMotor(motorName, position)
102
  )
103
  );
 
 
 
104
 
105
  return results.every((result) => result);
106
  }
 
62
  * Move motor to exact position
63
  */
64
  async moveMotor(motorName: string, targetPosition: number): Promise<boolean> {
65
+ const commandSentTimestamp = performance.now()/1000;
66
  const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
67
  if (!motorConfig) return false;
68
 
 
70
  motorConfig.minPosition,
71
  Math.min(motorConfig.maxPosition, targetPosition)
72
  );
73
+ const prevMotorConfigs = structuredClone(this.motorConfigs)
74
 
75
  try {
76
  await writeMotorPosition(
 
79
  Math.round(clampedPosition)
80
  );
81
  motorConfig.currentPosition = clampedPosition;
82
+ const positionChangedTimestamp = performance.now()/1000;
83
 
84
  // Notify UI of position change
85
  if (this.onStateUpdate) {
86
  this.onStateUpdate(this.buildTeleoperationState());
87
  }
88
 
89
+ this.dispatchMotorPositionChanged(prevMotorConfigs,this.motorConfigs, commandSentTimestamp, positionChangedTimestamp);
90
  return true;
91
  } catch (error) {
92
  console.warn(`Failed to move motor ${motorName}:`, error);
 
100
  async setMotorPositions(positions: {
101
  [motorName: string]: number;
102
  }): Promise<boolean> {
103
+ const commandSentTimestamp = performance.now()/1000;
104
+ const prevMotorConfigs = structuredClone(this.motorConfigs)
105
  const results = await Promise.all(
106
  Object.entries(positions).map(([motorName, position]) =>
107
  this.moveMotor(motorName, position)
108
  )
109
  );
110
+ const positionChangedTimestamp = performance.now()/1000;
111
+
112
+ this.dispatchMotorPositionChanged(prevMotorConfigs,this.motorConfigs, commandSentTimestamp, positionChangedTimestamp);
113
 
114
  return results.every((result) => result);
115
  }
packages/web/src/teleoperators/keyboard-teleoperator.ts CHANGED
@@ -171,7 +171,7 @@ export class KeyboardTeleoperator extends BaseWebTeleoperator {
171
  *
172
  * Keep it simple - this works!
173
  */
174
- private updateMotorPositions(): void {
175
  const now = Date.now();
176
 
177
  // Clear timed-out keys
@@ -219,25 +219,44 @@ export class KeyboardTeleoperator extends BaseWebTeleoperator {
219
  );
220
  }
221
 
 
 
 
 
222
  // Send motor commands and update positions
223
- Object.entries(targetPositions).forEach(([motorName, targetPosition]) => {
224
  const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
225
- if (motorConfig && targetPosition !== motorConfig.currentPosition) {
226
- writeMotorPosition(
227
- this.port,
228
- motorConfig.id,
229
- Math.round(targetPosition)
230
- )
231
- .then(() => {
232
- motorConfig.currentPosition = targetPosition;
233
- })
234
- .catch((error) => {
235
- console.warn(
236
- `Failed to write motor ${motorConfig.id} position:`,
237
- error
238
- );
239
- });
240
  }
241
- });
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
242
  }
243
  }
 
171
  *
172
  * Keep it simple - this works!
173
  */
174
+ private async updateMotorPositions(): Promise<void> {
175
  const now = Date.now();
176
 
177
  // Clear timed-out keys
 
219
  );
220
  }
221
 
222
+ const prevMotorConfigs = structuredClone(this.motorConfigs);
223
+ const commandSentTimestamp = performance.now()/1000;
224
+
225
+
226
  // Send motor commands and update positions
227
+ for (const [motorName, targetPosition] of Object.entries(targetPositions)) {
228
  const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
229
+ if (motorConfig && targetPosition !== motorConfig.currentPosition) {
230
+ try {
231
+ await writeMotorPosition(
232
+ this.port,
233
+ motorConfig.id,
234
+ Math.round(targetPosition)
235
+ );
236
+
237
+ motorConfig.currentPosition = targetPosition;
238
+ } catch (error) {
239
+ console.warn(
240
+ `Failed to write motor ${motorConfig.id} position:`,
241
+ error
242
+ );
243
+ }
244
  }
245
+ }
246
+
247
+
248
+ const positionChangedTimestamp = performance.now()/1000;
249
+
250
+ // Dispatch event for motor position change if something has changed
251
+ const somethingChanged = Object.entries(targetPositions).length > 0;
252
+
253
+ if(somethingChanged){
254
+ this.dispatchMotorPositionChanged(
255
+ prevMotorConfigs,
256
+ this.motorConfigs,
257
+ commandSentTimestamp,
258
+ positionChangedTimestamp
259
+ );
260
+ }
261
  }
262
  }
packages/web/src/threejs_display.ts ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //import * as THREE from 'three';
2
+
3
+
4
+ /**
5
+ * Module to display a 3D model of a robot, easily via threejs
6
+ */
7
+ export class LeRobotThreejsDisplay {
8
+ //scene: THREE.Scene;
9
+ constructor() {
10
+ //this.scene = new THREE.Scene();
11
+ }
12
+ }
packages/web/src/utils/record/generateREADME.ts ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ function generateREADME(metaInfo : string) {
2
+ return `\
3
+ ---
4
+ task_categories:
5
+ - robotics
6
+ tags:
7
+ - LeRobot
8
+ - tutorial
9
+ configs:
10
+ - config_name: default
11
+ data_files: data/*/*.parquet
12
+ ---
13
+
14
+ This dataset was created using [LeRobot.js](https://github.com/timpietrusky/lerobot.js) which is based on the [LeRobot](https://github.com/huggingface/lerobot) project
15
+
16
+ ## Dataset Description
17
+
18
+
19
+
20
+ - **Homepage:** [More Information Needed]
21
+ - **Paper:** [More Information Needed]
22
+ - **License:** apache-2.0
23
+
24
+ ## Dataset Structure
25
+
26
+ [meta/info.json](meta/info.json):
27
+ ${metaInfo}
28
+ `
29
+ }
30
+
31
+ export default generateREADME;
packages/web/src/utils/record/metadataInfo.ts ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ export interface VideoInfo {
2
+ height: number;
3
+ width: number;
4
+ channels: number;
5
+ codec: string;
6
+ pix_fmt: string;
7
+ is_depth_map: boolean;
8
+ has_audio: boolean;
9
+ }
10
+
11
+ /**
12
+ * Metadata parameters interface
13
+ */
14
+ interface MetadataParams {
15
+ total_episodes: number;
16
+ total_frames: number;
17
+ total_tasks: number;
18
+ chunks_size: number;
19
+ fps: number;
20
+ splits: { [key: string]: string };
21
+ features: { [key: string]: any };
22
+ videos_info: VideoInfo[];
23
+ data_files_size_in_mb: number;
24
+ video_files_size_in_mb: number;
25
+ }
26
+
27
+ /**
28
+ * Generates and returns a metadata information dictionary
29
+ * Needs some named parameters passed as parameters
30
+ */
31
+ function getMetadataInfo(params: MetadataParams) {
32
+ return {
33
+ "codebase_version": "v2.1",
34
+ "robot_type": "so100",
35
+ "total_episodes": params.total_episodes,
36
+ "total_frames": params.total_frames,
37
+ "total_tasks": params.total_tasks,
38
+ "total_videos": params.videos_info.length,
39
+ "total_chunks": 1,
40
+ "chunks_size": params.chunks_size,
41
+ "fps": params.fps,
42
+ "splits": {
43
+ "train": `0:${params.total_episodes}`
44
+ },
45
+ "data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
46
+ "video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
47
+ "features": {
48
+ "action": {
49
+ "dtype": "float32",
50
+ "shape": [
51
+ 6
52
+ ],
53
+ "names": [
54
+ "main_shoulder_pan",
55
+ "main_shoulder_lift",
56
+ "main_elbow_flex",
57
+ "main_wrist_flex",
58
+ "main_wrist_roll",
59
+ "main_gripper"
60
+ ],
61
+ "fps": params.fps
62
+ },
63
+ "observation.state": {
64
+ "dtype": "float32",
65
+ "shape": [
66
+ 6
67
+ ],
68
+ "names": [
69
+ "main_shoulder_pan",
70
+ "main_shoulder_lift",
71
+ "main_elbow_flex",
72
+ "main_wrist_flex",
73
+ "main_wrist_roll",
74
+ "main_gripper"
75
+ ],
76
+ "fps": params.fps
77
+ },
78
+ "observation.images.front": {
79
+ "dtype": "video",
80
+ "shape": [
81
+ 480,
82
+ 640,
83
+ 3
84
+ ],
85
+ "names": [
86
+ "height",
87
+ "width",
88
+ "channels"
89
+ ],
90
+ "info": {
91
+ "video.fps": params.fps,
92
+ "video.height": 480,
93
+ "video.width": 640,
94
+ "video.channels": 3,
95
+ "video.codec": "av1",
96
+ "video.pix_fmt": "yuv420p",
97
+ "video.is_depth_map": false,
98
+ "has_audio": false
99
+ }
100
+ },
101
+ "timestamp": {
102
+ "dtype": "float32",
103
+ "shape": [
104
+ 1
105
+ ],
106
+ "names": null,
107
+ "fps": params.fps
108
+ },
109
+ "frame_index": {
110
+ "dtype": "int64",
111
+ "shape": [
112
+ 1
113
+ ],
114
+ "names": null,
115
+ "fps": params.fps
116
+ },
117
+ "episode_index": {
118
+ "dtype": "int64",
119
+ "shape": [
120
+ 1
121
+ ],
122
+ "names": null,
123
+ "fps": params.fps
124
+ },
125
+ "index": {
126
+ "dtype": "int64",
127
+ "shape": [
128
+ 1
129
+ ],
130
+ "names": null,
131
+ "fps": params.fps
132
+ },
133
+ "task_index": {
134
+ "dtype": "int64",
135
+ "shape": [
136
+ 1
137
+ ],
138
+ "names": null,
139
+ "fps": params.fps
140
+ }
141
+ },
142
+ "data_files_size_in_mb": 100,
143
+ "video_files_size_in_mb": 500
144
+ }
145
+ }
146
+
147
+ export function getVideoInfo(width: number, height: number): VideoInfo {
148
+ return {
149
+ height,
150
+ width,
151
+ channels: 3,
152
+ codec: "h264",
153
+ pix_fmt: "yuv420p",
154
+ is_depth_map: false,
155
+ has_audio: false
156
+ };
157
+ }
158
+
159
+ export default getMetadataInfo;
packages/web/src/utils/record/stats.ts ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /**
2
+ * Calculates basic statistics (min, max, mean, std, count) for a numeric array
3
+ */
4
+ function calculateStats(values: number[]): { min: number[], max: number[], mean: number[], std: number[], count: number[] } {
5
+ const count = values.length;
6
+ if (count === 0) {
7
+ return {
8
+ min: [0],
9
+ max: [0],
10
+ mean: [0],
11
+ std: [0],
12
+ count: [0]
13
+ };
14
+ }
15
+
16
+ let min = Infinity;
17
+ let max = -Infinity;
18
+
19
+ for(let value of values){
20
+ min = Math.min(min, value);
21
+ max = Math.max(max, value);
22
+ }
23
+
24
+ const sum = values.reduce((acc, val) => acc + val, 0);
25
+ const mean = sum / count;
26
+
27
+ // Calculate standard deviation
28
+ const squareDiffs = values.map(value => Math.pow(value - mean, 2));
29
+ const avgSquareDiff = squareDiffs.reduce((acc, val) => acc + val, 0) / count;
30
+ const std = Math.sqrt(avgSquareDiff);
31
+
32
+ return {
33
+ min: [min],
34
+ max: [max],
35
+ mean: [mean],
36
+ std: [std],
37
+ count: [count]
38
+ };
39
+ }
40
+
41
+ /**
42
+ * Generates statistics for a dataset
43
+ * @param data The dataset to analyze
44
+ * @param cameraKeys Array of camera keys for dynamic observation.images entries
45
+ */
46
+ export function getStats(data: any[], cameraKeys: string[] = []): any {
47
+ // Extract timestamp and episode_index values
48
+ const timestamps = data.map(item => item.timestamp);
49
+ const episodeIndices = data.map(item => item.episode_index);
50
+
51
+ // Extract other common fields if they exist
52
+ const frameIndices = data.map(item => item.frame_index || 0);
53
+ const taskIndices = data.map(item => item.task_index || 0);
54
+
55
+ const stats: any = {
56
+ // Standard fields
57
+ "timestamp": calculateStats(timestamps),
58
+ "episode_index": calculateStats(episodeIndices),
59
+ "frame_index": calculateStats(frameIndices),
60
+ "task_index": calculateStats(taskIndices),
61
+ };
62
+
63
+ // Add observation.images entries for each camera key
64
+ cameraKeys.forEach(key => {
65
+ // In a real implementation, you would calculate actual stats from video data
66
+ // Since we don't have actual video frame data to analyze, we'll use placeholder values
67
+ stats[`observation.images.${key}`] = {
68
+ "min": [[[0.0]], [[0.0]], [[0.0]]], // R,G,B channels min
69
+ "max": [[[255.0]], [[255.0]], [[255.0]]], // R,G,B channels max
70
+ "mean": [[[127.5]], [[127.5]], [[127.5]]], // R,G,B channels mean
71
+ "std": [[[50.0]], [[50.0]], [[50.0]]], // R,G,B channels std
72
+ "count": [[[data.length * 3]]] // Number of pixels × 3 channels
73
+ };
74
+ });
75
+
76
+ return stats;
77
+ }
78
+
79
+ export default getStats;
packages/web/tests/record.test.ts ADDED
@@ -0,0 +1,208 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import { describe, it, expect, beforeEach } from "vitest";
2
+ import { vi } from "vitest";
3
+ import { LeRobotDatasetRecorder } from "../src/record";
4
+ import { WebTeleoperator } from "../../web/src/teleoperators/base-teleoperator";
5
+
6
+ // Mock the WebTeleoperator class
7
+ vi.mock("../../web/src/teleoperators/base-teleoperator", () => {
8
+ return {
9
+ WebTeleoperator: vi.fn().mockImplementation(() => {
10
+ return {
11
+ startRecording: vi.fn(),
12
+ stopRecording: vi.fn().mockResolvedValue([]),
13
+ clearRecording: vi.fn()
14
+ };
15
+ })
16
+ };
17
+ });
18
+
19
+ describe("LeRobotDatasetRecorder", () => {
20
+ let recorder: LeRobotDatasetRecorder;
21
+
22
+ beforeEach(() => {
23
+ // Create a new recorder instance before each test
24
+ // @ts-ignore
25
+ const mockTeleoperator = new WebTeleoperator() as unknown as WebTeleoperator;
26
+ const mockVideoStreams = {};
27
+ recorder = new LeRobotDatasetRecorder([mockTeleoperator], mockVideoStreams, 30);
28
+ });
29
+
30
+ describe("_interpolateAndCompleteLerobotData", () => {
31
+ it("should interpolate data to match the specified fps", async () => {
32
+ // Create test data with non-regular timestamps
33
+ const roughData = [
34
+ {
35
+ timestamp: 0,
36
+ action: {
37
+ shoulder_pan: 0,
38
+ shoulder_lift: 0,
39
+ elbow_flex: 0,
40
+ wrist_flex: 0,
41
+ wrist_roll: 0,
42
+ gripper: 0
43
+ },
44
+ "observation.state": {
45
+ shoulder_pan: 0,
46
+ shoulder_lift: 0,
47
+ elbow_flex: 0,
48
+ wrist_flex: 0,
49
+ wrist_roll: 0,
50
+ gripper: 0
51
+ },
52
+ episode_index: 0,
53
+ task_index: 0
54
+ },
55
+ {
56
+ timestamp: 0.5,
57
+ action: {
58
+ shoulder_pan: 10,
59
+ shoulder_lift: 20,
60
+ elbow_flex: 30,
61
+ wrist_flex: 40,
62
+ wrist_roll: 50,
63
+ gripper: 60
64
+ },
65
+ "observation.state": {
66
+ shoulder_pan: 15,
67
+ shoulder_lift: 25,
68
+ elbow_flex: 35,
69
+ wrist_flex: 45,
70
+ wrist_roll: 55,
71
+ gripper: 65
72
+ },
73
+ episode_index: 0,
74
+ task_index: 0
75
+ },
76
+ {
77
+ timestamp: 1.0,
78
+ action: {
79
+ shoulder_pan: 20,
80
+ shoulder_lift: 40,
81
+ elbow_flex: 60,
82
+ wrist_flex: 80,
83
+ wrist_roll: 100,
84
+ gripper: 120
85
+ },
86
+ "observation.state": {
87
+ shoulder_pan: 30,
88
+ shoulder_lift: 50,
89
+ elbow_flex: 70,
90
+ wrist_flex: 90,
91
+ wrist_roll: 110,
92
+ gripper: 130
93
+ },
94
+ episode_index: 1, // New episode
95
+ task_index: 0
96
+ },
97
+ {
98
+ timestamp: 1.5,
99
+ action: {
100
+ shoulder_pan: 25,
101
+ shoulder_lift: 50,
102
+ elbow_flex: 75,
103
+ wrist_flex: 100,
104
+ wrist_roll: 125,
105
+ gripper: 150
106
+ },
107
+ "observation.state": {
108
+ shoulder_pan: 35,
109
+ shoulder_lift: 55,
110
+ elbow_flex: 75,
111
+ wrist_flex: 95,
112
+ wrist_roll: 115,
113
+ gripper: 135
114
+ },
115
+ episode_index: 1, // New episode
116
+ task_index: 0
117
+ }
118
+ ];
119
+
120
+ // Set the FPS to 10 for this test
121
+ const fps = 10;
122
+
123
+ // Call the method under test
124
+ const result = await recorder._interpolateAndCompleteLerobotData(fps, roughData);
125
+
126
+ // log all the results, row by row
127
+ for (let i = 0; i < result.length; i++) {
128
+ console.log(result[i]);
129
+ }
130
+
131
+ // Verify the results
132
+ expect(result).toBeInstanceOf(Array);
133
+ expect(result.length).toBe(15); // 1.5 seconds at 10 fps = 15 frames
134
+
135
+ // Check the first frame
136
+ expect(result[0].timestamp).toBeCloseTo(0, 5);
137
+ expect(result[0].action).toEqual([0, 0, 0, 0, 0, 0]);
138
+ expect(result[0]["observation.state"]).toEqual([0, 0, 0, 0, 0, 0]);
139
+ expect(result[0].episode_index).toBe(0);
140
+ expect(result[0].task_index).toBe(0);
141
+ expect(result[0].frame_index).toBe(0);
142
+ expect(result[0].index).toBe(0);
143
+
144
+ // Check a middle frame (0.3 seconds)
145
+ const middleFrame = result[3];
146
+ expect(middleFrame.timestamp).toBeCloseTo(0.3, 5);
147
+ // At 0.3 seconds, we're 60% between 0 and 0.5 seconds
148
+ // So action.shoulder_pan should be 60% of 10 = 6
149
+ expect(middleFrame.action[0]).toBeCloseTo(6, 5);
150
+ expect(middleFrame.episode_index).toBe(0);
151
+ expect(middleFrame.frame_index).toBe(3);
152
+ expect(middleFrame.index).toBe(3);
153
+
154
+ // Check the frame right after the episode change
155
+ const episodeChangeFrame = result[5]; // 0.5 seconds
156
+ expect(episodeChangeFrame.timestamp).toBeCloseTo(0.5, 5);
157
+ expect(episodeChangeFrame.action[0]).toBeCloseTo(10, 5);
158
+ expect(episodeChangeFrame.episode_index).toBe(0);
159
+ expect(episodeChangeFrame.frame_index).toBe(5);
160
+
161
+ // Check the last frame before 1 second
162
+ const lastFrame = result[9]; // 0.9 seconds
163
+ expect(lastFrame.timestamp).toBeCloseTo(0.9, 5);
164
+ expect(lastFrame.episode_index).toBe(0);
165
+ expect(lastFrame.frame_index).toBe(9); // Frame index continues incrementing
166
+ expect(lastFrame.index).toBe(9);
167
+ });
168
+
169
+ it("should handle episode index changes correctly", async () => {
170
+ // Create test data with episode changes
171
+ const roughData = [
172
+ { timestamp: 0.0, action: { shoulder_pan: 0 }, "observation.state": { shoulder_pan: 0 }, episode_index: 0, task_index: 0 },
173
+ { timestamp: 0.3, action: { shoulder_pan: 30 }, "observation.state": { shoulder_pan: 30 }, episode_index: 0, task_index: 0 },
174
+ { timestamp: 0.5, action: { shoulder_pan: 50 }, "observation.state": { shoulder_pan: 50 }, episode_index: 1, task_index: 0 }, // Episode change
175
+ { timestamp: 0.8, action: { shoulder_pan: 80 }, "observation.state": { shoulder_pan: 80 }, episode_index: 1, task_index: 0 },
176
+ { timestamp: 1.0, action: { shoulder_pan: 100 }, "observation.state": { shoulder_pan: 100 }, episode_index: 2, task_index: 0 } // Another episode change
177
+ ];
178
+
179
+ const fps = 10;
180
+ const result = await recorder._interpolateAndCompleteLerobotData(fps, roughData);
181
+
182
+ // Check frame indices reset after episode changes
183
+ expect(result[0].episode_index).toBe(0);
184
+ expect(result[0].frame_index).toBe(0);
185
+
186
+ expect(result[5].episode_index).toBe(1); // After episode change
187
+ expect(result[5].frame_index).toBe(0); // Should reset to 0
188
+
189
+ expect(result[9].episode_index).toBe(1); // After second episode change
190
+ expect(result[9].frame_index).toBe(4); // Frame index continues incrementing
191
+ });
192
+
193
+ it("should handle empty or minimal data", async () => {
194
+ // Test with minimal data (just two points)
195
+ const minimalData = [
196
+ { timestamp: 0.0, action: { shoulder_pan: 0 }, "observation.state": { shoulder_pan: 0 }, episode_index: 0, task_index: 0 },
197
+ { timestamp: 0.1, action: { shoulder_pan: 10 }, "observation.state": { shoulder_pan: 10 }, episode_index: 0, task_index: 0 }
198
+ ];
199
+
200
+ const fps = 10;
201
+ const result = await recorder._interpolateAndCompleteLerobotData(fps, minimalData);
202
+
203
+ expect(result.length).toBe(1); // 0.1 seconds at 10fps = 1 frame
204
+ expect(result[0].timestamp).toBeCloseTo(0, 5);
205
+ expect(result[0].action[0]).toBeCloseTo(0, 5);
206
+ });
207
+ });
208
+ });
pnpm-lock.yaml CHANGED
The diff for this file is too large to render. See raw diff