Spaces:
Running
feat: added dataset recording; hf uploader, s3 uploader; runpod trainer (#6)
Browse files* Added recording options
- Added dataset recorder in record.ts
- Added dataset record show example in recorder.tsx
- Added huggingface upload options
- Added zip download options
- Added S3 upload options
* fixed huggingface upload
* fixed video recording
* feat: added react-router-dom to be able to navigate
* feat: show the recorder all the time; make camera config nicer
* - Added threejs display changes
- fixed storage to work episode wise
* feat(recorder): persist camera; better configuration
* docs: how to refactor the pr to make record fit into the lib
* fixed teleoperator, added frame wise tracking in ui
* fixed multi teleoperator recording, added statistical measurement
* refactored code to move recording into record.ts
* updated README to reflect recorder and export functions
* fixed merge errors
* removed 3d rendering
---------
Co-authored-by: Tim Pietrusky <[email protected]>
- docs/planning/007_record.md +483 -0
- examples/cyberpunk-standalone/package.json +10 -0
- examples/cyberpunk-standalone/pnpm-lock.yaml +0 -0
- examples/cyberpunk-standalone/src/App.tsx +248 -112
- examples/cyberpunk-standalone/src/components/recorder.tsx +1318 -0
- examples/cyberpunk-standalone/src/components/teleoperation-view.tsx +349 -301
- examples/cyberpunk-standalone/src/components/teleoperator-episodes-view.tsx +69 -0
- examples/cyberpunk-standalone/src/components/teleoperator-frames-view.tsx +78 -0
- examples/cyberpunk-standalone/src/components/teleoperator-joint-graph.tsx +137 -0
- examples/cyberpunk-standalone/src/main.tsx +11 -8
- package.json +12 -4
- packages/web/README.md +69 -0
- packages/web/package.json +13 -4
- packages/web/pnpm-lock.yaml +0 -0
- packages/web/src/calibrate.ts +0 -1
- packages/web/src/hf_uploader.ts +86 -0
- packages/web/src/index.ts +4 -0
- packages/web/src/record.ts +1412 -0
- packages/web/src/runpodTrainer.ts +60 -0
- packages/web/src/s3_uploader.ts +172 -0
- packages/web/src/teleoperate.ts +1 -2
- packages/web/src/teleoperators/base-teleoperator.ts +187 -9
- packages/web/src/teleoperators/direct-teleoperator.ts +9 -0
- packages/web/src/teleoperators/keyboard-teleoperator.ts +37 -18
- packages/web/src/threejs_display.ts +12 -0
- packages/web/src/utils/record/generateREADME.ts +31 -0
- packages/web/src/utils/record/metadataInfo.ts +159 -0
- packages/web/src/utils/record/stats.ts +79 -0
- packages/web/tests/record.test.ts +208 -0
- pnpm-lock.yaml +0 -0
@@ -0,0 +1,483 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# User Story 007: Clean Record API Implementation
|
2 |
+
|
3 |
+
## Story
|
4 |
+
|
5 |
+
**As a** robotics developer building teleoperation recording systems
|
6 |
+
**I want** to record robot motor positions and control data using a clean `record()` function API
|
7 |
+
**So that** I can capture teleoperation sessions for training AI models, analysis, and replay without dealing with complex class-based APIs or mixed concerns
|
8 |
+
|
9 |
+
## Background
|
10 |
+
|
11 |
+
A community contributor has provided a recording implementation in this PR branch, which includes a comprehensive `LeRobotDatasetRecorder` class with video recording, data export, and LeRobot dataset format support. However, the current implementation violates several of our core conventions and doesn't match the clean API patterns established by `calibrate()`, `teleoperate()`, and `findPort()`.
|
12 |
+
|
13 |
+
### Current Implementation Problems
|
14 |
+
|
15 |
+
The existing `LeRobotDatasetRecorder` implementation has several architectural issues:
|
16 |
+
|
17 |
+
- **Missing Standard Library Pattern**: Uses class instantiation instead of simple function call like our other APIs
|
18 |
+
- **Library vs Demo Separation Violation**: Mixes hardware recording (library concern) with video streams, export formats, and UI (demo concerns)
|
19 |
+
- **Teleoperator Integration Issues**: Recording logic deeply embedded in `BaseWebTeleoperator` with complex state management
|
20 |
+
- **Complex Constructor Anti-Pattern**: Requires pre-configured teleoperators and video streams, violating our "direct library usage" principle
|
21 |
+
- **Export API Complexity**: ZIP, HuggingFace, and S3 upload belong in demo code, not standard library
|
22 |
+
- **No Clean Process API**: Doesn't follow our consistent `start()/stop()/result` pattern
|
23 |
+
- **Redundant Event System**: Uses `dispatchMotorPositionChanged` events that aren't consumed and duplicate callback functionality
|
24 |
+
- **Artificial Polling**: 100ms polling in teleoperate instead of immediate callbacks when motors change
|
25 |
+
|
26 |
+
### Convention Alignment Needed
|
27 |
+
|
28 |
+
Our established patterns from `calibrate()`, `teleoperate()`, and `findPort()` follow these principles:
|
29 |
+
|
30 |
+
- **Simple Function API**: `const process = await record(config)`
|
31 |
+
- **Clean Process Objects**: Consistent `start()`, `stop()`, `getState()`, `result` interface
|
32 |
+
- **Hardware-Only Library**: Standard library handles only robotics hardware, not UI/storage/export
|
33 |
+
- **Demo Handles UI**: Examples handle video, export formats, browser storage, file downloads
|
34 |
+
- **Immediate Callbacks**: Real-time updates via callbacks, not polling or unused events
|
35 |
+
- **Direct Usage**: End users call library functions directly without complex setup
|
36 |
+
|
37 |
+
## Acceptance Criteria
|
38 |
+
|
39 |
+
### Core Functionality
|
40 |
+
|
41 |
+
- [ ] **Standard Library API**: Clean `record(config)` function matching our established patterns
|
42 |
+
- [ ] **Process Object Interface**: Consistent `RecordProcess` with `start()`, `stop()`, `getState()`, `result` methods
|
43 |
+
- [ ] **Hardware-Only Recording**: Library captures only robot motor positions and teleoperation data
|
44 |
+
- [ ] **Real-Time Callbacks**: Immediate `onDataUpdate` and `onStateUpdate` callbacks, no polling
|
45 |
+
- [ ] **Device-Agnostic**: Works with any robot type through configuration, not hardcoded values
|
46 |
+
- [ ] **Clean Teleoperator Integration**: Recording subscribes to teleoperation changes without embedding in teleoperator classes
|
47 |
+
|
48 |
+
### User Experience
|
49 |
+
|
50 |
+
- [ ] **Simple Integration**: Easy to add recording to existing teleoperation workflows
|
51 |
+
- [ ] **Consistent API**: Same patterns as `calibrate()` and `teleoperate()` for familiar developer experience
|
52 |
+
- [ ] **Immediate Feedback**: Real-time recording state and data updates for responsive UI
|
53 |
+
- [ ] **Error Handling**: Clear error messages for recording failures or invalid configurations
|
54 |
+
- [ ] **Resource Management**: Proper cleanup of recording resources on stop/disconnect
|
55 |
+
|
56 |
+
### Technical Requirements
|
57 |
+
|
58 |
+
- [ ] **Library/Demo Separation**: Move video, export, and storage logic to examples/demo layer
|
59 |
+
- [ ] **Remove Event System**: Eliminate unused `dispatchMotorPositionChanged` events, use callbacks only
|
60 |
+
- [ ] **Extract from Teleoperators**: Remove recording state and logic from `BaseWebTeleoperator`
|
61 |
+
- [ ] **TypeScript**: Fully typed with proper interfaces for recording configuration and data
|
62 |
+
- [ ] **No Code Duplication**: Reuse existing teleoperation and motor communication infrastructure
|
63 |
+
- [ ] **Performance**: Immediate callbacks when data changes, no unnecessary polling
|
64 |
+
|
65 |
+
## Expected User Flow
|
66 |
+
|
67 |
+
### Basic Robot Recording
|
68 |
+
|
69 |
+
```typescript
|
70 |
+
import { record } from "@lerobot/web";
|
71 |
+
|
72 |
+
// Clean API matching our conventions
|
73 |
+
const recordProcess = await record({
|
74 |
+
robot: connectedRobot,
|
75 |
+
options: {
|
76 |
+
fps: 30,
|
77 |
+
onDataUpdate: (data) => {
|
78 |
+
// Real-time recording data for UI feedback
|
79 |
+
console.log(`Recorded ${data.frameCount} frames`);
|
80 |
+
updateRecordingUI(data);
|
81 |
+
},
|
82 |
+
onStateUpdate: (state) => {
|
83 |
+
// Recording state changes
|
84 |
+
console.log(`Recording: ${state.isActive}`);
|
85 |
+
updateRecordingStatus(state);
|
86 |
+
},
|
87 |
+
},
|
88 |
+
});
|
89 |
+
|
90 |
+
// Consistent process interface
|
91 |
+
recordProcess.start();
|
92 |
+
|
93 |
+
// Recording runs automatically while teleoperation is active
|
94 |
+
setTimeout(() => {
|
95 |
+
recordProcess.stop();
|
96 |
+
}, 30000);
|
97 |
+
|
98 |
+
// Get pure robot recording data
|
99 |
+
const robotData = await recordProcess.result;
|
100 |
+
console.log("Episodes:", robotData.episodes);
|
101 |
+
console.log("Metadata:", robotData.metadata);
|
102 |
+
```
|
103 |
+
|
104 |
+
### Recording with Teleoperation
|
105 |
+
|
106 |
+
```typescript
|
107 |
+
import { teleoperate, record } from "@lerobot/web";
|
108 |
+
|
109 |
+
// Start teleoperation
|
110 |
+
const teleoperationProcess = await teleoperate({
|
111 |
+
robot: connectedRobot,
|
112 |
+
teleop: { type: "keyboard" },
|
113 |
+
calibrationData: calibrationData,
|
114 |
+
onStateUpdate: (state) => {
|
115 |
+
updateTeleoperationUI(state);
|
116 |
+
},
|
117 |
+
});
|
118 |
+
|
119 |
+
// Add recording to existing teleoperation
|
120 |
+
const recordProcess = await record({
|
121 |
+
robot: connectedRobot,
|
122 |
+
options: {
|
123 |
+
onDataUpdate: (data) => {
|
124 |
+
console.log(`Recording frame ${data.frameCount}`);
|
125 |
+
},
|
126 |
+
},
|
127 |
+
});
|
128 |
+
|
129 |
+
// Both run independently
|
130 |
+
teleoperationProcess.start();
|
131 |
+
recordProcess.start();
|
132 |
+
|
133 |
+
// Control independently
|
134 |
+
setTimeout(() => {
|
135 |
+
recordProcess.stop(); // Stop recording, keep teleoperation
|
136 |
+
}, 60000);
|
137 |
+
|
138 |
+
setTimeout(() => {
|
139 |
+
teleoperationProcess.stop(); // Stop teleoperation
|
140 |
+
}, 120000);
|
141 |
+
```
|
142 |
+
|
143 |
+
### Demo-Layer Dataset Export
|
144 |
+
|
145 |
+
```typescript
|
146 |
+
// In examples/demo - NOT in standard library
|
147 |
+
import { record } from "@lerobot/web";
|
148 |
+
import { DatasetExporter } from "./dataset-exporter"; // Demo code
|
149 |
+
|
150 |
+
const recordProcess = await record({ robot, options });
|
151 |
+
recordProcess.start();
|
152 |
+
|
153 |
+
// ... recording session ...
|
154 |
+
|
155 |
+
recordProcess.stop();
|
156 |
+
const robotData = await recordProcess.result;
|
157 |
+
|
158 |
+
// Demo handles complex export logic
|
159 |
+
const exporter = new DatasetExporter({
|
160 |
+
robotData,
|
161 |
+
videoStreams: cameraStreams, // Demo manages video
|
162 |
+
taskDescription: "Pick and place task",
|
163 |
+
});
|
164 |
+
|
165 |
+
// Export options handled by demo
|
166 |
+
await exporter.downloadZip();
|
167 |
+
await exporter.uploadToHuggingFace({ apiKey, repoName });
|
168 |
+
await exporter.uploadToS3({ credentials });
|
169 |
+
```
|
170 |
+
|
171 |
+
### Component Integration
|
172 |
+
|
173 |
+
```typescript
|
174 |
+
// React component - direct library usage like calibration
|
175 |
+
const [recordingState, setRecordingState] = useState<RecordingState>();
|
176 |
+
const [recordingData, setRecordingData] = useState<RecordingData>();
|
177 |
+
const recordProcessRef = useRef<RecordProcess | null>(null);
|
178 |
+
|
179 |
+
useEffect(() => {
|
180 |
+
const initRecording = async () => {
|
181 |
+
const process = await record({
|
182 |
+
robot,
|
183 |
+
options: {
|
184 |
+
onStateUpdate: setRecordingState,
|
185 |
+
onDataUpdate: setRecordingData,
|
186 |
+
},
|
187 |
+
});
|
188 |
+
recordProcessRef.current = process;
|
189 |
+
};
|
190 |
+
initRecording();
|
191 |
+
}, [robot]);
|
192 |
+
|
193 |
+
const handleStartRecording = () => {
|
194 |
+
recordProcessRef.current?.start();
|
195 |
+
};
|
196 |
+
|
197 |
+
const handleStopRecording = async () => {
|
198 |
+
recordProcessRef.current?.stop();
|
199 |
+
const data = await recordProcessRef.current?.result;
|
200 |
+
// Handle recorded data
|
201 |
+
};
|
202 |
+
```
|
203 |
+
|
204 |
+
## Implementation Details
|
205 |
+
|
206 |
+
### File Structure Refactoring
|
207 |
+
|
208 |
+
```
|
209 |
+
packages/web/src/
|
210 |
+
├── record.ts # NEW: Clean record() function
|
211 |
+
├── types/
|
212 |
+
│ └── recording.ts # NEW: Recording-specific types
|
213 |
+
├── utils/
|
214 |
+
│ └── recording-manager.ts # NEW: Internal recording logic
|
215 |
+
├── teleoperators/
|
216 |
+
│ └── base-teleoperator.ts # UPDATED: Remove recording logic
|
217 |
+
└── [MOVED TO EXAMPLES]
|
218 |
+
├── LeRobotDatasetRecorder.ts # Complex export logic
|
219 |
+
├── dataset-exporter.ts # Video + export functionality
|
220 |
+
└── upload-handlers.ts # HuggingFace, S3 upload logic
|
221 |
+
```
|
222 |
+
|
223 |
+
### Key Dependencies
|
224 |
+
|
225 |
+
#### No New Dependencies for Standard Library
|
226 |
+
|
227 |
+
- **Existing**: Reuse all current dependencies (motor communication, teleoperation integration)
|
228 |
+
- **Architecture Only**: Pure refactoring to clean up existing functionality
|
229 |
+
|
230 |
+
#### Demo Dependencies (Moved)
|
231 |
+
|
232 |
+
- **Video/Export**: `parquet-wasm`, `apache-arrow`, `jszip` - moved to examples
|
233 |
+
- **Upload**: `@huggingface/hub`, AWS SDK - moved to examples
|
234 |
+
|
235 |
+
### Core Functions to Implement
|
236 |
+
|
237 |
+
#### Clean Record API
|
238 |
+
|
239 |
+
```typescript
|
240 |
+
// record.ts - New clean API
|
241 |
+
interface RecordConfig {
|
242 |
+
robot: RobotConnection;
|
243 |
+
options?: {
|
244 |
+
fps?: number; // Default: 30
|
245 |
+
onDataUpdate?: (data: RecordingData) => void;
|
246 |
+
onStateUpdate?: (state: RecordingState) => void;
|
247 |
+
};
|
248 |
+
}
|
249 |
+
|
250 |
+
interface RecordProcess {
|
251 |
+
start(): void;
|
252 |
+
stop(): void;
|
253 |
+
getState(): RecordingState;
|
254 |
+
result: Promise<RobotRecordingData>;
|
255 |
+
}
|
256 |
+
|
257 |
+
interface RecordingState {
|
258 |
+
isActive: boolean;
|
259 |
+
frameCount: number;
|
260 |
+
episodeCount: number;
|
261 |
+
duration: number; // milliseconds
|
262 |
+
lastUpdate: number;
|
263 |
+
}
|
264 |
+
|
265 |
+
interface RecordingData {
|
266 |
+
frameCount: number;
|
267 |
+
currentEpisode: number;
|
268 |
+
recentFrames: MotorPositionFrame[]; // Last few frames for UI
|
269 |
+
}
|
270 |
+
|
271 |
+
interface RobotRecordingData {
|
272 |
+
episodes: MotorPositionFrame[][]; // Pure motor data only
|
273 |
+
metadata: {
|
274 |
+
fps: number;
|
275 |
+
robotType: string;
|
276 |
+
startTime: number;
|
277 |
+
endTime: number;
|
278 |
+
totalFrames: number;
|
279 |
+
totalEpisodes: number;
|
280 |
+
};
|
281 |
+
}
|
282 |
+
|
283 |
+
// Main function - matches our conventions
|
284 |
+
export async function record(config: RecordConfig): Promise<RecordProcess>;
|
285 |
+
```
|
286 |
+
|
287 |
+
#### Recording Manager (Internal)
|
288 |
+
|
289 |
+
```typescript
|
290 |
+
// utils/recording-manager.ts - Internal implementation
|
291 |
+
class RecordingManager {
|
292 |
+
private robot: RobotConnection;
|
293 |
+
private isActive: boolean = false;
|
294 |
+
private episodes: MotorPositionFrame[][] = [];
|
295 |
+
private currentEpisode: MotorPositionFrame[] = [];
|
296 |
+
private startTime: number = 0;
|
297 |
+
private frameCount: number = 0;
|
298 |
+
|
299 |
+
constructor(
|
300 |
+
robot: RobotConnection,
|
301 |
+
private options: RecordOptions,
|
302 |
+
private onDataUpdate?: (data: RecordingData) => void,
|
303 |
+
private onStateUpdate?: (state: RecordingState) => void
|
304 |
+
) {
|
305 |
+
this.robot = robot;
|
306 |
+
}
|
307 |
+
|
308 |
+
start(): void {
|
309 |
+
if (this.isActive) return;
|
310 |
+
|
311 |
+
this.isActive = true;
|
312 |
+
this.startTime = Date.now();
|
313 |
+
|
314 |
+
// Subscribe to teleoperation changes (NO events, just callbacks)
|
315 |
+
this.subscribeToRobotChanges();
|
316 |
+
|
317 |
+
this.notifyStateUpdate();
|
318 |
+
}
|
319 |
+
|
320 |
+
stop(): void {
|
321 |
+
if (!this.isActive) return;
|
322 |
+
|
323 |
+
this.isActive = false;
|
324 |
+
this.finishCurrentEpisode();
|
325 |
+
this.unsubscribeFromRobotChanges();
|
326 |
+
|
327 |
+
this.notifyStateUpdate();
|
328 |
+
}
|
329 |
+
|
330 |
+
private subscribeToRobotChanges(): void {
|
331 |
+
// Listen to existing teleoperation callbacks - no new events needed
|
332 |
+
// This integrates with the existing onStateUpdate mechanism
|
333 |
+
}
|
334 |
+
|
335 |
+
private recordFrame(motorConfigs: MotorConfig[]): void {
|
336 |
+
const frame: MotorPositionFrame = {
|
337 |
+
timestamp: Date.now() - this.startTime,
|
338 |
+
motorPositions: motorConfigs.map((config) => ({
|
339 |
+
id: config.id,
|
340 |
+
name: config.name,
|
341 |
+
position: config.currentPosition,
|
342 |
+
})),
|
343 |
+
frameIndex: this.frameCount++,
|
344 |
+
};
|
345 |
+
|
346 |
+
this.currentEpisode.push(frame);
|
347 |
+
|
348 |
+
if (this.onDataUpdate) {
|
349 |
+
this.onDataUpdate({
|
350 |
+
frameCount: this.frameCount,
|
351 |
+
currentEpisode: this.episodes.length,
|
352 |
+
recentFrames: this.currentEpisode.slice(-10), // Last 10 frames
|
353 |
+
});
|
354 |
+
}
|
355 |
+
}
|
356 |
+
|
357 |
+
getState(): RecordingState {
|
358 |
+
return {
|
359 |
+
isActive: this.isActive,
|
360 |
+
frameCount: this.frameCount,
|
361 |
+
episodeCount: this.episodes.length,
|
362 |
+
duration: this.isActive ? Date.now() - this.startTime : 0,
|
363 |
+
lastUpdate: Date.now(),
|
364 |
+
};
|
365 |
+
}
|
366 |
+
|
367 |
+
async getResult(): Promise<RobotRecordingData> {
|
368 |
+
return {
|
369 |
+
episodes: [...this.episodes],
|
370 |
+
metadata: {
|
371 |
+
fps: this.options.fps || 30,
|
372 |
+
robotType: this.robot.robotType || "unknown",
|
373 |
+
startTime: this.startTime,
|
374 |
+
endTime: Date.now(),
|
375 |
+
totalFrames: this.frameCount,
|
376 |
+
totalEpisodes: this.episodes.length,
|
377 |
+
},
|
378 |
+
};
|
379 |
+
}
|
380 |
+
}
|
381 |
+
```
|
382 |
+
|
383 |
+
#### Updated Teleoperate Integration
|
384 |
+
|
385 |
+
```typescript
|
386 |
+
// teleoperate.ts - Remove 100ms polling, add immediate callbacks
|
387 |
+
export async function teleoperate(
|
388 |
+
config: TeleoperateConfig
|
389 |
+
): Promise<TeleoperationProcess> {
|
390 |
+
const teleoperator = await createTeleoperatorProcess(config);
|
391 |
+
|
392 |
+
return {
|
393 |
+
start: () => {
|
394 |
+
teleoperator.start();
|
395 |
+
|
396 |
+
// NO MORE 100ms polling! Use immediate callbacks
|
397 |
+
if (config.onStateUpdate) {
|
398 |
+
teleoperator.setStateUpdateCallback(config.onStateUpdate);
|
399 |
+
}
|
400 |
+
},
|
401 |
+
// ... rest of interface
|
402 |
+
};
|
403 |
+
}
|
404 |
+
```
|
405 |
+
|
406 |
+
#### Clean Teleoperator Base
|
407 |
+
|
408 |
+
```typescript
|
409 |
+
// teleoperators/base-teleoperator.ts - Remove recording logic
|
410 |
+
export abstract class BaseWebTeleoperator extends WebTeleoperator {
|
411 |
+
protected port: MotorCommunicationPort;
|
412 |
+
public motorConfigs: MotorConfig[] = [];
|
413 |
+
protected isActive: boolean = false;
|
414 |
+
|
415 |
+
// REMOVED: All recording-related properties
|
416 |
+
// REMOVED: dispatchMotorPositionChanged events
|
417 |
+
// REMOVED: recordedMotorPositions, episodeIndex, etc.
|
418 |
+
|
419 |
+
private stateUpdateCallback?: (state: TeleoperationState) => void;
|
420 |
+
|
421 |
+
setStateUpdateCallback(callback: (state: TeleoperationState) => void): void {
|
422 |
+
this.stateUpdateCallback = callback;
|
423 |
+
}
|
424 |
+
|
425 |
+
protected motorPositionsChanged(): void {
|
426 |
+
// Call immediately when motors change - no events, no 100ms delay
|
427 |
+
if (this.stateUpdateCallback) {
|
428 |
+
const state = this.buildCurrentState();
|
429 |
+
this.stateUpdateCallback(state);
|
430 |
+
}
|
431 |
+
}
|
432 |
+
|
433 |
+
// Clean implementation without recording concerns
|
434 |
+
}
|
435 |
+
```
|
436 |
+
|
437 |
+
### Technical Considerations
|
438 |
+
|
439 |
+
#### Migration Strategy
|
440 |
+
|
441 |
+
**Preserve Existing Functionality:**
|
442 |
+
|
443 |
+
1. **Move Complex Logic**: `LeRobotDatasetRecorder` moves to `examples/` as demo code
|
444 |
+
2. **Extract Clean Core**: Create new `record()` function for standard library
|
445 |
+
3. **Update Examples**: Cyberpunk demo uses new API with demo-layer export functionality
|
446 |
+
4. **Remove Event System**: Clean up unused `dispatchMotorPositionChanged` events
|
447 |
+
5. **Fix Polling**: Replace 100ms polling with immediate callbacks
|
448 |
+
|
449 |
+
#### Performance Improvements
|
450 |
+
|
451 |
+
- **Remove Polling**: Eliminate artificial 100ms delays in favor of immediate callbacks
|
452 |
+
- **Event-Driven**: Only fire callbacks when robot state actually changes
|
453 |
+
- **Memory Efficiency**: No unused event listeners or redundant data structures
|
454 |
+
- **Responsive UI**: Immediate feedback for recording status and data updates
|
455 |
+
|
456 |
+
#### Future Extensibility
|
457 |
+
|
458 |
+
The clean architecture supports advanced recording features as demo enhancements:
|
459 |
+
|
460 |
+
```typescript
|
461 |
+
// Future: Advanced demo features (NOT in standard library)
|
462 |
+
class AdvancedDatasetExporter extends DatasetExporter {
|
463 |
+
// Video synchronization, multi-camera support
|
464 |
+
// Cloud storage, data preprocessing
|
465 |
+
// Visualization, playback, analysis tools
|
466 |
+
}
|
467 |
+
```
|
468 |
+
|
469 |
+
## Definition of Done
|
470 |
+
|
471 |
+
- [ ] **Clean Record API**: `record(config)` function implemented matching our established patterns
|
472 |
+
- [ ] **Process Interface**: `RecordProcess` with consistent `start()`, `stop()`, `getState()`, `result` methods
|
473 |
+
- [ ] **Hardware-Only Library**: Standard library captures only robot motor data, no video/export complexity
|
474 |
+
- [ ] **Demo Separation**: Video recording, export formats, and UI logic moved to examples layer
|
475 |
+
- [ ] **Remove Events**: `dispatchMotorPositionChanged` events eliminated, callbacks used exclusively
|
476 |
+
- [ ] **Fix Polling**: 100ms artificial polling replaced with immediate callbacks when motors change
|
477 |
+
- [ ] **Clean Teleoperators**: Recording logic extracted from `BaseWebTeleoperator` and teleoperator classes
|
478 |
+
- [ ] **TypeScript Coverage**: Full type safety with proper interfaces for all recording functionality
|
479 |
+
- [ ] **Performance**: Immediate, event-driven updates with no unnecessary polling or unused listeners
|
480 |
+
- [ ] **Integration**: Easy integration with existing teleoperation workflows using familiar patterns
|
481 |
+
- [ ] **Example Updates**: Cyberpunk demo updated to use new clean API with demo-layer export features
|
482 |
+
- [ ] **No Regression**: All existing recording functionality preserved through demo layer
|
483 |
+
- [ ] **Documentation**: Clear examples showing standard library vs demo separation
|
@@ -10,7 +10,10 @@
|
|
10 |
"preview": "vite preview"
|
11 |
},
|
12 |
"dependencies": {
|
|
|
|
|
13 |
"@hookform/resolvers": "^3.9.1",
|
|
|
14 |
"@lerobot/web": "file:../../packages/web",
|
15 |
"@radix-ui/react-accordion": "1.2.2",
|
16 |
"@radix-ui/react-alert-dialog": "1.1.4",
|
@@ -39,7 +42,11 @@
|
|
39 |
"@radix-ui/react-toggle": "1.1.1",
|
40 |
"@radix-ui/react-toggle-group": "1.1.1",
|
41 |
"@radix-ui/react-tooltip": "1.1.6",
|
|
|
|
|
|
|
42 |
"@types/react-syntax-highlighter": "^15.5.13",
|
|
|
43 |
"autoprefixer": "^10.4.20",
|
44 |
"class-variance-authority": "^0.7.1",
|
45 |
"clsx": "^2.1.1",
|
@@ -49,16 +56,19 @@
|
|
49 |
"input-otp": "1.4.1",
|
50 |
"lucide-react": "^0.454.0",
|
51 |
"next-themes": "^0.4.6",
|
|
|
52 |
"react": "^19",
|
53 |
"react-day-picker": "8.10.1",
|
54 |
"react-dom": "^19",
|
55 |
"react-hook-form": "^7.54.1",
|
56 |
"react-resizable-panels": "^2.1.7",
|
|
|
57 |
"react-syntax-highlighter": "^15.6.1",
|
58 |
"recharts": "2.15.0",
|
59 |
"sonner": "^1.7.1",
|
60 |
"tailwind-merge": "^2.5.5",
|
61 |
"tailwindcss-animate": "^1.0.7",
|
|
|
62 |
"vaul": "^0.9.6",
|
63 |
"zod": "^3.24.1"
|
64 |
},
|
|
|
10 |
"preview": "vite preview"
|
11 |
},
|
12 |
"dependencies": {
|
13 |
+
"@aws-sdk/client-s3": "^3.850.0",
|
14 |
+
"@aws-sdk/lib-storage": "^3.850.0",
|
15 |
"@hookform/resolvers": "^3.9.1",
|
16 |
+
"@huggingface/hub": "^2.4.0",
|
17 |
"@lerobot/web": "file:../../packages/web",
|
18 |
"@radix-ui/react-accordion": "1.2.2",
|
19 |
"@radix-ui/react-alert-dialog": "1.1.4",
|
|
|
42 |
"@radix-ui/react-toggle": "1.1.1",
|
43 |
"@radix-ui/react-toggle-group": "1.1.1",
|
44 |
"@radix-ui/react-tooltip": "1.1.6",
|
45 |
+
"@react-three/cannon": "^6.6.0",
|
46 |
+
"@react-three/drei": "^10.6.0",
|
47 |
+
"@react-three/fiber": "^9.2.0",
|
48 |
"@types/react-syntax-highlighter": "^15.5.13",
|
49 |
+
"apache-arrow": "^21.0.0",
|
50 |
"autoprefixer": "^10.4.20",
|
51 |
"class-variance-authority": "^0.7.1",
|
52 |
"clsx": "^2.1.1",
|
|
|
56 |
"input-otp": "1.4.1",
|
57 |
"lucide-react": "^0.454.0",
|
58 |
"next-themes": "^0.4.6",
|
59 |
+
"parquet-wasm": "^0.6.1",
|
60 |
"react": "^19",
|
61 |
"react-day-picker": "8.10.1",
|
62 |
"react-dom": "^19",
|
63 |
"react-hook-form": "^7.54.1",
|
64 |
"react-resizable-panels": "^2.1.7",
|
65 |
+
"react-router-dom": "^7.7.1",
|
66 |
"react-syntax-highlighter": "^15.6.1",
|
67 |
"recharts": "2.15.0",
|
68 |
"sonner": "^1.7.1",
|
69 |
"tailwind-merge": "^2.5.5",
|
70 |
"tailwindcss-animate": "^1.0.7",
|
71 |
+
"three": "^0.178.0",
|
72 |
"vaul": "^0.9.6",
|
73 |
"zod": "^3.24.1"
|
74 |
},
|
The diff for this file is too large to render.
See raw diff
|
|
@@ -1,4 +1,11 @@
|
|
1 |
import { useState, useEffect, useRef } from "react";
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2 |
import { ChevronLeft } from "lucide-react";
|
3 |
import { Header } from "@/components/header";
|
4 |
import { Footer } from "@/components/footer";
|
@@ -27,19 +34,15 @@ import {
|
|
27 |
} from "@/lib/unified-storage";
|
28 |
|
29 |
function App() {
|
30 |
-
const [view, setView] = useState<
|
31 |
-
"dashboard" | "calibrating" | "teleoperating"
|
32 |
-
>("dashboard");
|
33 |
const [robots, setRobots] = useState<RobotConnection[]>([]);
|
34 |
-
const [selectedRobot, setSelectedRobot] = useState<RobotConnection | null>(
|
35 |
-
null
|
36 |
-
);
|
37 |
const [editingRobot, setEditingRobot] = useState<RobotConnection | null>(
|
38 |
null
|
39 |
);
|
40 |
const [isConnecting, setIsConnecting] = useState(false);
|
41 |
const hardwareSectionRef = useRef<HTMLDivElement>(null);
|
42 |
const { toast } = useToast();
|
|
|
|
|
43 |
|
44 |
// Check browser support
|
45 |
const isSupported = isWebSerialSupported();
|
@@ -231,8 +234,7 @@ function App() {
|
|
231 |
return;
|
232 |
}
|
233 |
|
234 |
-
|
235 |
-
setView("calibrating");
|
236 |
};
|
237 |
|
238 |
const handleTeleoperate = (robot: RobotConnection) => {
|
@@ -245,122 +247,57 @@ function App() {
|
|
245 |
return;
|
246 |
}
|
247 |
|
248 |
-
|
249 |
-
setView("teleoperating");
|
250 |
};
|
251 |
|
252 |
-
const
|
253 |
-
|
254 |
-
setView("dashboard");
|
255 |
};
|
256 |
|
257 |
const scrollToHardware = () => {
|
258 |
hardwareSectionRef.current?.scrollIntoView({ behavior: "smooth" });
|
259 |
};
|
260 |
|
261 |
-
const renderView = () => {
|
262 |
-
switch (view) {
|
263 |
-
case "calibrating":
|
264 |
-
return selectedRobot && <CalibrationView robot={selectedRobot} />;
|
265 |
-
case "teleoperating":
|
266 |
-
return selectedRobot && <TeleoperationView robot={selectedRobot} />;
|
267 |
-
case "dashboard":
|
268 |
-
default:
|
269 |
-
return (
|
270 |
-
<div className="space-y-20">
|
271 |
-
<DeviceDashboard
|
272 |
-
robots={robots}
|
273 |
-
onCalibrate={handleCalibrate}
|
274 |
-
onTeleoperate={handleTeleoperate}
|
275 |
-
onRemove={handleRemoveRobot}
|
276 |
-
onEdit={setEditingRobot}
|
277 |
-
onFindNew={handleFindNewRobots}
|
278 |
-
isConnecting={isConnecting}
|
279 |
-
onScrollToHardware={scrollToHardware}
|
280 |
-
/>
|
281 |
-
<div>
|
282 |
-
<div className="mb-6">
|
283 |
-
<h2 className="text-3xl font-bold font-mono tracking-wider mb-2 uppercase">
|
284 |
-
install
|
285 |
-
</h2>
|
286 |
-
<p className="text-sm text-muted-foreground font-mono">
|
287 |
-
Choose your preferred development environment
|
288 |
-
</p>
|
289 |
-
</div>
|
290 |
-
<SetupCards />
|
291 |
-
</div>
|
292 |
-
<DocsSection />
|
293 |
-
<RoadmapSection />
|
294 |
-
<div ref={hardwareSectionRef}>
|
295 |
-
<HardwareSupportSection />
|
296 |
-
</div>
|
297 |
-
</div>
|
298 |
-
);
|
299 |
-
}
|
300 |
-
};
|
301 |
-
|
302 |
-
const PageHeader = () => {
|
303 |
-
return (
|
304 |
-
<div className="flex items-center justify-between mb-12">
|
305 |
-
<div className="flex items-center gap-4">
|
306 |
-
<div>
|
307 |
-
{view === "calibrating" && selectedRobot ? (
|
308 |
-
<h1 className="font-mono text-4xl font-bold tracking-wider">
|
309 |
-
<span className="text-muted-foreground uppercase">
|
310 |
-
calibrate:
|
311 |
-
</span>{" "}
|
312 |
-
<span
|
313 |
-
className="text-primary text-glitch uppercase"
|
314 |
-
data-text={selectedRobot.robotId}
|
315 |
-
>
|
316 |
-
{selectedRobot.robotId?.toUpperCase()}
|
317 |
-
</span>
|
318 |
-
</h1>
|
319 |
-
) : view === "teleoperating" && selectedRobot ? (
|
320 |
-
<h1 className="font-mono text-4xl font-bold tracking-wider">
|
321 |
-
<span className="text-muted-foreground uppercase">
|
322 |
-
teleoperate:
|
323 |
-
</span>{" "}
|
324 |
-
<span
|
325 |
-
className="text-primary text-glitch uppercase"
|
326 |
-
data-text={selectedRobot.robotId}
|
327 |
-
>
|
328 |
-
{selectedRobot.robotId?.toUpperCase()}
|
329 |
-
</span>
|
330 |
-
</h1>
|
331 |
-
) : (
|
332 |
-
<h1
|
333 |
-
className="font-mono text-4xl font-bold text-primary tracking-wider text-glitch uppercase"
|
334 |
-
data-text="dashboard"
|
335 |
-
>
|
336 |
-
DASHBOARD
|
337 |
-
</h1>
|
338 |
-
)}
|
339 |
-
<div className="h-6 flex items-center">
|
340 |
-
{view !== "dashboard" ? (
|
341 |
-
<button
|
342 |
-
onClick={handleCloseSubView}
|
343 |
-
className="flex items-center gap-2 text-sm text-muted-foreground font-mono hover:text-primary transition-colors"
|
344 |
-
>
|
345 |
-
<ChevronLeft className="w-4 h-4" />
|
346 |
-
<span className="uppercase">back to dashboard</span>
|
347 |
-
</button>
|
348 |
-
) : (
|
349 |
-
<p className="text-sm text-muted-foreground font-mono">{""} </p>
|
350 |
-
)}
|
351 |
-
</div>
|
352 |
-
</div>
|
353 |
-
</div>
|
354 |
-
</div>
|
355 |
-
);
|
356 |
-
};
|
357 |
-
|
358 |
return (
|
359 |
<div className="flex flex-col min-h-screen font-sans bg-gray-200 dark:bg-background">
|
360 |
<Header />
|
361 |
<main className="flex-grow container mx-auto py-12 px-4 md:px-6">
|
362 |
-
<
|
363 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
364 |
<EditRobotDialog
|
365 |
robot={editingRobot}
|
366 |
isOpen={!!editingRobot}
|
@@ -374,4 +311,203 @@ function App() {
|
|
374 |
);
|
375 |
}
|
376 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
377 |
export default App;
|
|
|
1 |
import { useState, useEffect, useRef } from "react";
|
2 |
+
import {
|
3 |
+
Routes,
|
4 |
+
Route,
|
5 |
+
useNavigate,
|
6 |
+
useParams,
|
7 |
+
useLocation,
|
8 |
+
} from "react-router-dom";
|
9 |
import { ChevronLeft } from "lucide-react";
|
10 |
import { Header } from "@/components/header";
|
11 |
import { Footer } from "@/components/footer";
|
|
|
34 |
} from "@/lib/unified-storage";
|
35 |
|
36 |
function App() {
|
|
|
|
|
|
|
37 |
const [robots, setRobots] = useState<RobotConnection[]>([]);
|
|
|
|
|
|
|
38 |
const [editingRobot, setEditingRobot] = useState<RobotConnection | null>(
|
39 |
null
|
40 |
);
|
41 |
const [isConnecting, setIsConnecting] = useState(false);
|
42 |
const hardwareSectionRef = useRef<HTMLDivElement>(null);
|
43 |
const { toast } = useToast();
|
44 |
+
const navigate = useNavigate();
|
45 |
+
const location = useLocation();
|
46 |
|
47 |
// Check browser support
|
48 |
const isSupported = isWebSerialSupported();
|
|
|
234 |
return;
|
235 |
}
|
236 |
|
237 |
+
navigate(`/device/${robot.serialNumber}/calibrate`);
|
|
|
238 |
};
|
239 |
|
240 |
const handleTeleoperate = (robot: RobotConnection) => {
|
|
|
247 |
return;
|
248 |
}
|
249 |
|
250 |
+
navigate(`/device/${robot.serialNumber}/control`);
|
|
|
251 |
};
|
252 |
|
253 |
+
const handleBackToDashboard = () => {
|
254 |
+
navigate("/");
|
|
|
255 |
};
|
256 |
|
257 |
const scrollToHardware = () => {
|
258 |
hardwareSectionRef.current?.scrollIntoView({ behavior: "smooth" });
|
259 |
};
|
260 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
261 |
return (
|
262 |
<div className="flex flex-col min-h-screen font-sans bg-gray-200 dark:bg-background">
|
263 |
<Header />
|
264 |
<main className="flex-grow container mx-auto py-12 px-4 md:px-6">
|
265 |
+
<Routes>
|
266 |
+
<Route
|
267 |
+
path="/"
|
268 |
+
element={
|
269 |
+
<DashboardPage
|
270 |
+
robots={robots}
|
271 |
+
onCalibrate={handleCalibrate}
|
272 |
+
onTeleoperate={handleTeleoperate}
|
273 |
+
onRemove={handleRemoveRobot}
|
274 |
+
onEdit={setEditingRobot}
|
275 |
+
onFindNew={handleFindNewRobots}
|
276 |
+
isConnecting={isConnecting}
|
277 |
+
onScrollToHardware={scrollToHardware}
|
278 |
+
hardwareSectionRef={hardwareSectionRef}
|
279 |
+
/>
|
280 |
+
}
|
281 |
+
/>
|
282 |
+
<Route
|
283 |
+
path="/device/:serialNumber/calibrate"
|
284 |
+
element={
|
285 |
+
<CalibratePage
|
286 |
+
robots={robots}
|
287 |
+
onBackToDashboard={handleBackToDashboard}
|
288 |
+
/>
|
289 |
+
}
|
290 |
+
/>
|
291 |
+
<Route
|
292 |
+
path="/device/:serialNumber/control"
|
293 |
+
element={
|
294 |
+
<ControlPage
|
295 |
+
robots={robots}
|
296 |
+
onBackToDashboard={handleBackToDashboard}
|
297 |
+
/>
|
298 |
+
}
|
299 |
+
/>
|
300 |
+
</Routes>
|
301 |
<EditRobotDialog
|
302 |
robot={editingRobot}
|
303 |
isOpen={!!editingRobot}
|
|
|
311 |
);
|
312 |
}
|
313 |
|
314 |
+
// Dashboard Page Component
|
315 |
+
function DashboardPage({
|
316 |
+
robots,
|
317 |
+
onCalibrate,
|
318 |
+
onTeleoperate,
|
319 |
+
onRemove,
|
320 |
+
onEdit,
|
321 |
+
onFindNew,
|
322 |
+
isConnecting,
|
323 |
+
onScrollToHardware,
|
324 |
+
hardwareSectionRef,
|
325 |
+
}: {
|
326 |
+
robots: RobotConnection[];
|
327 |
+
onCalibrate: (robot: RobotConnection) => void;
|
328 |
+
onTeleoperate: (robot: RobotConnection) => void;
|
329 |
+
onRemove: (robot: RobotConnection) => void;
|
330 |
+
onEdit: (robot: RobotConnection | null) => void;
|
331 |
+
onFindNew: () => void;
|
332 |
+
isConnecting: boolean;
|
333 |
+
onScrollToHardware: () => void;
|
334 |
+
hardwareSectionRef: React.RefObject<HTMLDivElement>;
|
335 |
+
}) {
|
336 |
+
return (
|
337 |
+
<div>
|
338 |
+
<PageHeader />
|
339 |
+
<div className="space-y-20">
|
340 |
+
<DeviceDashboard
|
341 |
+
robots={robots}
|
342 |
+
onCalibrate={onCalibrate}
|
343 |
+
onTeleoperate={onTeleoperate}
|
344 |
+
onRemove={onRemove}
|
345 |
+
onEdit={onEdit}
|
346 |
+
onFindNew={onFindNew}
|
347 |
+
isConnecting={isConnecting}
|
348 |
+
onScrollToHardware={onScrollToHardware}
|
349 |
+
/>
|
350 |
+
<div>
|
351 |
+
<div className="mb-6">
|
352 |
+
<h2 className="text-3xl font-bold font-mono tracking-wider mb-2 uppercase">
|
353 |
+
install
|
354 |
+
</h2>
|
355 |
+
<p className="text-sm text-muted-foreground font-mono">
|
356 |
+
Choose your preferred development environment
|
357 |
+
</p>
|
358 |
+
</div>
|
359 |
+
<SetupCards />
|
360 |
+
</div>
|
361 |
+
<DocsSection />
|
362 |
+
<RoadmapSection />
|
363 |
+
<div ref={hardwareSectionRef}>
|
364 |
+
<HardwareSupportSection />
|
365 |
+
</div>
|
366 |
+
</div>
|
367 |
+
</div>
|
368 |
+
);
|
369 |
+
}
|
370 |
+
|
371 |
+
// Calibrate Page Component
|
372 |
+
function CalibratePage({
|
373 |
+
robots,
|
374 |
+
onBackToDashboard,
|
375 |
+
}: {
|
376 |
+
robots: RobotConnection[];
|
377 |
+
onBackToDashboard: () => void;
|
378 |
+
}) {
|
379 |
+
const { serialNumber } = useParams<{ serialNumber: string }>();
|
380 |
+
const selectedRobot = robots.find(
|
381 |
+
(robot) => robot.serialNumber === serialNumber
|
382 |
+
);
|
383 |
+
|
384 |
+
if (!selectedRobot) {
|
385 |
+
return (
|
386 |
+
<div>
|
387 |
+
<PageHeader onBackToDashboard={onBackToDashboard} />
|
388 |
+
<div className="text-center py-20">
|
389 |
+
<p className="text-muted-foreground">
|
390 |
+
Device not found or not connected.
|
391 |
+
</p>
|
392 |
+
</div>
|
393 |
+
</div>
|
394 |
+
);
|
395 |
+
}
|
396 |
+
|
397 |
+
return (
|
398 |
+
<div>
|
399 |
+
<PageHeader
|
400 |
+
onBackToDashboard={onBackToDashboard}
|
401 |
+
selectedRobot={selectedRobot}
|
402 |
+
/>
|
403 |
+
<CalibrationView robot={selectedRobot} />
|
404 |
+
</div>
|
405 |
+
);
|
406 |
+
}
|
407 |
+
|
408 |
+
// Control Page Component
|
409 |
+
function ControlPage({
|
410 |
+
robots,
|
411 |
+
onBackToDashboard,
|
412 |
+
}: {
|
413 |
+
robots: RobotConnection[];
|
414 |
+
onBackToDashboard: () => void;
|
415 |
+
}) {
|
416 |
+
const { serialNumber } = useParams<{ serialNumber: string }>();
|
417 |
+
const selectedRobot = robots.find(
|
418 |
+
(robot) => robot.serialNumber === serialNumber
|
419 |
+
);
|
420 |
+
|
421 |
+
if (!selectedRobot) {
|
422 |
+
return (
|
423 |
+
<div>
|
424 |
+
<PageHeader onBackToDashboard={onBackToDashboard} />
|
425 |
+
<div className="text-center py-20">
|
426 |
+
<p className="text-muted-foreground">
|
427 |
+
Device not found or not connected.
|
428 |
+
</p>
|
429 |
+
</div>
|
430 |
+
</div>
|
431 |
+
);
|
432 |
+
}
|
433 |
+
|
434 |
+
return (
|
435 |
+
<div>
|
436 |
+
<PageHeader
|
437 |
+
onBackToDashboard={onBackToDashboard}
|
438 |
+
selectedRobot={selectedRobot}
|
439 |
+
/>
|
440 |
+
<TeleoperationView robot={selectedRobot} />
|
441 |
+
</div>
|
442 |
+
);
|
443 |
+
}
|
444 |
+
|
445 |
+
// Page Header Component
|
446 |
+
function PageHeader({
|
447 |
+
onBackToDashboard,
|
448 |
+
selectedRobot,
|
449 |
+
}: {
|
450 |
+
onBackToDashboard?: () => void;
|
451 |
+
selectedRobot?: RobotConnection | null;
|
452 |
+
}) {
|
453 |
+
const location = useLocation();
|
454 |
+
const isDashboard = location.pathname === "/";
|
455 |
+
const isCalibrating = location.pathname.includes("/calibrate");
|
456 |
+
const isTeleoperating = location.pathname.includes("/control");
|
457 |
+
|
458 |
+
return (
|
459 |
+
<div className="flex items-center justify-between mb-12">
|
460 |
+
<div className="flex items-center gap-4">
|
461 |
+
<div>
|
462 |
+
{isCalibrating && selectedRobot ? (
|
463 |
+
<h1 className="font-mono text-4xl font-bold tracking-wider">
|
464 |
+
<span className="text-muted-foreground uppercase">
|
465 |
+
calibrate:
|
466 |
+
</span>{" "}
|
467 |
+
<span
|
468 |
+
className="text-primary text-glitch uppercase"
|
469 |
+
data-text={selectedRobot.robotId}
|
470 |
+
>
|
471 |
+
{selectedRobot.robotId?.toUpperCase()}
|
472 |
+
</span>
|
473 |
+
</h1>
|
474 |
+
) : isTeleoperating && selectedRobot ? (
|
475 |
+
<h1 className="font-mono text-4xl font-bold tracking-wider">
|
476 |
+
<span className="text-muted-foreground uppercase">
|
477 |
+
teleoperate:
|
478 |
+
</span>{" "}
|
479 |
+
<span
|
480 |
+
className="text-primary text-glitch uppercase"
|
481 |
+
data-text={selectedRobot.robotId}
|
482 |
+
>
|
483 |
+
{selectedRobot.robotId?.toUpperCase()}
|
484 |
+
</span>
|
485 |
+
</h1>
|
486 |
+
) : (
|
487 |
+
<h1
|
488 |
+
className="font-mono text-4xl font-bold text-primary tracking-wider text-glitch uppercase"
|
489 |
+
data-text="dashboard"
|
490 |
+
>
|
491 |
+
DASHBOARD
|
492 |
+
</h1>
|
493 |
+
)}
|
494 |
+
<div className="h-6 flex items-center">
|
495 |
+
{!isDashboard && onBackToDashboard ? (
|
496 |
+
<button
|
497 |
+
onClick={onBackToDashboard}
|
498 |
+
className="flex items-center gap-2 text-sm text-muted-foreground font-mono hover:text-primary transition-colors"
|
499 |
+
>
|
500 |
+
<ChevronLeft className="w-4 h-4" />
|
501 |
+
<span className="uppercase">back to dashboard</span>
|
502 |
+
</button>
|
503 |
+
) : (
|
504 |
+
<p className="text-sm text-muted-foreground font-mono">{""} </p>
|
505 |
+
)}
|
506 |
+
</div>
|
507 |
+
</div>
|
508 |
+
</div>
|
509 |
+
</div>
|
510 |
+
);
|
511 |
+
}
|
512 |
+
|
513 |
export default App;
|
@@ -0,0 +1,1318 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
"use client";
|
2 |
+
|
3 |
+
import { useState, useEffect, useRef, useCallback, useMemo } from "react";
|
4 |
+
import { Button } from "@/components/ui/button";
|
5 |
+
import { Card } from "@/components/ui/card";
|
6 |
+
import {
|
7 |
+
Table,
|
8 |
+
TableBody,
|
9 |
+
TableCaption,
|
10 |
+
TableCell,
|
11 |
+
TableHead,
|
12 |
+
TableHeader,
|
13 |
+
TableRow,
|
14 |
+
} from "@/components/ui/table";
|
15 |
+
import { Input } from "@/components/ui/input";
|
16 |
+
import { Badge } from "@/components/ui/badge";
|
17 |
+
import {
|
18 |
+
Select,
|
19 |
+
SelectContent,
|
20 |
+
SelectItem,
|
21 |
+
SelectTrigger,
|
22 |
+
SelectValue,
|
23 |
+
} from "@/components/ui/select";
|
24 |
+
import { useToast } from "@/hooks/use-toast";
|
25 |
+
import {
|
26 |
+
Disc as Record,
|
27 |
+
Download,
|
28 |
+
Upload,
|
29 |
+
PlusCircle,
|
30 |
+
Square,
|
31 |
+
Camera,
|
32 |
+
Trash2,
|
33 |
+
Settings,
|
34 |
+
RefreshCw,
|
35 |
+
X,
|
36 |
+
Edit2,
|
37 |
+
Check,
|
38 |
+
} from "lucide-react";
|
39 |
+
import { LeRobotDatasetRecorder, LeRobotDatasetRow, NonIndexedLeRobotDatasetRow, LeRobotEpisode } from "@lerobot/web";
|
40 |
+
import { TeleoperatorEpisodesView } from "./teleoperator-episodes-view";
|
41 |
+
|
42 |
+
interface RecorderProps {
|
43 |
+
teleoperators: any[];
|
44 |
+
robot: any; // eslint-disable-line @typescript-eslint/no-explicit-any
|
45 |
+
onNeedsTeleoperation: () => Promise<boolean>;
|
46 |
+
videoStreams?: { [key: string]: MediaStream };
|
47 |
+
}
|
48 |
+
|
49 |
+
|
50 |
+
|
51 |
+
interface RecorderSettings {
|
52 |
+
huggingfaceApiKey: string;
|
53 |
+
cameraConfigs: {
|
54 |
+
[cameraName: string]: {
|
55 |
+
deviceId: string;
|
56 |
+
deviceLabel: string;
|
57 |
+
};
|
58 |
+
};
|
59 |
+
}
|
60 |
+
|
61 |
+
// Storage functions for recorder settings
|
62 |
+
const RECORDER_SETTINGS_KEY = "lerobot-recorder-settings";
|
63 |
+
|
64 |
+
function getRecorderSettings(): RecorderSettings {
|
65 |
+
try {
|
66 |
+
const stored = localStorage.getItem(RECORDER_SETTINGS_KEY);
|
67 |
+
if (stored) {
|
68 |
+
return JSON.parse(stored);
|
69 |
+
}
|
70 |
+
} catch (error) {
|
71 |
+
console.warn("Failed to load recorder settings:", error);
|
72 |
+
}
|
73 |
+
return {
|
74 |
+
huggingfaceApiKey: "",
|
75 |
+
cameraConfigs: {},
|
76 |
+
};
|
77 |
+
}
|
78 |
+
|
79 |
+
function saveRecorderSettings(settings: RecorderSettings): void {
|
80 |
+
try {
|
81 |
+
localStorage.setItem(RECORDER_SETTINGS_KEY, JSON.stringify(settings));
|
82 |
+
} catch (error) {
|
83 |
+
console.warn("Failed to save recorder settings:", error);
|
84 |
+
}
|
85 |
+
}
|
86 |
+
|
87 |
+
export function Recorder({
|
88 |
+
teleoperators,
|
89 |
+
robot,
|
90 |
+
onNeedsTeleoperation,
|
91 |
+
}: RecorderProps) {
|
92 |
+
const [isRecording, setIsRecording] = useState(false);
|
93 |
+
const [currentEpisode, setCurrentEpisode] = useState(0);
|
94 |
+
// Use huggingfaceApiKey from recorderSettings instead of separate state
|
95 |
+
const [cameraName, setCameraName] = useState("");
|
96 |
+
const [additionalCameras, setAdditionalCameras] = useState<{
|
97 |
+
[key: string]: MediaStream;
|
98 |
+
}>({});
|
99 |
+
const [availableCameras, setAvailableCameras] = useState<MediaDeviceInfo[]>(
|
100 |
+
[]
|
101 |
+
);
|
102 |
+
const [selectedCameraId, setSelectedCameraId] = useState<string>("");
|
103 |
+
const [previewStream, setPreviewStream] = useState<MediaStream | null>(null);
|
104 |
+
const [isLoadingCameras, setIsLoadingCameras] = useState(false);
|
105 |
+
const [cameraPermissionState, setCameraPermissionState] = useState<
|
106 |
+
"unknown" | "granted" | "denied"
|
107 |
+
>("unknown");
|
108 |
+
const [showCameraConfig, setShowCameraConfig] = useState(false);
|
109 |
+
const [showConfigure, setShowConfigure] = useState(false);
|
110 |
+
const [recorderSettings, setRecorderSettings] = useState<RecorderSettings>(
|
111 |
+
() => getRecorderSettings()
|
112 |
+
);
|
113 |
+
const [hasRecordedFrames, setHasRecordedFrames] = useState(false);
|
114 |
+
const [editingCameraName, setEditingCameraName] = useState<string | null>(
|
115 |
+
null
|
116 |
+
);
|
117 |
+
const [editingCameraNewName, setEditingCameraNewName] = useState("");
|
118 |
+
const [huggingfaceApiKey, setHuggingfaceApiKey] = useState("")
|
119 |
+
const recorderRef = useRef<LeRobotDatasetRecorder | null>(null);
|
120 |
+
const videoRef = useRef<HTMLVideoElement | null>(null);
|
121 |
+
const { toast } = useToast();
|
122 |
+
|
123 |
+
// Initialize the recorder when teleoperators are available
|
124 |
+
useEffect(() => {
|
125 |
+
if (teleoperators.length > 0) {
|
126 |
+
recorderRef.current = new LeRobotDatasetRecorder(
|
127 |
+
teleoperators,
|
128 |
+
additionalCameras,
|
129 |
+
30, // fps
|
130 |
+
"Robot teleoperation recording"
|
131 |
+
);
|
132 |
+
}
|
133 |
+
}, [teleoperators, additionalCameras]);
|
134 |
+
|
135 |
+
const handleStartRecording = async () => {
|
136 |
+
// If teleoperators aren't available, initialize teleoperation first
|
137 |
+
if (teleoperators.length === 0) {
|
138 |
+
toast({
|
139 |
+
title: "Initializing...",
|
140 |
+
description: `Setting up robot control for ${robot.robotId || "robot"}`,
|
141 |
+
});
|
142 |
+
|
143 |
+
const success = await onNeedsTeleoperation();
|
144 |
+
if (!success) {
|
145 |
+
toast({
|
146 |
+
title: "Recording Error",
|
147 |
+
description: "Failed to initialize robot control",
|
148 |
+
variant: "destructive",
|
149 |
+
});
|
150 |
+
return;
|
151 |
+
}
|
152 |
+
|
153 |
+
// Wait a moment for the recorder to initialize with new teleoperators
|
154 |
+
await new Promise((resolve) => setTimeout(resolve, 100));
|
155 |
+
}
|
156 |
+
|
157 |
+
if (!recorderRef.current) {
|
158 |
+
toast({
|
159 |
+
title: "Recording Error",
|
160 |
+
description: "Recorder not ready yet. Please try again.",
|
161 |
+
variant: "destructive",
|
162 |
+
});
|
163 |
+
return;
|
164 |
+
}
|
165 |
+
|
166 |
+
try {
|
167 |
+
// Set the episode index
|
168 |
+
recorderRef.current.setEpisodeIndex(currentEpisode);
|
169 |
+
recorderRef.current.setTaskIndex(0); // Default task index
|
170 |
+
|
171 |
+
// Start recording
|
172 |
+
recorderRef.current.startRecording();
|
173 |
+
setIsRecording(true);
|
174 |
+
setHasRecordedFrames(true);
|
175 |
+
|
176 |
+
toast({
|
177 |
+
title: "Recording Started",
|
178 |
+
description: `Episode ${currentEpisode} is now recording`,
|
179 |
+
});
|
180 |
+
} catch (error) {
|
181 |
+
const errorMessage =
|
182 |
+
error instanceof Error ? error.message : "Failed to start recording";
|
183 |
+
toast({
|
184 |
+
title: "Recording Error",
|
185 |
+
description: errorMessage,
|
186 |
+
variant: "destructive",
|
187 |
+
});
|
188 |
+
}
|
189 |
+
};
|
190 |
+
|
191 |
+
const handleStopRecording = async () => {
|
192 |
+
if (!recorderRef.current || !isRecording) {
|
193 |
+
return;
|
194 |
+
}
|
195 |
+
|
196 |
+
try {
|
197 |
+
const result = await recorderRef.current.stopRecording();
|
198 |
+
setIsRecording(false);
|
199 |
+
|
200 |
+
toast({
|
201 |
+
title: "Recording Stopped",
|
202 |
+
description: `Episode ${currentEpisode} completed with ${result.teleoperatorData.length} frames`,
|
203 |
+
});
|
204 |
+
} catch (error) {
|
205 |
+
const errorMessage =
|
206 |
+
error instanceof Error ? error.message : "Failed to stop recording";
|
207 |
+
toast({
|
208 |
+
title: "Recording Error",
|
209 |
+
description: errorMessage,
|
210 |
+
variant: "destructive",
|
211 |
+
});
|
212 |
+
}
|
213 |
+
};
|
214 |
+
|
215 |
+
const handleNextEpisode = () => {
|
216 |
+
// Make sure we're not recording
|
217 |
+
if (isRecording) {
|
218 |
+
handleStopRecording();
|
219 |
+
}
|
220 |
+
|
221 |
+
// Increment episode counter
|
222 |
+
setCurrentEpisode((prev) => prev + 1);
|
223 |
+
|
224 |
+
toast({
|
225 |
+
title: "New Episode",
|
226 |
+
description: `Ready to record episode ${currentEpisode + 1}`,
|
227 |
+
});
|
228 |
+
};
|
229 |
+
|
230 |
+
// Reset frames by clearing the recorder data
|
231 |
+
const handleResetFrames = useCallback(() => {
|
232 |
+
if (isRecording) {
|
233 |
+
handleStopRecording();
|
234 |
+
}
|
235 |
+
|
236 |
+
if (recorderRef.current) {
|
237 |
+
recorderRef.current.clearRecording();
|
238 |
+
setHasRecordedFrames(false);
|
239 |
+
|
240 |
+
toast({
|
241 |
+
title: "Frames Reset",
|
242 |
+
description: "All recorded frames have been cleared",
|
243 |
+
});
|
244 |
+
}
|
245 |
+
}, [isRecording, toast]);
|
246 |
+
|
247 |
+
// Load available cameras
|
248 |
+
const loadAvailableCameras = useCallback(
|
249 |
+
async (isAutoLoad = false) => {
|
250 |
+
if (isLoadingCameras) return;
|
251 |
+
|
252 |
+
setIsLoadingCameras(true);
|
253 |
+
|
254 |
+
try {
|
255 |
+
// Check if we already have permission
|
256 |
+
const permission = await navigator.permissions.query({
|
257 |
+
name: "camera" as PermissionName,
|
258 |
+
});
|
259 |
+
setCameraPermissionState(
|
260 |
+
permission.state === "granted"
|
261 |
+
? "granted"
|
262 |
+
: permission.state === "denied"
|
263 |
+
? "denied"
|
264 |
+
: "unknown"
|
265 |
+
);
|
266 |
+
|
267 |
+
let tempStream: MediaStream | null = null;
|
268 |
+
|
269 |
+
// Try to enumerate devices first (works if we have permission)
|
270 |
+
const devices = await navigator.mediaDevices.enumerateDevices();
|
271 |
+
const videoDevices = devices.filter(
|
272 |
+
(device) => device.kind === "videoinput"
|
273 |
+
);
|
274 |
+
|
275 |
+
// If devices have labels, we already have permission
|
276 |
+
const hasLabels = videoDevices.some((device) => device.label);
|
277 |
+
let finalVideoDevices = videoDevices;
|
278 |
+
|
279 |
+
if (!hasLabels && videoDevices.length > 0) {
|
280 |
+
// Need to request permission to get device labels
|
281 |
+
tempStream = await navigator.mediaDevices.getUserMedia({
|
282 |
+
video: true,
|
283 |
+
});
|
284 |
+
|
285 |
+
// Re-enumerate to get labels
|
286 |
+
const devicesWithLabels =
|
287 |
+
await navigator.mediaDevices.enumerateDevices();
|
288 |
+
const videoDevicesWithLabels = devicesWithLabels.filter(
|
289 |
+
(device) => device.kind === "videoinput"
|
290 |
+
);
|
291 |
+
finalVideoDevices = videoDevicesWithLabels;
|
292 |
+
setAvailableCameras(videoDevicesWithLabels);
|
293 |
+
|
294 |
+
if (!isAutoLoad) {
|
295 |
+
console.log(
|
296 |
+
`Found ${videoDevicesWithLabels.length} video devices:`,
|
297 |
+
videoDevicesWithLabels.map((d) => d.label || d.deviceId)
|
298 |
+
);
|
299 |
+
}
|
300 |
+
} else {
|
301 |
+
setAvailableCameras(videoDevices);
|
302 |
+
if (!isAutoLoad) {
|
303 |
+
console.log(
|
304 |
+
`Found ${videoDevices.length} video devices:`,
|
305 |
+
videoDevices.map((d) => d.label || d.deviceId)
|
306 |
+
);
|
307 |
+
}
|
308 |
+
}
|
309 |
+
|
310 |
+
// Auto-select and preview first camera if none selected
|
311 |
+
if (finalVideoDevices.length > 0 && !selectedCameraId) {
|
312 |
+
const firstCameraId = finalVideoDevices[0].deviceId;
|
313 |
+
|
314 |
+
// Stop temp stream since we'll create a fresh one with switchCameraPreview
|
315 |
+
if (tempStream) {
|
316 |
+
tempStream.getTracks().forEach((track) => track.stop());
|
317 |
+
}
|
318 |
+
|
319 |
+
setCameraPermissionState("granted");
|
320 |
+
|
321 |
+
// Use the same logic as manual camera switching
|
322 |
+
await switchCameraPreview(firstCameraId);
|
323 |
+
} else if (tempStream) {
|
324 |
+
// Stop temp stream if we didn't use it
|
325 |
+
tempStream.getTracks().forEach((track) => track.stop());
|
326 |
+
}
|
327 |
+
} catch (error) {
|
328 |
+
setCameraPermissionState("denied");
|
329 |
+
if (!isAutoLoad) {
|
330 |
+
toast({
|
331 |
+
title: "Camera Error",
|
332 |
+
description: `Failed to load cameras: ${
|
333 |
+
error instanceof Error ? error.message : String(error)
|
334 |
+
}`,
|
335 |
+
variant: "destructive",
|
336 |
+
});
|
337 |
+
}
|
338 |
+
} finally {
|
339 |
+
setIsLoadingCameras(false);
|
340 |
+
}
|
341 |
+
},
|
342 |
+
[selectedCameraId, toast]
|
343 |
+
);
|
344 |
+
|
345 |
+
// Switch camera preview
|
346 |
+
const switchCameraPreview = useCallback(
|
347 |
+
async (deviceId: string) => {
|
348 |
+
try {
|
349 |
+
// Stop current preview stream
|
350 |
+
if (previewStream) {
|
351 |
+
previewStream.getTracks().forEach((track) => track.stop());
|
352 |
+
}
|
353 |
+
|
354 |
+
// Start new stream with selected camera
|
355 |
+
const newStream = await navigator.mediaDevices.getUserMedia({
|
356 |
+
video: {
|
357 |
+
deviceId: { exact: deviceId },
|
358 |
+
width: { ideal: 1280 },
|
359 |
+
height: { ideal: 720 },
|
360 |
+
},
|
361 |
+
});
|
362 |
+
|
363 |
+
setPreviewStream(newStream);
|
364 |
+
setSelectedCameraId(deviceId);
|
365 |
+
} catch (error) {
|
366 |
+
toast({
|
367 |
+
title: "Camera Error",
|
368 |
+
description: `Failed to switch camera: ${
|
369 |
+
error instanceof Error ? error.message : String(error)
|
370 |
+
}`,
|
371 |
+
variant: "destructive",
|
372 |
+
});
|
373 |
+
}
|
374 |
+
},
|
375 |
+
[previewStream, toast]
|
376 |
+
);
|
377 |
+
|
378 |
+
// Add a new camera to the recorder
|
379 |
+
const handleAddCamera = useCallback(async () => {
|
380 |
+
if (!cameraName.trim()) {
|
381 |
+
toast({
|
382 |
+
title: "Camera Error",
|
383 |
+
description: "Please enter a camera name",
|
384 |
+
variant: "destructive",
|
385 |
+
});
|
386 |
+
return;
|
387 |
+
}
|
388 |
+
|
389 |
+
if (hasRecordedFrames) {
|
390 |
+
toast({
|
391 |
+
title: "Camera Error",
|
392 |
+
description: "Cannot add cameras after recording has started",
|
393 |
+
variant: "destructive",
|
394 |
+
});
|
395 |
+
return;
|
396 |
+
}
|
397 |
+
|
398 |
+
if (!selectedCameraId) {
|
399 |
+
toast({
|
400 |
+
title: "Camera Error",
|
401 |
+
description: "Please select a camera first",
|
402 |
+
variant: "destructive",
|
403 |
+
});
|
404 |
+
return;
|
405 |
+
}
|
406 |
+
|
407 |
+
try {
|
408 |
+
// Use the current preview stream (already running with correct camera)
|
409 |
+
if (!previewStream) {
|
410 |
+
throw new Error("No camera preview available");
|
411 |
+
}
|
412 |
+
|
413 |
+
// Clone the stream for recording (keep preview running)
|
414 |
+
const recordingStream = previewStream.clone();
|
415 |
+
|
416 |
+
// Add the new camera to our state
|
417 |
+
setAdditionalCameras((prev) => ({
|
418 |
+
...prev,
|
419 |
+
[cameraName]: recordingStream,
|
420 |
+
}));
|
421 |
+
|
422 |
+
// Save camera configuration to persistent storage
|
423 |
+
const selectedCamera = availableCameras.find(
|
424 |
+
(cam) => cam.deviceId === selectedCameraId
|
425 |
+
);
|
426 |
+
const newSettings = {
|
427 |
+
...recorderSettings,
|
428 |
+
cameraConfigs: {
|
429 |
+
...recorderSettings.cameraConfigs,
|
430 |
+
[cameraName]: {
|
431 |
+
deviceId: selectedCameraId,
|
432 |
+
deviceLabel:
|
433 |
+
selectedCamera?.label ||
|
434 |
+
`Camera ${selectedCameraId.slice(0, 8)}...`,
|
435 |
+
},
|
436 |
+
},
|
437 |
+
};
|
438 |
+
setRecorderSettings(newSettings);
|
439 |
+
saveRecorderSettings(newSettings);
|
440 |
+
|
441 |
+
setCameraName(""); // Clear the input
|
442 |
+
|
443 |
+
toast({
|
444 |
+
title: "Camera Added",
|
445 |
+
description: `Camera "${cameraName}" has been added to the recorder`,
|
446 |
+
});
|
447 |
+
} catch (error) {
|
448 |
+
toast({
|
449 |
+
title: "Camera Error",
|
450 |
+
description: `Failed to access camera: ${
|
451 |
+
error instanceof Error ? error.message : String(error)
|
452 |
+
}`,
|
453 |
+
variant: "destructive",
|
454 |
+
});
|
455 |
+
}
|
456 |
+
}, [
|
457 |
+
cameraName,
|
458 |
+
hasRecordedFrames,
|
459 |
+
selectedCameraId,
|
460 |
+
previewStream,
|
461 |
+
availableCameras,
|
462 |
+
recorderSettings,
|
463 |
+
toast,
|
464 |
+
]);
|
465 |
+
|
466 |
+
// Remove a camera from the recorder
|
467 |
+
const handleRemoveCamera = useCallback(
|
468 |
+
(name: string) => {
|
469 |
+
if (hasRecordedFrames) {
|
470 |
+
toast({
|
471 |
+
title: "Camera Error",
|
472 |
+
description: "Cannot remove cameras after recording has started",
|
473 |
+
variant: "destructive",
|
474 |
+
});
|
475 |
+
return;
|
476 |
+
}
|
477 |
+
|
478 |
+
setAdditionalCameras((prev) => {
|
479 |
+
const newCameras = { ...prev };
|
480 |
+
if (newCameras[name]) {
|
481 |
+
// Stop the stream tracks
|
482 |
+
newCameras[name].getTracks().forEach((track) => track.stop());
|
483 |
+
delete newCameras[name];
|
484 |
+
}
|
485 |
+
return newCameras;
|
486 |
+
});
|
487 |
+
|
488 |
+
// Remove camera configuration from persistent storage
|
489 |
+
const newSettings = {
|
490 |
+
...recorderSettings,
|
491 |
+
cameraConfigs: { ...recorderSettings.cameraConfigs },
|
492 |
+
};
|
493 |
+
delete newSettings.cameraConfigs[name];
|
494 |
+
setRecorderSettings(newSettings);
|
495 |
+
saveRecorderSettings(newSettings);
|
496 |
+
|
497 |
+
toast({
|
498 |
+
title: "Camera Removed",
|
499 |
+
description: `Camera "${name}" has been removed`,
|
500 |
+
});
|
501 |
+
},
|
502 |
+
[hasRecordedFrames, recorderSettings, toast]
|
503 |
+
);
|
504 |
+
|
505 |
+
// Camera name editing functions
|
506 |
+
const handleStartEditingCameraName = (cameraName: string) => {
|
507 |
+
setEditingCameraName(cameraName);
|
508 |
+
setEditingCameraNewName(cameraName);
|
509 |
+
};
|
510 |
+
|
511 |
+
const handleConfirmCameraNameEdit = (oldName: string) => {
|
512 |
+
if (editingCameraNewName.trim() && editingCameraNewName !== oldName) {
|
513 |
+
const stream = additionalCameras[oldName];
|
514 |
+
if (stream) {
|
515 |
+
// Update camera streams
|
516 |
+
setAdditionalCameras((prev) => {
|
517 |
+
const newCameras = { ...prev };
|
518 |
+
delete newCameras[oldName];
|
519 |
+
newCameras[editingCameraNewName.trim()] = stream;
|
520 |
+
return newCameras;
|
521 |
+
});
|
522 |
+
|
523 |
+
// Update camera configuration in persistent storage
|
524 |
+
const oldConfig = recorderSettings.cameraConfigs[oldName];
|
525 |
+
if (oldConfig) {
|
526 |
+
const newSettings = {
|
527 |
+
...recorderSettings,
|
528 |
+
cameraConfigs: { ...recorderSettings.cameraConfigs },
|
529 |
+
};
|
530 |
+
delete newSettings.cameraConfigs[oldName];
|
531 |
+
newSettings.cameraConfigs[editingCameraNewName.trim()] = oldConfig;
|
532 |
+
setRecorderSettings(newSettings);
|
533 |
+
saveRecorderSettings(newSettings);
|
534 |
+
}
|
535 |
+
}
|
536 |
+
}
|
537 |
+
setEditingCameraName(null);
|
538 |
+
setEditingCameraNewName("");
|
539 |
+
};
|
540 |
+
|
541 |
+
const handleCancelCameraNameEdit = () => {
|
542 |
+
setEditingCameraName(null);
|
543 |
+
setEditingCameraNewName("");
|
544 |
+
};
|
545 |
+
|
546 |
+
// Restore cameras from saved configurations
|
547 |
+
const restoreSavedCameras = useCallback(async () => {
|
548 |
+
const savedConfigs = recorderSettings.cameraConfigs;
|
549 |
+
if (!savedConfigs || Object.keys(savedConfigs).length === 0) {
|
550 |
+
return;
|
551 |
+
}
|
552 |
+
|
553 |
+
for (const [cameraName, config] of Object.entries(savedConfigs)) {
|
554 |
+
try {
|
555 |
+
// Check if this camera is still available
|
556 |
+
const isDeviceAvailable = availableCameras.some(
|
557 |
+
(cam) => cam.deviceId === config.deviceId
|
558 |
+
);
|
559 |
+
|
560 |
+
if (!isDeviceAvailable) {
|
561 |
+
console.warn(
|
562 |
+
`Saved camera "${cameraName}" (${config.deviceId}) is no longer available`
|
563 |
+
);
|
564 |
+
continue;
|
565 |
+
}
|
566 |
+
|
567 |
+
// Create stream for this saved camera
|
568 |
+
const stream = await navigator.mediaDevices.getUserMedia({
|
569 |
+
video: {
|
570 |
+
deviceId: { exact: config.deviceId },
|
571 |
+
width: { ideal: 1280 },
|
572 |
+
height: { ideal: 720 },
|
573 |
+
},
|
574 |
+
});
|
575 |
+
|
576 |
+
// Add to additional cameras
|
577 |
+
setAdditionalCameras((prev) => ({
|
578 |
+
...prev,
|
579 |
+
[cameraName]: stream,
|
580 |
+
}));
|
581 |
+
} catch (error) {
|
582 |
+
console.error(`Failed to restore camera "${cameraName}":`, error);
|
583 |
+
// Remove invalid configuration
|
584 |
+
const newSettings = {
|
585 |
+
...recorderSettings,
|
586 |
+
cameraConfigs: { ...recorderSettings.cameraConfigs },
|
587 |
+
};
|
588 |
+
delete newSettings.cameraConfigs[cameraName];
|
589 |
+
setRecorderSettings(newSettings);
|
590 |
+
saveRecorderSettings(newSettings);
|
591 |
+
}
|
592 |
+
}
|
593 |
+
}, [availableCameras, recorderSettings]);
|
594 |
+
|
595 |
+
// Auto-load cameras on component mount (only once)
|
596 |
+
useEffect(() => {
|
597 |
+
loadAvailableCameras(true);
|
598 |
+
// eslint-disable-next-line react-hooks/exhaustive-deps
|
599 |
+
}, []); // Empty dependency array to run only once
|
600 |
+
|
601 |
+
// Handle video stream assignment - runs when stream changes OR when settings panel opens
|
602 |
+
useEffect(() => {
|
603 |
+
if (videoRef.current && previewStream) {
|
604 |
+
videoRef.current.srcObject = previewStream;
|
605 |
+
}
|
606 |
+
}, [previewStream, showConfigure]); // Also depend on showConfigure so it re-runs when video element appears
|
607 |
+
|
608 |
+
// Cleanup preview stream on unmount
|
609 |
+
useEffect(() => {
|
610 |
+
return () => {
|
611 |
+
if (previewStream) {
|
612 |
+
previewStream.getTracks().forEach((track) => track.stop());
|
613 |
+
}
|
614 |
+
};
|
615 |
+
}, [previewStream]);
|
616 |
+
|
617 |
+
// Restore saved cameras when available cameras are loaded
|
618 |
+
useEffect(() => {
|
619 |
+
if (availableCameras.length > 0 && cameraPermissionState === "granted") {
|
620 |
+
restoreSavedCameras();
|
621 |
+
}
|
622 |
+
}, [availableCameras, cameraPermissionState, restoreSavedCameras]);
|
623 |
+
|
624 |
+
const handleDownloadZip = async () => {
|
625 |
+
if (!recorderRef.current) {
|
626 |
+
toast({
|
627 |
+
title: "Download Error",
|
628 |
+
description: "Recorder not initialized",
|
629 |
+
variant: "destructive",
|
630 |
+
});
|
631 |
+
return;
|
632 |
+
}
|
633 |
+
|
634 |
+
await recorderRef.current.exportForLeRobot("zip-download");
|
635 |
+
toast({
|
636 |
+
title: "Download Started",
|
637 |
+
description: "Your dataset is being downloaded as a ZIP file",
|
638 |
+
});
|
639 |
+
};
|
640 |
+
|
641 |
+
const handleUploadToHuggingFace = async () => {
|
642 |
+
if (!recorderRef.current) {
|
643 |
+
toast({
|
644 |
+
title: "Upload Error",
|
645 |
+
description: "Recorder not initialized",
|
646 |
+
variant: "destructive",
|
647 |
+
});
|
648 |
+
return;
|
649 |
+
}
|
650 |
+
|
651 |
+
if (!recorderSettings.huggingfaceApiKey) {
|
652 |
+
toast({
|
653 |
+
title: "Upload Error",
|
654 |
+
description: "Please enter your Hugging Face API key in Configure",
|
655 |
+
variant: "destructive",
|
656 |
+
});
|
657 |
+
return;
|
658 |
+
}
|
659 |
+
|
660 |
+
try {
|
661 |
+
toast({
|
662 |
+
title: "Upload Started",
|
663 |
+
description: "Uploading dataset to Hugging Face...",
|
664 |
+
});
|
665 |
+
|
666 |
+
// Generate a unique repository name
|
667 |
+
const repoName = `lerobot-recording-${Date.now()}`;
|
668 |
+
|
669 |
+
const uploader = await recorderRef.current.exportForLeRobot(
|
670 |
+
"huggingface",
|
671 |
+
{
|
672 |
+
repoName,
|
673 |
+
accessToken: recorderSettings.huggingfaceApiKey,
|
674 |
+
}
|
675 |
+
);
|
676 |
+
|
677 |
+
uploader.addEventListener("progress", (event: Event) => {
|
678 |
+
console.log(event);
|
679 |
+
});
|
680 |
+
} catch (error) {
|
681 |
+
const errorMessage =
|
682 |
+
error instanceof Error
|
683 |
+
? error.message
|
684 |
+
: "Failed to upload to Hugging Face";
|
685 |
+
toast({
|
686 |
+
title: "Upload Error",
|
687 |
+
description: errorMessage,
|
688 |
+
variant: "destructive",
|
689 |
+
});
|
690 |
+
}
|
691 |
+
};
|
692 |
+
|
693 |
+
// Helper function to format duration
|
694 |
+
const formatDuration = (seconds: number): string => {
|
695 |
+
const mins = Math.floor(seconds / 60);
|
696 |
+
const secs = Math.floor(seconds % 60);
|
697 |
+
return `${mins.toString().padStart(2, "0")}:${secs
|
698 |
+
.toString()
|
699 |
+
.padStart(2, "0")}`;
|
700 |
+
};
|
701 |
+
|
702 |
+
return (
|
703 |
+
<Card className="border-0 rounded-none mt-6">
|
704 |
+
<div className="p-4 border-b border-white/10">
|
705 |
+
<div className="flex items-center justify-between">
|
706 |
+
<div className="flex items-center gap-4">
|
707 |
+
<div className="w-1 h-8 bg-primary"></div>
|
708 |
+
<div>
|
709 |
+
<h3 className="text-xl font-bold text-foreground font-mono tracking-wider uppercase">
|
710 |
+
robot movement recorder
|
711 |
+
</h3>
|
712 |
+
<p className="text-sm text-muted-foreground font-mono">
|
713 |
+
dataset <span className="text-muted-foreground">recording</span>{" "}
|
714 |
+
interface
|
715 |
+
</p>
|
716 |
+
</div>
|
717 |
+
</div>
|
718 |
+
<div className="flex items-center gap-6">
|
719 |
+
<div className="border-l border-white/10 pl-6 flex items-center gap-4">
|
720 |
+
<Button
|
721 |
+
variant="outline"
|
722 |
+
size="lg"
|
723 |
+
className="gap-2"
|
724 |
+
onClick={() => setShowConfigure(!showConfigure)}
|
725 |
+
>
|
726 |
+
<Settings className="w-5 h-5" />
|
727 |
+
Configure
|
728 |
+
</Button>
|
729 |
+
|
730 |
+
<Button
|
731 |
+
variant={isRecording ? "destructive" : "default"}
|
732 |
+
size="lg"
|
733 |
+
className="gap-2"
|
734 |
+
onClick={
|
735 |
+
isRecording ? handleStopRecording : handleStartRecording
|
736 |
+
}
|
737 |
+
>
|
738 |
+
{isRecording ? (
|
739 |
+
<>
|
740 |
+
<Square className="w-5 h-5" />
|
741 |
+
Stop Recording
|
742 |
+
</>
|
743 |
+
) : (
|
744 |
+
<>
|
745 |
+
<Record className="w-5 h-5" />
|
746 |
+
Start Recording
|
747 |
+
</>
|
748 |
+
)}
|
749 |
+
</Button>
|
750 |
+
</div>
|
751 |
+
</div>
|
752 |
+
</div>
|
753 |
+
</div>
|
754 |
+
|
755 |
+
<div className="p-6 space-y-6">
|
756 |
+
{/* Recorder Settings - Toggleable Inline */}
|
757 |
+
{showConfigure && (
|
758 |
+
<div className="space-y-6">
|
759 |
+
{/* Hugging Face Settings */}
|
760 |
+
<div className="space-y-3">
|
761 |
+
<h3 className="text-lg font-semibold text-foreground">
|
762 |
+
Settings
|
763 |
+
</h3>
|
764 |
+
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
|
765 |
+
<div className="space-y-2">
|
766 |
+
<label className="text-sm text-muted-foreground">
|
767 |
+
Hugging Face API Key
|
768 |
+
</label>
|
769 |
+
<Input
|
770 |
+
placeholder="Enter your Hugging Face API key"
|
771 |
+
value={recorderSettings.huggingfaceApiKey}
|
772 |
+
onChange={(e) => {
|
773 |
+
const newSettings = {
|
774 |
+
...recorderSettings,
|
775 |
+
huggingfaceApiKey: e.target.value,
|
776 |
+
};
|
777 |
+
setRecorderSettings(newSettings);
|
778 |
+
saveRecorderSettings(newSettings);
|
779 |
+
}}
|
780 |
+
type="password"
|
781 |
+
className="bg-black/20 border-white/10"
|
782 |
+
/>
|
783 |
+
<p className="text-xs text-white/50">
|
784 |
+
Required to upload datasets to Hugging Face Hub
|
785 |
+
</p>
|
786 |
+
</div>
|
787 |
+
</div>
|
788 |
+
</div>
|
789 |
+
|
790 |
+
{/* Camera Configuration */}
|
791 |
+
<div className="space-y-4">
|
792 |
+
<h3 className="text-lg font-semibold text-foreground">
|
793 |
+
Camera Setup
|
794 |
+
</h3>
|
795 |
+
<div className="bg-black/40 border border-white/20 p-6">
|
796 |
+
<div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
797 |
+
{/* Left Column: Camera Selection & Adding */}
|
798 |
+
<div className="space-y-4">
|
799 |
+
{/* Camera Selection and Refresh */}
|
800 |
+
|
801 |
+
{/* Camera Access Request */}
|
802 |
+
{cameraPermissionState === "unknown" && (
|
803 |
+
<div className="space-y-3">
|
804 |
+
<div className="bg-black/60 border border-white/20 rounded-lg p-4 text-center">
|
805 |
+
<Camera className="w-8 h-8 mx-auto mb-2 opacity-50" />
|
806 |
+
<p className="text-sm text-white/70 mb-3">
|
807 |
+
Camera access needed to configure cameras
|
808 |
+
</p>
|
809 |
+
<Button
|
810 |
+
onClick={() => loadAvailableCameras(false)}
|
811 |
+
variant="outline"
|
812 |
+
className="gap-2"
|
813 |
+
disabled={isLoadingCameras}
|
814 |
+
>
|
815 |
+
<Camera className="w-4 h-4" />
|
816 |
+
{isLoadingCameras
|
817 |
+
? "Loading..."
|
818 |
+
: "Request Camera Access"}
|
819 |
+
</Button>
|
820 |
+
</div>
|
821 |
+
</div>
|
822 |
+
)}
|
823 |
+
|
824 |
+
{/* Camera Access Denied */}
|
825 |
+
{cameraPermissionState === "denied" && (
|
826 |
+
<div className="space-y-3">
|
827 |
+
<div className="bg-red-900/20 border border-red-500/20 rounded-lg p-4 text-center">
|
828 |
+
<Camera className="w-8 h-8 mx-auto mb-2 opacity-50 text-red-400" />
|
829 |
+
<p className="text-sm text-red-300 mb-1">
|
830 |
+
Camera access denied
|
831 |
+
</p>
|
832 |
+
<p className="text-xs text-red-400">
|
833 |
+
Please allow camera access in your browser settings
|
834 |
+
and refresh
|
835 |
+
</p>
|
836 |
+
</div>
|
837 |
+
</div>
|
838 |
+
)}
|
839 |
+
|
840 |
+
{/* Camera List with Refresh Button */}
|
841 |
+
{cameraPermissionState === "granted" &&
|
842 |
+
availableCameras.length > 0 && (
|
843 |
+
<div className="space-y-2">
|
844 |
+
<label className="text-sm text-white/70">
|
845 |
+
Select Camera:
|
846 |
+
</label>
|
847 |
+
<div className="flex items-center gap-2">
|
848 |
+
<Select
|
849 |
+
value={selectedCameraId}
|
850 |
+
onValueChange={switchCameraPreview}
|
851 |
+
disabled={hasRecordedFrames}
|
852 |
+
>
|
853 |
+
<SelectTrigger className="flex-1 bg-black/20 border-white/10">
|
854 |
+
<SelectValue placeholder="Choose a camera" />
|
855 |
+
</SelectTrigger>
|
856 |
+
<SelectContent>
|
857 |
+
{availableCameras.map((camera) => (
|
858 |
+
<SelectItem
|
859 |
+
key={camera.deviceId}
|
860 |
+
value={camera.deviceId}
|
861 |
+
>
|
862 |
+
{camera.label ||
|
863 |
+
`Camera ${camera.deviceId.slice(
|
864 |
+
0,
|
865 |
+
8
|
866 |
+
)}...`}
|
867 |
+
</SelectItem>
|
868 |
+
))}
|
869 |
+
</SelectContent>
|
870 |
+
</Select>
|
871 |
+
<Button
|
872 |
+
onClick={() => loadAvailableCameras(false)}
|
873 |
+
variant="ghost"
|
874 |
+
size="sm"
|
875 |
+
className="gap-2 text-white/70 hover:text-white"
|
876 |
+
disabled={isLoadingCameras}
|
877 |
+
>
|
878 |
+
<RefreshCw
|
879 |
+
className={`w-4 h-4 ${
|
880 |
+
isLoadingCameras ? "animate-spin" : ""
|
881 |
+
}`}
|
882 |
+
/>
|
883 |
+
Refresh
|
884 |
+
</Button>
|
885 |
+
</div>
|
886 |
+
</div>
|
887 |
+
)}
|
888 |
+
|
889 |
+
{/* Camera Name Input */}
|
890 |
+
{selectedCameraId && (
|
891 |
+
<div className="space-y-2">
|
892 |
+
<label className="text-sm text-white/70">
|
893 |
+
Camera Name:
|
894 |
+
</label>
|
895 |
+
<Input
|
896 |
+
placeholder="e.g., 'Overhead View', 'Side Angle', 'Close-up'"
|
897 |
+
value={cameraName}
|
898 |
+
onChange={(e) => setCameraName(e.target.value)}
|
899 |
+
className="bg-black/20 border-white/10"
|
900 |
+
disabled={hasRecordedFrames}
|
901 |
+
/>
|
902 |
+
<p className="text-xs text-white/50">
|
903 |
+
Give this camera a descriptive name for your recording
|
904 |
+
setup
|
905 |
+
</p>
|
906 |
+
</div>
|
907 |
+
)}
|
908 |
+
|
909 |
+
{/* Add Camera Button */}
|
910 |
+
{selectedCameraId && (
|
911 |
+
<div className="flex justify-end">
|
912 |
+
<Button
|
913 |
+
onClick={handleAddCamera}
|
914 |
+
className="gap-2"
|
915 |
+
disabled={
|
916 |
+
hasRecordedFrames ||
|
917 |
+
!cameraName.trim() ||
|
918 |
+
!selectedCameraId ||
|
919 |
+
!previewStream
|
920 |
+
}
|
921 |
+
>
|
922 |
+
<PlusCircle className="w-4 h-4" />
|
923 |
+
Add Camera to Recorder
|
924 |
+
</Button>
|
925 |
+
</div>
|
926 |
+
)}
|
927 |
+
</div>
|
928 |
+
|
929 |
+
{/* Right Column: Camera Preview */}
|
930 |
+
<div className="space-y-4">
|
931 |
+
<div className="aspect-video bg-black/60 border border-white/20 rounded-lg overflow-hidden">
|
932 |
+
{previewStream ? (
|
933 |
+
<video
|
934 |
+
ref={videoRef}
|
935 |
+
autoPlay
|
936 |
+
muted
|
937 |
+
playsInline
|
938 |
+
className="w-full h-full object-cover"
|
939 |
+
/>
|
940 |
+
) : (
|
941 |
+
<div className="w-full h-full flex items-center justify-center text-white/60">
|
942 |
+
<div className="text-center">
|
943 |
+
<Camera className="w-12 h-12 mx-auto mb-2 opacity-50" />
|
944 |
+
{cameraPermissionState === "unknown" ? (
|
945 |
+
<p className="text-sm">
|
946 |
+
Request camera access to preview
|
947 |
+
</p>
|
948 |
+
) : cameraPermissionState === "denied" ? (
|
949 |
+
<p className="text-sm">Camera access denied</p>
|
950 |
+
) : availableCameras.length === 0 ? (
|
951 |
+
<p className="text-sm">No cameras available</p>
|
952 |
+
) : !selectedCameraId ? (
|
953 |
+
<p className="text-sm">
|
954 |
+
Select a camera to preview
|
955 |
+
</p>
|
956 |
+
) : (
|
957 |
+
<p className="text-sm">Loading preview...</p>
|
958 |
+
)}
|
959 |
+
</div>
|
960 |
+
</div>
|
961 |
+
)}
|
962 |
+
</div>
|
963 |
+
</div>
|
964 |
+
</div>
|
965 |
+
</div>
|
966 |
+
</div>
|
967 |
+
</div>
|
968 |
+
)}
|
969 |
+
|
970 |
+
{/* Added Camera Previews */}
|
971 |
+
{Object.keys(additionalCameras).length > 0 && (
|
972 |
+
<div className="space-y-4">
|
973 |
+
<h3 className="text-lg font-semibold text-foreground">
|
974 |
+
Active Cameras
|
975 |
+
</h3>
|
976 |
+
<div className="grid grid-cols-2 md:grid-cols-3 lg:grid-cols-4 gap-4">
|
977 |
+
{Object.entries(additionalCameras).map(([cameraName, stream]) => (
|
978 |
+
<div
|
979 |
+
key={cameraName}
|
980 |
+
className="bg-black/40 border border-white/20 rounded-lg p-3 space-y-2"
|
981 |
+
>
|
982 |
+
<div className="aspect-video bg-black/60 border border-white/10 rounded overflow-hidden">
|
983 |
+
<video
|
984 |
+
autoPlay
|
985 |
+
muted
|
986 |
+
playsInline
|
987 |
+
className="w-full h-full object-cover"
|
988 |
+
ref={(video) => {
|
989 |
+
if (video && stream) {
|
990 |
+
video.srcObject = stream;
|
991 |
+
}
|
992 |
+
}}
|
993 |
+
/>
|
994 |
+
</div>
|
995 |
+
<div className="flex items-center justify-between">
|
996 |
+
{editingCameraName === cameraName ? (
|
997 |
+
<div className="flex items-center gap-1 flex-1">
|
998 |
+
<Input
|
999 |
+
value={editingCameraNewName}
|
1000 |
+
onChange={(e) =>
|
1001 |
+
setEditingCameraNewName(e.target.value)
|
1002 |
+
}
|
1003 |
+
className="text-xs h-6 bg-black/20 border-white/10"
|
1004 |
+
onKeyDown={(e) => {
|
1005 |
+
if (e.key === "Enter") {
|
1006 |
+
handleConfirmCameraNameEdit(cameraName);
|
1007 |
+
} else if (e.key === "Escape") {
|
1008 |
+
handleCancelCameraNameEdit();
|
1009 |
+
}
|
1010 |
+
}}
|
1011 |
+
autoFocus
|
1012 |
+
/>
|
1013 |
+
<button
|
1014 |
+
onClick={() =>
|
1015 |
+
handleConfirmCameraNameEdit(cameraName)
|
1016 |
+
}
|
1017 |
+
className="text-green-400 hover:text-green-300 p-1"
|
1018 |
+
>
|
1019 |
+
<Check className="w-3 h-3" />
|
1020 |
+
</button>
|
1021 |
+
<button
|
1022 |
+
onClick={handleCancelCameraNameEdit}
|
1023 |
+
className="text-red-400 hover:text-red-300 p-1"
|
1024 |
+
>
|
1025 |
+
<X className="w-3 h-3" />
|
1026 |
+
</button>
|
1027 |
+
</div>
|
1028 |
+
) : (
|
1029 |
+
<button
|
1030 |
+
onClick={() => handleStartEditingCameraName(cameraName)}
|
1031 |
+
className="text-sm font-medium text-white/90 truncate hover:text-white cursor-pointer flex items-center gap-1 flex-1"
|
1032 |
+
disabled={hasRecordedFrames}
|
1033 |
+
>
|
1034 |
+
{cameraName}
|
1035 |
+
<Edit2 className="w-3 h-3 opacity-50" />
|
1036 |
+
</button>
|
1037 |
+
)}
|
1038 |
+
<button
|
1039 |
+
onClick={() => handleRemoveCamera(cameraName)}
|
1040 |
+
className="text-red-400 hover:text-red-300 p-1 ml-2"
|
1041 |
+
disabled={hasRecordedFrames}
|
1042 |
+
title="Remove camera"
|
1043 |
+
>
|
1044 |
+
<X className="w-4 h-4" />
|
1045 |
+
</button>
|
1046 |
+
</div>
|
1047 |
+
</div>
|
1048 |
+
))}
|
1049 |
+
</div>
|
1050 |
+
</div>
|
1051 |
+
)}
|
1052 |
+
|
1053 |
+
{/* Episode Management & Dataset Actions */}
|
1054 |
+
<div className="flex justify-between items-center">
|
1055 |
+
<div className="flex items-center gap-2">
|
1056 |
+
<Button
|
1057 |
+
variant="outline"
|
1058 |
+
className="gap-2"
|
1059 |
+
onClick={handleResetFrames}
|
1060 |
+
disabled={isRecording || !hasRecordedFrames}
|
1061 |
+
>
|
1062 |
+
<Trash2 className="w-4 h-4" />
|
1063 |
+
Reset Frames
|
1064 |
+
</Button>
|
1065 |
+
<Button
|
1066 |
+
variant="outline"
|
1067 |
+
className="gap-2"
|
1068 |
+
onClick={handleNextEpisode}
|
1069 |
+
disabled={isRecording}
|
1070 |
+
>
|
1071 |
+
<PlusCircle className="w-4 h-4" />
|
1072 |
+
Next Episode
|
1073 |
+
</Button>
|
1074 |
+
</div>
|
1075 |
+
|
1076 |
+
<div className="flex items-center gap-2">
|
1077 |
+
<Button
|
1078 |
+
variant="outline"
|
1079 |
+
className="gap-2"
|
1080 |
+
onClick={handleDownloadZip}
|
1081 |
+
disabled={recorderRef.current?.teleoperatorData.length === 0 || isRecording}
|
1082 |
+
>
|
1083 |
+
<Download className="w-4 h-4" />
|
1084 |
+
Download as ZIP
|
1085 |
+
</Button>
|
1086 |
+
<Button
|
1087 |
+
variant="outline"
|
1088 |
+
className="gap-2"
|
1089 |
+
onClick={handleUploadToHuggingFace}
|
1090 |
+
disabled={
|
1091 |
+
recorderRef.current?.teleoperatorData.length === 0 ||
|
1092 |
+
isRecording ||
|
1093 |
+
!recorderSettings.huggingfaceApiKey
|
1094 |
+
}
|
1095 |
+
>
|
1096 |
+
<Upload className="w-4 h-4" />
|
1097 |
+
Upload to Hugging Face
|
1098 |
+
</Button>
|
1099 |
+
</div>
|
1100 |
+
</div>
|
1101 |
+
|
1102 |
+
<div className="border border-white/10 rounded-md overflow-hidden">
|
1103 |
+
<TeleoperatorEpisodesView teleoperatorData={recorderRef.current?.teleoperatorData} />
|
1104 |
+
</div>
|
1105 |
+
|
1106 |
+
{/* Camera Configuration */}
|
1107 |
+
<div className="border-t border-white/10 pt-4 mt-4">
|
1108 |
+
<div className="flex items-center justify-between mb-4">
|
1109 |
+
<h3 className="text-lg font-semibold">Camera Setup</h3>
|
1110 |
+
<Button
|
1111 |
+
onClick={() => setShowCameraConfig(!showCameraConfig)}
|
1112 |
+
variant="outline"
|
1113 |
+
size="sm"
|
1114 |
+
className="gap-2"
|
1115 |
+
>
|
1116 |
+
<Camera className="w-4 h-4" />
|
1117 |
+
{showCameraConfig ? "Hide Camera Config" : "Configure Cameras"}
|
1118 |
+
</Button>
|
1119 |
+
</div>
|
1120 |
+
|
1121 |
+
{showCameraConfig && (
|
1122 |
+
<div className="bg-black/40 border border-white/20 rounded-lg p-6 mb-4">
|
1123 |
+
<div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
1124 |
+
{/* Left Column: Camera Preview & Selection */}
|
1125 |
+
<div className="space-y-4">
|
1126 |
+
<h4 className="text-md font-semibold text-white/90">
|
1127 |
+
Camera Preview
|
1128 |
+
</h4>
|
1129 |
+
|
1130 |
+
{/* Camera Preview */}
|
1131 |
+
<div className="aspect-video bg-black/60 border border-white/20 rounded-lg overflow-hidden">
|
1132 |
+
{previewStream ? (
|
1133 |
+
<video
|
1134 |
+
ref={videoRef}
|
1135 |
+
autoPlay
|
1136 |
+
muted
|
1137 |
+
playsInline
|
1138 |
+
className="w-full h-full object-cover"
|
1139 |
+
/>
|
1140 |
+
) : (
|
1141 |
+
<div className="w-full h-full flex items-center justify-center text-white/60">
|
1142 |
+
<div className="text-center">
|
1143 |
+
<Camera className="w-12 h-12 mx-auto mb-2 opacity-50" />
|
1144 |
+
{isLoadingCameras ? (
|
1145 |
+
<p>Loading cameras...</p>
|
1146 |
+
) : cameraPermissionState === "denied" ? (
|
1147 |
+
<div>
|
1148 |
+
<p>Camera access denied</p>
|
1149 |
+
<p className="text-xs mt-1">
|
1150 |
+
Please allow camera access and refresh
|
1151 |
+
</p>
|
1152 |
+
</div>
|
1153 |
+
) : availableCameras.length === 0 ? (
|
1154 |
+
<p>Click "Load Cameras" to start</p>
|
1155 |
+
) : (
|
1156 |
+
<p>Select a camera to preview</p>
|
1157 |
+
)}
|
1158 |
+
</div>
|
1159 |
+
</div>
|
1160 |
+
)}
|
1161 |
+
</div>
|
1162 |
+
|
1163 |
+
{/* Camera Controls */}
|
1164 |
+
<div className="space-y-3">
|
1165 |
+
<Button
|
1166 |
+
onClick={() => loadAvailableCameras(false)}
|
1167 |
+
variant="outline"
|
1168 |
+
className="w-full gap-2"
|
1169 |
+
disabled={hasRecordedFrames || isLoadingCameras}
|
1170 |
+
>
|
1171 |
+
<Camera className="w-4 h-4" />
|
1172 |
+
{isLoadingCameras
|
1173 |
+
? "Loading..."
|
1174 |
+
: availableCameras.length > 0
|
1175 |
+
? "Refresh Cameras"
|
1176 |
+
: "Load Cameras"}
|
1177 |
+
</Button>
|
1178 |
+
|
1179 |
+
{availableCameras.length > 0 && (
|
1180 |
+
<div className="space-y-2">
|
1181 |
+
<label className="text-sm text-white/70">
|
1182 |
+
Select Camera:
|
1183 |
+
</label>
|
1184 |
+
<Select
|
1185 |
+
value={selectedCameraId}
|
1186 |
+
onValueChange={switchCameraPreview}
|
1187 |
+
disabled={hasRecordedFrames}
|
1188 |
+
>
|
1189 |
+
<SelectTrigger className="w-full bg-black/20 border-white/10">
|
1190 |
+
<SelectValue placeholder="Choose a camera" />
|
1191 |
+
</SelectTrigger>
|
1192 |
+
<SelectContent>
|
1193 |
+
{availableCameras.map((camera) => (
|
1194 |
+
<SelectItem
|
1195 |
+
key={camera.deviceId}
|
1196 |
+
value={camera.deviceId}
|
1197 |
+
>
|
1198 |
+
{camera.label ||
|
1199 |
+
`Camera ${camera.deviceId.slice(0, 8)}...`}
|
1200 |
+
</SelectItem>
|
1201 |
+
))}
|
1202 |
+
</SelectContent>
|
1203 |
+
</Select>
|
1204 |
+
</div>
|
1205 |
+
)}
|
1206 |
+
</div>
|
1207 |
+
</div>
|
1208 |
+
|
1209 |
+
{/* Right Column: Camera Naming & Adding */}
|
1210 |
+
<div className="space-y-4">
|
1211 |
+
<h4 className="text-md font-semibold text-white/90">
|
1212 |
+
Add to Recorder
|
1213 |
+
</h4>
|
1214 |
+
|
1215 |
+
<div className="space-y-4">
|
1216 |
+
<div className="space-y-2">
|
1217 |
+
<label className="text-sm text-white/70">
|
1218 |
+
Camera Name:
|
1219 |
+
</label>
|
1220 |
+
<Input
|
1221 |
+
placeholder="e.g., 'Overhead View', 'Side Angle', 'Close-up'"
|
1222 |
+
value={cameraName}
|
1223 |
+
onChange={(e) => setCameraName(e.target.value)}
|
1224 |
+
className="bg-black/20 border-white/10"
|
1225 |
+
disabled={hasRecordedFrames}
|
1226 |
+
/>
|
1227 |
+
<p className="text-xs text-white/50">
|
1228 |
+
Give this camera a descriptive name for your recording
|
1229 |
+
setup
|
1230 |
+
</p>
|
1231 |
+
</div>
|
1232 |
+
|
1233 |
+
<Button
|
1234 |
+
onClick={handleAddCamera}
|
1235 |
+
className="w-full gap-2"
|
1236 |
+
disabled={
|
1237 |
+
hasRecordedFrames ||
|
1238 |
+
!cameraName.trim() ||
|
1239 |
+
!selectedCameraId ||
|
1240 |
+
!previewStream
|
1241 |
+
}
|
1242 |
+
>
|
1243 |
+
<PlusCircle className="w-4 h-4" />
|
1244 |
+
Add Camera to Recorder
|
1245 |
+
</Button>
|
1246 |
+
</div>
|
1247 |
+
</div>
|
1248 |
+
</div>
|
1249 |
+
</div>
|
1250 |
+
)}
|
1251 |
+
|
1252 |
+
{/* Display added cameras */}
|
1253 |
+
{Object.keys(additionalCameras).length > 0 && (
|
1254 |
+
<div className="mb-4 space-y-2">
|
1255 |
+
<p className="text-sm text-muted-foreground">Added Cameras:</p>
|
1256 |
+
<div className="flex flex-wrap gap-2">
|
1257 |
+
{Object.keys(additionalCameras).map((name) => (
|
1258 |
+
<Badge
|
1259 |
+
key={name}
|
1260 |
+
variant="secondary"
|
1261 |
+
className="flex items-center gap-1 py-1 px-2"
|
1262 |
+
>
|
1263 |
+
{name}
|
1264 |
+
<button
|
1265 |
+
onClick={() => handleRemoveCamera(name)}
|
1266 |
+
className="ml-1 text-muted-foreground hover:text-destructive"
|
1267 |
+
disabled={hasRecordedFrames}
|
1268 |
+
>
|
1269 |
+
×
|
1270 |
+
</button>
|
1271 |
+
</Badge>
|
1272 |
+
))}
|
1273 |
+
</div>
|
1274 |
+
</div>
|
1275 |
+
)}
|
1276 |
+
</div>
|
1277 |
+
|
1278 |
+
<div className="flex items-center gap-4">
|
1279 |
+
<div className="flex items-center gap-2">
|
1280 |
+
<Button
|
1281 |
+
variant="outline"
|
1282 |
+
className="gap-2"
|
1283 |
+
onClick={handleDownloadZip}
|
1284 |
+
disabled={recorderRef.current?.teleoperatorData.length === 0 || isRecording}
|
1285 |
+
>
|
1286 |
+
<Download className="w-4 h-4" />
|
1287 |
+
Download as ZIP
|
1288 |
+
</Button>
|
1289 |
+
|
1290 |
+
{/* Reset Frames button moved to top bar */}
|
1291 |
+
</div>
|
1292 |
+
|
1293 |
+
<div className="flex items-center gap-2 flex-1">
|
1294 |
+
<Button
|
1295 |
+
variant="outline"
|
1296 |
+
className="gap-2"
|
1297 |
+
onClick={handleUploadToHuggingFace}
|
1298 |
+
disabled={
|
1299 |
+
recorderRef.current?.teleoperatorData.length === 0 || isRecording || !huggingfaceApiKey
|
1300 |
+
}
|
1301 |
+
>
|
1302 |
+
<Upload className="w-4 h-4" />
|
1303 |
+
Upload to HuggingFace
|
1304 |
+
</Button>
|
1305 |
+
|
1306 |
+
<Input
|
1307 |
+
placeholder="HuggingFace API Key"
|
1308 |
+
value={huggingfaceApiKey}
|
1309 |
+
onChange={(e) => setHuggingfaceApiKey(e.target.value)}
|
1310 |
+
className="flex-1 bg-black/20 border-white/10"
|
1311 |
+
type="password"
|
1312 |
+
/>
|
1313 |
+
</div>
|
1314 |
+
</div>
|
1315 |
+
</div>
|
1316 |
+
</Card>
|
1317 |
+
);
|
1318 |
+
}
|
@@ -1,6 +1,6 @@
|
|
1 |
"use client";
|
2 |
import { useState, useEffect, useMemo, useRef, useCallback } from "react";
|
3 |
-
import { Power, PowerOff, Keyboard } from "lucide-react";
|
4 |
import { Button } from "@/components/ui/button";
|
5 |
import { Card } from "@/components/ui/card";
|
6 |
import { Badge } from "@/components/ui/badge";
|
@@ -22,6 +22,11 @@ import {
|
|
22 |
} from "@lerobot/web";
|
23 |
import { getUnifiedRobotData } from "@/lib/unified-storage";
|
24 |
import VirtualKey from "@/components/VirtualKey";
|
|
|
|
|
|
|
|
|
|
|
25 |
|
26 |
interface TeleoperationViewProps {
|
27 |
robot: RobotConnection;
|
@@ -104,7 +109,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
104 |
}, [robot.serialNumber]);
|
105 |
|
106 |
// Lazy initialization function - only connects when user wants to start
|
107 |
-
const initializeTeleoperation = async () => {
|
108 |
if (!robot || !robot.robotType) {
|
109 |
return false;
|
110 |
}
|
@@ -130,15 +135,18 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
130 |
type: "direct",
|
131 |
},
|
132 |
calibrationData,
|
|
|
|
|
|
|
133 |
};
|
134 |
const directProcess = await teleoperate(directConfig);
|
135 |
|
136 |
keyboardProcessRef.current = keyboardProcess;
|
137 |
directProcessRef.current = directProcess;
|
138 |
-
setTeleopState(
|
139 |
|
140 |
// Initialize local motor positions from hardware state
|
141 |
-
const initialState =
|
142 |
const initialPositions: {
|
143 |
[motorName: string]: { position: number; timestamp: number };
|
144 |
} = {};
|
@@ -165,7 +173,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
165 |
});
|
166 |
return false;
|
167 |
}
|
168 |
-
};
|
169 |
|
170 |
// Cleanup on unmount
|
171 |
useEffect(() => {
|
@@ -247,7 +255,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
247 |
if (!success) return;
|
248 |
}
|
249 |
|
250 |
-
if (!keyboardProcessRef.current ||
|
251 |
toast({
|
252 |
title: "Teleoperation Error",
|
253 |
description: "Teleoperation not initialized",
|
@@ -257,8 +265,8 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
257 |
}
|
258 |
|
259 |
try {
|
260 |
-
keyboardProcessRef.current
|
261 |
-
directProcessRef.current
|
262 |
} catch (error) {
|
263 |
const errorMessage =
|
264 |
error instanceof Error
|
@@ -349,12 +357,20 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
349 |
const realMotorConfigs = teleopState?.motorConfigs || [];
|
350 |
const now = Date.now();
|
351 |
|
352 |
-
// If we have real motor configs, use them with local position overrides when
|
353 |
if (realMotorConfigs.length > 0) {
|
354 |
return realMotorConfigs.map((motor) => {
|
355 |
const localData = localMotorPositions[motor.name];
|
356 |
-
|
357 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
358 |
return {
|
359 |
...motor,
|
360 |
currentPosition: useLocalPosition
|
@@ -368,7 +384,13 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
368 |
return DEFAULT_MOTOR_CONFIGS.map((motor) => {
|
369 |
const calibratedMotor = calibrationData?.[motor.name];
|
370 |
const localData = localMotorPositions[motor.name];
|
371 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
372 |
|
373 |
return {
|
374 |
...motor,
|
@@ -392,332 +414,358 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
392 |
const keyStates = teleopState?.keyStates || {};
|
393 |
const controls = SO100_KEYBOARD_CONTROLS;
|
394 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
395 |
return (
|
396 |
-
|
397 |
-
<
|
398 |
-
<div className="
|
399 |
-
<div className="flex items-center
|
400 |
-
<div className="
|
401 |
-
|
402 |
-
<
|
403 |
-
|
404 |
-
|
405 |
-
|
406 |
-
|
407 |
-
|
408 |
-
|
409 |
-
|
410 |
-
|
411 |
-
</div>
|
412 |
-
<div className="flex items-center gap-6">
|
413 |
-
<div className="border-l border-white/10 pl-6 flex items-center gap-4">
|
414 |
-
{teleopState?.isActive ? (
|
415 |
-
<Button onClick={handleStop} variant="destructive" size="lg">
|
416 |
-
<PowerOff className="w-5 h-5 mr-2" /> Stop Control
|
417 |
-
</Button>
|
418 |
-
) : (
|
419 |
-
<Button
|
420 |
-
onClick={handleStart}
|
421 |
-
size="lg"
|
422 |
-
disabled={!robot.isConnected}
|
423 |
-
>
|
424 |
-
<Power className="w-5 h-5 mr-2" /> Control Robot
|
425 |
-
</Button>
|
426 |
-
)}
|
427 |
-
<div className="flex items-center gap-2">
|
428 |
-
<span className="text-sm font-mono text-muted-foreground uppercase">
|
429 |
-
status:
|
430 |
-
</span>
|
431 |
-
<Badge
|
432 |
-
variant="outline"
|
433 |
-
className={cn(
|
434 |
-
"border-primary/50 bg-primary/20 text-primary font-mono text-xs",
|
435 |
-
teleopState?.isActive && "animate-pulse-slow"
|
436 |
-
)}
|
437 |
-
>
|
438 |
-
{teleopState?.isActive ? "ACTIVE" : "STOPPED"}
|
439 |
-
</Badge>
|
440 |
</div>
|
441 |
</div>
|
442 |
-
|
443 |
-
|
444 |
-
|
445 |
-
|
446 |
-
|
447 |
-
|
448 |
-
|
449 |
-
|
450 |
-
|
451 |
-
|
452 |
-
|
453 |
-
|
454 |
-
|
455 |
-
|
456 |
-
|
457 |
-
|
458 |
-
|
459 |
-
|
460 |
-
|
461 |
-
|
462 |
-
|
463 |
-
disabled={!teleopState?.isActive}
|
464 |
-
className={!teleopState?.isActive ? "opacity-50" : ""}
|
465 |
-
/>
|
466 |
-
<span
|
467 |
className={cn(
|
468 |
-
"text-
|
469 |
-
teleopState?.isActive
|
470 |
-
? "text-accent"
|
471 |
-
: "text-muted-foreground"
|
472 |
)}
|
473 |
>
|
474 |
-
{
|
475 |
-
</
|
476 |
</div>
|
477 |
</div>
|
478 |
-
|
479 |
</div>
|
480 |
</div>
|
481 |
-
<div>
|
482 |
-
<
|
483 |
-
|
484 |
-
|
485 |
-
|
486 |
-
<div className="
|
487 |
-
|
488 |
-
<
|
489 |
-
label="
|
490 |
-
|
491 |
-
|
492 |
-
|
493 |
-
|
494 |
-
|
495 |
-
|
496 |
-
|
497 |
-
|
498 |
-
|
499 |
-
|
500 |
-
|
501 |
-
|
502 |
-
|
503 |
-
|
504 |
-
|
505 |
-
|
506 |
-
|
507 |
-
|
508 |
-
|
509 |
-
|
510 |
-
|
511 |
-
|
512 |
-
|
513 |
-
simulateKeyRelease(controls.shoulder_pan.negative)
|
514 |
-
}
|
515 |
-
disabled={!teleopState?.isActive}
|
516 |
-
/>
|
517 |
-
<VirtualKey
|
518 |
-
label="↓"
|
519 |
-
subLabel="Lift-"
|
520 |
-
isPressed={
|
521 |
-
!!keyStates[controls.shoulder_lift.negative]?.pressed
|
522 |
-
}
|
523 |
-
onMouseDown={() =>
|
524 |
-
simulateKeyPress(controls.shoulder_lift.negative)
|
525 |
-
}
|
526 |
-
onMouseUp={() =>
|
527 |
-
simulateKeyRelease(controls.shoulder_lift.negative)
|
528 |
-
}
|
529 |
-
disabled={!teleopState?.isActive}
|
530 |
-
/>
|
531 |
-
<VirtualKey
|
532 |
-
label="→"
|
533 |
-
subLabel="Pan+"
|
534 |
-
isPressed={
|
535 |
-
!!keyStates[controls.shoulder_pan.positive]?.pressed
|
536 |
-
}
|
537 |
-
onMouseDown={() =>
|
538 |
-
simulateKeyPress(controls.shoulder_pan.positive)
|
539 |
-
}
|
540 |
-
onMouseUp={() =>
|
541 |
-
simulateKeyRelease(controls.shoulder_pan.positive)
|
542 |
-
}
|
543 |
-
disabled={!teleopState?.isActive}
|
544 |
-
/>
|
545 |
</div>
|
546 |
-
|
547 |
-
|
548 |
-
|
549 |
-
|
550 |
-
|
551 |
-
|
552 |
-
|
553 |
-
|
554 |
-
|
555 |
-
|
556 |
-
onMouseUp={() =>
|
557 |
-
simulateKeyRelease(controls.elbow_flex.positive)
|
558 |
-
}
|
559 |
-
disabled={!teleopState?.isActive}
|
560 |
-
/>
|
561 |
-
<div className="flex gap-2">
|
562 |
-
<VirtualKey
|
563 |
-
label="A"
|
564 |
-
subLabel="Wrist+"
|
565 |
-
isPressed={
|
566 |
-
!!keyStates[controls.wrist_flex.positive]?.pressed
|
567 |
-
}
|
568 |
-
onMouseDown={() =>
|
569 |
-
simulateKeyPress(controls.wrist_flex.positive)
|
570 |
-
}
|
571 |
-
onMouseUp={() =>
|
572 |
-
simulateKeyRelease(controls.wrist_flex.positive)
|
573 |
-
}
|
574 |
-
disabled={!teleopState?.isActive}
|
575 |
-
/>
|
576 |
-
<VirtualKey
|
577 |
-
label="S"
|
578 |
-
subLabel="Elbow-"
|
579 |
-
isPressed={
|
580 |
-
!!keyStates[controls.elbow_flex.negative]?.pressed
|
581 |
-
}
|
582 |
-
onMouseDown={() =>
|
583 |
-
simulateKeyPress(controls.elbow_flex.negative)
|
584 |
-
}
|
585 |
-
onMouseUp={() =>
|
586 |
-
simulateKeyRelease(controls.elbow_flex.negative)
|
587 |
-
}
|
588 |
-
disabled={!teleopState?.isActive}
|
589 |
-
/>
|
590 |
<VirtualKey
|
591 |
-
label="
|
592 |
-
subLabel="
|
593 |
isPressed={
|
594 |
-
!!keyStates[controls.
|
595 |
}
|
596 |
onMouseDown={() =>
|
597 |
-
simulateKeyPress(controls.
|
598 |
}
|
599 |
onMouseUp={() =>
|
600 |
-
simulateKeyRelease(controls.
|
601 |
}
|
602 |
disabled={!teleopState?.isActive}
|
603 |
/>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
604 |
</div>
|
605 |
-
<
|
606 |
-
</div>
|
607 |
-
<div className="flex flex-col items-center gap-2">
|
608 |
-
<div className="flex gap-2">
|
609 |
-
<VirtualKey
|
610 |
-
label="Q"
|
611 |
-
subLabel="Roll+"
|
612 |
-
isPressed={
|
613 |
-
!!keyStates[controls.wrist_roll.positive]?.pressed
|
614 |
-
}
|
615 |
-
onMouseDown={() =>
|
616 |
-
simulateKeyPress(controls.wrist_roll.positive)
|
617 |
-
}
|
618 |
-
onMouseUp={() =>
|
619 |
-
simulateKeyRelease(controls.wrist_roll.positive)
|
620 |
-
}
|
621 |
-
disabled={!teleopState?.isActive}
|
622 |
-
/>
|
623 |
<VirtualKey
|
624 |
-
label="
|
625 |
-
subLabel="
|
626 |
isPressed={
|
627 |
-
!!keyStates[controls.
|
628 |
}
|
629 |
onMouseDown={() =>
|
630 |
-
simulateKeyPress(controls.
|
631 |
}
|
632 |
onMouseUp={() =>
|
633 |
-
simulateKeyRelease(controls.
|
634 |
}
|
635 |
disabled={!teleopState?.isActive}
|
636 |
/>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
637 |
</div>
|
638 |
-
<div className="flex gap-2">
|
639 |
-
<
|
640 |
-
|
641 |
-
|
642 |
-
|
643 |
-
|
644 |
-
|
645 |
-
|
646 |
-
|
647 |
-
|
648 |
-
|
649 |
-
|
650 |
-
|
651 |
-
|
652 |
-
|
653 |
-
|
654 |
-
|
655 |
-
|
656 |
-
|
657 |
-
|
658 |
-
|
659 |
-
|
660 |
-
|
661 |
-
|
662 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
663 |
</div>
|
664 |
-
<span className="font-bold text-sm font-sans">Roll/Grip</span>
|
665 |
</div>
|
666 |
-
|
667 |
-
|
668 |
-
|
669 |
-
|
670 |
-
|
671 |
-
|
672 |
-
|
673 |
-
|
674 |
-
</
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
675 |
</div>
|
676 |
-
<TooltipProvider>
|
677 |
-
<Tooltip>
|
678 |
-
<TooltipTrigger asChild>
|
679 |
-
<div
|
680 |
-
className={cn(
|
681 |
-
"w-10 h-6 border rounded-md flex items-center justify-center font-mono text-xs transition-all",
|
682 |
-
"select-none user-select-none",
|
683 |
-
!teleopState?.isActive &&
|
684 |
-
"opacity-50 cursor-not-allowed",
|
685 |
-
teleopState?.isActive &&
|
686 |
-
"cursor-pointer hover:bg-white/5",
|
687 |
-
keyStates[controls.stop]?.pressed
|
688 |
-
? "bg-destructive text-destructive-foreground border-destructive"
|
689 |
-
: "bg-background"
|
690 |
-
)}
|
691 |
-
onMouseDown={(e) => {
|
692 |
-
e.preventDefault();
|
693 |
-
if (teleopState?.isActive) {
|
694 |
-
simulateKeyPress(controls.stop);
|
695 |
-
}
|
696 |
-
}}
|
697 |
-
onMouseUp={(e) => {
|
698 |
-
e.preventDefault();
|
699 |
-
if (teleopState?.isActive) {
|
700 |
-
simulateKeyRelease(controls.stop);
|
701 |
-
}
|
702 |
-
}}
|
703 |
-
onMouseLeave={(e) => {
|
704 |
-
e.preventDefault();
|
705 |
-
if (teleopState?.isActive) {
|
706 |
-
simulateKeyRelease(controls.stop);
|
707 |
-
}
|
708 |
-
}}
|
709 |
-
>
|
710 |
-
ESC
|
711 |
-
</div>
|
712 |
-
</TooltipTrigger>
|
713 |
-
<TooltipContent>Emergency Stop</TooltipContent>
|
714 |
-
</Tooltip>
|
715 |
-
</TooltipProvider>
|
716 |
</div>
|
717 |
</div>
|
718 |
</div>
|
719 |
</div>
|
720 |
-
</
|
721 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
722 |
);
|
723 |
}
|
|
|
1 |
"use client";
|
2 |
import { useState, useEffect, useMemo, useRef, useCallback } from "react";
|
3 |
+
import { Power, PowerOff, Keyboard, Box } from "lucide-react";
|
4 |
import { Button } from "@/components/ui/button";
|
5 |
import { Card } from "@/components/ui/card";
|
6 |
import { Badge } from "@/components/ui/badge";
|
|
|
22 |
} from "@lerobot/web";
|
23 |
import { getUnifiedRobotData } from "@/lib/unified-storage";
|
24 |
import VirtualKey from "@/components/VirtualKey";
|
25 |
+
import { Recorder } from "@/components/recorder";
|
26 |
+
import { Canvas } from "@react-three/fiber";
|
27 |
+
import { Physics } from "@react-three/cannon";
|
28 |
+
import * as THREE from "three";
|
29 |
+
import { OrbitControls } from "@react-three/drei";
|
30 |
|
31 |
interface TeleoperationViewProps {
|
32 |
robot: RobotConnection;
|
|
|
109 |
}, [robot.serialNumber]);
|
110 |
|
111 |
// Lazy initialization function - only connects when user wants to start
|
112 |
+
const initializeTeleoperation = useCallback(async () => {
|
113 |
if (!robot || !robot.robotType) {
|
114 |
return false;
|
115 |
}
|
|
|
135 |
type: "direct",
|
136 |
},
|
137 |
calibrationData,
|
138 |
+
onStateUpdate: (state: TeleoperationState) => {
|
139 |
+
setTeleopState(state);
|
140 |
+
},
|
141 |
};
|
142 |
const directProcess = await teleoperate(directConfig);
|
143 |
|
144 |
keyboardProcessRef.current = keyboardProcess;
|
145 |
directProcessRef.current = directProcess;
|
146 |
+
setTeleopState(directProcess.getState());
|
147 |
|
148 |
// Initialize local motor positions from hardware state
|
149 |
+
const initialState = directProcess.getState();
|
150 |
const initialPositions: {
|
151 |
[motorName: string]: { position: number; timestamp: number };
|
152 |
} = {};
|
|
|
173 |
});
|
174 |
return false;
|
175 |
}
|
176 |
+
}, [robot, robot.robotType, calibrationData, toast]);
|
177 |
|
178 |
// Cleanup on unmount
|
179 |
useEffect(() => {
|
|
|
255 |
if (!success) return;
|
256 |
}
|
257 |
|
258 |
+
if (!(keyboardProcessRef.current || directProcessRef.current)) {
|
259 |
toast({
|
260 |
title: "Teleoperation Error",
|
261 |
description: "Teleoperation not initialized",
|
|
|
265 |
}
|
266 |
|
267 |
try {
|
268 |
+
keyboardProcessRef.current?.start();
|
269 |
+
directProcessRef.current?.start();
|
270 |
} catch (error) {
|
271 |
const errorMessage =
|
272 |
error instanceof Error
|
|
|
357 |
const realMotorConfigs = teleopState?.motorConfigs || [];
|
358 |
const now = Date.now();
|
359 |
|
360 |
+
// If we have real motor configs, use them with local position overrides when appropriate
|
361 |
if (realMotorConfigs.length > 0) {
|
362 |
return realMotorConfigs.map((motor) => {
|
363 |
const localData = localMotorPositions[motor.name];
|
364 |
+
|
365 |
+
// Use local position if it exists and either:
|
366 |
+
// 1. It's very recent (within 500ms) OR
|
367 |
+
// 2. The hardware position is not yet close to our requested position
|
368 |
+
const isRecent = localData && now - localData.timestamp < 500;
|
369 |
+
const isHardwareNotCaughtUp =
|
370 |
+
localData && Math.abs(motor.currentPosition - localData.position) > 5;
|
371 |
+
const useLocalPosition =
|
372 |
+
localData && (isRecent || isHardwareNotCaughtUp);
|
373 |
+
|
374 |
return {
|
375 |
...motor,
|
376 |
currentPosition: useLocalPosition
|
|
|
384 |
return DEFAULT_MOTOR_CONFIGS.map((motor) => {
|
385 |
const calibratedMotor = calibrationData?.[motor.name];
|
386 |
const localData = localMotorPositions[motor.name];
|
387 |
+
// Use local position if it exists and either:
|
388 |
+
// 1. It's very recent (within 500ms) OR
|
389 |
+
// 2. We don't have a hardware position yet that's close to our requested position
|
390 |
+
const isRecent = localData && now - localData.timestamp < 500;
|
391 |
+
const isHardwareNotCaughtUp =
|
392 |
+
localData && Math.abs(motor.currentPosition - localData.position) > 5;
|
393 |
+
const useLocalPosition = localData && (isRecent || isHardwareNotCaughtUp);
|
394 |
|
395 |
return {
|
396 |
...motor,
|
|
|
414 |
const keyStates = teleopState?.keyStates || {};
|
415 |
const controls = SO100_KEYBOARD_CONTROLS;
|
416 |
|
417 |
+
// Memoize teleoperators array to prevent unnecessary re-renders of the Recorder component
|
418 |
+
const memoizedTeleoperators = useMemo(() => {
|
419 |
+
if (!directProcessRef.current) return [];
|
420 |
+
|
421 |
+
return [
|
422 |
+
directProcessRef.current?.teleoperator,
|
423 |
+
].filter(Boolean);
|
424 |
+
}, [directProcessRef.current]);
|
425 |
+
|
426 |
return (
|
427 |
+
<>
|
428 |
+
<Card className="border-0 rounded-none">
|
429 |
+
<div className="p-4 border-b border-white/10">
|
430 |
+
<div className="flex items-center justify-between">
|
431 |
+
<div className="flex items-center gap-4">
|
432 |
+
<div className="w-1 h-8 bg-primary"></div>
|
433 |
+
<div>
|
434 |
+
<h3 className="text-xl font-bold text-foreground font-mono tracking-wider uppercase">
|
435 |
+
robot control
|
436 |
+
</h3>
|
437 |
+
<p className="text-sm text-muted-foreground font-mono">
|
438 |
+
manual{" "}
|
439 |
+
<span className="text-muted-foreground">teleoperate</span>{" "}
|
440 |
+
interface
|
441 |
+
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
442 |
</div>
|
443 |
</div>
|
444 |
+
<div className="flex items-center gap-6">
|
445 |
+
<div className="border-l border-white/10 pl-6 flex items-center gap-4">
|
446 |
+
{teleopState?.isActive ? (
|
447 |
+
<Button onClick={handleStop} variant="destructive" size="lg">
|
448 |
+
<PowerOff className="w-5 h-5 mr-2" /> Stop Control
|
449 |
+
</Button>
|
450 |
+
) : (
|
451 |
+
<Button
|
452 |
+
onClick={handleStart}
|
453 |
+
size="lg"
|
454 |
+
disabled={!robot.isConnected}
|
455 |
+
>
|
456 |
+
<Power className="w-5 h-5 mr-2" /> Control Robot
|
457 |
+
</Button>
|
458 |
+
)}
|
459 |
+
<div className="flex items-center gap-2">
|
460 |
+
<span className="text-sm font-mono text-muted-foreground uppercase">
|
461 |
+
status:
|
462 |
+
</span>
|
463 |
+
<Badge
|
464 |
+
variant="outline"
|
|
|
|
|
|
|
|
|
465 |
className={cn(
|
466 |
+
"border-primary/50 bg-primary/20 text-primary font-mono text-xs",
|
467 |
+
teleopState?.isActive && "animate-pulse-slow"
|
|
|
|
|
468 |
)}
|
469 |
>
|
470 |
+
{teleopState?.isActive ? "ACTIVE" : "STOPPED"}
|
471 |
+
</Badge>
|
472 |
</div>
|
473 |
</div>
|
474 |
+
</div>
|
475 |
</div>
|
476 |
</div>
|
477 |
+
<div className="pt-6 p-6 grid md:grid-cols-2 gap-8">
|
478 |
+
<div>
|
479 |
+
<h3 className="font-sans font-semibold mb-4 text-xl">
|
480 |
+
Motor Control
|
481 |
+
</h3>
|
482 |
+
<div className="space-y-6">
|
483 |
+
{motorConfigs.map((motor) => (
|
484 |
+
<div key={motor.name}>
|
485 |
+
<label className="text-sm font-mono text-muted-foreground">
|
486 |
+
{motor.name}
|
487 |
+
</label>
|
488 |
+
<div className="flex items-center gap-4">
|
489 |
+
<Slider
|
490 |
+
value={[motor.currentPosition]}
|
491 |
+
min={motor.minPosition}
|
492 |
+
max={motor.maxPosition}
|
493 |
+
step={1}
|
494 |
+
onValueChange={(val) => moveMotor(motor.name, val[0])}
|
495 |
+
disabled={!teleopState?.isActive}
|
496 |
+
className={!teleopState?.isActive ? "opacity-50" : ""}
|
497 |
+
/>
|
498 |
+
<span
|
499 |
+
className={cn(
|
500 |
+
"text-lg font-mono w-16 text-right",
|
501 |
+
teleopState?.isActive
|
502 |
+
? "text-accent"
|
503 |
+
: "text-muted-foreground"
|
504 |
+
)}
|
505 |
+
>
|
506 |
+
{Math.round(motor.currentPosition)}
|
507 |
+
</span>
|
508 |
+
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
509 |
</div>
|
510 |
+
))}
|
511 |
+
</div>
|
512 |
+
</div>
|
513 |
+
<div>
|
514 |
+
<h3 className="font-sans font-semibold mb-4 text-xl">
|
515 |
+
Keyboard Layout & Status
|
516 |
+
</h3>
|
517 |
+
<div className="p-4 bg-black/30 rounded-lg space-y-4">
|
518 |
+
<div className="flex justify-around items-end">
|
519 |
+
<div className="flex flex-col items-center gap-2">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
520 |
<VirtualKey
|
521 |
+
label="↑"
|
522 |
+
subLabel="Lift+"
|
523 |
isPressed={
|
524 |
+
!!keyStates[controls.shoulder_lift.positive]?.pressed
|
525 |
}
|
526 |
onMouseDown={() =>
|
527 |
+
simulateKeyPress(controls.shoulder_lift.positive)
|
528 |
}
|
529 |
onMouseUp={() =>
|
530 |
+
simulateKeyRelease(controls.shoulder_lift.positive)
|
531 |
}
|
532 |
disabled={!teleopState?.isActive}
|
533 |
/>
|
534 |
+
<div className="flex gap-2">
|
535 |
+
<VirtualKey
|
536 |
+
label="←"
|
537 |
+
subLabel="Pan-"
|
538 |
+
isPressed={
|
539 |
+
!!keyStates[controls.shoulder_pan.negative]?.pressed
|
540 |
+
}
|
541 |
+
onMouseDown={() =>
|
542 |
+
simulateKeyPress(controls.shoulder_pan.negative)
|
543 |
+
}
|
544 |
+
onMouseUp={() =>
|
545 |
+
simulateKeyRelease(controls.shoulder_pan.negative)
|
546 |
+
}
|
547 |
+
disabled={!teleopState?.isActive}
|
548 |
+
/>
|
549 |
+
<VirtualKey
|
550 |
+
label="↓"
|
551 |
+
subLabel="Lift-"
|
552 |
+
isPressed={
|
553 |
+
!!keyStates[controls.shoulder_lift.negative]?.pressed
|
554 |
+
}
|
555 |
+
onMouseDown={() =>
|
556 |
+
simulateKeyPress(controls.shoulder_lift.negative)
|
557 |
+
}
|
558 |
+
onMouseUp={() =>
|
559 |
+
simulateKeyRelease(controls.shoulder_lift.negative)
|
560 |
+
}
|
561 |
+
disabled={!teleopState?.isActive}
|
562 |
+
/>
|
563 |
+
<VirtualKey
|
564 |
+
label="→"
|
565 |
+
subLabel="Pan+"
|
566 |
+
isPressed={
|
567 |
+
!!keyStates[controls.shoulder_pan.positive]?.pressed
|
568 |
+
}
|
569 |
+
onMouseDown={() =>
|
570 |
+
simulateKeyPress(controls.shoulder_pan.positive)
|
571 |
+
}
|
572 |
+
onMouseUp={() =>
|
573 |
+
simulateKeyRelease(controls.shoulder_pan.positive)
|
574 |
+
}
|
575 |
+
disabled={!teleopState?.isActive}
|
576 |
+
/>
|
577 |
+
</div>
|
578 |
+
<span className="font-bold text-sm font-sans">Shoulder</span>
|
579 |
</div>
|
580 |
+
<div className="flex flex-col items-center gap-2">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
581 |
<VirtualKey
|
582 |
+
label="W"
|
583 |
+
subLabel="Elbow+"
|
584 |
isPressed={
|
585 |
+
!!keyStates[controls.elbow_flex.positive]?.pressed
|
586 |
}
|
587 |
onMouseDown={() =>
|
588 |
+
simulateKeyPress(controls.elbow_flex.positive)
|
589 |
}
|
590 |
onMouseUp={() =>
|
591 |
+
simulateKeyRelease(controls.elbow_flex.positive)
|
592 |
}
|
593 |
disabled={!teleopState?.isActive}
|
594 |
/>
|
595 |
+
<div className="flex gap-2">
|
596 |
+
<VirtualKey
|
597 |
+
label="A"
|
598 |
+
subLabel="Wrist+"
|
599 |
+
isPressed={
|
600 |
+
!!keyStates[controls.wrist_flex.positive]?.pressed
|
601 |
+
}
|
602 |
+
onMouseDown={() =>
|
603 |
+
simulateKeyPress(controls.wrist_flex.positive)
|
604 |
+
}
|
605 |
+
onMouseUp={() =>
|
606 |
+
simulateKeyRelease(controls.wrist_flex.positive)
|
607 |
+
}
|
608 |
+
disabled={!teleopState?.isActive}
|
609 |
+
/>
|
610 |
+
<VirtualKey
|
611 |
+
label="S"
|
612 |
+
subLabel="Elbow-"
|
613 |
+
isPressed={
|
614 |
+
!!keyStates[controls.elbow_flex.negative]?.pressed
|
615 |
+
}
|
616 |
+
onMouseDown={() =>
|
617 |
+
simulateKeyPress(controls.elbow_flex.negative)
|
618 |
+
}
|
619 |
+
onMouseUp={() =>
|
620 |
+
simulateKeyRelease(controls.elbow_flex.negative)
|
621 |
+
}
|
622 |
+
disabled={!teleopState?.isActive}
|
623 |
+
/>
|
624 |
+
<VirtualKey
|
625 |
+
label="D"
|
626 |
+
subLabel="Wrist-"
|
627 |
+
isPressed={
|
628 |
+
!!keyStates[controls.wrist_flex.negative]?.pressed
|
629 |
+
}
|
630 |
+
onMouseDown={() =>
|
631 |
+
simulateKeyPress(controls.wrist_flex.negative)
|
632 |
+
}
|
633 |
+
onMouseUp={() =>
|
634 |
+
simulateKeyRelease(controls.wrist_flex.negative)
|
635 |
+
}
|
636 |
+
disabled={!teleopState?.isActive}
|
637 |
+
/>
|
638 |
+
</div>
|
639 |
+
<span className="font-bold text-sm font-sans">
|
640 |
+
Elbow/Wrist
|
641 |
+
</span>
|
642 |
</div>
|
643 |
+
<div className="flex flex-col items-center gap-2">
|
644 |
+
<div className="flex gap-2">
|
645 |
+
<VirtualKey
|
646 |
+
label="Q"
|
647 |
+
subLabel="Roll+"
|
648 |
+
isPressed={
|
649 |
+
!!keyStates[controls.wrist_roll.positive]?.pressed
|
650 |
+
}
|
651 |
+
onMouseDown={() =>
|
652 |
+
simulateKeyPress(controls.wrist_roll.positive)
|
653 |
+
}
|
654 |
+
onMouseUp={() =>
|
655 |
+
simulateKeyRelease(controls.wrist_roll.positive)
|
656 |
+
}
|
657 |
+
disabled={!teleopState?.isActive}
|
658 |
+
/>
|
659 |
+
<VirtualKey
|
660 |
+
label="E"
|
661 |
+
subLabel="Roll-"
|
662 |
+
isPressed={
|
663 |
+
!!keyStates[controls.wrist_roll.negative]?.pressed
|
664 |
+
}
|
665 |
+
onMouseDown={() =>
|
666 |
+
simulateKeyPress(controls.wrist_roll.negative)
|
667 |
+
}
|
668 |
+
onMouseUp={() =>
|
669 |
+
simulateKeyRelease(controls.wrist_roll.negative)
|
670 |
+
}
|
671 |
+
disabled={!teleopState?.isActive}
|
672 |
+
/>
|
673 |
+
</div>
|
674 |
+
<div className="flex gap-2">
|
675 |
+
<VirtualKey
|
676 |
+
label="O"
|
677 |
+
subLabel="Grip+"
|
678 |
+
isPressed={
|
679 |
+
!!keyStates[controls.gripper.positive]?.pressed
|
680 |
+
}
|
681 |
+
onMouseDown={() =>
|
682 |
+
simulateKeyPress(controls.gripper.positive)
|
683 |
+
}
|
684 |
+
onMouseUp={() =>
|
685 |
+
simulateKeyRelease(controls.gripper.positive)
|
686 |
+
}
|
687 |
+
disabled={!teleopState?.isActive}
|
688 |
+
/>
|
689 |
+
<VirtualKey
|
690 |
+
label="C"
|
691 |
+
subLabel="Grip-"
|
692 |
+
isPressed={
|
693 |
+
!!keyStates[controls.gripper.negative]?.pressed
|
694 |
+
}
|
695 |
+
onMouseDown={() =>
|
696 |
+
simulateKeyPress(controls.gripper.negative)
|
697 |
+
}
|
698 |
+
onMouseUp={() =>
|
699 |
+
simulateKeyRelease(controls.gripper.negative)
|
700 |
+
}
|
701 |
+
disabled={!teleopState?.isActive}
|
702 |
+
/>
|
703 |
+
</div>
|
704 |
+
<span className="font-bold text-sm font-sans">Roll/Grip</span>
|
705 |
</div>
|
|
|
706 |
</div>
|
707 |
+
<div className="pt-4 border-t border-white/10">
|
708 |
+
<div className="flex justify-between items-center font-mono text-sm">
|
709 |
+
<div className="flex items-center gap-2 text-muted-foreground">
|
710 |
+
<Keyboard className="w-4 h-4" />
|
711 |
+
<span>
|
712 |
+
Active Keys:{" "}
|
713 |
+
{Object.values(keyStates).filter((k) => k.pressed).length}
|
714 |
+
</span>
|
715 |
+
</div>
|
716 |
+
<TooltipProvider>
|
717 |
+
<Tooltip>
|
718 |
+
<TooltipTrigger asChild>
|
719 |
+
<div
|
720 |
+
className={cn(
|
721 |
+
"w-10 h-6 border rounded-md flex items-center justify-center font-mono text-xs transition-all",
|
722 |
+
"select-none user-select-none",
|
723 |
+
!teleopState?.isActive &&
|
724 |
+
"opacity-50 cursor-not-allowed",
|
725 |
+
teleopState?.isActive &&
|
726 |
+
"cursor-pointer hover:bg-white/5",
|
727 |
+
keyStates[controls.stop]?.pressed
|
728 |
+
? "bg-destructive text-destructive-foreground border-destructive"
|
729 |
+
: "bg-background"
|
730 |
+
)}
|
731 |
+
onMouseDown={(e) => {
|
732 |
+
e.preventDefault();
|
733 |
+
if (teleopState?.isActive) {
|
734 |
+
simulateKeyPress(controls.stop);
|
735 |
+
}
|
736 |
+
}}
|
737 |
+
onMouseUp={(e) => {
|
738 |
+
e.preventDefault();
|
739 |
+
if (teleopState?.isActive) {
|
740 |
+
simulateKeyRelease(controls.stop);
|
741 |
+
}
|
742 |
+
}}
|
743 |
+
onMouseLeave={(e) => {
|
744 |
+
e.preventDefault();
|
745 |
+
if (teleopState?.isActive) {
|
746 |
+
simulateKeyRelease(controls.stop);
|
747 |
+
}
|
748 |
+
}}
|
749 |
+
>
|
750 |
+
ESC
|
751 |
+
</div>
|
752 |
+
</TooltipTrigger>
|
753 |
+
<TooltipContent>Emergency Stop</TooltipContent>
|
754 |
+
</Tooltip>
|
755 |
+
</TooltipProvider>
|
756 |
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
757 |
</div>
|
758 |
</div>
|
759 |
</div>
|
760 |
</div>
|
761 |
+
</Card>
|
762 |
+
|
763 |
+
{/* Robot Movement Recorder - Always show UI */}
|
764 |
+
<Recorder
|
765 |
+
teleoperators={memoizedTeleoperators}
|
766 |
+
robot={robot}
|
767 |
+
onNeedsTeleoperation={initializeTeleoperation}
|
768 |
+
/>
|
769 |
+
</>
|
770 |
);
|
771 |
}
|
@@ -0,0 +1,69 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
"use client";
|
2 |
+
|
3 |
+
import { useState } from "react";
|
4 |
+
import { LeRobotEpisode } from "@lerobot/web";
|
5 |
+
import { TeleoperatorFramesView } from "./teleoperator-frames-view";
|
6 |
+
import { Button } from "./ui/button";
|
7 |
+
|
8 |
+
interface TeleoperatorEpisodesViewProps {
|
9 |
+
teleoperatorData?: LeRobotEpisode[];
|
10 |
+
}
|
11 |
+
|
12 |
+
export function TeleoperatorEpisodesView({ teleoperatorData }: TeleoperatorEpisodesViewProps) {
|
13 |
+
// State to track which episodes are expanded
|
14 |
+
const [expandedEpisodes, setExpandedEpisodes] = useState<Record<number, boolean>>({});
|
15 |
+
|
16 |
+
// Toggle expanded state for an episode
|
17 |
+
const toggleEpisode = (index: number) => {
|
18 |
+
setExpandedEpisodes(prev => ({
|
19 |
+
...prev,
|
20 |
+
[index]: !prev[index]
|
21 |
+
}));
|
22 |
+
};
|
23 |
+
|
24 |
+
return (
|
25 |
+
<div className="flex flex-col gap-2">
|
26 |
+
<div className="text-sm text-center text-muted-foreground mb-2">List of recorded episodes</div>
|
27 |
+
<div className="flex flex-col gap-1">
|
28 |
+
{/* Header */}
|
29 |
+
<div className="flex flex-row font-medium text-sm">
|
30 |
+
<div className="flex-1 px-4 py-2">id</div>
|
31 |
+
<div className="flex-1 px-4 py-2">time length</div>
|
32 |
+
<div className="flex-1 px-4 py-2">frames</div>
|
33 |
+
</div>
|
34 |
+
|
35 |
+
{/* Body */}
|
36 |
+
{teleoperatorData && teleoperatorData.length > 0 ? (
|
37 |
+
teleoperatorData.map((episode: LeRobotEpisode, i: number) => (
|
38 |
+
<div key={i} className="flex flex-col border-t border-gray-700">
|
39 |
+
{/* Episode row */}
|
40 |
+
<div className="flex flex-row">
|
41 |
+
<div className="flex-1 px-4 py-2 font-mono">{i}</div>
|
42 |
+
<div className="flex-1 px-4 py-2 font-mono">{episode.timespan}</div>
|
43 |
+
<div className="flex-1 px-4 py-2 font-mono">{episode.length}</div>
|
44 |
+
<div className="px-4 py-2 cursor-pointer" onClick={() => toggleEpisode(i)}>
|
45 |
+
{expandedEpisodes[i] ?
|
46 |
+
<Button>hide frames</Button> :
|
47 |
+
<Button>show frames</Button>}
|
48 |
+
</div>
|
49 |
+
</div>
|
50 |
+
|
51 |
+
{/* Frames (collapsible) */}
|
52 |
+
{expandedEpisodes[i] && (
|
53 |
+
<TeleoperatorFramesView frames={episode.frames} />
|
54 |
+
)}
|
55 |
+
</div>
|
56 |
+
))
|
57 |
+
) : (
|
58 |
+
<div className="flex flex-row border-t border-gray-700">
|
59 |
+
<div
|
60 |
+
className="flex-1 px-4 py-4 text-center text-muted-foreground"
|
61 |
+
>
|
62 |
+
No episodes recorded yet. Click "Start Recording" to begin.
|
63 |
+
</div>
|
64 |
+
</div>
|
65 |
+
)}
|
66 |
+
</div>
|
67 |
+
</div>
|
68 |
+
);
|
69 |
+
}
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
"use client";
|
2 |
+
|
3 |
+
import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
|
4 |
+
import { TeleoperatorJointGraph } from "./teleoperator-joint-graph";
|
5 |
+
|
6 |
+
interface TeleoperatorFramesViewProps {
|
7 |
+
frames: NonIndexedLeRobotDatasetRow[];
|
8 |
+
}
|
9 |
+
|
10 |
+
export function TeleoperatorFramesView({ frames }: TeleoperatorFramesViewProps) {
|
11 |
+
// Joint names in the order they appear in the arrays
|
12 |
+
const jointNames = [
|
13 |
+
"shoulder_pan",
|
14 |
+
"shoulder_lift",
|
15 |
+
"elbow_flex",
|
16 |
+
"wrist_flex",
|
17 |
+
"wrist_roll",
|
18 |
+
"gripper"
|
19 |
+
];
|
20 |
+
|
21 |
+
// Helper function to format an object as a column of key-value pairs with joint names
|
22 |
+
const formatArrayAsColumn = (obj: Record<number, number>): string => {
|
23 |
+
return Object.entries(obj)
|
24 |
+
.map(([key, value]) => {
|
25 |
+
// Convert numeric key to joint name if possible
|
26 |
+
const index = parseInt(key);
|
27 |
+
const jointName = !isNaN(index) && index < jointNames.length ? jointNames[index] : key;
|
28 |
+
return `${jointName}: ${value}`;
|
29 |
+
})
|
30 |
+
.join('\n');
|
31 |
+
};
|
32 |
+
|
33 |
+
return (
|
34 |
+
<div className="ml-8 mr-4 mb-2">
|
35 |
+
{/* Joint visualization graph */}
|
36 |
+
<TeleoperatorJointGraph frames={frames} />
|
37 |
+
|
38 |
+
{/* Frames container with horizontal scroll */}
|
39 |
+
<div className="bg-gray-800/50 rounded-md overflow-hidden">
|
40 |
+
<div className="overflow-x-auto">
|
41 |
+
{/* Frames header */}
|
42 |
+
<table className="w-full min-w-max table-fixed">
|
43 |
+
<thead>
|
44 |
+
<tr className="text-xs font-medium bg-gray-800/80 text-gray-300">
|
45 |
+
<th className="w-16 px-2 py-1 text-left">Frame</th>
|
46 |
+
<th className="w-64 px-2 py-1 text-left">Timestamp</th>
|
47 |
+
<th className="w-[300px] px-2 py-1 text-left">Action</th>
|
48 |
+
<th className="w-[500px] px-2 py-1 text-left">State</th>
|
49 |
+
</tr>
|
50 |
+
</thead>
|
51 |
+
|
52 |
+
{/* Frame rows */}
|
53 |
+
<tbody className="max-h-60 overflow-y-auto">
|
54 |
+
{frames.map((frame: NonIndexedLeRobotDatasetRow, frameIndex: number) => (
|
55 |
+
<tr key={frameIndex} className="text-xs border-t border-gray-700/50">
|
56 |
+
<td className="w-16 px-2 py-1 font-mono whitespace-nowrap">{frameIndex}</td>
|
57 |
+
<td className="w-64 px-2 py-1 font-mono whitespace-nowrap">
|
58 |
+
{frame.timestamp}
|
59 |
+
</td>
|
60 |
+
<td className="w-[200px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
|
61 |
+
{Object.keys(frame.action).length > 0 ?
|
62 |
+
formatArrayAsColumn(frame.action) :
|
63 |
+
'-'}
|
64 |
+
</td>
|
65 |
+
<td className="w-[300px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
|
66 |
+
{Object.keys(frame["observation.state"]).length > 0 ?
|
67 |
+
formatArrayAsColumn(frame["observation.state"]) :
|
68 |
+
'-'}
|
69 |
+
</td>
|
70 |
+
</tr>
|
71 |
+
))}
|
72 |
+
</tbody>
|
73 |
+
</table>
|
74 |
+
</div>
|
75 |
+
</div>
|
76 |
+
</div>
|
77 |
+
);
|
78 |
+
}
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
import React from "react";
|
2 |
+
import {
|
3 |
+
LineChart,
|
4 |
+
Line,
|
5 |
+
XAxis,
|
6 |
+
YAxis,
|
7 |
+
CartesianGrid,
|
8 |
+
Tooltip,
|
9 |
+
Legend,
|
10 |
+
ResponsiveContainer
|
11 |
+
} from "recharts";
|
12 |
+
import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
|
13 |
+
|
14 |
+
interface TeleoperatorJointGraphProps {
|
15 |
+
frames: NonIndexedLeRobotDatasetRow[];
|
16 |
+
}
|
17 |
+
|
18 |
+
export function TeleoperatorJointGraph({ frames }: TeleoperatorJointGraphProps) {
|
19 |
+
// Skip rendering if no frames
|
20 |
+
if (!frames || frames.length === 0) {
|
21 |
+
return null;
|
22 |
+
}
|
23 |
+
|
24 |
+
// Use hardcoded joint names that match the LeRobot dataset format
|
25 |
+
const jointNames = [
|
26 |
+
"shoulder_pan",
|
27 |
+
"shoulder_lift",
|
28 |
+
"elbow_flex",
|
29 |
+
"wrist_flex",
|
30 |
+
"wrist_roll",
|
31 |
+
"gripper"
|
32 |
+
];
|
33 |
+
|
34 |
+
// Generate a color palette for the joints
|
35 |
+
const colors = [
|
36 |
+
"#8884d8", "#82ca9d", "#ffc658", "#ff8042", "#0088fe", "#00C49F",
|
37 |
+
"#FFBB28", "#FF8042", "#a4de6c", "#d0ed57"
|
38 |
+
];
|
39 |
+
|
40 |
+
// Prepare data for the chart - handling arrays
|
41 |
+
const chartData = frames.map((frame, index) => {
|
42 |
+
// Create base data point with index
|
43 |
+
const dataPoint: any = {
|
44 |
+
name: index,
|
45 |
+
timestamp: frame.timestamp
|
46 |
+
};
|
47 |
+
|
48 |
+
// Add action values (assuming action is an array)
|
49 |
+
if (Array.isArray(frame.action)) {
|
50 |
+
// Map each array index to the corresponding joint name
|
51 |
+
jointNames.forEach((jointName, i) => {
|
52 |
+
if (i < frame.action.length) {
|
53 |
+
dataPoint[`action_${jointName}`] = frame.action[i];
|
54 |
+
}
|
55 |
+
});
|
56 |
+
}
|
57 |
+
|
58 |
+
// Add observation state values (assuming observation.state is an array)
|
59 |
+
if (Array.isArray(frame["observation.state"])) {
|
60 |
+
// Map each array index to the corresponding joint name
|
61 |
+
jointNames.forEach((jointName, i) => {
|
62 |
+
if (i < frame["observation.state"].length) {
|
63 |
+
dataPoint[`state_${jointName}`] = frame["observation.state"][i];
|
64 |
+
}
|
65 |
+
});
|
66 |
+
}
|
67 |
+
|
68 |
+
return dataPoint;
|
69 |
+
});
|
70 |
+
|
71 |
+
// Create lines for each joint
|
72 |
+
const linesToRender = jointNames.flatMap(jointName => [
|
73 |
+
{
|
74 |
+
key: `action_${jointName}`,
|
75 |
+
dataKey: `action_${jointName}`,
|
76 |
+
name: `Action: ${jointName}`,
|
77 |
+
isDotted: true
|
78 |
+
},
|
79 |
+
{
|
80 |
+
key: `state_${jointName}`,
|
81 |
+
dataKey: `state_${jointName}`,
|
82 |
+
name: `State: ${jointName}`,
|
83 |
+
isDotted: false
|
84 |
+
}
|
85 |
+
]);
|
86 |
+
|
87 |
+
return (
|
88 |
+
<div className="w-full bg-gray-800/50 rounded-md p-4 mb-4">
|
89 |
+
<h3 className="text-sm font-medium text-gray-300 mb-2">Joint Positions Over Time</h3>
|
90 |
+
<ResponsiveContainer width="100%" height={300}>
|
91 |
+
<LineChart
|
92 |
+
data={chartData}
|
93 |
+
margin={{
|
94 |
+
top: 5,
|
95 |
+
right: 30,
|
96 |
+
left: 20,
|
97 |
+
bottom: 5
|
98 |
+
}}
|
99 |
+
>
|
100 |
+
<CartesianGrid strokeDasharray="3 3" stroke="#444" />
|
101 |
+
<XAxis
|
102 |
+
dataKey="name"
|
103 |
+
label={{ value: 'Frame Index', position: 'insideBottomRight', offset: -10 }}
|
104 |
+
stroke="#aaa"
|
105 |
+
/>
|
106 |
+
<YAxis stroke="#aaa" />
|
107 |
+
<Tooltip
|
108 |
+
contentStyle={{ backgroundColor: '#333', borderColor: '#555' }}
|
109 |
+
labelStyle={{ color: '#eee' }}
|
110 |
+
itemStyle={{ color: '#eee' }}
|
111 |
+
/>
|
112 |
+
<Legend />
|
113 |
+
|
114 |
+
{/* Render all lines */}
|
115 |
+
{linesToRender.map((lineConfig, index) => {
|
116 |
+
const jointName = lineConfig.dataKey.replace(/^(action|state)_/, '');
|
117 |
+
const jointIndex = jointNames.indexOf(jointName);
|
118 |
+
const colorIndex = jointIndex >= 0 ? jointIndex : index % colors.length;
|
119 |
+
|
120 |
+
return (
|
121 |
+
<Line
|
122 |
+
key={lineConfig.key}
|
123 |
+
type="monotone"
|
124 |
+
dataKey={lineConfig.dataKey}
|
125 |
+
name={lineConfig.name}
|
126 |
+
stroke={colors[colorIndex]}
|
127 |
+
strokeDasharray={lineConfig.isDotted ? "5 5" : undefined}
|
128 |
+
dot={false}
|
129 |
+
activeDot={{ r: 4 }}
|
130 |
+
/>
|
131 |
+
);
|
132 |
+
})}
|
133 |
+
</LineChart>
|
134 |
+
</ResponsiveContainer>
|
135 |
+
</div>
|
136 |
+
);
|
137 |
+
}
|
@@ -1,5 +1,6 @@
|
|
1 |
import { StrictMode } from "react";
|
2 |
import { createRoot } from "react-dom/client";
|
|
|
3 |
import "./global.css";
|
4 |
import App from "./App";
|
5 |
import { ThemeProvider } from "./components/theme-provider";
|
@@ -11,13 +12,15 @@ setTimeout(() => {
|
|
11 |
|
12 |
createRoot(document.getElementById("root")!).render(
|
13 |
<StrictMode>
|
14 |
-
<
|
15 |
-
|
16 |
-
|
17 |
-
|
18 |
-
|
19 |
-
|
20 |
-
|
21 |
-
|
|
|
|
|
22 |
</StrictMode>
|
23 |
);
|
|
|
1 |
import { StrictMode } from "react";
|
2 |
import { createRoot } from "react-dom/client";
|
3 |
+
import { BrowserRouter } from "react-router-dom";
|
4 |
import "./global.css";
|
5 |
import App from "./App";
|
6 |
import { ThemeProvider } from "./components/theme-provider";
|
|
|
12 |
|
13 |
createRoot(document.getElementById("root")!).render(
|
14 |
<StrictMode>
|
15 |
+
<BrowserRouter>
|
16 |
+
<ThemeProvider
|
17 |
+
attribute="class"
|
18 |
+
defaultTheme="system"
|
19 |
+
enableSystem
|
20 |
+
disableTransitionOnChange
|
21 |
+
>
|
22 |
+
<App />
|
23 |
+
</ThemeProvider>
|
24 |
+
</BrowserRouter>
|
25 |
</StrictMode>
|
26 |
);
|
@@ -34,14 +34,22 @@
|
|
34 |
"install-global": "pnpm run build && npm link"
|
35 |
},
|
36 |
"dependencies": {
|
37 |
-
"@
|
|
|
|
|
|
|
|
|
|
|
38 |
"log-update": "^6.1.0",
|
39 |
-
"
|
|
|
|
|
40 |
},
|
41 |
"devDependencies": {
|
42 |
"@changesets/cli": "^2.29.5",
|
43 |
-
"@types/node": "^22.
|
44 |
-
"
|
|
|
45 |
"typescript": "~5.8.3"
|
46 |
},
|
47 |
"repository": {
|
|
|
34 |
"install-global": "pnpm run build && npm link"
|
35 |
},
|
36 |
"dependencies": {
|
37 |
+
"@aws-sdk/client-s3": "^3.856.0",
|
38 |
+
"@aws-sdk/lib-storage": "^3.856.0",
|
39 |
+
"@huggingface/hub": "^2.4.0",
|
40 |
+
"@react-three/cannon": "^6.6.0",
|
41 |
+
"apache-arrow": "^21.0.0",
|
42 |
+
"jszip": "^3.10.1",
|
43 |
"log-update": "^6.1.0",
|
44 |
+
"parquet-wasm": "^0.6.1",
|
45 |
+
"serialport": "^12.0.0",
|
46 |
+
"three": "^0.178.0"
|
47 |
},
|
48 |
"devDependencies": {
|
49 |
"@changesets/cli": "^2.29.5",
|
50 |
+
"@types/node": "^22.17.0",
|
51 |
+
"@types/three": "^0.178.1",
|
52 |
+
"tsx": "^4.20.3",
|
53 |
"typescript": "~5.8.3"
|
54 |
},
|
55 |
"repository": {
|
@@ -312,6 +312,75 @@ await releaseMotors(robot, [1, 2, 3]);
|
|
312 |
- `robot: RobotConnection` - Connected robot
|
313 |
- `motorIds?: number[]` - Specific motor IDs (default: all motors for robot type)
|
314 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
315 |
## Browser Requirements
|
316 |
|
317 |
- **chromium 89+** with WebSerial and WebUSB API support
|
|
|
312 |
- `robot: RobotConnection` - Connected robot
|
313 |
- `motorIds?: number[]` - Specific motor IDs (default: all motors for robot type)
|
314 |
|
315 |
+
---
|
316 |
+
|
317 |
+
## Dataset Recording and Export
|
318 |
+
|
319 |
+
The LeRobot.js library provides functionality to record teleoperator data and export it in the LeRobot dataset format, compatible with machine learning models.
|
320 |
+
|
321 |
+
### `LeRobotDatasetRecorder`
|
322 |
+
|
323 |
+
Records teleoperator movements and camera streams, then exports them in the LeRobot dataset format.
|
324 |
+
|
325 |
+
```typescript
|
326 |
+
import { LeRobotDatasetRecorder } from "@lerobot/web";
|
327 |
+
|
328 |
+
// Create a recorder with teleoperator and video streams
|
329 |
+
const recorder = new LeRobotDatasetRecorder(
|
330 |
+
[teleoperator], // Array of teleoperators to record, currently only supports 1 teleoperator
|
331 |
+
{ "main": videoStream }, // Video streams by camera key
|
332 |
+
30, // Target FPS
|
333 |
+
"Pick and place task" // Task description
|
334 |
+
);
|
335 |
+
|
336 |
+
// Start recording
|
337 |
+
await recorder.startRecording();
|
338 |
+
|
339 |
+
// ... robot performs task ...
|
340 |
+
|
341 |
+
// Stop recording and get the data
|
342 |
+
const recordingData = await recorder.stopRecording();
|
343 |
+
|
344 |
+
// Export the dataset in various formats
|
345 |
+
// 1. As a downloadable zip file
|
346 |
+
await recorder.exportForLeRobot('zip-download');
|
347 |
+
|
348 |
+
// 2. Upload to Hugging Face
|
349 |
+
const hfUploader = await recorder.exportForLeRobot('huggingface', {
|
350 |
+
repoName: 'my-robot-dataset',
|
351 |
+
accessToken: 'hf_...',
|
352 |
+
});
|
353 |
+
|
354 |
+
// 3. Upload to S3
|
355 |
+
const s3Uploader = await recorder.exportForLeRobot('s3', {
|
356 |
+
bucketName: 'my-bucket',
|
357 |
+
accessKeyId: 'AKIA...',
|
358 |
+
secretAccessKey: '...',
|
359 |
+
region: 'us-east-1',
|
360 |
+
});
|
361 |
+
```
|
362 |
+
|
363 |
+
#### Key Features
|
364 |
+
|
365 |
+
- **Multi-source Recording**: Records teleoperator movements and synchronized video
|
366 |
+
- **Regular Interpolation**: Generates frames at consistent intervals with `episodes` getter
|
367 |
+
- **Multiple Export Formats**: Supports local download, Hugging Face, and S3 upload
|
368 |
+
- **LeRobot Dataset Format**: Follows the standard format for compatibility with ML models
|
369 |
+
|
370 |
+
> **Note:** The dataset statistical data currently generated is incorrect and needs to be updated in a future release.
|
371 |
+
|
372 |
+
#### Dataset Format
|
373 |
+
|
374 |
+
The exported dataset follows the LeRobot format with this structure:
|
375 |
+
|
376 |
+
```
|
377 |
+
/data/chunk-000/file-000.parquet # Teleoperator data
|
378 |
+
/videos/observation.images.{camera-key}/chunk-000/file-000.mp4 # Video data
|
379 |
+
/metadata.json # Dataset metadata
|
380 |
+
/statistics.json # Dataset statistics (currently incorrect)
|
381 |
+
/README.md # Dataset documentation
|
382 |
+
```
|
383 |
+
|
384 |
## Browser Requirements
|
385 |
|
386 |
- **chromium 89+** with WebSerial and WebUSB API support
|
@@ -36,6 +36,7 @@
|
|
36 |
],
|
37 |
"scripts": {
|
38 |
"build": "tsc --project tsconfig.build.json",
|
|
|
39 |
"prepublishOnly": "npm run build",
|
40 |
"dev": "vitest --watch",
|
41 |
"test": "vitest run",
|
@@ -43,10 +44,10 @@
|
|
43 |
"test:coverage": "vitest run --coverage"
|
44 |
},
|
45 |
"devDependencies": {
|
46 |
-
"vitest": "^2.
|
47 |
-
"
|
48 |
-
"
|
49 |
-
"
|
50 |
},
|
51 |
"peerDependencies": {
|
52 |
"typescript": ">=4.5.0"
|
@@ -59,5 +60,13 @@
|
|
59 |
"author": "Tim Pietrusky",
|
60 |
"publishConfig": {
|
61 |
"access": "public"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
62 |
}
|
63 |
}
|
|
|
36 |
],
|
37 |
"scripts": {
|
38 |
"build": "tsc --project tsconfig.build.json",
|
39 |
+
"watch": "tsc --project tsconfig.build.json --watch",
|
40 |
"prepublishOnly": "npm run build",
|
41 |
"dev": "vitest --watch",
|
42 |
"test": "vitest run",
|
|
|
44 |
"test:coverage": "vitest run --coverage"
|
45 |
},
|
46 |
"devDependencies": {
|
47 |
+
"@vitest/ui": "^2.1.9",
|
48 |
+
"jsdom": "^24.1.3",
|
49 |
+
"vite": "^6.3.5",
|
50 |
+
"vitest": "^2.1.9"
|
51 |
},
|
52 |
"peerDependencies": {
|
53 |
"typescript": ">=4.5.0"
|
|
|
60 |
"author": "Tim Pietrusky",
|
61 |
"publishConfig": {
|
62 |
"access": "public"
|
63 |
+
},
|
64 |
+
"dependencies": {
|
65 |
+
"@aws-sdk/client-s3": "^3.856.0",
|
66 |
+
"@aws-sdk/lib-storage": "^3.856.0",
|
67 |
+
"@huggingface/hub": "^2.4.0",
|
68 |
+
"apache-arrow": "^21.0.0",
|
69 |
+
"jszip": "^3.10.1",
|
70 |
+
"parquet-wasm": "^0.6.1"
|
71 |
}
|
72 |
}
|
The diff for this file is too large to render.
See raw diff
|
|
@@ -15,7 +15,6 @@ import {
|
|
15 |
writeHardwarePositionLimits,
|
16 |
} from "./utils/motor-calibration.js";
|
17 |
import { createSO100Config } from "./robots/so100_config.js";
|
18 |
-
import type { RobotConnection } from "./types/robot-connection.js";
|
19 |
import type { RobotHardwareConfig } from "./types/robot-config.js";
|
20 |
import type {
|
21 |
CalibrateConfig,
|
|
|
15 |
writeHardwarePositionLimits,
|
16 |
} from "./utils/motor-calibration.js";
|
17 |
import { createSO100Config } from "./robots/so100_config.js";
|
|
|
18 |
import type { RobotHardwareConfig } from "./types/robot-config.js";
|
19 |
import type {
|
20 |
CalibrateConfig,
|
@@ -0,0 +1,86 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
import * as hub from "@huggingface/hub";
|
2 |
+
import type { RepoDesignation } from "@huggingface/hub";
|
3 |
+
import type { ContentSource } from "@huggingface/hub";
|
4 |
+
|
5 |
+
type FileArray = Array<URL | File | { path: string; content: ContentSource }>;
|
6 |
+
/**
|
7 |
+
* Uploads a leRobot dataset to huggingface
|
8 |
+
*/
|
9 |
+
|
10 |
+
export class LeRobotHFUploader extends EventTarget {
|
11 |
+
private _repoDesignation: RepoDesignation;
|
12 |
+
private _uploaded : boolean;
|
13 |
+
private _created_repo : boolean;
|
14 |
+
|
15 |
+
constructor(username: string, repoName: string) {
|
16 |
+
super();
|
17 |
+
this._repoDesignation = {
|
18 |
+
name : `${username}/${repoName}`,
|
19 |
+
type : "dataset"
|
20 |
+
};
|
21 |
+
|
22 |
+
this._uploaded = false;
|
23 |
+
this._created_repo = false;
|
24 |
+
}
|
25 |
+
|
26 |
+
/**
|
27 |
+
* Returns whether the repository has been successfully created
|
28 |
+
*/
|
29 |
+
get createdRepo(): boolean {
|
30 |
+
return this._created_repo;
|
31 |
+
}
|
32 |
+
|
33 |
+
get uploaded() : boolean {
|
34 |
+
return this._uploaded;
|
35 |
+
}
|
36 |
+
|
37 |
+
|
38 |
+
/**
|
39 |
+
* Uploads the dataset to huggingface
|
40 |
+
*
|
41 |
+
* A referenceId is used to be able to track progress of the upload,
|
42 |
+
* this provides a progressEvent from huggingface.js (see : https://github.com/huggingface/huggingface.js/blob/main/packages/hub/README.md#usage)
|
43 |
+
*
|
44 |
+
* both this and huggingface.js have pretty bad documentation, some exploration will be required (sorry!, I have university, and work and I already feel guilty procastinating those to to write this)
|
45 |
+
*
|
46 |
+
* @param dataset The dataset to upload
|
47 |
+
* @param accessToken The access token for huggingface
|
48 |
+
* @param referenceId The reference id for the upload, to track it (optional)
|
49 |
+
*/
|
50 |
+
async createRepoAndUploadFiles(files : FileArray, accessToken : string, referenceId : string = "") {
|
51 |
+
await hub.createRepo({
|
52 |
+
repo: this._repoDesignation,
|
53 |
+
accessToken: accessToken,
|
54 |
+
license: "mit"
|
55 |
+
});
|
56 |
+
|
57 |
+
this._created_repo = true;
|
58 |
+
this.dispatchEvent(new CustomEvent("repoCreated", { detail: this._repoDesignation }));
|
59 |
+
|
60 |
+
const uploadPromises : Promise<void>[] = [];
|
61 |
+
uploadPromises.push(this.uploadFilesWithProgress(files, accessToken, referenceId));
|
62 |
+
|
63 |
+
await Promise.all(uploadPromises);
|
64 |
+
}
|
65 |
+
|
66 |
+
/**
|
67 |
+
* Uploads files to huggingface with progress events
|
68 |
+
*
|
69 |
+
* @param files The files to upload
|
70 |
+
* @param accessToken The access token for huggingface
|
71 |
+
* @param referenceId The reference id for the upload, to track it (optional)
|
72 |
+
*/
|
73 |
+
async uploadFilesWithProgress(files : FileArray, accessToken : string, referenceId : string = "") {
|
74 |
+
for await (const progressEvent of hub.uploadFilesWithProgress({
|
75 |
+
repo: this._repoDesignation,
|
76 |
+
accessToken: accessToken,
|
77 |
+
files: files,
|
78 |
+
})) {
|
79 |
+
this.dispatchEvent(new CustomEvent("progress", { detail: {
|
80 |
+
progressEvent,
|
81 |
+
repoDesignation: this._repoDesignation,
|
82 |
+
referenceId
|
83 |
+
} }));
|
84 |
+
}
|
85 |
+
}
|
86 |
+
}
|
@@ -58,3 +58,7 @@ export {
|
|
58 |
SO100_KEYBOARD_CONTROLS,
|
59 |
} from "./robots/so100_config.js";
|
60 |
export { KEYBOARD_TELEOPERATOR_DEFAULTS } from "./teleoperators/index.js";
|
|
|
|
|
|
|
|
|
|
58 |
SO100_KEYBOARD_CONTROLS,
|
59 |
} from "./robots/so100_config.js";
|
60 |
export { KEYBOARD_TELEOPERATOR_DEFAULTS } from "./teleoperators/index.js";
|
61 |
+
|
62 |
+
// Record
|
63 |
+
export { LeRobotDatasetRecorder } from "./record.js";
|
64 |
+
export { LeRobotHFUploader } from "./hf_uploader.js";
|
@@ -0,0 +1,1412 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
import { WebTeleoperator } from "./teleoperators/base-teleoperator";
|
2 |
+
import { MotorConfig } from "./types/teleoperation";
|
3 |
+
import * as parquet from 'parquet-wasm';
|
4 |
+
import * as arrow from 'apache-arrow';
|
5 |
+
import JSZip from 'jszip';
|
6 |
+
import getMetadataInfo from './utils/record/metadataInfo';
|
7 |
+
import type { VideoInfo } from './utils/record/metadataInfo';
|
8 |
+
import getStats from './utils/record/stats';
|
9 |
+
import generateREADME from './utils/record/generateREADME';
|
10 |
+
import { LeRobotHFUploader } from './hf_uploader';
|
11 |
+
import { LeRobotS3Uploader } from './s3_uploader';
|
12 |
+
|
13 |
+
// declare a type leRobot action that's basically an array of numbers
|
14 |
+
interface LeRobotAction {
|
15 |
+
[key: number]: number;
|
16 |
+
}
|
17 |
+
|
18 |
+
export class LeRobotEpisode {
|
19 |
+
// we assume that the frames are ordered
|
20 |
+
public frames : NonIndexedLeRobotDatasetRow[]
|
21 |
+
|
22 |
+
/**
|
23 |
+
* Optional start time of the episode
|
24 |
+
* If not set, defaults to the timestamp of the first frame
|
25 |
+
*/
|
26 |
+
private _startTime?: number;
|
27 |
+
|
28 |
+
/**
|
29 |
+
* Optional end time of the episode
|
30 |
+
* If not set, defaults to the timestamp of the last frame
|
31 |
+
*/
|
32 |
+
private _endTime?: number;
|
33 |
+
|
34 |
+
/**
|
35 |
+
* Creates a new LeRobotEpisode
|
36 |
+
*
|
37 |
+
* @param frames Optional array of frames to initialize the episode with
|
38 |
+
* @param startTime Optional explicit start time for the episode
|
39 |
+
* @param endTime Optional explicit end time for the episode
|
40 |
+
*/
|
41 |
+
constructor(frames?: NonIndexedLeRobotDatasetRow[], startTime?: number, endTime?: number){
|
42 |
+
this.frames = frames || [];
|
43 |
+
this._startTime = startTime;
|
44 |
+
this._endTime = endTime;
|
45 |
+
}
|
46 |
+
|
47 |
+
/**
|
48 |
+
* Adds a new frame to the episode
|
49 |
+
* Ensures frames are always in chronological order
|
50 |
+
*
|
51 |
+
* @param frame The frame to add
|
52 |
+
* @throws Error if the frame's timestamp is before the last frame's timestamp
|
53 |
+
*/
|
54 |
+
add(frame : NonIndexedLeRobotDatasetRow){
|
55 |
+
const lastFrame = this.frames.at(-1);
|
56 |
+
if (lastFrame && frame.timestamp < lastFrame.timestamp) {
|
57 |
+
throw new Error(`Frame timestamp ${frame.timestamp} is before last frame timestamp ${lastFrame.timestamp}`);
|
58 |
+
}
|
59 |
+
this.frames.push(frame);
|
60 |
+
}
|
61 |
+
|
62 |
+
/**
|
63 |
+
* Gets the start time of the episode
|
64 |
+
* If not explicitly set, returns the timestamp of the first frame
|
65 |
+
* If no frames exist, throws an error
|
66 |
+
*/
|
67 |
+
get startTime(): number {
|
68 |
+
if (this._startTime !== undefined) {
|
69 |
+
return this._startTime;
|
70 |
+
}
|
71 |
+
|
72 |
+
const firstFrame = this.frames.at(0);
|
73 |
+
if (!firstFrame) {
|
74 |
+
throw new Error("Cannot determine start time: no frames in episode");
|
75 |
+
}
|
76 |
+
|
77 |
+
return firstFrame.timestamp;
|
78 |
+
}
|
79 |
+
|
80 |
+
/**
|
81 |
+
* Sets an explicit start time for the episode
|
82 |
+
*/
|
83 |
+
set startTime(value: number) {
|
84 |
+
this._startTime = value;
|
85 |
+
}
|
86 |
+
|
87 |
+
/**
|
88 |
+
* Gets the end time of the episode
|
89 |
+
* If not explicitly set, returns the timestamp of the last frame
|
90 |
+
* If no frames exist, throws an error
|
91 |
+
*/
|
92 |
+
get endTime(): number {
|
93 |
+
if (this._endTime !== undefined) {
|
94 |
+
return this._endTime;
|
95 |
+
}
|
96 |
+
|
97 |
+
const lastFrame = this.frames.at(-1);
|
98 |
+
if (!lastFrame) {
|
99 |
+
throw new Error("Cannot determine end time: no frames in episode");
|
100 |
+
}
|
101 |
+
|
102 |
+
return lastFrame.timestamp;
|
103 |
+
}
|
104 |
+
|
105 |
+
/**
|
106 |
+
* Sets an explicit end time for the episode
|
107 |
+
*/
|
108 |
+
set endTime(value: number) {
|
109 |
+
this._endTime = value;
|
110 |
+
}
|
111 |
+
|
112 |
+
/**
|
113 |
+
* The time difference between the start and end time of the episode, in seconds
|
114 |
+
*/
|
115 |
+
get timespan(){
|
116 |
+
const hasNoFrames = this.frames.length === 0
|
117 |
+
if(hasNoFrames) return 0
|
118 |
+
|
119 |
+
return this.endTime - this.startTime;
|
120 |
+
}
|
121 |
+
|
122 |
+
/**
|
123 |
+
* The number of frames in the episode
|
124 |
+
*/
|
125 |
+
get length(){
|
126 |
+
return this.frames.length;
|
127 |
+
}
|
128 |
+
|
129 |
+
/**
|
130 |
+
* Creates a new LeRobotEpisode with frames interpolated at regular intervals
|
131 |
+
*
|
132 |
+
* @param fps The desired frames per second for the interpolated episode
|
133 |
+
* @param startIndex The desired starting index for the episode frames, useful when storing multiple episodes
|
134 |
+
* @returns A new LeRobotEpisode with interpolated frames
|
135 |
+
*/
|
136 |
+
getInterpolatedRegularEpisode(fps: number, startIndex: number = 0): LeRobotEpisode {
|
137 |
+
if (this.frames.length === 0) {
|
138 |
+
return new LeRobotEpisode([], this._startTime, this._endTime);
|
139 |
+
}
|
140 |
+
|
141 |
+
const actualStartTime = this._startTime !== undefined ? this._startTime : this.frames[0].timestamp;
|
142 |
+
const actualEndTime = this._endTime !== undefined ? this._endTime : this.frames[this.frames.length - 1].timestamp;
|
143 |
+
const timeDifference = actualEndTime - actualStartTime;
|
144 |
+
|
145 |
+
const numFrames = Math.max(1, Math.floor(timeDifference * fps));
|
146 |
+
const interpolatedFrames: NonIndexedLeRobotDatasetRow[] = [];
|
147 |
+
|
148 |
+
const firstFrame = this.frames[0];
|
149 |
+
const lastFrame = this.frames[this.frames.length - 1];
|
150 |
+
|
151 |
+
for (let i = 0; i < numFrames; i++) {
|
152 |
+
const timestamp = actualStartTime + (i / fps);
|
153 |
+
let frameToAdd: NonIndexedLeRobotDatasetRow;
|
154 |
+
|
155 |
+
if (timestamp < firstFrame.timestamp) {
|
156 |
+
frameToAdd = { ...firstFrame, timestamp };
|
157 |
+
frameToAdd.frame_index = i;
|
158 |
+
frameToAdd.index = startIndex + i;
|
159 |
+
} else if (timestamp > lastFrame.timestamp) {
|
160 |
+
frameToAdd = { ...lastFrame, timestamp };
|
161 |
+
frameToAdd.frame_index = i;
|
162 |
+
frameToAdd.index = startIndex + i;
|
163 |
+
} else {
|
164 |
+
let lowerIndex = 0;
|
165 |
+
for (let j = 0; j < this.frames.length - 1; j++) {
|
166 |
+
if (this.frames[j].timestamp <= timestamp && this.frames[j + 1].timestamp > timestamp) {
|
167 |
+
lowerIndex = j;
|
168 |
+
break;
|
169 |
+
}
|
170 |
+
}
|
171 |
+
|
172 |
+
const lowerFrame = this.frames[lowerIndex];
|
173 |
+
const upperFrame = this.frames[lowerIndex + 1];
|
174 |
+
|
175 |
+
frameToAdd = LeRobotEpisode.interpolateFrames(
|
176 |
+
lowerFrame,
|
177 |
+
upperFrame,
|
178 |
+
timestamp
|
179 |
+
);
|
180 |
+
|
181 |
+
frameToAdd.frame_index = i;
|
182 |
+
frameToAdd.episode_index = lowerFrame.episode_index;
|
183 |
+
frameToAdd.index = startIndex + i;
|
184 |
+
frameToAdd.task_index = lowerFrame.task_index;
|
185 |
+
}
|
186 |
+
|
187 |
+
interpolatedFrames.push(frameToAdd);
|
188 |
+
}
|
189 |
+
|
190 |
+
return new LeRobotEpisode(interpolatedFrames, actualStartTime, actualEndTime);
|
191 |
+
}
|
192 |
+
|
193 |
+
/**
|
194 |
+
* Interpolates between two frames to create a new frame at the specified timestamp
|
195 |
+
*
|
196 |
+
* @param frame1 The first frame
|
197 |
+
* @param frame2 The second frame
|
198 |
+
* @param targetTimestamp The timestamp at which to interpolate
|
199 |
+
* @returns A new interpolated frame
|
200 |
+
*/
|
201 |
+
static interpolateFrames(
|
202 |
+
frame1: NonIndexedLeRobotDatasetRow,
|
203 |
+
frame2: NonIndexedLeRobotDatasetRow,
|
204 |
+
targetTimestamp: number
|
205 |
+
): NonIndexedLeRobotDatasetRow {
|
206 |
+
if (targetTimestamp < frame1.timestamp || targetTimestamp > frame2.timestamp) {
|
207 |
+
throw new Error("Target timestamp must be between the timestamps of the two frames");
|
208 |
+
}
|
209 |
+
|
210 |
+
const timeRange = frame2.timestamp - frame1.timestamp;
|
211 |
+
const interpolationFactor = (targetTimestamp - frame1.timestamp) / timeRange;
|
212 |
+
|
213 |
+
// Interpolate action array
|
214 |
+
const interpolatedAction = LeRobotEpisode.interpolateArrays(
|
215 |
+
frame1.action,
|
216 |
+
frame2.action,
|
217 |
+
interpolationFactor
|
218 |
+
);
|
219 |
+
|
220 |
+
// Interpolate observation.state array
|
221 |
+
const interpolatedObservationState = LeRobotEpisode.interpolateArrays(
|
222 |
+
frame1["observation.state"],
|
223 |
+
frame2["observation.state"],
|
224 |
+
interpolationFactor
|
225 |
+
);
|
226 |
+
|
227 |
+
// Create the interpolated frame
|
228 |
+
return {
|
229 |
+
timestamp: targetTimestamp,
|
230 |
+
action: interpolatedAction,
|
231 |
+
"observation.state": interpolatedObservationState,
|
232 |
+
episode_index: frame1.episode_index,
|
233 |
+
task_index: frame1.task_index,
|
234 |
+
// Optional properties are not interpolated
|
235 |
+
frame_index: frame1.frame_index,
|
236 |
+
index: frame1.index
|
237 |
+
};
|
238 |
+
}
|
239 |
+
|
240 |
+
/**
|
241 |
+
* Helper method to interpolate between two arrays
|
242 |
+
*
|
243 |
+
* @param array1 First array of values
|
244 |
+
* @param array2 Second array of values
|
245 |
+
* @param factor Interpolation factor (0-1)
|
246 |
+
* @returns Interpolated array
|
247 |
+
*/
|
248 |
+
private static interpolateArrays(array1: any, array2: any, factor: number): any {
|
249 |
+
// Handle different types of inputs
|
250 |
+
if (Array.isArray(array1) && Array.isArray(array2)) {
|
251 |
+
// For arrays, interpolate each element
|
252 |
+
return array1.map((value, index) => {
|
253 |
+
return value + (array2[index] - value) * factor;
|
254 |
+
});
|
255 |
+
} else if (typeof array1 === 'object' && typeof array2 === 'object') {
|
256 |
+
// For objects, interpolate each property
|
257 |
+
const result: any = {};
|
258 |
+
for (const key of Object.keys(array1)) {
|
259 |
+
if (key in array2) {
|
260 |
+
result[key] = array1[key] + (array2[key] - array1[key]) * factor;
|
261 |
+
} else {
|
262 |
+
result[key] = array1[key];
|
263 |
+
}
|
264 |
+
}
|
265 |
+
return result;
|
266 |
+
} else {
|
267 |
+
// For primitive values
|
268 |
+
return array1 + (array2 - array1) * factor;
|
269 |
+
}
|
270 |
+
}
|
271 |
+
}
|
272 |
+
|
273 |
+
/**
|
274 |
+
* Base interface for LeRobot dataset rows with common fields
|
275 |
+
*/
|
276 |
+
export interface NonIndexedLeRobotDatasetRow {
|
277 |
+
timestamp: number;
|
278 |
+
action: LeRobotAction;
|
279 |
+
"observation.state": LeRobotAction;
|
280 |
+
|
281 |
+
// properties are optional for back-converstion from normal rows
|
282 |
+
episode_index : number,
|
283 |
+
task_index : number
|
284 |
+
frame_index? : number
|
285 |
+
index? : number
|
286 |
+
}
|
287 |
+
|
288 |
+
/**
|
289 |
+
* Represents a complete row in the LeRobot dataset format after indexing
|
290 |
+
* Used in the final exported dataset
|
291 |
+
*/
|
292 |
+
export interface LeRobotDatasetRow extends NonIndexedLeRobotDatasetRow {
|
293 |
+
frame_index: number;
|
294 |
+
index: number;
|
295 |
+
}
|
296 |
+
|
297 |
+
/**
|
298 |
+
* A mechanism to store and record, the video of all associated cameras
|
299 |
+
* as well as the teleoperator data
|
300 |
+
*
|
301 |
+
* follows the lerobot dataset format https://github.com/huggingface/lerobot/blob/cf86b9300dc83fdad408cfe4787b7b09b55f12cf/README.md#the-lerobotdataset-format
|
302 |
+
*/
|
303 |
+
export class LeRobotDatasetRecorder {
|
304 |
+
teleoperators: WebTeleoperator[];
|
305 |
+
videoStreams: { [key: string]: MediaStream };
|
306 |
+
mediaRecorders: { [key: string]: MediaRecorder };
|
307 |
+
videoChunks: { [key: string]: Blob[] };
|
308 |
+
videoBlobs: { [key: string]: Blob };
|
309 |
+
teleoperatorData: LeRobotEpisode[];
|
310 |
+
private _isRecording: boolean;
|
311 |
+
private episodeIndex: number;
|
312 |
+
private taskIndex: number;
|
313 |
+
fps: number;
|
314 |
+
taskDescription: string;
|
315 |
+
|
316 |
+
constructor(
|
317 |
+
teleoperators: WebTeleoperator[],
|
318 |
+
videoStreams: { [key: string]: MediaStream },
|
319 |
+
fps: number,
|
320 |
+
taskDescription: string = "Default task description"
|
321 |
+
) {
|
322 |
+
this.teleoperators = [];
|
323 |
+
|
324 |
+
if(teleoperators.length > 1)
|
325 |
+
throw Error(`
|
326 |
+
Currently, only 1 teleoperator can be recorded at a time!
|
327 |
+
|
328 |
+
Note : Do not attempt to create 2 different recorders via 2 different teleoperators, this would not work either
|
329 |
+
`)
|
330 |
+
|
331 |
+
this.addTeleoperator(teleoperators[0])
|
332 |
+
this.mediaRecorders = {};
|
333 |
+
this.videoChunks = {};
|
334 |
+
this.videoBlobs = {};
|
335 |
+
this.videoStreams = {};
|
336 |
+
this.teleoperatorData = [];
|
337 |
+
this._isRecording = false;
|
338 |
+
this.fps = fps;
|
339 |
+
this.taskDescription = taskDescription;
|
340 |
+
|
341 |
+
for(const [key, stream] of Object.entries(videoStreams)) {
|
342 |
+
this.addVideoStream(key, stream);
|
343 |
+
}
|
344 |
+
}
|
345 |
+
|
346 |
+
get isRecording() : boolean {
|
347 |
+
return this._isRecording;
|
348 |
+
}
|
349 |
+
|
350 |
+
get currentEpisode() : LeRobotEpisode | undefined {
|
351 |
+
return this.teleoperatorData.at(-1);
|
352 |
+
}
|
353 |
+
|
354 |
+
/**
|
355 |
+
* Adds a new video stream to be recorded
|
356 |
+
* @param key The key to identify this video stream
|
357 |
+
* @param stream The media stream to record from
|
358 |
+
*/
|
359 |
+
addVideoStream(key: string, stream: MediaStream) {
|
360 |
+
console.log("Adding video stream", key);
|
361 |
+
if (this._isRecording) {
|
362 |
+
throw new Error("Cannot add video streams while recording");
|
363 |
+
}
|
364 |
+
|
365 |
+
// Add to video streams dictionary
|
366 |
+
this.videoStreams[key] = stream;
|
367 |
+
|
368 |
+
// Initialize MediaRecorder for this stream
|
369 |
+
this.mediaRecorders[key] = new MediaRecorder(stream, {
|
370 |
+
mimeType: 'video/mp4'
|
371 |
+
});
|
372 |
+
|
373 |
+
// add a video chunk array for this stream
|
374 |
+
this.videoChunks[key] = [];
|
375 |
+
}
|
376 |
+
|
377 |
+
/**
|
378 |
+
* Add a new teleoperator and set up state update callbacks
|
379 |
+
* for recording joint position data in the LeRobot dataset format
|
380 |
+
*
|
381 |
+
* @param teleoperator The teleoperator to add callbacks to
|
382 |
+
*/
|
383 |
+
addTeleoperator(teleoperator : WebTeleoperator) {
|
384 |
+
teleoperator.addOnStateUpdateCallback(params => {
|
385 |
+
if(this._isRecording){
|
386 |
+
if(!this.currentEpisode) throw Error("There is no current episode while recording, something is wrong!, this means that no frames exist on the recorder for some reason")
|
387 |
+
|
388 |
+
// Create a frame with the current state data
|
389 |
+
// Using the normalized configs for consistent data ranges
|
390 |
+
const frame : NonIndexedLeRobotDatasetRow = {
|
391 |
+
timestamp : params.commandSentTimestamp,
|
392 |
+
// For observation state, use the current motor positions
|
393 |
+
"observation.state" : this.convertMotorConfigToArray(params.newMotorConfigsNormalized),
|
394 |
+
// For action, use the target positions that were commanded
|
395 |
+
action : this.convertMotorConfigToArray(params.previousMotorConfigsNormalized),
|
396 |
+
episode_index : this.episodeIndex,
|
397 |
+
task_index : this.taskIndex
|
398 |
+
}
|
399 |
+
|
400 |
+
// Add the frame to the current episode
|
401 |
+
this.currentEpisode.add(frame);
|
402 |
+
}
|
403 |
+
});
|
404 |
+
|
405 |
+
this.teleoperators.push(teleoperator)
|
406 |
+
}
|
407 |
+
|
408 |
+
/**
|
409 |
+
* Starts recording for all teleoperators and video streams
|
410 |
+
*/
|
411 |
+
startRecording() {
|
412 |
+
console.log("Starting recording");
|
413 |
+
if (this._isRecording) {
|
414 |
+
console.warn('Recording already in progress');
|
415 |
+
return;
|
416 |
+
}
|
417 |
+
|
418 |
+
this._isRecording = true;
|
419 |
+
|
420 |
+
// add a new episode
|
421 |
+
this.teleoperatorData.push(new LeRobotEpisode())
|
422 |
+
|
423 |
+
// Start recording video streams
|
424 |
+
Object.entries(this.videoStreams).forEach(([key, stream]) => {
|
425 |
+
// Create a media recorder for this stream
|
426 |
+
const mediaRecorder = new MediaRecorder(stream, {
|
427 |
+
mimeType: 'video/mp4'
|
428 |
+
});
|
429 |
+
|
430 |
+
// Handle data available events
|
431 |
+
mediaRecorder.ondataavailable = (event) => {
|
432 |
+
console.log("data available for", key);
|
433 |
+
if (event.data && event.data.size > 0) {
|
434 |
+
this.videoChunks[key].push(event.data);
|
435 |
+
}
|
436 |
+
};
|
437 |
+
|
438 |
+
// Save the recorder and start recording
|
439 |
+
this.mediaRecorders[key] = mediaRecorder;
|
440 |
+
mediaRecorder.start(1000); // Capture in 1-second chunks
|
441 |
+
|
442 |
+
console.log(`Started recording video stream: ${key}`);
|
443 |
+
});
|
444 |
+
}
|
445 |
+
|
446 |
+
setEpisodeIndex(index: number): void {
|
447 |
+
this.episodeIndex = index
|
448 |
+
}
|
449 |
+
|
450 |
+
setTaskIndex(index: number): void {
|
451 |
+
this.taskIndex = index
|
452 |
+
}
|
453 |
+
|
454 |
+
/**
|
455 |
+
* teleoperatorData by default only contains data
|
456 |
+
* for the episodes in a non-regularized manner
|
457 |
+
*
|
458 |
+
* this function returns episodes in a regularized manner, wherein
|
459 |
+
* the frames in each are interpolated through so that all frames are spaced
|
460 |
+
* equally through each other
|
461 |
+
*/
|
462 |
+
get episodes() : LeRobotEpisode[]{
|
463 |
+
const regularizedEpisodes : LeRobotEpisode[] = [];
|
464 |
+
let lastFrameIndex = 0;
|
465 |
+
|
466 |
+
for(let i=0; i<this.teleoperatorData.length; i++){
|
467 |
+
let episode = this.teleoperatorData[i];
|
468 |
+
const regularizedEpisode = episode.getInterpolatedRegularEpisode(this.fps, lastFrameIndex);
|
469 |
+
regularizedEpisodes.push(regularizedEpisode)
|
470 |
+
|
471 |
+
lastFrameIndex += regularizedEpisode.frames?.at(-1)?.index || 0
|
472 |
+
}
|
473 |
+
|
474 |
+
return regularizedEpisodes
|
475 |
+
}
|
476 |
+
|
477 |
+
/**
|
478 |
+
* Stops recording for all teleoperators and video streams
|
479 |
+
* @returns An object containing teleoperator data and video blobs
|
480 |
+
*/
|
481 |
+
async stopRecording() {
|
482 |
+
if (!this._isRecording) {
|
483 |
+
console.warn('No recording in progress');
|
484 |
+
return { teleoperatorData: [], videoBlobs: {} };
|
485 |
+
}
|
486 |
+
|
487 |
+
this._isRecording = false;
|
488 |
+
|
489 |
+
// Stop all media recorders
|
490 |
+
const stopPromises = Object.entries(this.mediaRecorders).map(([key, recorder]) => {
|
491 |
+
return new Promise<void>((resolve) => {
|
492 |
+
// Only do this if the recorder is active
|
493 |
+
if (recorder.state === 'inactive') {
|
494 |
+
resolve();
|
495 |
+
return;
|
496 |
+
}
|
497 |
+
|
498 |
+
// When the recorder stops, create a blob
|
499 |
+
recorder.onstop = () => {
|
500 |
+
// Combine all chunks into a single blob
|
501 |
+
const chunks = this.videoChunks[key] || [];
|
502 |
+
const blob = new Blob(chunks, { type: 'video/mp4' });
|
503 |
+
this.videoBlobs[key] = blob;
|
504 |
+
resolve();
|
505 |
+
};
|
506 |
+
|
507 |
+
// Stop the recorder
|
508 |
+
recorder.stop();
|
509 |
+
});
|
510 |
+
});
|
511 |
+
|
512 |
+
// Wait for all recorders to stop
|
513 |
+
await Promise.all(stopPromises);
|
514 |
+
return {
|
515 |
+
teleoperatorData: this.episodes,
|
516 |
+
videoBlobs: this.videoBlobs
|
517 |
+
};
|
518 |
+
}
|
519 |
+
|
520 |
+
/**
|
521 |
+
* Clears the teleoperator data and video blobs
|
522 |
+
*/
|
523 |
+
clearRecording() {
|
524 |
+
this.teleoperatorData = [];
|
525 |
+
this.videoBlobs = {};
|
526 |
+
}
|
527 |
+
|
528 |
+
/**
|
529 |
+
* Action is a dictionary of motor positions, timestamp1 and timestamp2 are when the actions occurred
|
530 |
+
* reqTimestamp must be between timestamp1 and timestamp2
|
531 |
+
*
|
532 |
+
* the keys in action1 and action2 must match, this will loop through the dictionary
|
533 |
+
* and interpolate each action to the required timestamp
|
534 |
+
*
|
535 |
+
* @param action1 Motor positions at timestamp1
|
536 |
+
* @param action2 Motor positions at timestamp2
|
537 |
+
* @param timestamp1 The timestamp of action1
|
538 |
+
* @param timestamp2 The timestamp of action2
|
539 |
+
* @param reqTimestamp The timestamp at which to interpolate
|
540 |
+
* @returns The interpolated action
|
541 |
+
*/
|
542 |
+
_actionInterpolatate(action1 : any, action2 : any, timestamp1 : number, timestamp2 : number, reqTimestamp : number) : any {
|
543 |
+
if(reqTimestamp < timestamp1 || reqTimestamp > timestamp2) throw new Error("reqTimestamp must be between timestamp1 and timestamp2");
|
544 |
+
if(timestamp2 < timestamp1) throw new Error("timestamp2 must be greater than timestamp1");
|
545 |
+
|
546 |
+
const numActions = Object.keys(action1).length;
|
547 |
+
const interpolatedAction : any = {};
|
548 |
+
const timeRange = timestamp2 - timestamp1;
|
549 |
+
|
550 |
+
for(let i = 0; i < numActions; i++){
|
551 |
+
const key = Object.keys(action1)[i];
|
552 |
+
interpolatedAction[key] = action1[key] + (action2[key] - action1[key]) * (reqTimestamp - timestamp1) / timeRange;
|
553 |
+
}
|
554 |
+
|
555 |
+
return interpolatedAction;
|
556 |
+
}
|
557 |
+
|
558 |
+
/**
|
559 |
+
* Converts an action object to an array of numbers
|
560 |
+
* follows the same pattern as https://huggingface.co/datasets/lerobot/svla_so100_pickplace
|
561 |
+
* I am not really sure if the array can be in a different order
|
562 |
+
* but I am not going to risk it tbh 😛
|
563 |
+
*
|
564 |
+
* @param action The action object to convert
|
565 |
+
* @returns An array of numbers
|
566 |
+
*/
|
567 |
+
convertActionToArray(action : any) : number[] {
|
568 |
+
return [
|
569 |
+
action["shoulder_pan"],
|
570 |
+
action["shoulder_lift"],
|
571 |
+
action["elbow_flex"],
|
572 |
+
action["wrist_flex"],
|
573 |
+
action["wrist_roll"],
|
574 |
+
action["gripper"]
|
575 |
+
]
|
576 |
+
}
|
577 |
+
|
578 |
+
/**
|
579 |
+
* Converts an array of MotorConfig objects to an action object
|
580 |
+
* following the same joint order as convertActionToArray
|
581 |
+
*
|
582 |
+
* @param motorConfigs Array of MotorConfig objects
|
583 |
+
* @returns An action object with joint positions
|
584 |
+
*/
|
585 |
+
convertMotorConfigToArray(motorConfigs: MotorConfig[]) : number[] {
|
586 |
+
// Create a map for quick lookup of motor positions by name
|
587 |
+
const motorMap: Record<string, number> = {};
|
588 |
+
for (const config of motorConfigs) {
|
589 |
+
motorMap[config.name] = config.currentPosition;
|
590 |
+
}
|
591 |
+
|
592 |
+
// Define required joint names
|
593 |
+
const requiredJoints = [
|
594 |
+
"shoulder_pan",
|
595 |
+
"shoulder_lift",
|
596 |
+
"elbow_flex",
|
597 |
+
"wrist_flex",
|
598 |
+
"wrist_roll",
|
599 |
+
"gripper"
|
600 |
+
];
|
601 |
+
|
602 |
+
// Check that all required joints are present
|
603 |
+
const missingJoints = requiredJoints.filter(joint => motorMap[joint] === undefined);
|
604 |
+
if (missingJoints.length > 0) {
|
605 |
+
throw new Error(`Missing required joints in motor configs: ${missingJoints.join(", ")}. Available joints: ${Object.keys(motorMap).join(", ")}`);
|
606 |
+
}
|
607 |
+
|
608 |
+
// Return in the same order as convertActionToArray
|
609 |
+
return [
|
610 |
+
motorMap["shoulder_pan"],
|
611 |
+
motorMap["shoulder_lift"],
|
612 |
+
motorMap["elbow_flex"],
|
613 |
+
motorMap["wrist_flex"],
|
614 |
+
motorMap["wrist_roll"],
|
615 |
+
motorMap["gripper"]
|
616 |
+
];
|
617 |
+
}
|
618 |
+
|
619 |
+
/**
|
620 |
+
* Finds the closest timestamp to the target timestamp
|
621 |
+
*
|
622 |
+
* the data must have timestamps in ascending order
|
623 |
+
* uses binary search to get the closest timestamp
|
624 |
+
*
|
625 |
+
* @param data The data to search through
|
626 |
+
* @param targetTimestamp The target timestamp
|
627 |
+
* @returns The closest timestamp in the data's index
|
628 |
+
*/
|
629 |
+
_findClosestTimestampBefore(data : any[], targetTimestamp : number): number{
|
630 |
+
let firstIndex = 0;
|
631 |
+
let lastIndex = data.length - 1;
|
632 |
+
|
633 |
+
while (firstIndex <= lastIndex) {
|
634 |
+
const middleIndex = Math.floor((firstIndex + lastIndex) / 2);
|
635 |
+
const middleTimestamp = data[middleIndex].timestamp;
|
636 |
+
|
637 |
+
if (middleTimestamp === targetTimestamp) {
|
638 |
+
return middleIndex;
|
639 |
+
} else if (middleTimestamp < targetTimestamp) {
|
640 |
+
firstIndex = middleIndex + 1;
|
641 |
+
} else {
|
642 |
+
lastIndex = middleIndex - 1;
|
643 |
+
}
|
644 |
+
}
|
645 |
+
|
646 |
+
return lastIndex;
|
647 |
+
}
|
648 |
+
|
649 |
+
/**
|
650 |
+
* Takes non-regularly spaced lerobot-ish data and interpolates it to a regularly spaced dataset
|
651 |
+
* also adds additional
|
652 |
+
* - frame_index
|
653 |
+
* - episode_index
|
654 |
+
* - index columns
|
655 |
+
*
|
656 |
+
* to match lerobot dataset requirements
|
657 |
+
*/
|
658 |
+
_interpolateAndCompleteLerobotData(fps : number, frameData : NonIndexedLeRobotDatasetRow[], lastFrameIndex : number) : LeRobotDatasetRow[]{
|
659 |
+
const interpolatedData : LeRobotDatasetRow[] = [];
|
660 |
+
const minTimestamp = frameData[0].timestamp;
|
661 |
+
const maxTimestamp = frameData[frameData.length - 1].timestamp;
|
662 |
+
const timeDifference = maxTimestamp - minTimestamp;
|
663 |
+
const numFrames = Math.floor(timeDifference * fps);
|
664 |
+
const firstFrame = frameData[0]
|
665 |
+
|
666 |
+
|
667 |
+
console.log("frames before interpolation", numFrames, frameData[0].timestamp, frameData[frameData.length - 1].timestamp, fps)
|
668 |
+
|
669 |
+
interpolatedData.push({
|
670 |
+
timestamp: firstFrame.timestamp,
|
671 |
+
action: this.convertActionToArray(firstFrame.action),
|
672 |
+
"observation.state": this.convertActionToArray(firstFrame["observation.state"]),
|
673 |
+
episode_index: firstFrame.episode_index,
|
674 |
+
task_index: firstFrame.task_index,
|
675 |
+
frame_index : 0,
|
676 |
+
index: lastFrameIndex
|
677 |
+
})
|
678 |
+
|
679 |
+
// start from 1 as the first frame is pushed already (see above)
|
680 |
+
for(let i = 1; i < numFrames; i++){
|
681 |
+
const timestamp = (i / fps)
|
682 |
+
const closestIndex = this._findClosestTimestampBefore(frameData, timestamp);
|
683 |
+
const nextIndex = closestIndex + 1;
|
684 |
+
const closestItemData = frameData[closestIndex];
|
685 |
+
const nextItemData = frameData[nextIndex];
|
686 |
+
const action = this._actionInterpolatate(closestItemData.action, nextItemData.action, closestItemData.timestamp, nextItemData.timestamp, timestamp);
|
687 |
+
const observation_state = this._actionInterpolatate(closestItemData["observation.state"], nextItemData["observation.state"], closestItemData.timestamp, nextItemData.timestamp, timestamp);
|
688 |
+
|
689 |
+
interpolatedData.push({
|
690 |
+
timestamp: timestamp,
|
691 |
+
action: this.convertActionToArray(action),
|
692 |
+
"observation.state": this.convertActionToArray(observation_state),
|
693 |
+
episode_index: closestItemData.episode_index,
|
694 |
+
task_index: closestItemData.task_index,
|
695 |
+
frame_index : i,
|
696 |
+
index: lastFrameIndex + i
|
697 |
+
});
|
698 |
+
|
699 |
+
}
|
700 |
+
|
701 |
+
return interpolatedData;
|
702 |
+
}
|
703 |
+
|
704 |
+
/**
|
705 |
+
* converts all the frames of a recording into lerobot dataset frame style
|
706 |
+
*
|
707 |
+
* NOTE : This does not interpolate the data, you are only working with raw data
|
708 |
+
* that is called by the teleop when things are actively "changing"
|
709 |
+
* @param episodeRough
|
710 |
+
*/
|
711 |
+
_convertToLeRobotDataFormatFrames(episodeRough : any[]) : NonIndexedLeRobotDatasetRow[]{
|
712 |
+
const properFormatFrames : NonIndexedLeRobotDatasetRow[] = [];
|
713 |
+
|
714 |
+
const firstTimestamp = episodeRough[0].commandSentTimestamp;
|
715 |
+
for(let i=0; i<episodeRough.length; i++){
|
716 |
+
const frameRough = episodeRough[i]
|
717 |
+
|
718 |
+
properFormatFrames.push({
|
719 |
+
timestamp: frameRough.commandSentTimestamp - firstTimestamp, // so timestamps start from 0, and are in seconds
|
720 |
+
action: frameRough.previousMotorConfigsNormalized,
|
721 |
+
"observation.state": frameRough.newMotorConfigsNormalized,
|
722 |
+
episode_index: frameRough.episodeIndex,
|
723 |
+
task_index: frameRough.taskIndex
|
724 |
+
});
|
725 |
+
}
|
726 |
+
|
727 |
+
return properFormatFrames
|
728 |
+
}
|
729 |
+
|
730 |
+
/**
|
731 |
+
* Converts teleoperator data to a parquet blob
|
732 |
+
* @private
|
733 |
+
* @returns Array of objects containing parquet file content and path
|
734 |
+
*/
|
735 |
+
private async _exportEpisodesToBlob(episodes: LeRobotEpisode[]): Promise<{content: Blob, path: string}[]> {
|
736 |
+
// combine all the frames
|
737 |
+
let data : NonIndexedLeRobotDatasetRow[] = [];
|
738 |
+
const episodeBlobs : any[] = []
|
739 |
+
|
740 |
+
for(let i=0; i<episodes.length; i++){
|
741 |
+
const episode = episodes[i]
|
742 |
+
data = episode.frames
|
743 |
+
const { tableFromArrays, vectorFromArray } = arrow;
|
744 |
+
|
745 |
+
const timestamps = data.map((row: any) => row.timestamp);
|
746 |
+
const actions = data.map((row: any) => row.action);
|
747 |
+
const observationStates = data.map((row: any) => row["observation.state"]);
|
748 |
+
const episodeIndexes = data.map((row: any) => row.episode_index);
|
749 |
+
const taskIndexes = data.map((row: any) => row.task_index);
|
750 |
+
const frameIndexes = data.map((row: any) => row.frame_index);
|
751 |
+
const indexes = data.map((row: any) => row.index);
|
752 |
+
|
753 |
+
const table = tableFromArrays({
|
754 |
+
timestamp: timestamps,
|
755 |
+
// @ts-ignore, this works, idk why
|
756 |
+
action: vectorFromArray(actions, new arrow.List(new arrow.Field("item", new arrow.Float32()))),
|
757 |
+
// @ts-ignore, this works, idk why
|
758 |
+
"observation.state": vectorFromArray(observationStates, new arrow.List(new arrow.Field("item", new arrow.Float32()))),
|
759 |
+
episode_index: episodeIndexes,
|
760 |
+
task_index: taskIndexes,
|
761 |
+
frame_index: frameIndexes,
|
762 |
+
index: indexes
|
763 |
+
});
|
764 |
+
|
765 |
+
const wasmUrl = "https://cdn.jsdelivr.net/npm/[email protected]/esm/parquet_wasm_bg.wasm";
|
766 |
+
const initWasm = parquet.default;
|
767 |
+
await initWasm(wasmUrl);
|
768 |
+
|
769 |
+
const wasmTable = parquet.Table.fromIPCStream(arrow.tableToIPC(table, "stream"));
|
770 |
+
const writerProperties = new parquet.WriterPropertiesBuilder()
|
771 |
+
.setCompression(parquet.Compression.UNCOMPRESSED)
|
772 |
+
.build();
|
773 |
+
|
774 |
+
const parquetUint8Array = parquet.writeParquet(wasmTable, writerProperties);
|
775 |
+
const numpadded = i.toString().padStart(6, "0")
|
776 |
+
const content = new Blob([parquetUint8Array])
|
777 |
+
|
778 |
+
episodeBlobs.push({
|
779 |
+
content, path: `data/chunk-000/episode_${numpadded}.parquet`
|
780 |
+
})
|
781 |
+
|
782 |
+
}
|
783 |
+
|
784 |
+
return episodeBlobs
|
785 |
+
}
|
786 |
+
|
787 |
+
/**
|
788 |
+
* Exports the teleoperator data in lerobot format
|
789 |
+
* @param format The format to return the data in ('json' or 'blob')
|
790 |
+
* @returns Either an array of data objects or a Uint8Array blob depending on format
|
791 |
+
*/
|
792 |
+
exportEpisodes(format: 'json' | 'blob' = 'json') {
|
793 |
+
if(this._isRecording) throw new Error("This can only be called after recording has stopped!");
|
794 |
+
const data = this.episodes;
|
795 |
+
|
796 |
+
if (format === 'json') {
|
797 |
+
return data;
|
798 |
+
} else {
|
799 |
+
return this._exportEpisodesToBlob(data);
|
800 |
+
}
|
801 |
+
}
|
802 |
+
|
803 |
+
/**
|
804 |
+
* Exports the media (video) data as blobs
|
805 |
+
* @returns A dictionary of video blobs with the same keys as videoStreams
|
806 |
+
*/
|
807 |
+
async exportMediaData(): Promise<{ [key: string]: Blob }> {
|
808 |
+
if(this._isRecording) throw new Error("This can only be called after recording has stopped!");
|
809 |
+
return this.videoBlobs;
|
810 |
+
}
|
811 |
+
|
812 |
+
/**
|
813 |
+
* Generates metadata for the dataset
|
814 |
+
* @returns Metadata object for the LeRobot dataset
|
815 |
+
*/
|
816 |
+
async generateMetadata(data : any[]): Promise<any> {
|
817 |
+
// Calculate total episodes, frames, and tasks
|
818 |
+
let total_episodes = 0;
|
819 |
+
const total_frames = data.length;
|
820 |
+
let total_tasks = 0;
|
821 |
+
|
822 |
+
for (const row of data) {
|
823 |
+
total_episodes = Math.max(total_episodes, row.episode_index);
|
824 |
+
total_tasks = Math.max(total_tasks, row.task_index);
|
825 |
+
}
|
826 |
+
|
827 |
+
// Create video info objects for each video stream
|
828 |
+
const videos_info: VideoInfo[] = Object.keys(this.videoBlobs).map(key => {
|
829 |
+
// Default values - in a production environment, you would extract
|
830 |
+
// these from the actual video metadata using the key to identify the video source
|
831 |
+
console.log(`Generating metadata for video stream: ${key}`);
|
832 |
+
return {
|
833 |
+
height: 480,
|
834 |
+
width: 640,
|
835 |
+
channels: 3,
|
836 |
+
codec: 'h264',
|
837 |
+
pix_fmt: 'yuv420p',
|
838 |
+
is_depth_map: false,
|
839 |
+
has_audio: false
|
840 |
+
};
|
841 |
+
});
|
842 |
+
|
843 |
+
// Calculate approximate file sizes in MB
|
844 |
+
const data_files_size_in_mb = Math.round(data.length * 0.001); // Estimate
|
845 |
+
|
846 |
+
// Calculate video size by summing the sizes of all video blobs and converting to MB
|
847 |
+
let video_files_size_in_mb = 0;
|
848 |
+
for (const blob of Object.values(this.videoBlobs)) {
|
849 |
+
video_files_size_in_mb += blob.size / (1024 * 1024);
|
850 |
+
}
|
851 |
+
video_files_size_in_mb = Math.round(video_files_size_in_mb);
|
852 |
+
|
853 |
+
// Generate and return the metadata
|
854 |
+
return getMetadataInfo({
|
855 |
+
total_episodes,
|
856 |
+
total_frames,
|
857 |
+
total_tasks,
|
858 |
+
chunks_size: 1000, // Default chunk size
|
859 |
+
fps: this.fps,
|
860 |
+
splits: { "train": `0:${total_episodes}` }, // All episodes in train split
|
861 |
+
features: {}, // Additional features can be added here
|
862 |
+
videos_info,
|
863 |
+
data_files_size_in_mb,
|
864 |
+
video_files_size_in_mb
|
865 |
+
});
|
866 |
+
}
|
867 |
+
|
868 |
+
/**
|
869 |
+
* Generates statistics for the dataset
|
870 |
+
* @returns Statistics object for the LeRobot dataset
|
871 |
+
*/
|
872 |
+
async getStatistics(data : any[]): Promise<any> {
|
873 |
+
|
874 |
+
// Get camera keys from the video blobs
|
875 |
+
const cameraKeys = Object.keys(this.videoBlobs);
|
876 |
+
|
877 |
+
// Generate stats using the data and camera keys
|
878 |
+
return getStats(data, cameraKeys);
|
879 |
+
}
|
880 |
+
|
881 |
+
/**
|
882 |
+
* Creates a tasks.parquet file containing task description
|
883 |
+
* @returns A Uint8Array blob containing the parquet data
|
884 |
+
*/
|
885 |
+
async createTasksParquet(): Promise<Uint8Array> {
|
886 |
+
// Create a simple data structure with the task description
|
887 |
+
const tasksData = [{
|
888 |
+
task_index: 0,
|
889 |
+
__index_level_0__: this.taskDescription
|
890 |
+
}];
|
891 |
+
|
892 |
+
// Create Arrow table from the data
|
893 |
+
const taskIndexArr = arrow.vectorFromArray(tasksData.map(d => d.task_index), new arrow.Int32());
|
894 |
+
const descriptionArr = arrow.vectorFromArray(tasksData.map(d => d.__index_level_0__), new arrow.Utf8());
|
895 |
+
|
896 |
+
const table = arrow.tableFromArrays({
|
897 |
+
// @ts-ignore, this works, idk why
|
898 |
+
task_index: taskIndexArr,
|
899 |
+
// @ts-ignore, this works, idk why
|
900 |
+
__index_level_0__: descriptionArr
|
901 |
+
});
|
902 |
+
|
903 |
+
// Initialize the WASM module
|
904 |
+
const wasmUrl = "https://cdn.jsdelivr.net/npm/[email protected]/esm/parquet_wasm_bg.wasm";
|
905 |
+
const initWasm = parquet.default;
|
906 |
+
await initWasm(wasmUrl);
|
907 |
+
|
908 |
+
// Convert Arrow table to Parquet WASM table
|
909 |
+
const wasmTable = parquet.Table.fromIPCStream(arrow.tableToIPC(table, "stream"));
|
910 |
+
|
911 |
+
// Set compression properties
|
912 |
+
const writerProperties = new parquet.WriterPropertiesBuilder()
|
913 |
+
.setCompression(parquet.Compression.UNCOMPRESSED)
|
914 |
+
.build();
|
915 |
+
|
916 |
+
// Write the Parquet file
|
917 |
+
return parquet.writeParquet(wasmTable, writerProperties);
|
918 |
+
}
|
919 |
+
|
920 |
+
/**
|
921 |
+
* Creates the episodes statistics parquet file
|
922 |
+
* @returns A Uint8Array blob containing the parquet data
|
923 |
+
*/
|
924 |
+
async getEpisodeStatistics(data : any[]): Promise<Uint8Array> {
|
925 |
+
const { vectorFromArray } = arrow;
|
926 |
+
const statistics = await this.getStatistics(data);
|
927 |
+
|
928 |
+
// Calculate total episodes and frames
|
929 |
+
let total_episodes = 0;
|
930 |
+
|
931 |
+
for(let row of data){
|
932 |
+
total_episodes = Math.max(total_episodes, row.episode_index)
|
933 |
+
}
|
934 |
+
|
935 |
+
total_episodes += 1; // +1 since episodes start from 0
|
936 |
+
|
937 |
+
const episodes: any[] = [];
|
938 |
+
|
939 |
+
// we'll create one row per episode
|
940 |
+
for (let episode_index = 0; episode_index < total_episodes; episode_index++) {
|
941 |
+
// Get data for this episode only
|
942 |
+
const episodeData = data.filter(row => row.episode_index === episode_index);
|
943 |
+
|
944 |
+
// Extract timestamps for this episode
|
945 |
+
const timestamps = episodeData.map(row => row.timestamp);
|
946 |
+
let min_timestamp = Infinity;
|
947 |
+
let max_timestamp = -Infinity;
|
948 |
+
|
949 |
+
for(let timestamp of timestamps){
|
950 |
+
min_timestamp = Math.min(min_timestamp, timestamp);
|
951 |
+
max_timestamp = Math.max(max_timestamp, timestamp);
|
952 |
+
}
|
953 |
+
|
954 |
+
|
955 |
+
|
956 |
+
// Camera keys from video blobs
|
957 |
+
const cameraKeys = Object.keys(this.videoBlobs);
|
958 |
+
|
959 |
+
// Create entry for this episode
|
960 |
+
const episodeEntry: any = {
|
961 |
+
// Basic episode information
|
962 |
+
episode_index: episode_index,
|
963 |
+
"data/chunk_index": 0,
|
964 |
+
"data/file_index": 0,
|
965 |
+
dataset_from_index: 0,
|
966 |
+
dataset_to_index: episodeData.length - 1,
|
967 |
+
length: episodeData.length,
|
968 |
+
tasks: [0], // Task index 0, could be extended for multiple tasks
|
969 |
+
|
970 |
+
// Meta information
|
971 |
+
"meta/episodes/chunk_index": 0,
|
972 |
+
"meta/episodes/file_index": 0,
|
973 |
+
};
|
974 |
+
|
975 |
+
// Add video information for each camera
|
976 |
+
cameraKeys.forEach(key => {
|
977 |
+
episodeEntry[`videos/observation.images.${key}/chunk_index`] = 0;
|
978 |
+
episodeEntry[`videos/observation.images.${key}/file_index`] = 0;
|
979 |
+
episodeEntry[`videos/observation.images.${key}/from_timestamp`] = min_timestamp;
|
980 |
+
episodeEntry[`videos/observation.images.${key}/to_timestamp`] = max_timestamp;
|
981 |
+
});
|
982 |
+
|
983 |
+
// Add statistics for each field
|
984 |
+
// This is a simplified approach - in a real implementation, you'd calculate
|
985 |
+
// these values for each episode individually
|
986 |
+
|
987 |
+
// Add timestamp statistics
|
988 |
+
episodeEntry["stats/timestamp/min"] = [statistics.timestamp.min];
|
989 |
+
episodeEntry["stats/timestamp/max"] = [statistics.timestamp.max];
|
990 |
+
episodeEntry["stats/timestamp/mean"] = [statistics.timestamp.mean];
|
991 |
+
episodeEntry["stats/timestamp/std"] = [statistics.timestamp.std];
|
992 |
+
episodeEntry["stats/timestamp/count"] = [statistics.timestamp.count];
|
993 |
+
|
994 |
+
// Add frame_index statistics
|
995 |
+
episodeEntry["stats/frame_index/min"] = [statistics.frame_index.min];
|
996 |
+
episodeEntry["stats/frame_index/max"] = [statistics.frame_index.max];
|
997 |
+
episodeEntry["stats/frame_index/mean"] = [statistics.frame_index.mean];
|
998 |
+
episodeEntry["stats/frame_index/std"] = [statistics.frame_index.std];
|
999 |
+
episodeEntry["stats/frame_index/count"] = [statistics.frame_index.count];
|
1000 |
+
|
1001 |
+
// Add episode_index statistics
|
1002 |
+
episodeEntry["stats/episode_index/min"] = [statistics.episode_index.min];
|
1003 |
+
episodeEntry["stats/episode_index/max"] = [statistics.episode_index.max];
|
1004 |
+
episodeEntry["stats/episode_index/mean"] = [statistics.episode_index.mean];
|
1005 |
+
episodeEntry["stats/episode_index/std"] = [statistics.episode_index.std];
|
1006 |
+
episodeEntry["stats/episode_index/count"] = [statistics.episode_index.count];
|
1007 |
+
|
1008 |
+
// Add task_index statistics
|
1009 |
+
episodeEntry["stats/task_index/min"] = [statistics.task_index.min];
|
1010 |
+
episodeEntry["stats/task_index/max"] = [statistics.task_index.max];
|
1011 |
+
episodeEntry["stats/task_index/mean"] = [statistics.task_index.mean];
|
1012 |
+
episodeEntry["stats/task_index/std"] = [statistics.task_index.std];
|
1013 |
+
episodeEntry["stats/task_index/count"] = [statistics.task_index.count];
|
1014 |
+
|
1015 |
+
// Add index statistics
|
1016 |
+
episodeEntry["stats/index/min"] = [0];
|
1017 |
+
episodeEntry["stats/index/max"] = [episodeData.length - 1];
|
1018 |
+
episodeEntry["stats/index/mean"] = [episodeData.length / 2];
|
1019 |
+
episodeEntry["stats/index/std"] = [episodeData.length / 4]; // Approximate std
|
1020 |
+
episodeEntry["stats/index/count"] = [episodeData.length];
|
1021 |
+
|
1022 |
+
// Add action statistics (placeholder)
|
1023 |
+
episodeEntry["stats/action/min"] = [0.0];
|
1024 |
+
episodeEntry["stats/action/max"] = [1.0];
|
1025 |
+
episodeEntry["stats/action/mean"] = [0.5];
|
1026 |
+
episodeEntry["stats/action/std"] = [0.2];
|
1027 |
+
episodeEntry["stats/action/count"] = [episodeData.length];
|
1028 |
+
|
1029 |
+
// Add observation.state statistics (placeholder)
|
1030 |
+
episodeEntry["stats/observation.state/min"] = [0.0];
|
1031 |
+
episodeEntry["stats/observation.state/max"] = [1.0];
|
1032 |
+
episodeEntry["stats/observation.state/mean"] = [0.5];
|
1033 |
+
episodeEntry["stats/observation.state/std"] = [0.2];
|
1034 |
+
episodeEntry["stats/observation.state/count"] = [episodeData.length];
|
1035 |
+
|
1036 |
+
// Add observation.images statistics for each camera
|
1037 |
+
cameraKeys.forEach(key => {
|
1038 |
+
// Get the image statistics from the overall statistics
|
1039 |
+
const imageStats = statistics[`observation.images.${key}`] || {
|
1040 |
+
min: [[[0.0]], [[0.0]], [[0.0]]],
|
1041 |
+
max: [[[255.0]], [[255.0]], [[255.0]]],
|
1042 |
+
mean: [[[127.5]], [[127.5]], [[127.5]]],
|
1043 |
+
std: [[[50.0]], [[50.0]], [[50.0]]],
|
1044 |
+
count: [[[episodeData.length * 3]]]
|
1045 |
+
};
|
1046 |
+
|
1047 |
+
episodeEntry[`stats/observation.images.${key}/min`] = imageStats.min;
|
1048 |
+
episodeEntry[`stats/observation.images.${key}/max`] = imageStats.max;
|
1049 |
+
episodeEntry[`stats/observation.images.${key}/mean`] = imageStats.mean;
|
1050 |
+
episodeEntry[`stats/observation.images.${key}/std`] = imageStats.std;
|
1051 |
+
episodeEntry[`stats/observation.images.${key}/count`] = imageStats.count;
|
1052 |
+
});
|
1053 |
+
|
1054 |
+
episodes.push(episodeEntry);
|
1055 |
+
}
|
1056 |
+
|
1057 |
+
// Create vector arrays for each column
|
1058 |
+
const columns: any = {};
|
1059 |
+
|
1060 |
+
// Define column names and default types
|
1061 |
+
const columnNames = [
|
1062 |
+
"episode_index", "data/chunk_index", "data/file_index", "dataset_from_index", "dataset_to_index",
|
1063 |
+
"length", "meta/episodes/chunk_index", "meta/episodes/file_index", "tasks"
|
1064 |
+
];
|
1065 |
+
|
1066 |
+
// Add camera-specific columns
|
1067 |
+
const cameraKeys = Object.keys(this.videoBlobs);
|
1068 |
+
cameraKeys.forEach(key => {
|
1069 |
+
columnNames.push(
|
1070 |
+
`videos/observation.images.${key}/chunk_index`,
|
1071 |
+
`videos/observation.images.${key}/file_index`,
|
1072 |
+
`videos/observation.images.${key}/from_timestamp`,
|
1073 |
+
`videos/observation.images.${key}/to_timestamp`
|
1074 |
+
);
|
1075 |
+
});
|
1076 |
+
|
1077 |
+
// Add statistic columns for each field
|
1078 |
+
const statFields = ["timestamp", "frame_index", "episode_index", "task_index", "index", "action", "observation.state"];
|
1079 |
+
statFields.forEach(field => {
|
1080 |
+
columnNames.push(
|
1081 |
+
`stats/${field}/min`,
|
1082 |
+
`stats/${field}/max`,
|
1083 |
+
`stats/${field}/mean`,
|
1084 |
+
`stats/${field}/std`,
|
1085 |
+
`stats/${field}/count`
|
1086 |
+
);
|
1087 |
+
});
|
1088 |
+
|
1089 |
+
// Add image statistic columns for each camera
|
1090 |
+
cameraKeys.forEach(key => {
|
1091 |
+
columnNames.push(
|
1092 |
+
`stats/observation.images.${key}/min`,
|
1093 |
+
`stats/observation.images.${key}/max`,
|
1094 |
+
`stats/observation.images.${key}/mean`,
|
1095 |
+
`stats/observation.images.${key}/std`,
|
1096 |
+
`stats/observation.images.${key}/count`
|
1097 |
+
);
|
1098 |
+
});
|
1099 |
+
|
1100 |
+
// Create vector arrays for each column
|
1101 |
+
columnNames.forEach(columnName => {
|
1102 |
+
const values = episodes.map(ep => ep[columnName] || 0);
|
1103 |
+
|
1104 |
+
// Check if the column is an array type and needs special handling
|
1105 |
+
if (columnName.includes('stats/') || columnName === 'tasks') {
|
1106 |
+
// Handle different types of array columns based on their naming pattern
|
1107 |
+
if (columnName.includes('/count')) {
|
1108 |
+
// Bigint arrays for count fields
|
1109 |
+
// @ts-ignore
|
1110 |
+
columns[columnName] = vectorFromArray(values.map(v => Number(v)), new arrow.List(new arrow.Field("item", new arrow.Int64())));
|
1111 |
+
} else if (columnName.includes('/min') || columnName.includes('/max') ||
|
1112 |
+
columnName.includes('/mean') || columnName.includes('/std')) {
|
1113 |
+
// Double arrays for min, max, mean, std fields
|
1114 |
+
if (columnName.includes('observation.images') &&
|
1115 |
+
(columnName.includes('/min') || columnName.includes('/max') ||
|
1116 |
+
columnName.includes('/mean') || columnName.includes('/std'))) {
|
1117 |
+
// These are 3D arrays [[[value]]]
|
1118 |
+
// For 3D arrays, we need nested Lists
|
1119 |
+
// @ts-ignore
|
1120 |
+
columns[columnName] = vectorFromArray(values, new arrow.List(new arrow.Field("item",
|
1121 |
+
new arrow.List(new arrow.Field("subitem",
|
1122 |
+
new arrow.List(new arrow.Field("value", new arrow.Float64())))))));
|
1123 |
+
} else {
|
1124 |
+
// These are normal arrays [value]
|
1125 |
+
// @ts-ignore
|
1126 |
+
columns[columnName] = vectorFromArray(values, new arrow.List(new arrow.Field("item", new arrow.Float64())));
|
1127 |
+
}
|
1128 |
+
} else {
|
1129 |
+
// Default to Float64 List for other array types
|
1130 |
+
// @ts-ignore
|
1131 |
+
columns[columnName] = vectorFromArray(values, new arrow.List(new arrow.Field("item", new arrow.Float64())));
|
1132 |
+
}
|
1133 |
+
} else {
|
1134 |
+
// For non-array columns, use regular vectorFromArray
|
1135 |
+
// @ts-ignore
|
1136 |
+
columns[columnName] = vectorFromArray(values);
|
1137 |
+
}
|
1138 |
+
});
|
1139 |
+
|
1140 |
+
// Create the table with all columns
|
1141 |
+
const table = arrow.tableFromArrays(columns);
|
1142 |
+
|
1143 |
+
// Initialize the WASM module
|
1144 |
+
const wasmUrl = "https://cdn.jsdelivr.net/npm/[email protected]/esm/parquet_wasm_bg.wasm";
|
1145 |
+
const initWasm = parquet.default;
|
1146 |
+
await initWasm(wasmUrl);
|
1147 |
+
|
1148 |
+
// Convert Arrow table to Parquet WASM table
|
1149 |
+
const wasmTable = parquet.Table.fromIPCStream(arrow.tableToIPC(table, "stream"));
|
1150 |
+
|
1151 |
+
// Set compression properties
|
1152 |
+
const writerProperties = new parquet.WriterPropertiesBuilder()
|
1153 |
+
.setCompression(parquet.Compression.UNCOMPRESSED)
|
1154 |
+
.build();
|
1155 |
+
|
1156 |
+
// Write the Parquet file
|
1157 |
+
return parquet.writeParquet(wasmTable, writerProperties);
|
1158 |
+
}
|
1159 |
+
|
1160 |
+
generateREADME(metaInfo : string) {
|
1161 |
+
return generateREADME(metaInfo);
|
1162 |
+
}
|
1163 |
+
|
1164 |
+
/**
|
1165 |
+
* Creates an array of path and blob content objects for the LeRobot dataset
|
1166 |
+
*
|
1167 |
+
* @returns An array of {path, content} objects representing the dataset files
|
1168 |
+
* @private
|
1169 |
+
*/
|
1170 |
+
async _exportForLeRobotBlobs() {
|
1171 |
+
const teleoperatorDataJson = await this.exportEpisodes('json') as any[];
|
1172 |
+
const parquetEpisodeDataFiles = await this._exportEpisodesToBlob(teleoperatorDataJson)
|
1173 |
+
const videoBlobs = await this.exportMediaData();
|
1174 |
+
const metadata = await this.generateMetadata(teleoperatorDataJson);
|
1175 |
+
const statistics = await this.getStatistics(teleoperatorDataJson);
|
1176 |
+
const tasksParquet = await this.createTasksParquet();
|
1177 |
+
const episodesParquet = await this.getEpisodeStatistics(teleoperatorDataJson);
|
1178 |
+
const readme = this.generateREADME(JSON.stringify(metadata));
|
1179 |
+
|
1180 |
+
// Create the blob array with proper paths
|
1181 |
+
const blobArray = [
|
1182 |
+
...parquetEpisodeDataFiles,
|
1183 |
+
{
|
1184 |
+
path: "meta/info.json",
|
1185 |
+
content: new Blob([JSON.stringify(metadata, null, 2)], { type: "application/json" })
|
1186 |
+
},
|
1187 |
+
{
|
1188 |
+
path: "meta/stats.json",
|
1189 |
+
content: new Blob([JSON.stringify(statistics, null, 2)], { type: "application/json" })
|
1190 |
+
},
|
1191 |
+
{
|
1192 |
+
path: "meta/tasks.parquet",
|
1193 |
+
content: new Blob([tasksParquet])
|
1194 |
+
},
|
1195 |
+
{
|
1196 |
+
path: "meta/episodes/chunk-000/file-000.parquet",
|
1197 |
+
content: new Blob([episodesParquet])
|
1198 |
+
},
|
1199 |
+
{
|
1200 |
+
path: "README.md",
|
1201 |
+
content: new Blob([readme], { type: "text/markdown" })
|
1202 |
+
}
|
1203 |
+
];
|
1204 |
+
|
1205 |
+
// Add video blobs with proper paths
|
1206 |
+
for (const [key, blob] of Object.entries(videoBlobs)) {
|
1207 |
+
blobArray.push({
|
1208 |
+
path: `videos/chunk-000/observation.images.${key}/episode_000000.mp4`,
|
1209 |
+
content: blob
|
1210 |
+
});
|
1211 |
+
}
|
1212 |
+
|
1213 |
+
return blobArray;
|
1214 |
+
}
|
1215 |
+
|
1216 |
+
/**
|
1217 |
+
* Creates a ZIP file from the dataset blobs
|
1218 |
+
*
|
1219 |
+
* @returns A Blob containing the ZIP file
|
1220 |
+
* @private
|
1221 |
+
*/
|
1222 |
+
async _exportForLeRobotZip() {
|
1223 |
+
const blobArray = await this._exportForLeRobotBlobs();
|
1224 |
+
const zip = new JSZip();
|
1225 |
+
|
1226 |
+
// Add all blobs to the zip with their paths
|
1227 |
+
for (const item of blobArray) {
|
1228 |
+
// Split the path to handle directories
|
1229 |
+
const pathParts = item.path.split('/');
|
1230 |
+
const fileName = pathParts.pop() || '';
|
1231 |
+
let currentFolder = zip;
|
1232 |
+
|
1233 |
+
// Create nested folders as needed
|
1234 |
+
if (pathParts.length > 0) {
|
1235 |
+
for (const part of pathParts) {
|
1236 |
+
currentFolder = currentFolder.folder(part) || currentFolder;
|
1237 |
+
}
|
1238 |
+
}
|
1239 |
+
|
1240 |
+
// Add file to the current folder
|
1241 |
+
currentFolder.file(fileName, item.content);
|
1242 |
+
}
|
1243 |
+
|
1244 |
+
// Generate the zip file
|
1245 |
+
return await zip.generateAsync({ type: "blob" });
|
1246 |
+
}
|
1247 |
+
|
1248 |
+
/**
|
1249 |
+
* Uploads the LeRobot dataset to Hugging Face
|
1250 |
+
*
|
1251 |
+
* @param username Hugging Face username
|
1252 |
+
* @param repoName Repository name for the dataset
|
1253 |
+
* @param accessToken Hugging Face access token
|
1254 |
+
* @returns The LeRobotHFUploader instance used for upload
|
1255 |
+
*/
|
1256 |
+
async _exportForLeRobotHuggingface(username: string, repoName: string, accessToken: string) {
|
1257 |
+
// Create the blobs array for upload
|
1258 |
+
const blobArray = await this._exportForLeRobotBlobs();
|
1259 |
+
|
1260 |
+
// Create the uploader
|
1261 |
+
const uploader = new LeRobotHFUploader(username, repoName);
|
1262 |
+
|
1263 |
+
// Convert blobs to File objects for HF uploader
|
1264 |
+
const files = blobArray.map(item => {
|
1265 |
+
return {
|
1266 |
+
path: item.path,
|
1267 |
+
content: item.content
|
1268 |
+
};
|
1269 |
+
});
|
1270 |
+
|
1271 |
+
// Generate a unique reference ID for tracking the upload
|
1272 |
+
const referenceId = `lerobot-upload-${Date.now()}`;
|
1273 |
+
|
1274 |
+
try {
|
1275 |
+
// Start the upload process
|
1276 |
+
uploader.createRepoAndUploadFiles(files, accessToken, referenceId);
|
1277 |
+
console.log(`Successfully uploaded dataset to ${username}/${repoName}`);
|
1278 |
+
return uploader;
|
1279 |
+
} catch (error) {
|
1280 |
+
console.error("Error uploading to Hugging Face:", error);
|
1281 |
+
throw error;
|
1282 |
+
}
|
1283 |
+
}
|
1284 |
+
|
1285 |
+
/**
|
1286 |
+
* Uploads the LeRobot dataset to Amazon S3
|
1287 |
+
*
|
1288 |
+
* @param bucketName S3 bucket name
|
1289 |
+
* @param accessKeyId AWS access key ID
|
1290 |
+
* @param secretAccessKey AWS secret access key
|
1291 |
+
* @param region AWS region (default: us-east-1)
|
1292 |
+
* @param prefix Optional prefix (folder) to upload files to within the bucket
|
1293 |
+
* @returns The LeRobotS3Uploader instance used for upload
|
1294 |
+
*/
|
1295 |
+
async _exportForLeRobotS3(bucketName: string, accessKeyId: string, secretAccessKey: string, region: string = "us-east-1", prefix: string = "") {
|
1296 |
+
// Create the blobs array for upload
|
1297 |
+
const blobArray = await this._exportForLeRobotBlobs();
|
1298 |
+
|
1299 |
+
// Create the uploader
|
1300 |
+
const uploader = new LeRobotS3Uploader(bucketName, region);
|
1301 |
+
|
1302 |
+
// Convert blobs to File objects for S3 uploader
|
1303 |
+
const files = blobArray.map(item => {
|
1304 |
+
return {
|
1305 |
+
path: item.path,
|
1306 |
+
content: item.content
|
1307 |
+
};
|
1308 |
+
});
|
1309 |
+
|
1310 |
+
// Generate a unique reference ID for tracking the upload
|
1311 |
+
const referenceId = `lerobot-s3-upload-${Date.now()}`;
|
1312 |
+
|
1313 |
+
try {
|
1314 |
+
// Start the upload process
|
1315 |
+
uploader.checkBucketAndUploadFiles(files, accessKeyId, secretAccessKey, prefix, referenceId);
|
1316 |
+
console.log(`Successfully uploaded dataset to S3 bucket: ${bucketName}`);
|
1317 |
+
return uploader;
|
1318 |
+
} catch (error) {
|
1319 |
+
console.error("Error uploading to S3:", error);
|
1320 |
+
throw error;
|
1321 |
+
}
|
1322 |
+
}
|
1323 |
+
|
1324 |
+
/**
|
1325 |
+
* Exports the LeRobot dataset in various formats
|
1326 |
+
*
|
1327 |
+
* @param format The export format - 'blobs', 'zip', 'zip-download', 'huggingface', or 's3'
|
1328 |
+
* @param options Additional options for specific formats
|
1329 |
+
* @param options.username Hugging Face username (if not provided for "huggingface" format, it will use the default username)
|
1330 |
+
* @param options.repoName Hugging Face repository name (required for 'huggingface' format)
|
1331 |
+
* @param options.accessToken Hugging Face access token (required for 'huggingface' format)
|
1332 |
+
* @param options.bucketName S3 bucket name (required for 's3' format)
|
1333 |
+
* @param options.accessKeyId AWS access key ID (required for 's3' format)
|
1334 |
+
* @param options.secretAccessKey AWS secret access key (required for 's3' format)
|
1335 |
+
* @param options.region AWS region (optional for 's3' format, default: us-east-1)
|
1336 |
+
* @param options.prefix S3 prefix/folder (optional for 's3' format)
|
1337 |
+
* @returns The exported data in the requested format or the uploader instance for 'huggingface'/'s3' formats
|
1338 |
+
*/
|
1339 |
+
async exportForLeRobot(format: 'blobs' | 'zip' | 'zip-download' | 'huggingface' | 's3' = 'zip-download', options?: {
|
1340 |
+
username?: string;
|
1341 |
+
repoName?: string;
|
1342 |
+
accessToken?: string;
|
1343 |
+
bucketName?: string;
|
1344 |
+
accessKeyId?: string;
|
1345 |
+
secretAccessKey?: string;
|
1346 |
+
region?: string;
|
1347 |
+
prefix?: string;
|
1348 |
+
}) {
|
1349 |
+
switch (format) {
|
1350 |
+
case 'blobs':
|
1351 |
+
return this._exportForLeRobotBlobs();
|
1352 |
+
|
1353 |
+
case 'zip':
|
1354 |
+
return this._exportForLeRobotZip();
|
1355 |
+
|
1356 |
+
case 'huggingface':
|
1357 |
+
// Validate required options for Hugging Face upload
|
1358 |
+
if (!options || !options.repoName || !options.accessToken) {
|
1359 |
+
throw new Error('Hugging Face upload requires repoName, and accessToken options');
|
1360 |
+
}
|
1361 |
+
|
1362 |
+
if (!options.username) {
|
1363 |
+
const hub = await import("@huggingface/hub");
|
1364 |
+
const {name: username} = await hub.whoAmI({accessToken: options.accessToken});
|
1365 |
+
options.username = username;
|
1366 |
+
}
|
1367 |
+
|
1368 |
+
return this._exportForLeRobotHuggingface(
|
1369 |
+
options.username,
|
1370 |
+
options.repoName,
|
1371 |
+
options.accessToken
|
1372 |
+
);
|
1373 |
+
|
1374 |
+
case 's3':
|
1375 |
+
// Validate required options for S3 upload
|
1376 |
+
if (!options || !options.bucketName || !options.accessKeyId || !options.secretAccessKey) {
|
1377 |
+
throw new Error('S3 upload requires bucketName, accessKeyId, and secretAccessKey options');
|
1378 |
+
}
|
1379 |
+
|
1380 |
+
return this._exportForLeRobotS3(
|
1381 |
+
options.bucketName,
|
1382 |
+
options.accessKeyId,
|
1383 |
+
options.secretAccessKey,
|
1384 |
+
options.region,
|
1385 |
+
options.prefix
|
1386 |
+
);
|
1387 |
+
|
1388 |
+
case 'zip-download':
|
1389 |
+
default:
|
1390 |
+
// Get the zip blob
|
1391 |
+
const zipContent = await this._exportForLeRobotZip();
|
1392 |
+
|
1393 |
+
// Create a URL for the zip file
|
1394 |
+
const url = URL.createObjectURL(zipContent);
|
1395 |
+
|
1396 |
+
// Create a download link and trigger the download
|
1397 |
+
const link = document.createElement('a');
|
1398 |
+
link.href = url;
|
1399 |
+
link.download = `lerobot_dataset_${new Date().toISOString().replace(/[:.]/g, '-')}.zip`;
|
1400 |
+
document.body.appendChild(link);
|
1401 |
+
link.click();
|
1402 |
+
|
1403 |
+
// Clean up
|
1404 |
+
setTimeout(() => {
|
1405 |
+
document.body.removeChild(link);
|
1406 |
+
URL.revokeObjectURL(url);
|
1407 |
+
}, 100);
|
1408 |
+
|
1409 |
+
return zipContent;
|
1410 |
+
}
|
1411 |
+
}
|
1412 |
+
}
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
/**
|
2 |
+
* Class to train a model using runpod
|
3 |
+
*/
|
4 |
+
class LeRobotRunpodTrainer extends EventTarget {
|
5 |
+
api_token: string;
|
6 |
+
constructor(api_token: string) {
|
7 |
+
super();
|
8 |
+
this.api_token = api_token;
|
9 |
+
}
|
10 |
+
|
11 |
+
async _startPod(api_token: string) {
|
12 |
+
const podOptions = {
|
13 |
+
allowedCudaVersions: ["12.8"],
|
14 |
+
cloudType: "SECURE",
|
15 |
+
computeType: "GPU",
|
16 |
+
containerDiskInGb: 50,
|
17 |
+
countryCodes: ["<string>"],
|
18 |
+
cpuFlavorIds: ["cpu3c"],
|
19 |
+
cpuFlavorPriority: "availability",
|
20 |
+
dataCenterIds: ["EU-RO-1","CA-MTL-1"],
|
21 |
+
dataCenterPriority: "availability",
|
22 |
+
dockerEntrypoint: [],
|
23 |
+
dockerStartCmd: [],
|
24 |
+
env: {"ENV_VAR":"value"},
|
25 |
+
globalNetworking: true,
|
26 |
+
gpuCount: 1,
|
27 |
+
gpuTypeIds: ["NVIDIA GeForce RTX 4090"],
|
28 |
+
gpuTypePriority: "availability",
|
29 |
+
imageName: "runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04",
|
30 |
+
interruptible: false,
|
31 |
+
locked: false,
|
32 |
+
minDiskBandwidthMBps: 123,
|
33 |
+
minDownloadMbps: 123,
|
34 |
+
minRAMPerGPU: 8,
|
35 |
+
minUploadMbps: 123,
|
36 |
+
minVCPUPerGPU: 2,
|
37 |
+
name: "my pod",
|
38 |
+
ports: ["8888/http","22/tcp"],
|
39 |
+
supportPublicIp: true,
|
40 |
+
templateId: null,
|
41 |
+
vcpuCount: 2,
|
42 |
+
volumeInGb: 20,
|
43 |
+
volumeMountPath: "/workspace"
|
44 |
+
}
|
45 |
+
const options = {
|
46 |
+
method: 'POST',
|
47 |
+
headers: {Authorization: 'Bearer ' + api_token, 'Content-Type': 'application/json'},
|
48 |
+
body: JSON.stringify(podOptions)
|
49 |
+
};
|
50 |
+
|
51 |
+
const response = await fetch('https://rest.runpod.io/v1/pods', options);
|
52 |
+
const data = await response.json();
|
53 |
+
return data;
|
54 |
+
}
|
55 |
+
|
56 |
+
async start() {
|
57 |
+
await this._startPod(this.api_token);
|
58 |
+
this.dispatchEvent(new CustomEvent("deployed_training_pod"));
|
59 |
+
}
|
60 |
+
}
|
@@ -0,0 +1,172 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
// Note: To use this module, you'll need to install the AWS SDK:
|
2 |
+
// npm install @aws-sdk/client-s3 @aws-sdk/lib-storage
|
3 |
+
|
4 |
+
import { S3Client, HeadBucketCommand } from "@aws-sdk/client-s3";
|
5 |
+
import { Upload } from "@aws-sdk/lib-storage";
|
6 |
+
|
7 |
+
// Define ContentSource type locally to avoid HuggingFace dependency
|
8 |
+
type ContentSource = Blob | ArrayBuffer | Uint8Array | string;
|
9 |
+
type FileArray = Array<URL | File | { path: string; content: ContentSource }>;
|
10 |
+
|
11 |
+
/**
|
12 |
+
* Uploads a leRobot dataset to Amazon S3
|
13 |
+
*/
|
14 |
+
export class LeRobotS3Uploader extends EventTarget {
|
15 |
+
private _bucketName: string;
|
16 |
+
private _region: string;
|
17 |
+
private _uploaded: boolean;
|
18 |
+
private _bucket_exists: boolean;
|
19 |
+
private _s3Client: S3Client | null;
|
20 |
+
|
21 |
+
constructor(bucketName: string, region: string = "us-east-1") {
|
22 |
+
super();
|
23 |
+
this._bucketName = bucketName;
|
24 |
+
this._region = region;
|
25 |
+
this._uploaded = false;
|
26 |
+
this._bucket_exists = false;
|
27 |
+
this._s3Client = null;
|
28 |
+
}
|
29 |
+
|
30 |
+
/**
|
31 |
+
* Returns whether the bucket has been successfully checked/created
|
32 |
+
*/
|
33 |
+
get bucketExists(): boolean {
|
34 |
+
return this._bucket_exists;
|
35 |
+
}
|
36 |
+
|
37 |
+
get uploaded(): boolean {
|
38 |
+
return this._uploaded;
|
39 |
+
}
|
40 |
+
|
41 |
+
/**
|
42 |
+
* Initialize the S3 client with credentials
|
43 |
+
*
|
44 |
+
* @param accessKeyId AWS access key ID
|
45 |
+
* @param secretAccessKey AWS secret access key
|
46 |
+
*/
|
47 |
+
initializeClient(accessKeyId: string, secretAccessKey: string): void {
|
48 |
+
this._s3Client = new S3Client({
|
49 |
+
region: this._region,
|
50 |
+
credentials: {
|
51 |
+
accessKeyId,
|
52 |
+
secretAccessKey
|
53 |
+
}
|
54 |
+
});
|
55 |
+
}
|
56 |
+
|
57 |
+
/**
|
58 |
+
* Checks if the bucket exists and uploads files to it
|
59 |
+
*
|
60 |
+
* @param files The files to upload
|
61 |
+
* @param accessKeyId AWS access key ID
|
62 |
+
* @param secretAccessKey AWS secret access key
|
63 |
+
* @param prefix Optional prefix (folder) to upload files to within the bucket
|
64 |
+
* @param referenceId The reference id for the upload, to track it (optional)
|
65 |
+
*/
|
66 |
+
async checkBucketAndUploadFiles(
|
67 |
+
files: FileArray,
|
68 |
+
accessKeyId: string,
|
69 |
+
secretAccessKey: string,
|
70 |
+
prefix: string = "",
|
71 |
+
referenceId: string = ""
|
72 |
+
): Promise<void> {
|
73 |
+
// Initialize the client if not already done
|
74 |
+
if (!this._s3Client) {
|
75 |
+
this.initializeClient(accessKeyId, secretAccessKey);
|
76 |
+
}
|
77 |
+
|
78 |
+
// Check if bucket exists
|
79 |
+
try {
|
80 |
+
await this._s3Client!.send(new HeadBucketCommand({ Bucket: this._bucketName }));
|
81 |
+
this._bucket_exists = true;
|
82 |
+
this.dispatchEvent(new CustomEvent("bucketExists", {
|
83 |
+
detail: { bucketName: this._bucketName }
|
84 |
+
}));
|
85 |
+
} catch (error) {
|
86 |
+
throw new Error(`Bucket ${this._bucketName} does not exist or you don't have permission to access it`);
|
87 |
+
}
|
88 |
+
|
89 |
+
// Upload files
|
90 |
+
const uploadPromises: Promise<void>[] = [];
|
91 |
+
for (const file of files) {
|
92 |
+
uploadPromises.push(this.uploadFileWithProgress([file], prefix, referenceId));
|
93 |
+
}
|
94 |
+
|
95 |
+
await Promise.all(uploadPromises);
|
96 |
+
this._uploaded = true;
|
97 |
+
}
|
98 |
+
|
99 |
+
/**
|
100 |
+
* Uploads files to S3 with progress events
|
101 |
+
*
|
102 |
+
* @param files The files to upload
|
103 |
+
* @param prefix Optional prefix (folder) to upload files to within the bucket
|
104 |
+
* @param referenceId The reference id for the upload, to track it (optional)
|
105 |
+
*/
|
106 |
+
async uploadFileWithProgress(
|
107 |
+
files: FileArray,
|
108 |
+
prefix: string = "",
|
109 |
+
referenceId: string = ""
|
110 |
+
): Promise<void> {
|
111 |
+
if (!this._s3Client) {
|
112 |
+
throw new Error("S3 client not initialized. Call initializeClient first.");
|
113 |
+
}
|
114 |
+
|
115 |
+
for (const file of files) {
|
116 |
+
let key: string;
|
117 |
+
let body: any;
|
118 |
+
|
119 |
+
if (file instanceof URL) {
|
120 |
+
const response = await fetch(file);
|
121 |
+
body = await response.blob();
|
122 |
+
key = `${prefix}${prefix ? '/' : ''}${file.pathname.split('/').pop()}`;
|
123 |
+
} else if (file instanceof File) {
|
124 |
+
body = file;
|
125 |
+
key = `${prefix}${prefix ? '/' : ''}${file.name}`;
|
126 |
+
} else {
|
127 |
+
body = file.content;
|
128 |
+
key = `${prefix}${prefix ? '/' : ''}${file.path}`;
|
129 |
+
}
|
130 |
+
|
131 |
+
const upload = new Upload({
|
132 |
+
client: this._s3Client,
|
133 |
+
params: {
|
134 |
+
Bucket: this._bucketName,
|
135 |
+
Key: key,
|
136 |
+
Body: body,
|
137 |
+
},
|
138 |
+
});
|
139 |
+
|
140 |
+
// Set up progress tracking
|
141 |
+
upload.on('httpUploadProgress', (progress) => {
|
142 |
+
this.dispatchEvent(new CustomEvent("progress", {
|
143 |
+
detail: {
|
144 |
+
progressEvent: progress,
|
145 |
+
bucketName: this._bucketName,
|
146 |
+
key,
|
147 |
+
referenceId
|
148 |
+
}
|
149 |
+
}));
|
150 |
+
});
|
151 |
+
|
152 |
+
await upload.done();
|
153 |
+
}
|
154 |
+
}
|
155 |
+
|
156 |
+
/**
|
157 |
+
* Generates a pre-signed URL for downloading a file from S3
|
158 |
+
*
|
159 |
+
* @param _key The key (path) of the file in the S3 bucket
|
160 |
+
* @param _expiresIn The number of seconds until the URL expires (default: 3600 = 1 hour)
|
161 |
+
* @returns A pre-signed URL for downloading the file
|
162 |
+
*/
|
163 |
+
async generatePresignedUrl(_key: string, _expiresIn: number = 3600): Promise<string> {
|
164 |
+
if (!this._s3Client) {
|
165 |
+
throw new Error("S3 client not initialized. Call initializeClient first.");
|
166 |
+
}
|
167 |
+
|
168 |
+
// Note: This requires the @aws-sdk/s3-request-presigner package
|
169 |
+
// Implementation would go here
|
170 |
+
throw new Error("generatePresignedUrl not implemented. Requires @aws-sdk/s3-request-presigner package.");
|
171 |
+
}
|
172 |
+
}
|
@@ -146,7 +146,6 @@ export async function teleoperate(
|
|
146 |
config: TeleoperateConfig
|
147 |
): Promise<TeleoperationProcess> {
|
148 |
const teleoperator = await createTeleoperatorProcess(config);
|
149 |
-
const motorConfigs = teleoperator.motorConfigs;
|
150 |
|
151 |
return {
|
152 |
start: () => {
|
@@ -173,7 +172,7 @@ export async function teleoperate(
|
|
173 |
},
|
174 |
getState: () => buildTeleoperationStateFromTeleoperator(teleoperator),
|
175 |
teleoperator,
|
176 |
-
disconnect: () => teleoperator.disconnect()
|
177 |
};
|
178 |
}
|
179 |
|
|
|
146 |
config: TeleoperateConfig
|
147 |
): Promise<TeleoperationProcess> {
|
148 |
const teleoperator = await createTeleoperatorProcess(config);
|
|
|
149 |
|
150 |
return {
|
151 |
start: () => {
|
|
|
172 |
},
|
173 |
getState: () => buildTeleoperationStateFromTeleoperator(teleoperator),
|
174 |
teleoperator,
|
175 |
+
disconnect: () => teleoperator.disconnect()
|
176 |
};
|
177 |
}
|
178 |
|
@@ -6,17 +6,63 @@
|
|
6 |
import type { MotorConfig } from "../types/teleoperation.js";
|
7 |
import type { MotorCommunicationPort } from "../utils/motor-communication.js";
|
8 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
9 |
/**
|
10 |
* Base interface that all Web teleoperators must implement
|
11 |
*/
|
12 |
-
export
|
13 |
-
|
14 |
-
|
15 |
-
|
16 |
-
|
17 |
-
|
18 |
-
|
19 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
20 |
}
|
21 |
|
22 |
/**
|
@@ -31,14 +77,26 @@ export type TeleoperatorSpecificState = {
|
|
31 |
/**
|
32 |
* Base abstract class with common functionality for all teleoperators
|
33 |
*/
|
34 |
-
export abstract class BaseWebTeleoperator
|
35 |
protected port: MotorCommunicationPort;
|
36 |
public motorConfigs: MotorConfig[] = [];
|
37 |
protected isActive: boolean = false;
|
|
|
|
|
|
|
|
|
38 |
|
39 |
constructor(port: MotorCommunicationPort, motorConfigs: MotorConfig[]) {
|
|
|
40 |
this.port = port;
|
41 |
this.motorConfigs = motorConfigs;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
42 |
}
|
43 |
|
44 |
abstract initialize(): Promise<void>;
|
@@ -53,10 +111,130 @@ export abstract class BaseWebTeleoperator implements WebTeleoperator {
|
|
53 |
}
|
54 |
}
|
55 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
56 |
onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void {
|
57 |
this.motorConfigs = motorConfigs;
|
58 |
}
|
59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
60 |
get isActiveTeleoperator(): boolean {
|
61 |
return this.isActive;
|
62 |
}
|
|
|
6 |
import type { MotorConfig } from "../types/teleoperation.js";
|
7 |
import type { MotorCommunicationPort } from "../utils/motor-communication.js";
|
8 |
|
9 |
+
/**
|
10 |
+
* Normalizes a value from one range to another
|
11 |
+
* @param value The value to normalize
|
12 |
+
* @param minVal The minimum value of the original range
|
13 |
+
* @param maxVal The maximum value of the original range
|
14 |
+
* @param minNorm The minimum value of the normalized range
|
15 |
+
* @param maxNorm The maximum value of the normalized range
|
16 |
+
* @returns The normalized value
|
17 |
+
*/
|
18 |
+
function normalizeValue(value: number, minVal: number, maxVal: number, minNorm: number, maxNorm: number): number {
|
19 |
+
const range = maxVal - minVal;
|
20 |
+
const normRange = maxNorm - minNorm;
|
21 |
+
const normalized = (value - minVal) / range;
|
22 |
+
return normalized * normRange + minNorm;
|
23 |
+
}
|
24 |
+
|
25 |
+
/**
|
26 |
+
* Type definition for state update callback parameters
|
27 |
+
*/
|
28 |
+
interface StateUpdateCallbackParams {
|
29 |
+
previousMotorConfigs: MotorConfig[];
|
30 |
+
newMotorConfigs: MotorConfig[];
|
31 |
+
previousMotorConfigsNormalized: MotorConfig[];
|
32 |
+
newMotorConfigsNormalized: MotorConfig[];
|
33 |
+
commandSentTimestamp: number;
|
34 |
+
positionChangedTimestamp: number;
|
35 |
+
}
|
36 |
+
|
37 |
+
/**
|
38 |
+
* Type definition for state update callback function
|
39 |
+
*/
|
40 |
+
type StateUpdateCallback = (params: StateUpdateCallbackParams) => void;
|
41 |
+
|
42 |
+
/**
|
43 |
+
* Type definition for array of state update callbacks
|
44 |
+
*/
|
45 |
+
type StateUpdateCallbackArray = Array<StateUpdateCallback>;
|
46 |
+
|
47 |
/**
|
48 |
* Base interface that all Web teleoperators must implement
|
49 |
*/
|
50 |
+
export abstract class WebTeleoperator {
|
51 |
+
protected onStateUpdateCallbacks: StateUpdateCallbackArray = [];
|
52 |
+
public motorConfigs: MotorConfig[] = [];
|
53 |
+
abstract initialize(): Promise<void>;
|
54 |
+
abstract start(): void;
|
55 |
+
abstract stop(): void;
|
56 |
+
|
57 |
+
abstract recordingTaskIndex : number;
|
58 |
+
abstract recordingEpisodeIndex : number;
|
59 |
+
abstract setEpisodeIndex(index: number): void;
|
60 |
+
abstract setTaskIndex(index: number): void;
|
61 |
+
|
62 |
+
abstract disconnect(): Promise<void>;
|
63 |
+
abstract getState(): TeleoperatorSpecificState;
|
64 |
+
abstract onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void;
|
65 |
+
abstract addOnStateUpdateCallback(fn : StateUpdateCallback): void;
|
66 |
}
|
67 |
|
68 |
/**
|
|
|
77 |
/**
|
78 |
* Base abstract class with common functionality for all teleoperators
|
79 |
*/
|
80 |
+
export abstract class BaseWebTeleoperator extends WebTeleoperator {
|
81 |
protected port: MotorCommunicationPort;
|
82 |
public motorConfigs: MotorConfig[] = [];
|
83 |
protected isActive: boolean = false;
|
84 |
+
public isRecording: boolean = false;
|
85 |
+
public recordingTaskIndex : number;
|
86 |
+
public recordingEpisodeIndex : number;
|
87 |
+
public recordedMotorPositionEpisodes : any[];
|
88 |
|
89 |
constructor(port: MotorCommunicationPort, motorConfigs: MotorConfig[]) {
|
90 |
+
super();
|
91 |
this.port = port;
|
92 |
this.motorConfigs = motorConfigs;
|
93 |
+
|
94 |
+
// store episode positions
|
95 |
+
this.recordedMotorPositionEpisodes = []
|
96 |
+
|
97 |
+
this.isRecording = false;
|
98 |
+
this.recordingTaskIndex = 0;
|
99 |
+
this.recordingEpisodeIndex = 0;
|
100 |
}
|
101 |
|
102 |
abstract initialize(): Promise<void>;
|
|
|
111 |
}
|
112 |
}
|
113 |
|
114 |
+
setEpisodeIndex(index: number): void {
|
115 |
+
this.recordingEpisodeIndex = index;
|
116 |
+
|
117 |
+
// create a new empty array at that position on the array
|
118 |
+
this.recordedMotorPositionEpisodes[this.recordingEpisodeIndex] = []
|
119 |
+
}
|
120 |
+
|
121 |
+
setTaskIndex(index: number): void {
|
122 |
+
this.recordingTaskIndex = index;
|
123 |
+
}
|
124 |
+
|
125 |
onMotorConfigsUpdate(motorConfigs: MotorConfig[]): void {
|
126 |
this.motorConfigs = motorConfigs;
|
127 |
}
|
128 |
|
129 |
+
normalizeMotorConfigPosition(motorConfig: MotorConfig){
|
130 |
+
let minNormPosition;
|
131 |
+
let maxNormPosition;
|
132 |
+
|
133 |
+
/**
|
134 |
+
* This follows the guide at https://github.com/huggingface/lerobot/blob/cf86b9300dc83fdad408cfe4787b7b09b55f12cf/src/lerobot/robots/so100_follower/so100_follower.py#L49
|
135 |
+
* Meaning, for everything except gripper, it normalizes the positions to between -100 and 100
|
136 |
+
* and for gripper it normalizes between 0 - 100
|
137 |
+
*/
|
138 |
+
if(["shoulder_pan", "shoulder_lift", "elbow_flex", "wrist_flex", "wrist_roll"].includes(motorConfig.name)) {
|
139 |
+
minNormPosition = -100;
|
140 |
+
maxNormPosition = 100;
|
141 |
+
} else {
|
142 |
+
minNormPosition = 0;
|
143 |
+
maxNormPosition = 100;
|
144 |
+
}
|
145 |
+
|
146 |
+
return normalizeValue(motorConfig.currentPosition, motorConfig.minPosition, motorConfig.maxPosition, minNormPosition, maxNormPosition)
|
147 |
+
}
|
148 |
+
|
149 |
+
/**
|
150 |
+
* Normalize an entire list of motor configurations
|
151 |
+
*
|
152 |
+
* This follows the guide at https://github.com/huggingface/lerobot/blob/cf86b9300dc83fdad408cfe4787b7b09b55f12cf/src/lerobot/robots/so100_follower/so100_follower.py#L49
|
153 |
+
* Meaning, for everything except gripper, it normalizes the positions to between -100 and 100
|
154 |
+
* and for gripper it normalizes between 0 - 100
|
155 |
+
*/
|
156 |
+
normalizeMotorConfigs(motorConfigs : MotorConfig[]) : MotorConfig[] {
|
157 |
+
// Create a deep copy of the motor configs
|
158 |
+
const normalizedConfigs = JSON.parse(JSON.stringify(motorConfigs)) as MotorConfig[];
|
159 |
+
|
160 |
+
// Normalize the current position values
|
161 |
+
for(let i = 0; i < normalizedConfigs.length; i++) {
|
162 |
+
const config = normalizedConfigs[i];
|
163 |
+
|
164 |
+
if(config.name === "gripper") {
|
165 |
+
config.currentPosition = normalizeValue(motorConfigs[i].currentPosition, motorConfigs[i].minPosition, motorConfigs[i].maxPosition, 0, 100);
|
166 |
+
} else {
|
167 |
+
config.currentPosition = normalizeValue(motorConfigs[i].currentPosition, motorConfigs[i].minPosition, motorConfigs[i].maxPosition, -100, 100);
|
168 |
+
}
|
169 |
+
|
170 |
+
// Also normalize min/max positions for consistency
|
171 |
+
if(config.name === "gripper") {
|
172 |
+
config.minPosition = 0;
|
173 |
+
config.maxPosition = 100;
|
174 |
+
} else {
|
175 |
+
config.minPosition = -100;
|
176 |
+
config.maxPosition = 100;
|
177 |
+
}
|
178 |
+
}
|
179 |
+
|
180 |
+
return normalizedConfigs;
|
181 |
+
}
|
182 |
+
|
183 |
+
/**
|
184 |
+
* Dispatches a motor position changed event
|
185 |
+
* Gets the motor positions, normalized
|
186 |
+
*
|
187 |
+
* This follows the guide at https://github.com/huggingface/lerobot/blob/cf86b9300dc83fdad408cfe4787b7b09b55f12cf/src/lerobot/robots/so100_follower/so100_follower.py#L49
|
188 |
+
* Meaning, for everything except gripper, it normalizes the positions to between -100 and 100
|
189 |
+
* and for gripper it normalizes between 0 - 100
|
190 |
+
*
|
191 |
+
* @param motorName The name of the motor that changed
|
192 |
+
* @param motorConfig The motor configuration
|
193 |
+
* @param previousPosition The previous position of the motor
|
194 |
+
* @param currentPosition The current position of the motor
|
195 |
+
* @param timestamp The timestamp of the event
|
196 |
+
*/
|
197 |
+
dispatchMotorPositionChanged(prevMotorConfigs: MotorConfig[], newMotorConfigs: MotorConfig[], commandSentTimestamp: number, positionChangedTimestamp: number): void {
|
198 |
+
// Call all registered state update callbacks
|
199 |
+
const callbackParams: StateUpdateCallbackParams = {
|
200 |
+
previousMotorConfigs: prevMotorConfigs,
|
201 |
+
newMotorConfigs: newMotorConfigs,
|
202 |
+
previousMotorConfigsNormalized: this.normalizeMotorConfigs(prevMotorConfigs),
|
203 |
+
newMotorConfigsNormalized: this.normalizeMotorConfigs(newMotorConfigs),
|
204 |
+
commandSentTimestamp: commandSentTimestamp,
|
205 |
+
positionChangedTimestamp: positionChangedTimestamp
|
206 |
+
};
|
207 |
+
|
208 |
+
// call all the onStateUpdateCallbacks
|
209 |
+
this.onStateUpdateCallbacks.forEach(callback => callback(callbackParams));
|
210 |
+
|
211 |
+
// if recording, store the changes
|
212 |
+
if(this.isRecording) {
|
213 |
+
const data = {
|
214 |
+
previousMotorConfigs : prevMotorConfigs,
|
215 |
+
newMotorConfigs,
|
216 |
+
previousMotorConfigsNormalized : this.normalizeMotorConfigs(prevMotorConfigs),
|
217 |
+
newMotorConfigsNormalized : this.normalizeMotorConfigs(newMotorConfigs),
|
218 |
+
commandSentTimestamp,
|
219 |
+
positionChangedTimestamp,
|
220 |
+
episodeIndex: this.recordingEpisodeIndex,
|
221 |
+
taskIndex: this.recordingTaskIndex,
|
222 |
+
}
|
223 |
+
|
224 |
+
const episodes = this.recordedMotorPositionEpisodes[this.recordingEpisodeIndex]
|
225 |
+
episodes.push(data)
|
226 |
+
}
|
227 |
+
}
|
228 |
+
|
229 |
+
/**
|
230 |
+
* Adds an onstateupdate callback
|
231 |
+
* to return a response for the user with motor config states and timestamps
|
232 |
+
* @param fn Callback function that receives state update parameters
|
233 |
+
*/
|
234 |
+
addOnStateUpdateCallback(fn: StateUpdateCallback): void {
|
235 |
+
this.onStateUpdateCallbacks.push(fn);
|
236 |
+
}
|
237 |
+
|
238 |
get isActiveTeleoperator(): boolean {
|
239 |
return this.isActive;
|
240 |
}
|
@@ -62,6 +62,7 @@ export class DirectTeleoperator extends BaseWebTeleoperator {
|
|
62 |
* Move motor to exact position
|
63 |
*/
|
64 |
async moveMotor(motorName: string, targetPosition: number): Promise<boolean> {
|
|
|
65 |
const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
|
66 |
if (!motorConfig) return false;
|
67 |
|
@@ -69,6 +70,7 @@ export class DirectTeleoperator extends BaseWebTeleoperator {
|
|
69 |
motorConfig.minPosition,
|
70 |
Math.min(motorConfig.maxPosition, targetPosition)
|
71 |
);
|
|
|
72 |
|
73 |
try {
|
74 |
await writeMotorPosition(
|
@@ -77,12 +79,14 @@ export class DirectTeleoperator extends BaseWebTeleoperator {
|
|
77 |
Math.round(clampedPosition)
|
78 |
);
|
79 |
motorConfig.currentPosition = clampedPosition;
|
|
|
80 |
|
81 |
// Notify UI of position change
|
82 |
if (this.onStateUpdate) {
|
83 |
this.onStateUpdate(this.buildTeleoperationState());
|
84 |
}
|
85 |
|
|
|
86 |
return true;
|
87 |
} catch (error) {
|
88 |
console.warn(`Failed to move motor ${motorName}:`, error);
|
@@ -96,11 +100,16 @@ export class DirectTeleoperator extends BaseWebTeleoperator {
|
|
96 |
async setMotorPositions(positions: {
|
97 |
[motorName: string]: number;
|
98 |
}): Promise<boolean> {
|
|
|
|
|
99 |
const results = await Promise.all(
|
100 |
Object.entries(positions).map(([motorName, position]) =>
|
101 |
this.moveMotor(motorName, position)
|
102 |
)
|
103 |
);
|
|
|
|
|
|
|
104 |
|
105 |
return results.every((result) => result);
|
106 |
}
|
|
|
62 |
* Move motor to exact position
|
63 |
*/
|
64 |
async moveMotor(motorName: string, targetPosition: number): Promise<boolean> {
|
65 |
+
const commandSentTimestamp = performance.now()/1000;
|
66 |
const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
|
67 |
if (!motorConfig) return false;
|
68 |
|
|
|
70 |
motorConfig.minPosition,
|
71 |
Math.min(motorConfig.maxPosition, targetPosition)
|
72 |
);
|
73 |
+
const prevMotorConfigs = structuredClone(this.motorConfigs)
|
74 |
|
75 |
try {
|
76 |
await writeMotorPosition(
|
|
|
79 |
Math.round(clampedPosition)
|
80 |
);
|
81 |
motorConfig.currentPosition = clampedPosition;
|
82 |
+
const positionChangedTimestamp = performance.now()/1000;
|
83 |
|
84 |
// Notify UI of position change
|
85 |
if (this.onStateUpdate) {
|
86 |
this.onStateUpdate(this.buildTeleoperationState());
|
87 |
}
|
88 |
|
89 |
+
this.dispatchMotorPositionChanged(prevMotorConfigs,this.motorConfigs, commandSentTimestamp, positionChangedTimestamp);
|
90 |
return true;
|
91 |
} catch (error) {
|
92 |
console.warn(`Failed to move motor ${motorName}:`, error);
|
|
|
100 |
async setMotorPositions(positions: {
|
101 |
[motorName: string]: number;
|
102 |
}): Promise<boolean> {
|
103 |
+
const commandSentTimestamp = performance.now()/1000;
|
104 |
+
const prevMotorConfigs = structuredClone(this.motorConfigs)
|
105 |
const results = await Promise.all(
|
106 |
Object.entries(positions).map(([motorName, position]) =>
|
107 |
this.moveMotor(motorName, position)
|
108 |
)
|
109 |
);
|
110 |
+
const positionChangedTimestamp = performance.now()/1000;
|
111 |
+
|
112 |
+
this.dispatchMotorPositionChanged(prevMotorConfigs,this.motorConfigs, commandSentTimestamp, positionChangedTimestamp);
|
113 |
|
114 |
return results.every((result) => result);
|
115 |
}
|
@@ -171,7 +171,7 @@ export class KeyboardTeleoperator extends BaseWebTeleoperator {
|
|
171 |
*
|
172 |
* Keep it simple - this works!
|
173 |
*/
|
174 |
-
private updateMotorPositions(): void {
|
175 |
const now = Date.now();
|
176 |
|
177 |
// Clear timed-out keys
|
@@ -219,25 +219,44 @@ export class KeyboardTeleoperator extends BaseWebTeleoperator {
|
|
219 |
);
|
220 |
}
|
221 |
|
|
|
|
|
|
|
|
|
222 |
// Send motor commands and update positions
|
223 |
-
|
224 |
const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
|
225 |
-
if (motorConfig && targetPosition !== motorConfig.currentPosition) {
|
226 |
-
|
227 |
-
|
228 |
-
|
229 |
-
|
230 |
-
|
231 |
-
|
232 |
-
|
233 |
-
|
234 |
-
|
235 |
-
|
236 |
-
|
237 |
-
|
238 |
-
|
239 |
-
|
240 |
}
|
241 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
242 |
}
|
243 |
}
|
|
|
171 |
*
|
172 |
* Keep it simple - this works!
|
173 |
*/
|
174 |
+
private async updateMotorPositions(): Promise<void> {
|
175 |
const now = Date.now();
|
176 |
|
177 |
// Clear timed-out keys
|
|
|
219 |
);
|
220 |
}
|
221 |
|
222 |
+
const prevMotorConfigs = structuredClone(this.motorConfigs);
|
223 |
+
const commandSentTimestamp = performance.now()/1000;
|
224 |
+
|
225 |
+
|
226 |
// Send motor commands and update positions
|
227 |
+
for (const [motorName, targetPosition] of Object.entries(targetPositions)) {
|
228 |
const motorConfig = this.motorConfigs.find((m) => m.name === motorName);
|
229 |
+
if (motorConfig && targetPosition !== motorConfig.currentPosition) {
|
230 |
+
try {
|
231 |
+
await writeMotorPosition(
|
232 |
+
this.port,
|
233 |
+
motorConfig.id,
|
234 |
+
Math.round(targetPosition)
|
235 |
+
);
|
236 |
+
|
237 |
+
motorConfig.currentPosition = targetPosition;
|
238 |
+
} catch (error) {
|
239 |
+
console.warn(
|
240 |
+
`Failed to write motor ${motorConfig.id} position:`,
|
241 |
+
error
|
242 |
+
);
|
243 |
+
}
|
244 |
}
|
245 |
+
}
|
246 |
+
|
247 |
+
|
248 |
+
const positionChangedTimestamp = performance.now()/1000;
|
249 |
+
|
250 |
+
// Dispatch event for motor position change if something has changed
|
251 |
+
const somethingChanged = Object.entries(targetPositions).length > 0;
|
252 |
+
|
253 |
+
if(somethingChanged){
|
254 |
+
this.dispatchMotorPositionChanged(
|
255 |
+
prevMotorConfigs,
|
256 |
+
this.motorConfigs,
|
257 |
+
commandSentTimestamp,
|
258 |
+
positionChangedTimestamp
|
259 |
+
);
|
260 |
+
}
|
261 |
}
|
262 |
}
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
//import * as THREE from 'three';
|
2 |
+
|
3 |
+
|
4 |
+
/**
|
5 |
+
* Module to display a 3D model of a robot, easily via threejs
|
6 |
+
*/
|
7 |
+
export class LeRobotThreejsDisplay {
|
8 |
+
//scene: THREE.Scene;
|
9 |
+
constructor() {
|
10 |
+
//this.scene = new THREE.Scene();
|
11 |
+
}
|
12 |
+
}
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
function generateREADME(metaInfo : string) {
|
2 |
+
return `\
|
3 |
+
---
|
4 |
+
task_categories:
|
5 |
+
- robotics
|
6 |
+
tags:
|
7 |
+
- LeRobot
|
8 |
+
- tutorial
|
9 |
+
configs:
|
10 |
+
- config_name: default
|
11 |
+
data_files: data/*/*.parquet
|
12 |
+
---
|
13 |
+
|
14 |
+
This dataset was created using [LeRobot.js](https://github.com/timpietrusky/lerobot.js) which is based on the [LeRobot](https://github.com/huggingface/lerobot) project
|
15 |
+
|
16 |
+
## Dataset Description
|
17 |
+
|
18 |
+
|
19 |
+
|
20 |
+
- **Homepage:** [More Information Needed]
|
21 |
+
- **Paper:** [More Information Needed]
|
22 |
+
- **License:** apache-2.0
|
23 |
+
|
24 |
+
## Dataset Structure
|
25 |
+
|
26 |
+
[meta/info.json](meta/info.json):
|
27 |
+
${metaInfo}
|
28 |
+
`
|
29 |
+
}
|
30 |
+
|
31 |
+
export default generateREADME;
|
@@ -0,0 +1,159 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
export interface VideoInfo {
|
2 |
+
height: number;
|
3 |
+
width: number;
|
4 |
+
channels: number;
|
5 |
+
codec: string;
|
6 |
+
pix_fmt: string;
|
7 |
+
is_depth_map: boolean;
|
8 |
+
has_audio: boolean;
|
9 |
+
}
|
10 |
+
|
11 |
+
/**
|
12 |
+
* Metadata parameters interface
|
13 |
+
*/
|
14 |
+
interface MetadataParams {
|
15 |
+
total_episodes: number;
|
16 |
+
total_frames: number;
|
17 |
+
total_tasks: number;
|
18 |
+
chunks_size: number;
|
19 |
+
fps: number;
|
20 |
+
splits: { [key: string]: string };
|
21 |
+
features: { [key: string]: any };
|
22 |
+
videos_info: VideoInfo[];
|
23 |
+
data_files_size_in_mb: number;
|
24 |
+
video_files_size_in_mb: number;
|
25 |
+
}
|
26 |
+
|
27 |
+
/**
|
28 |
+
* Generates and returns a metadata information dictionary
|
29 |
+
* Needs some named parameters passed as parameters
|
30 |
+
*/
|
31 |
+
function getMetadataInfo(params: MetadataParams) {
|
32 |
+
return {
|
33 |
+
"codebase_version": "v2.1",
|
34 |
+
"robot_type": "so100",
|
35 |
+
"total_episodes": params.total_episodes,
|
36 |
+
"total_frames": params.total_frames,
|
37 |
+
"total_tasks": params.total_tasks,
|
38 |
+
"total_videos": params.videos_info.length,
|
39 |
+
"total_chunks": 1,
|
40 |
+
"chunks_size": params.chunks_size,
|
41 |
+
"fps": params.fps,
|
42 |
+
"splits": {
|
43 |
+
"train": `0:${params.total_episodes}`
|
44 |
+
},
|
45 |
+
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
|
46 |
+
"video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
|
47 |
+
"features": {
|
48 |
+
"action": {
|
49 |
+
"dtype": "float32",
|
50 |
+
"shape": [
|
51 |
+
6
|
52 |
+
],
|
53 |
+
"names": [
|
54 |
+
"main_shoulder_pan",
|
55 |
+
"main_shoulder_lift",
|
56 |
+
"main_elbow_flex",
|
57 |
+
"main_wrist_flex",
|
58 |
+
"main_wrist_roll",
|
59 |
+
"main_gripper"
|
60 |
+
],
|
61 |
+
"fps": params.fps
|
62 |
+
},
|
63 |
+
"observation.state": {
|
64 |
+
"dtype": "float32",
|
65 |
+
"shape": [
|
66 |
+
6
|
67 |
+
],
|
68 |
+
"names": [
|
69 |
+
"main_shoulder_pan",
|
70 |
+
"main_shoulder_lift",
|
71 |
+
"main_elbow_flex",
|
72 |
+
"main_wrist_flex",
|
73 |
+
"main_wrist_roll",
|
74 |
+
"main_gripper"
|
75 |
+
],
|
76 |
+
"fps": params.fps
|
77 |
+
},
|
78 |
+
"observation.images.front": {
|
79 |
+
"dtype": "video",
|
80 |
+
"shape": [
|
81 |
+
480,
|
82 |
+
640,
|
83 |
+
3
|
84 |
+
],
|
85 |
+
"names": [
|
86 |
+
"height",
|
87 |
+
"width",
|
88 |
+
"channels"
|
89 |
+
],
|
90 |
+
"info": {
|
91 |
+
"video.fps": params.fps,
|
92 |
+
"video.height": 480,
|
93 |
+
"video.width": 640,
|
94 |
+
"video.channels": 3,
|
95 |
+
"video.codec": "av1",
|
96 |
+
"video.pix_fmt": "yuv420p",
|
97 |
+
"video.is_depth_map": false,
|
98 |
+
"has_audio": false
|
99 |
+
}
|
100 |
+
},
|
101 |
+
"timestamp": {
|
102 |
+
"dtype": "float32",
|
103 |
+
"shape": [
|
104 |
+
1
|
105 |
+
],
|
106 |
+
"names": null,
|
107 |
+
"fps": params.fps
|
108 |
+
},
|
109 |
+
"frame_index": {
|
110 |
+
"dtype": "int64",
|
111 |
+
"shape": [
|
112 |
+
1
|
113 |
+
],
|
114 |
+
"names": null,
|
115 |
+
"fps": params.fps
|
116 |
+
},
|
117 |
+
"episode_index": {
|
118 |
+
"dtype": "int64",
|
119 |
+
"shape": [
|
120 |
+
1
|
121 |
+
],
|
122 |
+
"names": null,
|
123 |
+
"fps": params.fps
|
124 |
+
},
|
125 |
+
"index": {
|
126 |
+
"dtype": "int64",
|
127 |
+
"shape": [
|
128 |
+
1
|
129 |
+
],
|
130 |
+
"names": null,
|
131 |
+
"fps": params.fps
|
132 |
+
},
|
133 |
+
"task_index": {
|
134 |
+
"dtype": "int64",
|
135 |
+
"shape": [
|
136 |
+
1
|
137 |
+
],
|
138 |
+
"names": null,
|
139 |
+
"fps": params.fps
|
140 |
+
}
|
141 |
+
},
|
142 |
+
"data_files_size_in_mb": 100,
|
143 |
+
"video_files_size_in_mb": 500
|
144 |
+
}
|
145 |
+
}
|
146 |
+
|
147 |
+
export function getVideoInfo(width: number, height: number): VideoInfo {
|
148 |
+
return {
|
149 |
+
height,
|
150 |
+
width,
|
151 |
+
channels: 3,
|
152 |
+
codec: "h264",
|
153 |
+
pix_fmt: "yuv420p",
|
154 |
+
is_depth_map: false,
|
155 |
+
has_audio: false
|
156 |
+
};
|
157 |
+
}
|
158 |
+
|
159 |
+
export default getMetadataInfo;
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
/**
|
2 |
+
* Calculates basic statistics (min, max, mean, std, count) for a numeric array
|
3 |
+
*/
|
4 |
+
function calculateStats(values: number[]): { min: number[], max: number[], mean: number[], std: number[], count: number[] } {
|
5 |
+
const count = values.length;
|
6 |
+
if (count === 0) {
|
7 |
+
return {
|
8 |
+
min: [0],
|
9 |
+
max: [0],
|
10 |
+
mean: [0],
|
11 |
+
std: [0],
|
12 |
+
count: [0]
|
13 |
+
};
|
14 |
+
}
|
15 |
+
|
16 |
+
let min = Infinity;
|
17 |
+
let max = -Infinity;
|
18 |
+
|
19 |
+
for(let value of values){
|
20 |
+
min = Math.min(min, value);
|
21 |
+
max = Math.max(max, value);
|
22 |
+
}
|
23 |
+
|
24 |
+
const sum = values.reduce((acc, val) => acc + val, 0);
|
25 |
+
const mean = sum / count;
|
26 |
+
|
27 |
+
// Calculate standard deviation
|
28 |
+
const squareDiffs = values.map(value => Math.pow(value - mean, 2));
|
29 |
+
const avgSquareDiff = squareDiffs.reduce((acc, val) => acc + val, 0) / count;
|
30 |
+
const std = Math.sqrt(avgSquareDiff);
|
31 |
+
|
32 |
+
return {
|
33 |
+
min: [min],
|
34 |
+
max: [max],
|
35 |
+
mean: [mean],
|
36 |
+
std: [std],
|
37 |
+
count: [count]
|
38 |
+
};
|
39 |
+
}
|
40 |
+
|
41 |
+
/**
|
42 |
+
* Generates statistics for a dataset
|
43 |
+
* @param data The dataset to analyze
|
44 |
+
* @param cameraKeys Array of camera keys for dynamic observation.images entries
|
45 |
+
*/
|
46 |
+
export function getStats(data: any[], cameraKeys: string[] = []): any {
|
47 |
+
// Extract timestamp and episode_index values
|
48 |
+
const timestamps = data.map(item => item.timestamp);
|
49 |
+
const episodeIndices = data.map(item => item.episode_index);
|
50 |
+
|
51 |
+
// Extract other common fields if they exist
|
52 |
+
const frameIndices = data.map(item => item.frame_index || 0);
|
53 |
+
const taskIndices = data.map(item => item.task_index || 0);
|
54 |
+
|
55 |
+
const stats: any = {
|
56 |
+
// Standard fields
|
57 |
+
"timestamp": calculateStats(timestamps),
|
58 |
+
"episode_index": calculateStats(episodeIndices),
|
59 |
+
"frame_index": calculateStats(frameIndices),
|
60 |
+
"task_index": calculateStats(taskIndices),
|
61 |
+
};
|
62 |
+
|
63 |
+
// Add observation.images entries for each camera key
|
64 |
+
cameraKeys.forEach(key => {
|
65 |
+
// In a real implementation, you would calculate actual stats from video data
|
66 |
+
// Since we don't have actual video frame data to analyze, we'll use placeholder values
|
67 |
+
stats[`observation.images.${key}`] = {
|
68 |
+
"min": [[[0.0]], [[0.0]], [[0.0]]], // R,G,B channels min
|
69 |
+
"max": [[[255.0]], [[255.0]], [[255.0]]], // R,G,B channels max
|
70 |
+
"mean": [[[127.5]], [[127.5]], [[127.5]]], // R,G,B channels mean
|
71 |
+
"std": [[[50.0]], [[50.0]], [[50.0]]], // R,G,B channels std
|
72 |
+
"count": [[[data.length * 3]]] // Number of pixels × 3 channels
|
73 |
+
};
|
74 |
+
});
|
75 |
+
|
76 |
+
return stats;
|
77 |
+
}
|
78 |
+
|
79 |
+
export default getStats;
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
import { describe, it, expect, beforeEach } from "vitest";
|
2 |
+
import { vi } from "vitest";
|
3 |
+
import { LeRobotDatasetRecorder } from "../src/record";
|
4 |
+
import { WebTeleoperator } from "../../web/src/teleoperators/base-teleoperator";
|
5 |
+
|
6 |
+
// Mock the WebTeleoperator class
|
7 |
+
vi.mock("../../web/src/teleoperators/base-teleoperator", () => {
|
8 |
+
return {
|
9 |
+
WebTeleoperator: vi.fn().mockImplementation(() => {
|
10 |
+
return {
|
11 |
+
startRecording: vi.fn(),
|
12 |
+
stopRecording: vi.fn().mockResolvedValue([]),
|
13 |
+
clearRecording: vi.fn()
|
14 |
+
};
|
15 |
+
})
|
16 |
+
};
|
17 |
+
});
|
18 |
+
|
19 |
+
describe("LeRobotDatasetRecorder", () => {
|
20 |
+
let recorder: LeRobotDatasetRecorder;
|
21 |
+
|
22 |
+
beforeEach(() => {
|
23 |
+
// Create a new recorder instance before each test
|
24 |
+
// @ts-ignore
|
25 |
+
const mockTeleoperator = new WebTeleoperator() as unknown as WebTeleoperator;
|
26 |
+
const mockVideoStreams = {};
|
27 |
+
recorder = new LeRobotDatasetRecorder([mockTeleoperator], mockVideoStreams, 30);
|
28 |
+
});
|
29 |
+
|
30 |
+
describe("_interpolateAndCompleteLerobotData", () => {
|
31 |
+
it("should interpolate data to match the specified fps", async () => {
|
32 |
+
// Create test data with non-regular timestamps
|
33 |
+
const roughData = [
|
34 |
+
{
|
35 |
+
timestamp: 0,
|
36 |
+
action: {
|
37 |
+
shoulder_pan: 0,
|
38 |
+
shoulder_lift: 0,
|
39 |
+
elbow_flex: 0,
|
40 |
+
wrist_flex: 0,
|
41 |
+
wrist_roll: 0,
|
42 |
+
gripper: 0
|
43 |
+
},
|
44 |
+
"observation.state": {
|
45 |
+
shoulder_pan: 0,
|
46 |
+
shoulder_lift: 0,
|
47 |
+
elbow_flex: 0,
|
48 |
+
wrist_flex: 0,
|
49 |
+
wrist_roll: 0,
|
50 |
+
gripper: 0
|
51 |
+
},
|
52 |
+
episode_index: 0,
|
53 |
+
task_index: 0
|
54 |
+
},
|
55 |
+
{
|
56 |
+
timestamp: 0.5,
|
57 |
+
action: {
|
58 |
+
shoulder_pan: 10,
|
59 |
+
shoulder_lift: 20,
|
60 |
+
elbow_flex: 30,
|
61 |
+
wrist_flex: 40,
|
62 |
+
wrist_roll: 50,
|
63 |
+
gripper: 60
|
64 |
+
},
|
65 |
+
"observation.state": {
|
66 |
+
shoulder_pan: 15,
|
67 |
+
shoulder_lift: 25,
|
68 |
+
elbow_flex: 35,
|
69 |
+
wrist_flex: 45,
|
70 |
+
wrist_roll: 55,
|
71 |
+
gripper: 65
|
72 |
+
},
|
73 |
+
episode_index: 0,
|
74 |
+
task_index: 0
|
75 |
+
},
|
76 |
+
{
|
77 |
+
timestamp: 1.0,
|
78 |
+
action: {
|
79 |
+
shoulder_pan: 20,
|
80 |
+
shoulder_lift: 40,
|
81 |
+
elbow_flex: 60,
|
82 |
+
wrist_flex: 80,
|
83 |
+
wrist_roll: 100,
|
84 |
+
gripper: 120
|
85 |
+
},
|
86 |
+
"observation.state": {
|
87 |
+
shoulder_pan: 30,
|
88 |
+
shoulder_lift: 50,
|
89 |
+
elbow_flex: 70,
|
90 |
+
wrist_flex: 90,
|
91 |
+
wrist_roll: 110,
|
92 |
+
gripper: 130
|
93 |
+
},
|
94 |
+
episode_index: 1, // New episode
|
95 |
+
task_index: 0
|
96 |
+
},
|
97 |
+
{
|
98 |
+
timestamp: 1.5,
|
99 |
+
action: {
|
100 |
+
shoulder_pan: 25,
|
101 |
+
shoulder_lift: 50,
|
102 |
+
elbow_flex: 75,
|
103 |
+
wrist_flex: 100,
|
104 |
+
wrist_roll: 125,
|
105 |
+
gripper: 150
|
106 |
+
},
|
107 |
+
"observation.state": {
|
108 |
+
shoulder_pan: 35,
|
109 |
+
shoulder_lift: 55,
|
110 |
+
elbow_flex: 75,
|
111 |
+
wrist_flex: 95,
|
112 |
+
wrist_roll: 115,
|
113 |
+
gripper: 135
|
114 |
+
},
|
115 |
+
episode_index: 1, // New episode
|
116 |
+
task_index: 0
|
117 |
+
}
|
118 |
+
];
|
119 |
+
|
120 |
+
// Set the FPS to 10 for this test
|
121 |
+
const fps = 10;
|
122 |
+
|
123 |
+
// Call the method under test
|
124 |
+
const result = await recorder._interpolateAndCompleteLerobotData(fps, roughData);
|
125 |
+
|
126 |
+
// log all the results, row by row
|
127 |
+
for (let i = 0; i < result.length; i++) {
|
128 |
+
console.log(result[i]);
|
129 |
+
}
|
130 |
+
|
131 |
+
// Verify the results
|
132 |
+
expect(result).toBeInstanceOf(Array);
|
133 |
+
expect(result.length).toBe(15); // 1.5 seconds at 10 fps = 15 frames
|
134 |
+
|
135 |
+
// Check the first frame
|
136 |
+
expect(result[0].timestamp).toBeCloseTo(0, 5);
|
137 |
+
expect(result[0].action).toEqual([0, 0, 0, 0, 0, 0]);
|
138 |
+
expect(result[0]["observation.state"]).toEqual([0, 0, 0, 0, 0, 0]);
|
139 |
+
expect(result[0].episode_index).toBe(0);
|
140 |
+
expect(result[0].task_index).toBe(0);
|
141 |
+
expect(result[0].frame_index).toBe(0);
|
142 |
+
expect(result[0].index).toBe(0);
|
143 |
+
|
144 |
+
// Check a middle frame (0.3 seconds)
|
145 |
+
const middleFrame = result[3];
|
146 |
+
expect(middleFrame.timestamp).toBeCloseTo(0.3, 5);
|
147 |
+
// At 0.3 seconds, we're 60% between 0 and 0.5 seconds
|
148 |
+
// So action.shoulder_pan should be 60% of 10 = 6
|
149 |
+
expect(middleFrame.action[0]).toBeCloseTo(6, 5);
|
150 |
+
expect(middleFrame.episode_index).toBe(0);
|
151 |
+
expect(middleFrame.frame_index).toBe(3);
|
152 |
+
expect(middleFrame.index).toBe(3);
|
153 |
+
|
154 |
+
// Check the frame right after the episode change
|
155 |
+
const episodeChangeFrame = result[5]; // 0.5 seconds
|
156 |
+
expect(episodeChangeFrame.timestamp).toBeCloseTo(0.5, 5);
|
157 |
+
expect(episodeChangeFrame.action[0]).toBeCloseTo(10, 5);
|
158 |
+
expect(episodeChangeFrame.episode_index).toBe(0);
|
159 |
+
expect(episodeChangeFrame.frame_index).toBe(5);
|
160 |
+
|
161 |
+
// Check the last frame before 1 second
|
162 |
+
const lastFrame = result[9]; // 0.9 seconds
|
163 |
+
expect(lastFrame.timestamp).toBeCloseTo(0.9, 5);
|
164 |
+
expect(lastFrame.episode_index).toBe(0);
|
165 |
+
expect(lastFrame.frame_index).toBe(9); // Frame index continues incrementing
|
166 |
+
expect(lastFrame.index).toBe(9);
|
167 |
+
});
|
168 |
+
|
169 |
+
it("should handle episode index changes correctly", async () => {
|
170 |
+
// Create test data with episode changes
|
171 |
+
const roughData = [
|
172 |
+
{ timestamp: 0.0, action: { shoulder_pan: 0 }, "observation.state": { shoulder_pan: 0 }, episode_index: 0, task_index: 0 },
|
173 |
+
{ timestamp: 0.3, action: { shoulder_pan: 30 }, "observation.state": { shoulder_pan: 30 }, episode_index: 0, task_index: 0 },
|
174 |
+
{ timestamp: 0.5, action: { shoulder_pan: 50 }, "observation.state": { shoulder_pan: 50 }, episode_index: 1, task_index: 0 }, // Episode change
|
175 |
+
{ timestamp: 0.8, action: { shoulder_pan: 80 }, "observation.state": { shoulder_pan: 80 }, episode_index: 1, task_index: 0 },
|
176 |
+
{ timestamp: 1.0, action: { shoulder_pan: 100 }, "observation.state": { shoulder_pan: 100 }, episode_index: 2, task_index: 0 } // Another episode change
|
177 |
+
];
|
178 |
+
|
179 |
+
const fps = 10;
|
180 |
+
const result = await recorder._interpolateAndCompleteLerobotData(fps, roughData);
|
181 |
+
|
182 |
+
// Check frame indices reset after episode changes
|
183 |
+
expect(result[0].episode_index).toBe(0);
|
184 |
+
expect(result[0].frame_index).toBe(0);
|
185 |
+
|
186 |
+
expect(result[5].episode_index).toBe(1); // After episode change
|
187 |
+
expect(result[5].frame_index).toBe(0); // Should reset to 0
|
188 |
+
|
189 |
+
expect(result[9].episode_index).toBe(1); // After second episode change
|
190 |
+
expect(result[9].frame_index).toBe(4); // Frame index continues incrementing
|
191 |
+
});
|
192 |
+
|
193 |
+
it("should handle empty or minimal data", async () => {
|
194 |
+
// Test with minimal data (just two points)
|
195 |
+
const minimalData = [
|
196 |
+
{ timestamp: 0.0, action: { shoulder_pan: 0 }, "observation.state": { shoulder_pan: 0 }, episode_index: 0, task_index: 0 },
|
197 |
+
{ timestamp: 0.1, action: { shoulder_pan: 10 }, "observation.state": { shoulder_pan: 10 }, episode_index: 0, task_index: 0 }
|
198 |
+
];
|
199 |
+
|
200 |
+
const fps = 10;
|
201 |
+
const result = await recorder._interpolateAndCompleteLerobotData(fps, minimalData);
|
202 |
+
|
203 |
+
expect(result.length).toBe(1); // 0.1 seconds at 10fps = 1 frame
|
204 |
+
expect(result[0].timestamp).toBeCloseTo(0, 5);
|
205 |
+
expect(result[0].action[0]).toBeCloseTo(0, 5);
|
206 |
+
});
|
207 |
+
});
|
208 |
+
});
|
The diff for this file is too large to render.
See raw diff
|
|