pi0-lerobot

MVP of Multiview Video → Camera parameters + 3D keypoints. The basic pipeline as of right now looks like this: 1. Capture – Using 4 iPhones and an Insta360 Go. iPhone videos are captured via Final Cut Pro Multicam for easy sync and the exocentric view; the Insta360 Go is used for the egocentric view. 2. Sync – Custom Gradio app using two @rerundotio viewers and callbacks for easily aligning frame timestamps so the ego and exo views are aligned. 3. Calibrate – Use VGGT from Meta to get intrinsics/extrinsics for sparse cameras. 4. Estimate 3D – Use RTMLib whole‑body keypoint estimator on each frame, then triangulate in 3D. What's missing? 1. No temporal coherence: I’m estimating keypoints one frame at a time and one camera at a time. This leads to a lot of jittering. For now, I plan on adding a One Euro Filter to help with jittering. Long t

I've been trying to wrap my head around fwd/bwd kinematics for imitation learning, so I built a fully‑differentiable kinematic hand skeleton in JAX and visualized it with reruns new callback system in a Jupyter Notebook. This shows each joint angle and how it impacts the kinematic skeleton.

Continuing with my robot training collection pipeline, one of the challenges has been obtaining calibrated cameras from sparse multi-view inputs. I previously worked with Dust3r but found its accuracy insufficient for generating reliable camera parameters. Recently, Meta released VGGT—Visual Geometry Grounded Transformer—which improves both inference speed and accuracy. With assumptions such as non-metric scale and the use of pinhole cameras, I believe fully automatic camera calibration is now achievable. Using @rerun, I established a baseline from the HoCAP dataset and conducted a qualitative comparison among the ground-truth calibrated cameras, Dust3r, and VGGT. The improvements are evident in both the camera parameters and the multi-view depth map/point cloud. I will eventually add quantitative comparisons, such as Relative Rotation Ac

I’m developing a straightforward method to collect first-person (ego) and third-person (exo) data for robotic training. I’ve been using the HO-cap dataset to establish a baseline, and here are some updates I’ve made: * added in MANO parameters * added TSDF fused mesh from realsense camera to better visualize dataset and potentially set the stage for a gsplat integration * Built a consistent interface between different ego+exo datasets (in this case assembly101 and hocap) From here the plan is to include the egocentric perspective now that the exocentric part is done! The full MVP masterplan laid out 1. Build a strong opensource baseline on a permissively licensed dataset - [x] Exocentric data <- we are here - [ ] Egocentric data <- up next - [ ] Pose estimation algorithm - [ ] Mano fitting algorithm 2. Build an opensource data