Agentic World Representation IEI Lab

Agentic World Representation

Shaoqiu Zhang1,* Kunqi Xu1,* Jingchen Nie1 Jianglong Ye2 Xueyan Zou1,†
1 Tsinghua University 2 University of California, San Diego
* Co-first authors† Project Lead

  1. RGBD video stream

    RGBDepth

  2. Neural simulation object

    Point cloudsSDF / Normal

  3. Keypoint optimization

    −∇qL → Δq
    Loss53.40

  4. Real robot deployment

Neural simulation objects

Explore the workspace, move a keypoint, and see the geometry respond.

Loading workspace…
Workspace

Optimize a grasp through geometry

Choose a hand and an object. Optimize the grasp, lift, and hold through two gentle shakes.

Object
Hand
XHand · CupIteration 0
Loading hand…

Finger joint angles degrees

    Grasp optimization loss 0

    From simulation to the real robot

    OursMuJoCoπ0.5

    In-distribution

    Out-of-distribution