<aside> 💡

Use VLMs to track pose of the manipulator with the object, learn parameters, optimize the ideal parameters that enable manipulator to pick up.

Lets say goal is to determine the object properties. specifically we have simulation arm, and physical arm. sim arm starts at some initial joints, and we manipulate the object and feel the force, like if its heavy object our physical arm is slower, so its joint positions are in lower points, but sim is higher because they havent learned object intrinsics, so this diff in joint positions will serve as error, and we backpropogate the update the gradient of the loss wrt object parameters such as mass, etc.

</aside>

CUDA Toolkit 12.6, Driver 12.5

Python 3.10

colcon build / catkin_make
source /opt/ros/noetic/setup.sh
roscore

                        FLOW AND DEFORMATION BEHAVIOR
                                  |
             ---------------------------------------------------
             |                                                   |
          SOLIDS                                             FLUIDS
      (rigid or deformable)                           (can flow under stress)
             |                                                   |
   ------------------                         -------------------------------------
   |                |                         |                                    |
Elastic          Plastic                  Newtonian                          Non-Newtonian
(spring back    (deform irreversibly)       (constant viscosity)            (variable viscosity)
under stress)                                                        (viscosity changes with shear rate)
                                           |                                    |
                                ------------------              -------------------------------
                                |                |              |              |              |
                              Water         Simple oils   Bingham      Power Law       Herschel-Bulkley
                                                        (yield stress) (shear-thin    (yield stress, 
                                                                       or shear-thick)  non-linear flow)
                                                       

Todos:

• [ ] Robot arm to move
  • [ ] Obtain sensor data from robot
    • [ ] Maybe ask Yips lab for sensor colelction?
    • [x] Find sensor data
  • [ ] Setup the robot arm with the force scale
    • [x] Ask annabella to replace the gripper.
    • [ ] first experiment iteraiton
      • [ ] Use human arm to poke material
      • [ ] setup camera and find all details for simulation emulation
      • [ ] 3D print materials
      • [ ] put on frame of the base robot
    • [ ] Secod experiment iteration
      • [ ] Write script that hardcodes robot arm movement
  • [ ] think about how friction will work (both objects in contact, need reference friciton)
• [ ] palying with nvidia warp and taking some pictures
  • [ ] Start on comparison
    • [x] test out some more confounding variables
    • [x] Manually build key frame extraction for side by side comparison
    • [ ] Start ablation studies
      • [x] Remove the 2nd Lame param
      • [x] Remove the 1st Lame param
      • [x] remove the damping param
      • [ ] Plot all of the values predicted, as we ablate them.
    • [x] Prompt LLM to pick the most salient parameters that can be visualized.
  • [ ] 2nd upgraded simulation environment
    • [x] Identify surface points to apply force
    • [x] add vector visualization to show the force direction
    • [x] Use One simple block.
    • [x] build frame extraction script
  • [ ] 3rd upgraded simulation
    • [ ] Chnage area of force

Meetings

GOAL:

federico, hsyics gaussian, rendering to estimate softbody parameter.

• MPM
• switching llm and rendering.