Make the robot's next action visible
A live grasp preview, planning scene, and stage controls let the operator inspect a target and follow execution. The interface exposes useful system state alongside the motion controls.
PROJECT / 01 · Perception · Planning · Physical execution
From a text prompt to a physical grasp
An integrated UR16e workflow connecting language-guided object selection, RGB-D perception, grasp generation, motion planning, and an operator interface.
01 / THE CHALLENGE
Connecting the whole system
A grasp prediction is only one part of a working manipulation system. The selected object, depth measurements, robot coordinate frames, motion plan, gripper, and operator controls all have to agree before the robot moves.
02 / MY CONTRIBUTION
What I worked on
- Integrated GroundingDINO object identification and GraspSAM grasp generation with the robot's RGB-D and MoveIt 2 workflow.
- Developed C++ ROS 2 nodes and services for approach, grasp, retreat, homing, and gripper control.
- Built an operator dashboard with RViz, live grasp previews, depth and confidence information, execution status, and logs.
- Connected staged execution and home overrides to support supervised physical demonstrations.
RECORDED DEMONSTRATIONS
Inside the system
Recorded operator interface
03 / SYSTEM WORKFLOW
How the pieces connect
- 01
Select
Text prompt and GroundingDINO localization
- 02
Perceive
GraspSAM candidates and RGB-D measurements
- 03
Plan
Robot-frame targets and MoveIt 2 motion planning
- 04
Execute
Operator review, staged motion, and gripper control
04 / ENGINEERING DECISIONS
Decisions behind the implementation
Treat coordinate frames as part of the system
Camera measurements must be connected to robot frames before they become motion targets. That connection is part of the perception-to-planning integration.
Separate the execution stages
Approach, grasp, retreat, and home are explicit operations. This makes physical testing and recovery easier to observe and debug.
PROJECT GALLERY
Perception and operator controls
05 / OUTCOME
What the work demonstrates
Delivered an integrated physical robot workflow and operator interface for the DOE system integration and baseline demonstration milestone. The demonstrated workflow uses operator supervision and staged execution.
Work developed at FIU's Applied Research Center with the project team. This page describes my software, perception, planning, and operator-interface contributions.