Perception and vision execution

EROBS uses camera measurements to calculate an approach pose or align a tool with a detected feature. Vision-enabled task presets run through the unified vision-task server: prepare → detect → calculate a target → execute → report.

For task selection and examples, see vision-guided tasks. This page explains how those requests run inside EROBS.

Task flow

action_servers/vision_task_server.py receives /beambot_vision_task goals and delegates to stages/vision_task_stages.py. Paths on this page are relative to src/beambot/beambot/ unless stated otherwise.

Prepare. Check flange offsets and sample-ROI settings. If scan_pose is set, perform any requested gripper pre-opening and move there. Allow the arm to settle before detection.
Detect. Select the requested detector and obtain a marker pose, sample pickup point, or spincoater angle. Multi-position marker detection can add scan movements here.
Calculate a target. Convert the measurement into a Cartesian pose or six-joint configuration. With approach_pose and detect_only, return the calculated pose instead.
Execute. Build and run the motion, including any requested gripper operation or retreat.
Report. Return success or an error. A failed stage stops the remaining sequence.

Open the full-size diagram

Detection and camera capture

vision/vision_task_detection.py maps detector names to these operations:

Detector	Measurement	Target calculation
`marker`	ArUco marker pose in `base_link`; checks a local cache first, but only batch tag scans fill it, so the unified server normally captures	`approach_pose`
`sample_roi`	Pickup point inside a marker-anchored region of interest (ROI), projected from the image onto the marker plane with a sample-thickness adjustment	`approach_pose`
`spincoater_pocket`	Pocket angle from a flash-lit 2D image	`j6_snap`
`spincoater_sample`	Sample angle from 2D image segmentation	`j6_snap`

sample_roi does not use depth at the sample pixel, which is often sparse on thin or poorly reflecting samples. The projection assumes the sample lies flat on the marker plane.

Shared image processing lives in vision/detection/image_detection.py and vision/detection/spincoater_detection.py. The reusable learned object detector is in vision/detection/yolo_object_detection.py. See the vision function reference for individual functions.

vision/vision_engine.py coordinates capture and transforms. The vision servers read camera.type, camera.frame, and camera.marker_dictionary from the beamline configuration and pass them to it; it raises an error if any is missing. Camera adapters live in vision/camera/; Zivid capture uses vision/camera/zivid.py, with presets under src/beambot/config/zivid/. The retained ZED adapter has known limitations; see camera support.

The unified server warms the spincoater sample model in a background thread. The model is models/spincoater_sample_seg.pt in the installed beambot package share directory. A warmup failure is logged but does not prevent startup.

Capture time and coordinates

Marker and sample-ROI detections start in the camera frame. VisionEngine uses ROS TF to transform them into base_link before calculating an approach.

The Zivid adapter records a timestamp immediately before requesting capture. This approximates capture time; it is not the hardware exposure timestamp. The engine requests the transform at that time and uses the latest transform only when no timestamp is supplied. An unavailable transform fails the detection.

For a wrist-mounted camera, the timestamp connects the measurement to the arm’s position during capture. Using the arm’s later position could shift the target. See frame conventions and camera calibration.

Goal computation and execution

vision/motion_target.py provides two target calculations, selected by goal_computer.

Cartesian approach

approach_pose combines the detected pose with marker-relative offsets, a base-frame Z offset, and a downward tool orientation using marker-derived yaw. An optional flange-relative displacement is applied afterwards. A zero z_offset selects the tool’s configured default.

The resulting CartesianTarget contains the approach pose, working frame, and optional gripper state and retreat pose. The executor attempts inverse kinematics and plans the approach. When a gripper operation or retreat is requested, it combines them with the approach in one MTC task.

Spincoater alignment

j6_snap starts from a named six-joint pose and adjusts joint 6 using the detected angle and calibration offset. Square symmetry limits the correction to (−45°, 45°]. For example, a detected angle of 80° with no calibration offset produces a −10° correction.

The resulting JointTarget specifies all six joints, so reaching it can move the whole arm. The corrected joint move, optional forward contact, and optional gripper operation each use a separate planning and execution cycle.

Execution limits

Setting or result	Current behavior
`detect_only` with `approach_pose`	Returns the calculated approach pose without executing it. Earlier gripper opening and scan movements can still occur.
`detect_only` with `j6_snap`	Ignored; the target can be executed.
Path constraints	Applied to the named `scan_pose` move, Cartesian approach/retreat, and joint-target positioning/contact moves. Multi-position `scan_positions` moves use a separate helper without constraints.
`motion_kind` result	On success, reports `NONE` when a detection-only pose is returned; otherwise reports `CARTESIAN_POSE`, including joint-target execution.
`vacuum_ok` result	True on successful requests because the vacuum check is disabled; it does not confirm a successful grasp.

Separate scan and mapping paths

action_servers/vision_scan_server.py serves /beambot_vision_scan. It averages marker measurements across scan positions and stores them in its own VisionEngine. Its cache is separate from the unified task server’s cache: a standalone scan does not supply detections to a later Vision MoveTo task.

Tool exchange resets the unified vision-task server’s TF buffer. The standalone scan server has no corresponding reset service.

Pick/place requests with use_vision: false use action_servers/sample_pick_place_server.py. Vision-enabled presets use the unified server described above.

Optional occupancy mapping starts from src/beambot/launch/octomap_mapping.launch.py. mapping/pointcloud_relay.py feeds clouds to the Octomap server; mapping/octomap_to_planning_scene.py forwards maps to MoveIt. The bridge copies the incoming map without transforming it and publishes updates without waiting for acknowledgement. It forwards at most one update per min_update_interval (0.5 s by default) and drops the rest, so the planning scene can miss the latest map change. Check the map frame and placement when enabling this separate workflow; see Octomap configuration.