Vision-guided tasks

A vision task uses camera measurements to locate a marker or sample, calculate a target, and optionally move the arm or operate the gripper.

Before starting

For CMS, complete the Zivid camera setup, check calibration and the selected tool, and launch EROBS with enable_vision:=true. Vision is disabled by default.

Mock robot hardware does not simulate camera detections. Vision tasks still need camera services and do not support Dry Run.

Choose a task

Task	Use it to
Vision MoveTo (`vision_moveto`)	Approach a marker or a pickup point in a marker-anchored sample region.
Pick Sample / Place Sample (`pick_sample` / `place_sample`)	Add grasp or release to a vision-guided approach. Keep Use Vision selected.
Pick Spincoater / Place Spincoater	Adjust joint 6 of a calibrated pose to the detected angle, then perform the configured contact and gripper action.
Vision Scan (`vision_scan`)	Capture and average marker measurements from several saved positions.
`vision_task` in JSON	Specify the detector and target calculation directly.

Add the task in the GUI and double-click its row to configure it. For marker or sample-region tasks, choose Detection Type (marker or sample_roi) and the marker ID. For sample_roi in Vision MoveTo, Strategy selects a center, edge, or corner; Edge Inset sets the inward edge offset in millimetres. The inset must be finite and nonnegative.

Set the marker dictionary in the beamline’s camera.marker_dictionary. The current GUI’s Dictionary field does not change it.

Task presets supply defaults that explicit JSON fields can override. See vision task fields for detector choices, scan/retreat poses, and spincoater settings.

Understand the offsets

Marker and sample-region tasks use approach_pose to calculate the tool target. These offsets use metres in different coordinate frames:

Field	Direction of the offset
`marker_offset_x/y/z`	Along the marker’s axes; rotates with the marker.
`z_offset`	Along the robot base’s Z axis.
`offset_direction` and `offset_distance`	Along the current flange’s axes; rotates with the wrist.

The approach uses a straight-down roll/pitch and marker-derived yaw. z_offset: 0 means use the tool’s configured default; it does not cancel a nonzero default.

Spincoater tasks use the separate j6_snap target calculation.

Check detection before an approach

This example requests the approach target for marker 5 without executing the computed approach. Set start_gripper and tag_id to match your installation.

{
  "start_gripper": "epick",
  "tasks": [
    {
      "task_type": "vision_task",
      "detector": "marker",
      "goal_computer": "approach_pose",
      "tag_id": 5,
      "detect_only": true
    }
  ]
}

Load the JSON with File → Load JSON, leave Dry Run cleared, and press Execute. The request uses the camera and initializes MoveIt. The action result contains the computed position and quaternion in base_link when available; an empty result array means no pose was returned. The GUI does not display this pose: read it from the orchestrator log (“Stored detected pose”) or the vision server log (“Detect-only: returning approach pose”), or use an action client that prints the result.

Detect Only controls the computed approach, not every action in the task:

Adding scan_pose or scan_positions can introduce arm movements before detection.
Pick presets can open the gripper before moving to a scan pose.
j6_snap ignores detect_only and can execute spincoater movements.

The example above includes no scan moves or pre-opening. A successful detection does not validate the later approach path or grasp.

Scans and execution limits

Use known saved poses for scan positions. Missing names are skipped; vision_scan fails if none resolve. Vision Scan and Vision MoveTo keep separate caches. Running a scan does not replace the later task’s detection step.

For multi-position marker detection within one task, supply scan_positions in that task’s JSON. See the detection fields.

Path constraints apply to the named scan_pose move, approach and retreat, and the j6_snap positioning and contact moves. Multi-position scan_positions moves and standalone Vision Scan do not apply them. See constraint fields.

If a task fails

Read the Activity Log and server logs to identify whether capture, detection, coordinate transforms, target calculation, planning, or execution failed. Check camera availability, marker visibility and ID, transforms, and saved pose names before changing offsets. Use troubleshooting for recovery and perception architecture for the processing stages.