Cameras and detection

EROBS uses camera measurements to locate markers, select sample pickup points, and align the tool with a spincoater. Zivid is the integrated CMS camera. A ZED adapter is also present, with the limitations described below.

For task settings and a detection-only example, see Vision-guided tasks. Perception architecture explains how a measurement becomes a motion target.

Camera setup

Complete the optional Zivid installation and check the camera calibration. The vendor SDK is a separate host dependency; the ROS drivers are imported through src/vision/vision.repos.

The beamline’s camera block selects the adapter, coordinate frame, and marker dictionary. CMS uses type: zivid, frame: zivid_optical_frame, and marker_dictionary: aruco4x4_50.

Camera	EROBS integration
Zivid	Main bringup starts the camera and vision servers with `enable_vision:=true`. Vision is disabled by default.
ZED	The adapter reads streamed images and point clouds. Main bringup does not start a ZED driver; changing `camera.type` alone does not configure one.

Mock UR hardware does not simulate camera measurements. The CMS robot model also requires zivid_description for camera geometry when live vision is disabled.

For vendor settings and service details, use the official Zivid ROS driver documentation.

Zivid capture and detection

The adapter in src/beambot/beambot/vision/camera/zivid.py requests captures when needed. Different tasks use different measurements:

Task or workflow	Camera data and processing
Marker detection	`/capture_and_detect_markers` returns native 3D marker poses.
Sample region (`sample_roi`)	The same capture supplies a marker pose and image. Image processing selects a pickup pixel, then projects it onto the marker plane with the configured sample thickness. It does not read sample depth from the point cloud.
Spincoater alignment	`/capture_2d` publishes a flash-lit image on `/color/image_color`. Pocket detection uses image geometry; sample detection uses a segmentation model.
Survey mapping	`/capture` publishes a point cloud on `/points/xyzrgba`. The survey waits for a new cloud after capture; see Survey mapping.

Spincoater sample detection requires Ultralytics and models/spincoater_sample_seg.pt in the installed beambot share directory. Native marker detection and pocket detection do not use that learned model.

The 2D helper subscribes before triggering capture, discards an older frame, and waits for both the service result and an image within one timeout. Temporary subscriptions and clients are cleaned up on success or failure. These waits require a running ROS executor to deliver callbacks; see the vision function reference when using the helpers directly.

Capture settings

The files are in src/beambot/config/zivid/ and are installed with beambot:

zivid_settings.yml — main bringup’s 2D capture settings.
scene_capture_noproj.yml — main bringup’s 3D capture settings.
manufacturing_specular.yml — survey capture’s default preset for reflective surfaces.
manufacturing_specular_fullres.yml — a full-resolution variant for explicit selection; neither main bringup nor the survey selects it by default.

Despite its name, scene_capture_noproj.yml uses nonzero projector brightness for 3D acquisition. Its color acquisition uses zero brightness.

The manufacturing presets were serialized for Zivid SDK 2.17.1 and the CMS camera model. Check compatibility when changing the camera or SDK.

Capture time and coordinates

A camera measurement is not yet a tool target. EROBS transforms it into base_link, then applies the task’s offsets and tool settings.

For Zivid, the adapter records a timestamp immediately before requesting capture. This approximates capture time; it is not the hardware exposure timestamp. VisionEngine uses it to look up the camera-to-base transform. An unavailable transform fails the detection.

This matters because the camera moves with the wrist. Using a later arm position could shift the calculated target. See capture time and coordinates for the transform behavior.

ZED limitations

The adapter in vision/camera/zed.py, relative to src/beambot/beambot/, reads images and registered point clouds under /zed/zed_node/. It uses the shared OpenCV ArUco helper to find markers and looks up their positions in the cloud.

This path is not verified end to end:

Marker orientation is not estimated; the adapter returns an identity quaternion.
Missing or invalid depth can leave a detected marker at the default zero position instead of rejecting it.
Image and cloud messages are collected independently, without timestamp synchronization.
The adapter has no sample_roi implementation, and spincoater detection explicitly uses Zivid capture.

These limits prevent treating ZED as a drop-in replacement for CMS vision tasks.

Recorded camera check

A detection-only task returns the approach target, including the tool offset, not the raw marker center. Physical calibration accuracy and vision-guided grasping have not been validated.