Perception and mapping

Camera acquisition, feature detection, motion-target calculation, and scene mapping. Includes survey capture and offline point-cloud merging.

Read the guide · All subsystems · Symbol index

Start here

get_camera — Find the selected camera backend and the interface used for acquisition and detection.
get_detector — Find the detector dispatch table when adding or investigating a configured detection type.
VisionTaskStages.run — Follow a vision request through pre-scan motion, detection, target computation, and optional target motion.
main — Trace the offline route from recorded clouds and transforms to a filtered PLY map.

Modules

Module	Responsibility	Entries
Camera backend selection	Define `DetectionResult` and select camera backends on demand.	1
Zivid camera backend	Zivid triggered capture and native marker/sample-ROI detection. The helpers rely on an already running executor to deliver service futures and image callbacks; their waits do not spin the node. Pre-capture timestamps approximate acquisition timing for later TF lookup.	7
ZED camera backend	ZED backend using streamed RGB images and registered point clouds. It estimates marker centers but not marker orientation; unavailable depth can leave a returned marker at the default zero position.	5
Image and marker geometry	Detect markers and samples in supplied images and compute marker-relative pickup geometry without ROS imports. Requires NumPy/OpenCV and calibrated inputs.	5
Spincoater image detection	Classical pocket detection and YOLO segmentation for a red-painted spincoater chuck. Functions consume supplied flash-lit BGR images; the camera adapter performs acquisition. Segmentation weights are loaded lazily from the installed beambot package share directory.	5
YOLO object detection	Detect and annotate objects in supplied images with a lazily loaded Ultralytics model.	5
Vision task detector adapters	Select detector functions for `VisionTaskStages`. The `DETECTORS` dictionary maps `marker`, `sample_roi`, `spincoater_pocket`, and `spincoater_sample` to their adapters.	6
Vision motion targets	Convert vision detections into Cartesian or joint targets for the vision-task coordinator.	4
Shared vision engine	Shared vision engine for camera acquisition, capture-time TF, multi-view scans, approach geometry, and configured collision objects. It also inherits MTC helpers from BaseStages. Some functions capture or move hardware; others only compute geometry. Its scan cache belongs to this engine instance and is not shared with other servers.	22
Marker scan action server	Expose multi-position marker scans backed by a server-local VisionEngine and tag cache.	4
Vision task action server	Expose generic vision-task execution and TF reset, and warm the spincoater inference model in a daemon thread. motion_kind is inferred from the detected-pose result rather than the executed target type.	7
Vision task coordination	Coordinate detector selection, target computation and motion for generic vision tasks. Detection-only requests can still perform pre-scan motion; vacuum verification is disabled.	8
Point-cloud filtering relay	ROS point-cloud relay with optional Open3D statistical filtering and voxel reduction. The default voxel size is 0.02 metres. Noise filtering runs only when downsampling is enabled.	4
Octomap scene bridge	ROS bridge from Octomap messages to differential MoveIt PlanningScene messages. The bridge assumes incoming maps already use the intended frame; it does not transform their geometry.	3
Octomap pipeline launch	Launch the live point-cloud-to-Octomap pipeline and its MoveIt planning-scene bridge.	1
Image point selector	Show a Tk image window for pixel selection and return the result to the MCP process as JSON. Display configuration is probed by `main()`, not at import.	9
Survey viewpoint teaching	Record operator-selected survey viewpoints from `/joint_states` and save them as a YAML pose registry.	7
Survey capture	Move through taught viewpoints and record camera clouds, TF, and joint states for a survey. The `--dry-run` option still commands motion.	12
Survey point-cloud merger	Merge recorded survey clouds into a binary PLY file using TF or URDF transforms, range filtering, and voxel averaging. Processing is offline but requires ROS Python dependencies.	13