🚧 Work in progress: this documentation is still being written and reviewed. Check details against the source code.

Image and marker geometry

Detect markers and samples in supplied images and compute marker-relative pickup geometry without ROS imports. Requires NumPy/OpenCV and calibrated inputs.

beambot.vision.detection.image_detection · Source

Perception and mapping · Guide · Symbol index

How it is used

Camera backends call these functions to detect ArUco corners, read cloud depth, or find a sample inside a marker-relative image region. They operate on supplied image/cloud data and do not acquire frames or command the robot.

For sample pickup, detect_sample_in_roi() finds the image point and sample_roi_pickup_camera_xyz() projects it onto the calibrated marker plane. SampleRoiDetectionParams holds crop geometry, contour filters, marker size, and sample thickness.

Related data types:

• SampleRoiDetectionParams: ROI geometry and filtering settings for marker-relative sample detection. Distances use millimetres; marker size and sample thickness affect the camera-space pickup estimate.

Start here

Callable	Purpose
detect_aruco_markers	Convert an RGB image to grayscale and detect ArUco corners/IDs with the available OpenCV API.
get_3d_position	Read XYZ at a pixel, then search expanding square borders for the first usable neighboring point.
detect_sample_in_roi	Locate the largest contour passing filters in a marker-relative image crop, fit a rectangle, and choose a pickup point by strategy.
sample_roi_pickup_camera_xyz	Project a pickup pixel into camera coordinates using marker image axes and a marker quaternion, without reading sample depth.

Functions

detect_aruco_markers()

Convert an RGB image to grayscale and detect ArUco corners/IDs with the available OpenCV API.

Function · Source

detect_aruco_markers(
    rgb_image: np.ndarray,
    dictionary_id: int,
) -> tuple[tuple[np.ndarray, ...], np.ndarray | None]

Parameter	Meaning
rgb_image	RGB NumPy image, converted with COLOR_RGB2GRAY.
dictionary_id	OpenCV ArUco dictionary constant, such as cv2.aruco.DICT_4X4_50.

Returns: (corners, ids); OpenCV supplies the corner arrays and returns ids=None when nothing is detected. Rejected candidates are discarded.

Errors: AttributeError in the newer API path triggers the legacy Dictionary_get/DetectorParameters_create fallback. Image/dictionary errors otherwise propagate.

Side effects: Allocates grayscale/detection arrays; no ROS, file, or camera operation.

Referenced in source (2)

• detect_markers — beambot.vision.camera.zed
• _detect_aruco_markers — src.beambot.mcp_server.beambot_mcp_server

get_3d_position()

Read XYZ at a pixel, then search expanding square borders for the first usable neighboring point.

Function · Source

get_3d_position(
    cloud,
    cx: int,
    cy: int,
    search_radius: int = 10,
) -> tuple[float, float, float] | None

Parameter	Meaning
cloud	Image-aligned PointCloud2-like object with width, height, point_step, row_step, and data. XYZ must be the first three little-endian float32 values in each point.
cx	Horizontal pixel index.
cy	Vertical pixel index.
search_radius	Largest square-border radius in pixels; zero checks only the requested pixel.

Returns: (x,y,z) in metres, or None. Search order selects the first valid point on square borders, not a Euclidean-nearest or averaged point.

Errors: Out-of-bounds pixels, short buffers, NaNs, and all-zero XYZ are skipped. Infinity is not rejected. Unsupported layouts or invalid argument types can raise.

Side effects: Reads cloud bytes without modifying them; no TF transform is applied.

detect_sample_in_roi()

Locate the largest contour passing filters in a marker-relative image crop, fit a rectangle, and choose a pickup point by strategy.

Function · Source

detect_sample_in_roi(
    rgb_image: np.ndarray,
    marker_corners: np.ndarray,
    px_per_mm: float,
    strategy: str = 'farthest_edge',
    edge_inset_mm: float = 0.0,
    params: SampleRoiDetectionParams | None = None,
) -> dict[str, Any] | None

Parameter	Meaning
rgb_image	Despite the name, this implementation expects a BGR or grayscale image; three-channel crops use COLOR_BGR2GRAY.
marker_corners	Finite (4,2) pixel array ordered top-left, top-right, bottom-right, bottom-left.
px_per_mm	Finite positive pixels-per-millimetre scale.
strategy	center, farthest_edge, nearest_edge, farthest_corner, or nearest_corner, measured relative to the marker center.
edge_inset_mm	Finite nonnegative distance toward the fitted sample center; it is not clamped to the center or contour.
params	SampleRoiDetectionParams; None supplies defaults for ROI geometry and contour filtering.

Returns: Dictionary containing pickup_px, center_px, sample_size_mm, sample_angle, sample_area_px, offset_from_center_mm, roi, strategy, and edge_inset_mm; or None. Pixels are full-image coordinates; angle is OpenCV’s fitted rectangle angle in degrees.

Errors: Unsupported strategies, invalid numeric geometry, degenerate marker axes, an empty crop, or no accepted contour return None. Image/OpenCV errors and invalid non-geometry filter fields can propagate.

Side effects: Allocates image crops, edge masks, and contour geometry. The input image is not annotated or written to disk.

sample_roi_pickup_camera_xyz()

Project a pickup pixel into camera coordinates using marker image axes and a marker quaternion, without reading sample depth.

Function · Source

sample_roi_pickup_camera_xyz(
    pickup_px: tuple[float, float],
    marker_corners: np.ndarray,
    marker_position: tuple[float, float, float],
    marker_quat_xyzw: tuple[float, float, float, float],
    px_per_mm: float,
    sample_thickness_mm: float = 0.0,
) -> tuple[float, float, float] | None

Parameter	Meaning
pickup_px	Pickup (x,y) in full-image pixels.
marker_corners	Marker corners ordered TL/TR/BR/BL in a finite (4,2) array.
marker_position	Marker position (x,y,z) in camera-frame metres.
marker_quat_xyzw	Marker-to-camera quaternion in x,y,z,w order; normalization is handled by its squared norm.
px_per_mm	Finite positive pixel scale.
sample_thickness_mm	Sample height in millimetres, applied along negative marker-local Z.

Returns: Camera-frame XYZ tuple in metres, or None for rejected/nonfinite geometry. The projection assumes a near-top-down view and uses normalized image axes.

Errors: Invalid corner geometry, zero quaternion norm, or nonfinite output returns None. Array conversion, incompatible shapes, and malformed quaternion/position inputs can raise.

Side effects: Performs numeric calculation only; does not access a point cloud or mutate inputs.

Internal callbacks and helpers

get_3d_position.get_xyz_at()

Decode a single candidate point using the enclosing cloud layout.

Internal helper · Source

get_xyz_at(u: int, v: int)

Parameter	Meaning
u	Candidate column index.
v	Candidate row index.

Returns: XYZ tuple, or None for bounds, unpacking, NaN, or all-zero failures.

Errors: struct.error is converted to None; other errors are not caught.

Side effects: Reads the cloud buffer through the enclosing function’s captured variables.

Referenced in source (1)

• get_3d_position — beambot.vision.detection.image_detection