RobVision Setup

Prerequisites

• Minimum RobFlow Frontend Version: v5.5.0
• Minimum RobFlow Backend Version: v5.5.0

Configuration

Open RobFlow

1. Open a web browser, preferably Chrome.
2. Enter the Robot's IP address in the address bar to launch RobFlow.

Access Robot Configuration

1. Navigate to the Robot Configuration section.
2. Select the RobVision Devices tab

Add RobVision Device

1. Click on "Add Device".
2. Enter the following details:
   • Name: [custom name]
   • Host IP: 192.168.1.11
3. Close the Window

Camera-robot Calibration

Camera-robot calibration determines the spatial relationship between the camera and the Robot, enabling the system to translate object positions detected by the camera into the Robot's coordinate system.

Camera mounted on Robot

Camera Selection

1. Open RobVision.
2. Select the connected camera.

Calibration Tool

Position the calibration tool on the table and don’t move it during the entire calibration.

Start New Calibration

1. Click on Start New Calibration.
2. Choose the method Robot Mount.

Robot Positioning and Image Capture

1. Move the Robot so the camera can fully capture the calibration board.
2. Verify the visibility of the calibration board by capturing an image (Capture button).
3. When the board is fully visible, click Calibrate.
4. Repeat this process until 9 different poses are added.
5. Verify the Calibration step: The calibration step was successful when the corners of the chessboard are marked.

6. If the Calibration step was successful, Press Next to proceed to the next calibration step

Complete Calibration

1. Close the window in the bottom right corner.
2. Press the reload button.

Static camera

Camera Selection

1. Open RobVision.
2. Select the connected camera configured for static mount calibration.

Calibration Tool

Mount the calibration tool (chessboard) securely on the Robot, after the last joint, ensuring it is stable and does not shift during movements.

Start New Calibration

1. Click on Start New Calibration.
2. Choose the method Static Mount.

Robot Positioning and Image Capture

1. Move the Robot to different positions so the static camera can clearly view the calibration board from various angles.
2. Verify the visibility of the calibration board by capturing an image (Capture button).
3. When the board is fully visible, click Calibrate.
4. Verify the Calibration step: The calibration step was successful when the corners of the chessboard are marked.

5. If the Calibration step was successful, Press Next to proceed to the next calibration step

Complete Calibration

1. Close the window in the bottom right corner.
2. Press the reload button

Calibration Assessment

The calibration accuracy should be within 3 mm or less. If it exceeds this threshold, reset the calibration and add new poses to improve accuracy. Ensure the chessboard is positioned close enough to the camera for successful calibration.

If a consistent offset is observed, you can fine-tune the accuracy by applying an offset in the grasping pose

Important: Ensure the Robot's poses vary significantly in position and orientation, and ensure the calibration board is always fully visible to the camera for optimal results.

Recalibration

Follow these steps to safely and accurately perform the automatic recalibration.
Exercise caution, especially during the first recalibration, to avoid potential collisions.

1. Adjust Speed
   Set the Robot's global speed to 20% to ensure a safe recalibration process.
2. Start Automatic Recalibration
   Click Recalibrate to initiate the automatic calibration process.
   The Robot will automatically move to the previously saved poses and capture new images for calibration.
3. Review Results
   Press the reload button to view the calibration results.

Solution Setup

1. Open RobVision
2. Launch RobVision and go to the Solution tab.
3. Create New Solution
   Click on the Add New Solution button to add a new solution.
   Enter a name for the solution.
4. Select Camera
   Choose the camera to use.
5. Select Object
   Choose the object to be grasped.
6. Region of Interest

Optionally, you can limit the Field of View by defining one or more Regions of Interest (ROIs). Every grasping pose outside all ROIs will be rejected. To set an ROI, click Add Region of Interest. A dialog will open, allowing you to specify the ROI.

The region of interest is defined as a cuboid in the camera's coordinate system (RGB frame, as shown below). The cuboid is specified using the following parameters. To help set the ROI, a 3D Rendering of the last camera capture will be shown and the ROI can be set in that visualization.

• Reference Position: The position of one corner of the cuboid.
• Vector to the Opposite Corner: The offset from the reference position to the opposite corner.

7. Grasping Pose

Every object is going to be gripped at the predicted keypoint. To move this grasping position, click edit grasping pose. A 3D editor will open that lets you move the grasping position around the 3D model of the object. To get to a more exact position, exact numbers can also be put in. Since the Grasping Pose Selector needs a 3D model of the object, only CAD picking projects are supported. For different pick poses for label picking projects, use the detection only mode and adjust the picking pose with Variable Nodes.

In this example the cube is the object to be gripped, the round object symbolizes the gripper.

Advanced Settings

Pose Filters

1. Max Tilt:

The tilt angle is the difference between the orientation of the camera's z-axis at the moment of image capture and the orientation of the z-axis of the calculated grasping pose. If this angle exceeds the defined maximum tilt, the system will reject the proposed grasping pose.

Imagine the camera is mounted above a table, and the Robot picks from that table. Limiting the tilt angle is essential to avoid Robot arm collisions. A minimal tilt (left image) shows the Robot picking straight down, aligned with the camera's z-axis. A small tilt (middle image) could be fine, but a large tilt (right image) risks a collision.

2. Distance Filter

With the distance filter activated, detected objects are not going to be picked, if another object is within the specified distance to the grasping position of the object.

Pose Estimator Settings

1. Grasping Pose Override

The Grasping Pose Override feature allows you to predefine the desired orientation for the grasping pose. This can significantly speed up your Robot program, as frequent end effector orientation changes can slow down overall motion. To use this feature, first position the Robot's end effector in the correct orientation. Then, press Apply Current Pose to save this configuration.

2. Picking Priority

Here you can choose in which order the objects are going to be picked if more than one object gets detected. Distance to Center prioritizes objects that are closer to the center of the image. Confidence prioritizes objects with higher confidence values from the detection model.

Axis Priority is a powerful option that lets you define the sorting order based on object positions along specific axes — for example, if objects are aligned along a conveyor belt or arranged in a row. Objects are sorted according to their position along the selected axes, following the defined priority order.

For each axis, the algorithm compares object positions (in either the camera or Robot frame). If the difference between two objects is larger than the specified tolerance, sorting is determined by the selected direction:

• Ascending: objects with smaller coordinate values come first.
• Descending: objects with larger coordinate values come first.

If two objects are within the tolerance on one axis, the next axis in the priority list is evaluated. If all axes are within tolerance, the objects are considered equal in priority.

This allows for fine-grained control over object picking behavior, ideal for structured setups where spatial order matters.

In this example, sorting starts with the Z axis (first priority). All objects are at the same height, so the algorithm moves to the X axis. Objects 1, 2, and 3 have nearly the same X position — within the defined tolerance — so they’re considered equal on that axis. The next priority axis is Y, where Object 1 has the lowest Y value, so it comes first.

Small variations due to imperfect parts, size differences, or measurement noise are handled by the tolerance setting, which prevents insignificant deviations from affecting the sorting order.

3. Pick from defined surface

Alternatively, instead of relying on the depth sensor to estimate the grasping pose height, you can define a fixed plane representing the object's surface. When this plane is enabled, the system no longer estimates height from the depth sensor. This means stacked objects will not be recognized, as the system will only consider objects lying on the defined plane.
To define this plane, you will teach the system three positions. These positions should represent the plane where you want the Robot to grasp the objects (i.e., the top surface). One method is to place three objects at the corners of the desired plane and move the Robot's TCP (Tool Center Point) to each object's position. Ensure the height of these positions is the desired grasping height for the objects..

4. Pixel radius depth estimation

This value sets how many pixels around the grasping point are going to be considered for the depth estimation. More pixels usually lead to a better average, especially in noisy depth data. The object geometry and size needs to be considered here.

5. Pose estimation refinement

You can activate Pose estimation refinement to do a point cloud matching based refinement (ICP) after the initial pose estimation. This is only relevant for objects with CAD data and might lead to better results, but In most cases the initial pose estimation is sufficient.

Model Config

1. Confidence Threshold

This sets the confidence threshold for the detection model. With a higher value, the model needs to be more confident to detect an object, leading to more objects with higher uncertainty being filtered.

Length Measurement

1. Length Measurement

With activated length measurement every object that is detected is going to be measured along its main two axes, leading to a width and height value. Those values can be accessed through a Variable in the Flow.