2.5.1 Perception 2D

For determining the type, and possibly the distance of objects, a YOLO26 (‘You Only Look Once-26’) object-detection model was used. A dataset was created by recording videos using the USB camera included in the MIRTE Master hardware package, from which one in every fifteen frames was extracted for the training dataset. A figure of four frames in different environments is shown below.

Why YOLO26 Was Used¶

The YOLO object-detection family is well established, with YOLO26 Ultralytics, 2026 being the most recent generation. YOLO26 was chosen because the robot has to detect and classify objects from a live camera on the MIRTE Master platform, which has limited processing power because it uses an Orange Pi 3B. Object detection with limited processing power requires a lightweight model with fast inference times while still maintaining accuracy. YOLO26 is suitable for this because it was designed with deployment on low-power devices in mind. It uses NMS-free inference in its post-processing pipeline, simplifies bounding-box predictions, and supports deployment formats commonly used for low-power devices Sapkota et al., 2026. In this case, the RK3566 NPU of the Orange Pi 3B can be used to improve performance by reducing the total inference time Limaran et al., 2026. For this model, the Nano variant of YOLO26 was chosen to minimize inference times. Another benefit of YOLO26 over the older variants is its support for oriented bounding boxes. Unlike standard bounding boxes, oriented bounding boxes also estimate the rotation of recognized objects Sapkota & Karkee, 2026. This could be useful for complex grasping tasks where the orientation of the gripper is crucial. However, this project only uses standard object detection.

Training¶

The training data was annotated manually using Label Studio. Initially, roughly 280 frames were captured in four different environments. Due to a large amount of blurring within the images, about 100 pictures were deleted. Of the remaining 180 pictures, 60 were imported into Label Studio and labelled per colour: green, red, purple, white, light-grey, dark-grey and black. In total, 1206 bounding boxes were drawn. A test of the perception model performed during training is shown in the figure below.

The curves in the figure below show that the model converged during training. This means that the model actually learns while training instead of making random or unstable decisions.

The normalized confusion matrix shows how well classes are recognized by the model. As shown in the figure below, most classes are classified correctly, as indicated by the strong diagonal values. However, light-grey objects were among the least correctly identified objects, because changes in exposure and lighting often caused the model to classify them as background. Some objects were missed (classified as background), a small number of objects were misclassified, and some objects, such as dark-grey and white ones, were confused with light-grey.

A figure of the F1 confidence curve is shown below. This curve shows how the F1 metric, which combines precision (how many objects were correctly predicted) and recall (how many objects were correctly found), changes with the confidence threshold of the model. The curve shows that a good starting confidence threshold would be 0.3–0.35.