This section shows the practices employed to process the point cloud received from the camera topic /camera/depth/points. Even though the inner workings of the specific functions used are outside the scope of this project, their influence is highlighted here to show their importance.
For completeness, a point cloud is included below to give an impression of what the input of the pipeline looks like as received from the point cloud topic.
Source
from IPython.display import HTML
HTML("""
<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>
<model-viewer
src="https://matt-rbt.github.io/Lab-Cleanup-Robot-using-the-Mirte-Master-Platform/models/pointcloud-compressed.glb"
camera-controls
auto-rotate
style="width: 640px; height: 640px; background: #1c1917;">
</model-viewer>
""")
# IMPORTANT:
# The model path is an absolute GitHub Pages URL on purpose.
# A relative path like "./models/floor_with_cubes.glb" works locally in VS Code,
# but fails on the deployed MyST site because the page is served from /coverageplanners/.
# Do not change this to a relative path unless you also test the deployed site.To do this, a sample point cloud has been saved from the Gazebo simulation and is used as the input to a filtering pipeline built with the Open3D Python library:
steps:
Crop the point cloud to include only points below a given height.
Copy a downsampled version of the point cloud to speed up future calculations.
Iteratively detect planes and remove the corresponding points from both point clouds. From there, the downsampled point cloud will not be used anymore.
Detect clusters from the remaining points.
Create bounding boxes for these clusters.
Reject boxes that overlap with the costmap.
Any remaining bounding boxes are published as detected objects.
The entire pipeline, including the individual effect of each step, is shown below:
First, the original point cloud is created from the camera’s depth image, as shown in Figure 1.
Figure 1:The point cloud as received from the camera topic including the eventual bounding boxes
Next, this point cloud is downsampled. Note that the resolution afterwards is still quite high. This is because the objects are very small, and reducing the resolution dramatically would lose details of these objects, reducing bounding-box accuracy. An example is shown below in Figure 2.
Figure 2:The downsampled point cloud
In the downsampled point cloud, planes are detected using RANSAC plane segmentation Fischler & Bolles, 1981. With this, large planes such as walls and tabletops are discarded from the point cloud, as can be seen in Figure 3.
Figure 3:The plane-segmented point cloud
Now, the remaining points are cross-referenced with the costmap obtained during mapping. Any point that lies on the costmap is excluded from the point cloud, resulting in the point cloud depicted in Figure 4.
Figure 4:The point cloud containing only points belonging to free objects
Finally, the remaining points are converted to clusters using DBSCAN Ester et al., 1996. This enables the construction of a bounding box for each cluster, each of which represents an object. Bounding boxes, however faint, can be made out in Figure 5 below.
Figure 5:Bounding boxes (in front of the robot) are drawn around the remaining clusters.
Information about these bounding boxes is then published on a ROS 2 topic, which is later used to approach and pick up the corresponding objects.
- Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6), 381–395. 10.1145/358669.358692
- Ester, M., Kriegel, H.-P., Sander, J., & Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, 226–231.