Giving every point a name
This is my bachelor’s thesis in Computer Science. I did it at IRALAB (opens in a new tab), the Informatics and Robotics for Automation laboratory of the University of Milano-Bicocca (opens in a new tab), between March and July 2024. Its real title, in Italian, means “Multi-instance semantic segmentation of point clouds”.
My supervisor was Domenico Giorgio Sorrenti and my co-supervisor Matteo Vaghi. My thanks to both.
The problem
A self-driving car has to understand what is around it, and it uses different sensors to do so. The camera sees what is in the scene: a neural network can tell, pixel by pixel, where the road, the cars and the pedestrians are. But an image is flat. The LiDAR does the opposite: it measures precisely where everything is in space and returns a three-dimensional cloud of points, in which, however, a point is just a point, with no meaning.
The thesis brings the two together. Segmentation is run on the camera images and then back-projected onto the point cloud, so that every LiDAR point knows what it is. The result is meant to feed into a larger SLAM pipeline, that is, a system with which the vehicle builds a map of an environment it doesn’t know while working out where it is.


“Multi-instance” because the segmentation I use is panoptic. Semantic segmentation says that certain pixels are “car”; panoptic segmentation also says which car, that is, it keeps two cars parked side by side apart.
The pipeline
I designed the pipeline and wrote it in Python. For every LiDAR scan it goes through these steps.
- It loads the LiDAR scan and the camera image taken at the same instant.
- It projects the point cloud onto the image plane. That takes two changes of reference frame, from the LiDAR to the camera and from the camera to the image: it does them with the calibration matrices of the two sensors, in a single product in homogeneous coordinates. The LiDAR sees all 360 degrees and the camera doesn’t, so only the points that fall inside the frame remain.
- It runs EfficientPS on the image, a convolutional network for panoptic segmentation, in a version pre-trained on Cityscapes.
- It gives every projected point the label of the pixel it lands on.
- It removes the dynamic objects, such as cars and pedestrians. On a map, things that move are noise, and SLAM works better without them.
- It goes back to three dimensions. There is actually nothing to compute: it keeps the original coordinates of every projected point, so the label goes straight onto the 3D point. The labelled cloud is saved in binary format, to take up less space.



How far you can trust it
The quality of the result depends entirely on the network that segments the images: if it gets a pixel wrong, the point that lands on it is wrong too. Training a new one needed more computing power than we had in the lab, so I used the pre-trained model released by the authors of EfficientPS and went and measured how reliable it was, counting the correctly classified pixels on three datasets of very different cities.
| Dataset | Images | Accuracy |
|---|---|---|
| Mapillary Vistas | 20,000 | 0.79 |
| Indian Driving Dataset | 7,884 | 0.79 |
| KITTI Panoptic Segmentation | 1,055 | 0.74 |
A little under 80%, more or less everywhere. But the number alone is a poor account of how the model does. Every dataset carves the world into classes in its own way, and when you convert another dataset’s labels into the ones the model knows, whatever finds no matching class ends up as “unlabeled”. The model assigns a class anyway, though, and each time it is counted as an error.


The real error lies elsewhere: the model tends to make the objects it recognises a little too large, and gets the borders between one area and the next wrong. And the metric itself has a limit, because it counts pixels: a car in the foreground weighs far more than a distant pedestrian, even though the pedestrian matters more to whoever is driving.
The engineering
- Everything runs in a Docker environment: Docker and an NVIDIA GPU are all you need to repeat the experiments on another machine.
- The code is a Python library, which I named IRAfunesta. It contains the pipeline and two functions that act as an interface to EfficientPS: one loads the model, the other runs inference on an image.
- I worked with six autonomous-driving datasets (KITTI, KITTI Panoptic Segmentation, KITTI-360, Cityscapes, Mapillary Vistas and the Indian Driving Dataset) and wrote the mappings that convert labels from one format to another.
- Along with the code I wrote the documentation: how to install the environment, how to prepare the dataset, how to use the library.
For anyone looking for keywords: Python, PyTorch, NumPy, OpenCV, Pillow, Matplotlib, Docker, NVIDIA CUDA, Ubuntu, Git.
What’s missing
It’s a starting point, and the thesis says so. For now I give the points only their class, because what SLAM needs is to know which objects to remove; the network already tells the individual instances apart, though, and adding them to the cloud is a short step. The biggest improvement would come from a model trained on a purpose-built dataset, and from adding other sensors, such as radar.
The driving scenes on this page come from two public research datasets: KITTI-360 (opens in a new tab), released under a CC BY-NC-SA 3.0 licence, and Mapillary Vistas (opens in a new tab), in its research edition.