We have been struggling with the task of detecting garbage in the environment for almost 5 months. In our previous post we described some metrics to compare performance achieved by different models. Now is the time to present our first results.

Let’s get back to the data

In the Detect Waste in Pomerania project we intended to base our research on TACO dataset - an open image dataset of waste in the environment. As we wrote in one of our first posts, we converted TACO categories to detect waste categories according to the obligatory segregation rules of the city of Gdańsk. Additionally, around 3k images were tagged by our partner Epinote with bounding boxes. In this way, we started with waste detection performed for 6 classes - paper, glass, metal and plastic, bio, non-recyclable, and other - on an extended TACO dataset (almost 4.5k images).

Unfortunately, there are types of waste that do not fall into any of above-mentioned categories - very crowded photos with garbage obscured by various objects, where it was not possible to tag individual garbage, and trash that could not be clearly identified due to their state of degradation. We have created another category for it - “unknown”.

Detecting only metal and plastic in 7-class detection challenge

We conducted our experiments on an EfficientDet network, written in PyTorch. EfficientDet is a family of object detection models that came out of the Google Brain team. It is based on the EfficientNet paper, which achieves one of the best performances on the image classification task. In our case EffcientDet-D2 gave us the best possible results. Smaller architectures got lower scores, whereas bigger ones quickly overfitted. We hypothesize this is because the dataset itself is small (roughly 4.5k images) and highly imbalanced.

The best result was recorded for the “metal and plastic” class. It is not surprising at all, considering the fact that a given class makes up almost 45% of our imbalanced dataset. Generalizing, our model treated almost every object it found as a plastic one. The worst results were achieved for the smallest class - “bio” with only 69 instances (0.5% of the dataset). The “other” class was also not very successful due to the large variety of other item’s kinds (from rubble to medicines).

Sometimes less is more: 1-class detection task

Detection combines two tasks: localization and classification of the objects in the image. We have noticed that our model performs quite well when localizing waste but works poorly with classification. Moreover, we have massive data from OpenLitterMap, with thousands of images labeled for the multi-label classification task (few categories in one image). We have decided to try single class detection instead, and separate classification. This allowed us to obtain an Average Precision (AP) on one litter class above 50%, which is a much better result than the one reported in the article describing the detection on one-class TACO (16%).

Future works

Good results, obtained for detection in a single class, prompted us to dive into these studies also on other datasets - representing the detection or segmentation of waste in various environments. This is the first post in a series describing our results for garbage detection on public datasets. Look out for more soon.