Data-Centric Land Cover Classification Challenge

In supervised semantic segmentation, obtaining fully accurate and reliable labels is often challenging and costly. Labels are frequently contaminated by noise due to factors such as limited information, human labeling errors, or encoding mistakes. In the real-world, label noise is ubiquitous and can significantly degrade the performance of trained machine learning models. Therefore, it is essential to develop methods capable of automatically identifying noisy samples prior to training, allowing machine learning models to be optimized based on the cleanest and most reliable data available.

Challenge Summary

The data-centric land cover classification challenge, as part of the Workshop on Machine Vision for Earth Observation and Environment Monitoring and the British Machine Vision Conference (BMVC) 2024, aims at novel data-centric approaches towards identifying noisy labels for a semantic segmentation task.

Participants of this challenge are asked to develop an AI-based system that ranks the samples based on their levels of label noise. Success is measured by comparing the submitted ranking with an undisclosed official one and calculating the Kendall Tau score.

Please, check for more info on Kaggle.

Dataset

The challenge will use this dataset, which is composed of 5,000 256x256 images and their corresponding (noisy) labels. The labels consist of two classes: background (encoded as 0 in the label data) and building (encoded as 1 in the label data).

More info on Kaggle.

Submission

After generating the ranking (from least to most noisy), you must create a file with a header and the following format:

id,imageid
0,1_26_72_418_418.png
1,2_19_53_0_627.png
2,2_22_42_627_0.png
etc.

The id is NOT relevant and is NOT used in the evaluation process. The imageid column should have the name of labels/images (for example, 1_26_72_418_418.png) in order from least to most noisy.

After creating this file, please submit it on Kaggle.

Timeline

Participants may submit multiple solutions. All submitted solutions will be listed on the leaderboard.

Results, Presentation, Awards, and Prizes

The final results of this challenge will be presented during the Workshop. The authors of the top-ranked methods will be invited to present their approaches at the Workshop in Glasgow/UK, on 28 November 2024. These authors will also be invited to co-author a journal paper which will summarize the outcome of this challenge and will be submitted with open access to IEEE JSTARS.

Organizers

Keiller Nogueira, University of Liverpool, UK
Ronny Hänsch, German Aerospace Center (DLR), Germany
June Moh Goo, University College London (UCL), UK
Zichao Zeng, University College London (UCL), UK
Pallavi Jain, Inria, France
Zhipeng Liu, University of Exeter, UK