SpaceNet: Accelerating geospatial machine learning Building Foot Print Extraction: Round 2 | SpaceNet on AWS
Building Foot Print Extraction Road Extraction and Routing

SpaceNet Round 2 Challenge Implementations

The Problem

The commercialization of the geospatial industry has led to an explosive amount of data being collected to characterize our changing planet. One area for innovation is the application of computer vision and deep learning to extract information from satellite imagery at scale. CosmiQ Works, DigitalGlobe and NVIDIA have partnered to release the SpaceNet data set to the public to enable developers and data scientists.

Today, map features such as roads, building footprints, and points of interest are primarily created through manual techniques. We believe that advancing automated feature extraction techniques will serve important downstream uses of map data including humanitarian and disaster response, as recently observed by the need to map buildings in Haiti during the response to Hurricane Matthew. Furthermore, we think that solving this challenge is an important stepping stone to unleashing the power of advanced computer vision algorithms applied to a variety of remote sensing data applications in both the public and private sector. For more information see the SpaceNet Challenge Round 2 Competition Page

Prize List

Prize USD
1st $6,000
2nd $3,000
3rd $1,500
Best F-score, Las Vegas $1,000
Best F-score, Paris $1,000
Best F-score, Shanghai $1,000
Best F-score, Khartoum $1,000
Early Incentive $1000
Total Prizes $15,500

The Metric

In SpaceNet Challenge, the metric for ranking entries is based on the Jaccard Index, also called the Intersection-over-Union (IoU). For more information read the full article on The DownlinQ.

and see below

Evaluation Metric

The evaluation metric for this competition is an F1 score with the matching algorithm inspired by Algorithm 2 in the ILSVRC paper applied to the detection of building footprints. For each building there is a geospatially defined polygon label to represent the footprint of the building. A SpaceNet entry will generate polygons to represent proposed building footprints. Each proposed building footprint is either a “true positive” or a “false positive”.

There is at most one “true positive” per labeled polygon. The measure of proximity between labeled polygons and proposed polygons is the Jaccard similarity or the “Intersection over Union (IoU)”, defined as:

alt text

The value of IoU is between 0 and 1, where closer polygons have higher IoU values.

The F1 score is the harmonic mean of precision and recall, combining the accuracy in the precision measure and the completeness in the recall measure. For this competition, the number of true positives and false positives are aggregated over all of the test imagery and the F1 score is computed from the aggregated counts.

For example, suppose there are N polygon labels for building footprints that are considered ground truth and suppose there are M proposed polygons by an entry in the SpaceNet competition. Let tp denote the number of true positives of the M proposed polygons. The F1 score is calculated as follows:

alt text

The F1 score is between 0 and 1, where larger numbers are better scores.


Winning solutions:

The top three solutions have been open sourced at the SpaceNet Challenge Github Repository

Competitor Final Score
1. XD_XD 0.687740
2. wleite 0.642982
3. nofto 0.579014

Click Here For the full leaderboard and submission history

The Data

AOI Area of Raster (Sq. Km) Building Labels (Polygons)
AOI_2_Vegas 216 151,367
AOI_3_Paris 1,030 23,816
AOI_4_Shanghai 1,000 92,015
AOI_5_Khartoum 765 710,960

For Round 2 results please see the Round 1 Site