Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (2% of outliers version#02)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1630 objects, 32 outliers (1.96%)

Download raw algorithm results (10.0 MB) Download raw algorithm evaluation table (57.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.32500 0.31148 0.27322 0.25867 0.38462 0.37229 0.80422
KNN 6 0.34375 0.33061 0.24990 0.23488 0.39286 0.38070 0.72091
KNN 8 0.34375 0.33061 0.24011 0.22489 0.41509 0.40338 0.69564
KNNW 1 0.37216 0.35959 0.33998 0.32676 0.41667 0.40499 0.81930
KNNW 7 0.31250 0.29873 0.30065 0.28665 0.42553 0.41403 0.77262
LOF 4 0.37500 0.36248 0.26911 0.25448 0.38554 0.37324 0.78420
LOF 9 0.25000 0.23498 0.23109 0.21570 0.31008 0.29626 0.83432
LOF 70 0.31250 0.29873 0.24376 0.22862 0.40000 0.38798 0.71471
SimplifiedLOF 5 0.34375 0.33061 0.24704 0.23196 0.37778 0.36532 0.78389
SimplifiedLOF 7 0.31250 0.29873 0.24179 0.22661 0.40000 0.38798 0.81094
SimplifiedLOF 9 0.31250 0.29873 0.26156 0.24677 0.36364 0.35089 0.84400
SimplifiedLOF 15 0.31250 0.29873 0.30864 0.29479 0.38384 0.37150 0.83419
LoOP 12 0.46875 0.45811 0.35497 0.34206 0.51724 0.50757 0.86192
LoOP 14 0.50000 0.48999 0.36336 0.35061 0.51724 0.50757 0.86159
LoOP 17 0.50000 0.48999 0.37618 0.36369 0.50847 0.49863 0.85931
LDOF 11 0.40625 0.39436 0.26254 0.24777 0.40625 0.39436 0.82597
LDOF 15 0.40625 0.39436 0.29963 0.28560 0.43478 0.42346 0.85959
LDOF 31 0.40625 0.39436 0.28937 0.27514 0.46575 0.45506 0.83714
LDOF 52 0.40625 0.39436 0.30625 0.29236 0.43836 0.42711 0.80145
ODIN 48 0.34615 0.33306 0.20750 0.19163 0.37037 0.35776 0.81372
ODIN 98 0.41903 0.40740 0.26168 0.24689 0.43478 0.42346 0.78462
ODIN 99 0.41903 0.40740 0.26175 0.24696 0.43478 0.42346 0.78481
FastABOD 8 0.04545 0.02634 0.11178 0.09400 0.29565 0.28155 0.80228
FastABOD 17 0.46875 0.45811 0.39087 0.37867 0.51429 0.50456 0.78402
FastABOD 18 0.46875 0.45811 0.39344 0.38129 0.52174 0.51216 0.78391
FastABOD 26 0.46875 0.45811 0.37244 0.35988 0.53521 0.52590 0.79161
KDEOS 11 0.31250 0.29873 0.12825 0.11079 0.31250 0.29873 0.75332
KDEOS 13 0.31250 0.29873 0.14923 0.13220 0.33898 0.32575 0.75246
KDEOS 15 0.31250 0.29873 0.15988 0.14305 0.33846 0.32521 0.79953
KDEOS 16 0.31250 0.29873 0.15895 0.14211 0.31746 0.30379 0.80663
LDF 1 0.04372 0.02457 0.03453 0.01519 0.08802 0.06976 0.62654
LDF 2 0.04301 0.02385 0.03797 0.01870 0.08983 0.07161 0.67067
LDF 6 0.03468 0.01535 0.03524 0.01592 0.08532 0.06701 0.71288
LDF 99 0.00000 -0.02003 0.03110 0.01170 0.11538 0.09767 0.59399
INFLO 6 0.40625 0.39436 0.26114 0.24634 0.41270 0.40094 0.77118
INFLO 15 0.31250 0.29873 0.30577 0.29186 0.38710 0.37482 0.84906
COF 4 0.28125 0.26686 0.12424 0.10670 0.30769 0.29383 0.63936
COF 53 0.18750 0.17123 0.16564 0.14893 0.23810 0.22284 0.55278

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2867 objects, 57 outliers (1.99%)

Download raw algorithm results (12.0 MB) Download raw algorithm evaluation table (67.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.47826 0.46768 0.51006 0.50012 0.56522 0.55640 0.91593
KNN 4 0.57895 0.57041 0.59326 0.58501 0.65306 0.64602 0.90087
KNN 5 0.57895 0.57041 0.59345 0.58521 0.67391 0.66730 0.89566
KNN 7 0.57895 0.57041 0.58528 0.57687 0.69474 0.68854 0.88668
KNNW 3 0.47368 0.46301 0.51212 0.50222 0.53333 0.52387 0.91780
KNNW 6 0.57895 0.57041 0.59817 0.59002 0.66667 0.65991 0.91260
KNNW 8 0.57895 0.57041 0.60571 0.59771 0.68085 0.67438 0.90765
KNNW 10 0.57895 0.57041 0.60264 0.59458 0.68817 0.68185 0.90247
LOF 8 0.05063 0.03138 0.05417 0.03498 0.12796 0.11027 0.82167
LOF 9 0.05113 0.03188 0.05435 0.03517 0.12265 0.10486 0.82353
SimplifiedLOF 9 0.04993 0.03065 0.04981 0.03054 0.10270 0.08449 0.80030
SimplifiedLOF 11 0.04881 0.02952 0.04917 0.02988 0.10410 0.08592 0.79952
LoOP 1 0.13225 0.11465 0.06051 0.04145 0.14433 0.12697 0.60178
LoOP 24 0.05263 0.03341 0.09702 0.07870 0.16087 0.14385 0.86580
LoOP 30 0.05263 0.03341 0.09209 0.07367 0.18378 0.16723 0.86909
LoOP 73 0.08772 0.06921 0.09250 0.07409 0.20392 0.18777 0.85613
LDOF 73 0.12281 0.10501 0.07591 0.05716 0.16828 0.15141 0.82608
LDOF 74 0.12281 0.10501 0.07858 0.05989 0.17508 0.15835 0.82930
LDOF 77 0.10526 0.08711 0.07668 0.05795 0.17712 0.16043 0.82431
ODIN 40 0.29057 0.27618 0.21097 0.19496 0.36025 0.34727 0.89277
ODIN 95 0.37568 0.36302 0.26854 0.25370 0.42647 0.41484 0.88909
ODIN 100 0.37568 0.36302 0.26963 0.25481 0.42647 0.41484 0.88943
FastABOD 29 0.00820 -0.01192 0.06335 0.04435 0.17672 0.16002 0.81127
FastABOD 73 0.00901 -0.01109 0.06798 0.04908 0.17787 0.16120 0.80801
FastABOD 85 0.00901 -0.01109 0.06909 0.05021 0.18655 0.17005 0.80923
FastABOD 93 0.00901 -0.01109 0.07017 0.05131 0.18655 0.17005 0.81012
KDEOS 11 0.00000 -0.02028 0.03995 0.02047 0.09375 0.07537 0.75069
KDEOS 73 0.01754 -0.00238 0.03415 0.01455 0.07937 0.06069 0.69528
LDF 2 0.06061 0.04155 0.02691 0.00717 0.10127 0.08304 0.44413
LDF 9 0.00469 -0.01549 0.02553 0.00576 0.05234 0.03312 0.55778
INFLO 8 0.05071 0.03146 0.05151 0.03227 0.12431 0.10655 0.79862
INFLO 9 0.05120 0.03196 0.05256 0.03334 0.11793 0.10004 0.81034
COF 36 0.03509 0.01551 0.06855 0.04965 0.16268 0.14569 0.77886
COF 74 0.05263 0.03341 0.07790 0.05920 0.19231 0.17592 0.77344
COF 77 0.03509 0.01551 0.07217 0.05335 0.19592 0.17961 0.76104
COF 81 0.07018 0.05131 0.07077 0.05192 0.18182 0.16522 0.77018

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO