Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (2% of outliers version#07)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1630 objects, 32 outliers (1.96%)

Download raw algorithm results (10.2 MB) Download raw algorithm evaluation table (56.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.40909 0.39726 0.42028 0.40867 0.48000 0.46959 0.86789
KNN 2 0.40937 0.39755 0.38320 0.37085 0.48148 0.47110 0.85074
KNN 3 0.40625 0.39436 0.37824 0.36578 0.50000 0.48999 0.83188
KNNW 2 0.46875 0.45811 0.42854 0.41710 0.47458 0.46405 0.87635
KNNW 4 0.43750 0.42624 0.45125 0.44026 0.50980 0.49999 0.86154
LOF 3 0.40625 0.39436 0.33775 0.32449 0.41270 0.40094 0.85628
LOF 6 0.28125 0.26686 0.32793 0.31447 0.37209 0.35952 0.89579
LOF 13 0.37500 0.36248 0.35817 0.34531 0.39286 0.38070 0.82267
SimplifiedLOF 4 0.37500 0.36248 0.33096 0.31756 0.37500 0.36248 0.85124
SimplifiedLOF 6 0.28125 0.26686 0.33986 0.32664 0.34783 0.33477 0.89366
SimplifiedLOF 13 0.37500 0.36248 0.39725 0.38518 0.42857 0.41713 0.85906
LoOP 10 0.40625 0.39436 0.42948 0.41806 0.50704 0.49717 0.90667
LoOP 12 0.50000 0.48999 0.45252 0.44156 0.50794 0.49808 0.89862
LoOP 17 0.53125 0.52186 0.44506 0.43395 0.54237 0.53321 0.87758
LoOP 19 0.53125 0.52186 0.42808 0.41662 0.54839 0.53934 0.87140
LDOF 9 0.34375 0.33061 0.29998 0.28597 0.34375 0.33061 0.86270
LDOF 28 0.43750 0.42624 0.38039 0.36798 0.44444 0.43332 0.82754
ODIN 7 0.16667 0.14998 0.11152 0.09372 0.23077 0.21537 0.84198
ODIN 93 0.35938 0.34655 0.25091 0.23591 0.39583 0.38373 0.79100
ODIN 96 0.36719 0.35452 0.26100 0.24620 0.39583 0.38373 0.79089
ODIN 100 0.36719 0.35452 0.26115 0.24635 0.39583 0.38373 0.79137
FastABOD 23 0.43750 0.42624 0.36636 0.35367 0.48649 0.47620 0.85077
FastABOD 24 0.43750 0.42624 0.36874 0.35610 0.48649 0.47620 0.84872
FastABOD 94 0.50000 0.48999 0.34603 0.33293 0.50000 0.48999 0.82658
FastABOD 95 0.50000 0.48999 0.34144 0.32826 0.50794 0.49808 0.82124
KDEOS 6 0.18750 0.17123 0.10120 0.08321 0.20588 0.18998 0.76993
KDEOS 10 0.09375 0.07560 0.06593 0.04723 0.12969 0.11226 0.77468
KDEOS 73 0.15625 0.13935 0.08036 0.06194 0.21212 0.19634 0.71871
LDF 4 0.04489 0.02576 0.04450 0.02536 0.09355 0.07540 0.77922
LDF 100 0.02083 0.00123 0.03545 0.01614 0.10753 0.08966 0.63631
INFLO 9 0.34375 0.33061 0.36794 0.35529 0.36735 0.35468 0.88409
INFLO 13 0.40625 0.39436 0.39928 0.38725 0.42857 0.41713 0.84479
INFLO 51 0.34375 0.33061 0.35958 0.34676 0.43137 0.41999 0.77124
COF 4 0.18750 0.17123 0.10049 0.08248 0.20408 0.18814 0.69423

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2867 objects, 57 outliers (1.99%)

Download raw algorithm results (12.0 MB) Download raw algorithm evaluation table (68.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.48131 0.47079 0.47399 0.46332 0.51765 0.50786 0.88637
KNN 2 0.46930 0.45853 0.47797 0.46738 0.50575 0.49572 0.89252
KNN 3 0.46093 0.44999 0.47819 0.46760 0.54321 0.53394 0.88218
KNN 8 0.45614 0.44511 0.45283 0.44173 0.58140 0.57290 0.84068
KNNW 6 0.43860 0.42721 0.46941 0.45865 0.51765 0.50786 0.87639
KNNW 8 0.47368 0.46301 0.47659 0.46597 0.51765 0.50786 0.87368
KNNW 12 0.47368 0.46301 0.49356 0.48328 0.56818 0.55942 0.85956
KNNW 17 0.47368 0.46301 0.48436 0.47390 0.58427 0.57584 0.83977
LOF 7 0.05057 0.03131 0.04931 0.03003 0.11549 0.09754 0.76060
LOF 9 0.04970 0.03042 0.05082 0.03157 0.10966 0.09160 0.80092
SimplifiedLOF 8 0.05065 0.03139 0.04661 0.02727 0.09358 0.07520 0.75254
SimplifiedLOF 9 0.04853 0.02923 0.04681 0.02748 0.09722 0.07891 0.78094
LoOP 2 0.22807 0.21241 0.15643 0.13932 0.23214 0.21657 0.73178
LoOP 14 0.08739 0.06888 0.12437 0.10661 0.18539 0.16887 0.85798
LDOF 73 0.12281 0.10501 0.06936 0.05048 0.14500 0.12766 0.78748
LDOF 77 0.12281 0.10501 0.07486 0.05610 0.16111 0.14409 0.79705
LDOF 93 0.12281 0.10501 0.06936 0.05048 0.13203 0.11442 0.80332
ODIN 86 0.32491 0.31122 0.24994 0.23472 0.41509 0.40323 0.89281
ODIN 89 0.32749 0.31384 0.24726 0.23199 0.42038 0.40862 0.89319
ODIN 91 0.32749 0.31384 0.24745 0.23219 0.42038 0.40862 0.89356
ODIN 98 0.32749 0.31384 0.24805 0.23279 0.42308 0.41137 0.89310
FastABOD 28 0.02190 0.00206 0.05517 0.03600 0.14675 0.12944 0.77781
FastABOD 74 0.00901 -0.01109 0.06003 0.04096 0.15172 0.13452 0.77253
FastABOD 86 0.00901 -0.01109 0.05989 0.04082 0.16038 0.14335 0.76876
KDEOS 10 0.01695 -0.00299 0.04087 0.02141 0.09864 0.08036 0.75247
KDEOS 73 0.01754 -0.00238 0.03143 0.01178 0.07447 0.05569 0.65127
LDF 2 0.03876 0.01926 0.02882 0.00912 0.11715 0.09925 0.50063
LDF 8 0.04969 0.03041 0.03262 0.01300 0.08537 0.06681 0.64167
INFLO 7 0.05065 0.03140 0.04822 0.02891 0.11141 0.09339 0.77008
COF 13 0.00000 -0.02028 0.05641 0.03727 0.13035 0.11271 0.78127
COF 77 0.08772 0.06921 0.06036 0.04130 0.19251 0.17613 0.71561
COF 84 0.12281 0.10501 0.05976 0.04069 0.17143 0.15462 0.72967

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO