Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (10% of outliers version#05)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1775 objects, 177 outliers (9.97%)

Download raw algorithm results (13.1 MB) Download raw algorithm evaluation table (73.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.51635 0.46278 0.54359 0.49303 0.52987 0.47780 0.82143
KNN 6 0.53296 0.48123 0.55416 0.50478 0.54603 0.49575 0.80875
KNN 8 0.52004 0.46687 0.55454 0.50520 0.55147 0.50179 0.79623
KNN 10 0.50835 0.45390 0.54408 0.49358 0.55224 0.50264 0.78041
KNNW 9 0.51412 0.46031 0.54727 0.49713 0.52326 0.47045 0.82455
KNNW 11 0.54237 0.49168 0.56223 0.51374 0.54237 0.49168 0.82177
KNNW 19 0.51977 0.46658 0.57516 0.52810 0.55000 0.50016 0.80090
KNNW 39 0.52542 0.47286 0.56414 0.51586 0.56522 0.51706 0.76925
LOF 68 0.53107 0.47913 0.56518 0.51702 0.55019 0.50036 0.80067
LOF 99 0.56497 0.51679 0.59522 0.55039 0.60351 0.55959 0.79123
LOF 100 0.56497 0.51679 0.59524 0.55041 0.60417 0.56032 0.79054
SimplifiedLOF 68 0.52542 0.47286 0.56589 0.51780 0.53595 0.48455 0.81318
SimplifiedLOF 99 0.57627 0.52934 0.59690 0.55225 0.59756 0.55299 0.80276
SimplifiedLOF 100 0.57627 0.52934 0.59825 0.55375 0.59939 0.55502 0.80232
LoOP 33 0.48588 0.42893 0.39992 0.33345 0.48864 0.43200 0.72679
LoOP 41 0.45763 0.39755 0.41214 0.34703 0.48930 0.43273 0.74688
LoOP 100 0.44068 0.37873 0.48363 0.42643 0.46735 0.40836 0.79987
LDOF 32 0.45763 0.39755 0.35539 0.28399 0.45763 0.39755 0.72556
LDOF 54 0.44633 0.38500 0.42733 0.36390 0.47134 0.41278 0.76103
LDOF 100 0.44068 0.37873 0.47751 0.41964 0.46106 0.40136 0.79934
ODIN 36 0.29379 0.21556 0.20463 0.11653 0.33266 0.25874 0.66643
ODIN 38 0.29034 0.21174 0.20315 0.11489 0.33538 0.26176 0.66641
ODIN 100 0.27307 0.19255 0.20824 0.12054 0.32634 0.25172 0.68076
FastABOD 15 0.46893 0.41010 0.43236 0.36948 0.50505 0.45023 0.80448
FastABOD 18 0.48023 0.42265 0.44463 0.38311 0.51713 0.46365 0.79901
FastABOD 22 0.49718 0.44148 0.43756 0.37526 0.50602 0.45131 0.80007
KDEOS 66 0.24859 0.16536 0.17414 0.08267 0.26425 0.18275 0.64423
LDF 90 0.40678 0.34107 0.22628 0.14058 0.43478 0.37218 0.71164
LDF 99 0.40113 0.33480 0.23381 0.14895 0.45000 0.38908 0.71726
LDF 100 0.40678 0.34107 0.23731 0.15283 0.45000 0.38908 0.71806
INFLO 69 0.46328 0.40383 0.52327 0.47046 0.49808 0.44249 0.80880
INFLO 100 0.53107 0.47913 0.56853 0.52074 0.55873 0.50985 0.80739
COF 5 0.19209 0.10260 0.15808 0.06483 0.23343 0.14852 0.60999
COF 7 0.20904 0.12143 0.16002 0.06699 0.26070 0.17881 0.59171
COF 52 0.22034 0.13398 0.15224 0.05833 0.22472 0.13885 0.52951
COF 81 0.20339 0.11515 0.16960 0.07762 0.21605 0.12922 0.53613

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 3122 objects, 312 outliers (9.99%)

Download raw algorithm results (13.7 MB) Download raw algorithm evaluation table (74.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.47079 0.41203 0.52366 0.47077 0.48060 0.42293 0.86037
KNN 8 0.49231 0.43594 0.54021 0.48916 0.49240 0.43604 0.85829
KNN 10 0.46980 0.41093 0.54792 0.49772 0.51190 0.45771 0.83872
KNNW 13 0.45192 0.39107 0.49980 0.44426 0.47717 0.41912 0.85878
KNNW 23 0.46154 0.40175 0.54409 0.49347 0.48584 0.42875 0.84220
KNNW 24 0.46795 0.40887 0.54378 0.49312 0.48869 0.43192 0.84017
KNNW 63 0.48718 0.43024 0.51446 0.46055 0.48718 0.43024 0.78715
LOF 9 0.13870 0.04306 0.14912 0.05465 0.28334 0.20377 0.66554
LOF 10 0.13113 0.03465 0.14461 0.04964 0.28371 0.20417 0.66261
SimplifiedLOF 9 0.13569 0.03972 0.13178 0.03538 0.24233 0.15820 0.61411
SimplifiedLOF 10 0.13113 0.03465 0.13281 0.03652 0.24512 0.16130 0.62404
LoOP 76 0.24359 0.15960 0.17812 0.08686 0.29670 0.21861 0.68891
LoOP 77 0.24038 0.15604 0.17820 0.08695 0.29937 0.22158 0.68961
LoOP 80 0.23077 0.14536 0.17595 0.08445 0.30496 0.22779 0.68902
LoOP 83 0.23397 0.14892 0.17778 0.08649 0.30090 0.22328 0.69157
LDOF 75 0.22756 0.14180 0.16558 0.07293 0.26606 0.18456 0.66284
LDOF 100 0.21474 0.12755 0.17208 0.08016 0.27985 0.19989 0.67914
ODIN 16 0.22628 0.14037 0.20610 0.11795 0.35039 0.27827 0.73162
ODIN 37 0.37747 0.30834 0.24988 0.16660 0.40169 0.33525 0.72449
ODIN 79 0.39204 0.32454 0.25592 0.17331 0.39735 0.33044 0.72147
ODIN 80 0.39204 0.32454 0.25674 0.17422 0.39735 0.33044 0.72155
FastABOD 28 0.16987 0.07770 0.18260 0.09185 0.32929 0.25482 0.73758
FastABOD 87 0.14744 0.05277 0.18641 0.09608 0.35174 0.27976 0.74201
FastABOD 100 0.14744 0.05277 0.19031 0.10041 0.34762 0.27519 0.74596
KDEOS 4 0.08469 -0.01694 0.10643 0.00722 0.24959 0.16628 0.54998
KDEOS 9 0.12500 0.02785 0.12265 0.02524 0.23235 0.14711 0.59214
KDEOS 10 0.11538 0.01716 0.12562 0.02854 0.23467 0.14969 0.60386
KDEOS 11 0.11218 0.01360 0.12476 0.02758 0.23225 0.14701 0.60701
LDF 2 0.22051 0.13396 0.12702 0.03009 0.24212 0.15797 0.45245
LDF 29 0.10256 0.00292 0.10534 0.00601 0.18616 0.09579 0.53014
INFLO 9 0.13870 0.04306 0.13831 0.04263 0.26460 0.18295 0.64448
INFLO 10 0.13113 0.03465 0.13605 0.04013 0.26782 0.18652 0.64533
COF 80 0.24359 0.15960 0.15846 0.06502 0.26139 0.17938 0.64933
COF 83 0.24359 0.15960 0.16466 0.07191 0.26846 0.18724 0.66935
COF 84 0.24038 0.15604 0.16501 0.07230 0.27201 0.19118 0.66638
COF 87 0.23077 0.14536 0.16119 0.06806 0.27846 0.19835 0.65980

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO