Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (10% of outliers version#08)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1775 objects, 177 outliers (9.97%)

Download raw algorithm results (13.2 MB) Download raw algorithm evaluation table (72.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.45032 0.38944 0.48382 0.42665 0.45591 0.39565 0.82369
KNN 5 0.47690 0.41896 0.51582 0.46219 0.48257 0.42526 0.82267
KNN 38 0.44633 0.38500 0.47347 0.41515 0.49612 0.44031 0.71473
KNNW 7 0.45763 0.39755 0.49530 0.43940 0.48338 0.42616 0.83173
KNNW 10 0.49153 0.43521 0.51745 0.46400 0.49337 0.43725 0.82206
KNNW 11 0.49153 0.43521 0.52003 0.46687 0.50146 0.44624 0.81743
KNNW 48 0.44633 0.38500 0.50374 0.44878 0.50382 0.44886 0.74412
LOF 52 0.44633 0.38500 0.48701 0.43019 0.47500 0.41685 0.78978
LOF 91 0.51412 0.46031 0.53535 0.48388 0.54194 0.49120 0.77676
LOF 93 0.51412 0.46031 0.53905 0.48799 0.55873 0.50985 0.77672
LOF 100 0.51412 0.46031 0.54193 0.49119 0.55414 0.50476 0.77097
SimplifiedLOF 52 0.43503 0.37245 0.48645 0.42957 0.45181 0.39109 0.80266
SimplifiedLOF 100 0.50847 0.45403 0.53853 0.48742 0.54915 0.49922 0.78387
LoOP 22 0.46328 0.40383 0.40364 0.33759 0.47229 0.41384 0.73769
LoOP 26 0.44068 0.37873 0.40136 0.33505 0.47678 0.41883 0.72716
LoOP 100 0.36158 0.29087 0.43418 0.37150 0.42029 0.35608 0.78085
LDOF 26 0.41808 0.35362 0.35836 0.28729 0.43243 0.36957 0.73765
LDOF 31 0.42373 0.35990 0.36223 0.29158 0.42735 0.36392 0.72586
LDOF 100 0.36723 0.29714 0.42811 0.36477 0.41877 0.35439 0.77525
ODIN 13 0.27146 0.19077 0.22112 0.13485 0.35398 0.28243 0.72014
ODIN 24 0.33164 0.25761 0.24035 0.15620 0.37753 0.30858 0.70559
ODIN 26 0.33785 0.26451 0.24326 0.15944 0.37727 0.30830 0.70257
ODIN 32 0.34599 0.27355 0.23734 0.15286 0.36754 0.29749 0.69439
FastABOD 22 0.45763 0.39755 0.41805 0.35360 0.47385 0.41557 0.81361
FastABOD 23 0.46328 0.40383 0.41832 0.35389 0.47447 0.41627 0.81318
FastABOD 24 0.45198 0.39128 0.41863 0.35424 0.47020 0.41152 0.81288
FastABOD 26 0.45763 0.39755 0.41413 0.34923 0.47619 0.41817 0.80883
KDEOS 60 0.20904 0.12143 0.16640 0.07407 0.25042 0.16739 0.64987
KDEOS 61 0.20339 0.11515 0.15920 0.06607 0.25129 0.16836 0.64891
KDEOS 70 0.22034 0.13398 0.15861 0.06541 0.23478 0.15002 0.64925
KDEOS 72 0.20904 0.12143 0.15888 0.06572 0.23810 0.15370 0.65027
LDF 100 0.23164 0.14653 0.17575 0.08446 0.35477 0.28330 0.67307
INFLO 55 0.37288 0.30342 0.44929 0.38829 0.40845 0.34293 0.78854
INFLO 99 0.44633 0.38500 0.49490 0.43895 0.48551 0.42852 0.78585
INFLO 100 0.44633 0.38500 0.49674 0.44100 0.48375 0.42657 0.78701
COF 5 0.20904 0.12143 0.17057 0.07870 0.24965 0.16654 0.62514
COF 7 0.24294 0.15908 0.16760 0.07540 0.25688 0.17457 0.58935

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 3122 objects, 312 outliers (9.99%)

Download raw algorithm results (13.7 MB) Download raw algorithm evaluation table (74.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.44616 0.38467 0.49658 0.44068 0.46076 0.40089 0.83940
KNN 12 0.44413 0.38241 0.49909 0.44348 0.47419 0.41581 0.79818
KNN 14 0.45384 0.39320 0.49392 0.43773 0.48440 0.42716 0.78730
KNNW 14 0.41987 0.35546 0.45576 0.39533 0.44186 0.37989 0.82952
KNNW 26 0.45833 0.39819 0.49307 0.43679 0.46584 0.40653 0.81116
KNNW 30 0.46474 0.40531 0.49098 0.43447 0.47452 0.41618 0.80451
KNNW 32 0.47115 0.41243 0.48941 0.43272 0.47115 0.41243 0.80144
LOF 9 0.13408 0.03793 0.14795 0.05334 0.28397 0.20446 0.66720
LOF 11 0.13746 0.04169 0.14280 0.04762 0.27328 0.19259 0.65218
SimplifiedLOF 17 0.13403 0.03788 0.13323 0.03700 0.24987 0.16658 0.62841
LoOP 76 0.22436 0.13824 0.17592 0.08442 0.30162 0.22408 0.68781
LoOP 81 0.21154 0.12399 0.17446 0.08280 0.31552 0.23952 0.68947
LoOP 87 0.21154 0.12399 0.17018 0.07804 0.32598 0.25114 0.68728
LDOF 75 0.23397 0.14892 0.16792 0.07554 0.29422 0.21586 0.66845
LDOF 100 0.22436 0.13824 0.17307 0.08126 0.31066 0.23412 0.68453
ODIN 23 0.26678 0.18537 0.21878 0.13204 0.39614 0.32909 0.72104
ODIN 37 0.35196 0.28000 0.23839 0.15383 0.41867 0.35413 0.71348
ODIN 45 0.41414 0.34909 0.24062 0.15630 0.41471 0.34972 0.70725
ODIN 92 0.39235 0.32488 0.25045 0.16723 0.39933 0.33264 0.70667
FastABOD 74 0.16026 0.06702 0.18092 0.08998 0.32428 0.24925 0.72216
FastABOD 91 0.13462 0.03853 0.18122 0.09031 0.33837 0.26491 0.72306
FastABOD 100 0.13782 0.04209 0.18181 0.09097 0.33608 0.26237 0.72370
KDEOS 7 0.12821 0.03141 0.11284 0.01434 0.23474 0.14977 0.56186
KDEOS 10 0.10577 0.00648 0.12061 0.02297 0.23852 0.15398 0.58982
KDEOS 12 0.09615 -0.00420 0.12350 0.02618 0.23636 0.15158 0.60879
LDF 1 0.21090 0.12328 0.11800 0.02007 0.21417 0.12692 0.42596
LDF 4 0.20833 0.12043 0.11628 0.01816 0.21914 0.13243 0.46438
LDF 5 0.20833 0.12043 0.12205 0.02457 0.21019 0.12250 0.46183
LDF 30 0.12680 0.02985 0.10656 0.00736 0.18513 0.09465 0.53008
INFLO 11 0.13746 0.04169 0.13782 0.04209 0.26548 0.18392 0.64758
COF 41 0.14744 0.05277 0.14668 0.05193 0.25263 0.16965 0.65817
COF 92 0.23077 0.14536 0.15795 0.06446 0.26726 0.18590 0.65245
COF 94 0.22115 0.13468 0.15737 0.06381 0.27220 0.19139 0.65714
COF 97 0.24359 0.15960 0.15627 0.06259 0.25836 0.17601 0.64810

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO