Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (10% of outliers version#09)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1775 objects, 177 outliers (9.97%)

Download raw algorithm results (13.0 MB) Download raw algorithm evaluation table (73.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.45289 0.39229 0.47245 0.41402 0.45432 0.39388 0.78896
KNN 7 0.49605 0.44023 0.51088 0.45671 0.51449 0.46072 0.75920
KNN 15 0.47700 0.41907 0.49557 0.43970 0.54015 0.48921 0.71778
KNNW 8 0.47458 0.41638 0.50151 0.44629 0.48916 0.43258 0.78982
KNNW 12 0.51977 0.46658 0.52365 0.47089 0.52149 0.46849 0.78017
KNNW 17 0.50847 0.45403 0.53162 0.47974 0.52805 0.47578 0.76456
KNNW 26 0.49718 0.44148 0.52779 0.47548 0.55290 0.50338 0.74243
LOF 32 0.42938 0.36617 0.43433 0.37167 0.44595 0.38458 0.77035
LOF 89 0.48023 0.42265 0.52159 0.46860 0.55762 0.50862 0.74033
LOF 98 0.49153 0.43521 0.52526 0.47268 0.55556 0.50633 0.73629
LOF 99 0.49153 0.43521 0.52636 0.47390 0.55556 0.50633 0.73577
SimplifiedLOF 35 0.45763 0.39755 0.47182 0.41332 0.48485 0.42779 0.78934
SimplifiedLOF 77 0.48588 0.42893 0.52839 0.47616 0.55396 0.50455 0.75726
SimplifiedLOF 95 0.49718 0.44148 0.53321 0.48150 0.55019 0.50036 0.75190
SimplifiedLOF 99 0.49718 0.44148 0.53471 0.48318 0.55000 0.50016 0.74949
LoOP 21 0.48023 0.42265 0.40462 0.33868 0.48295 0.42568 0.75103
LoOP 64 0.42373 0.35990 0.44225 0.38047 0.44860 0.38752 0.77503
LoOP 94 0.43503 0.37245 0.48014 0.42255 0.49829 0.44272 0.77127
LoOP 99 0.45763 0.39755 0.48593 0.42899 0.49829 0.44272 0.77006
LDOF 36 0.42938 0.36617 0.37339 0.30398 0.43017 0.36705 0.77561
LDOF 67 0.46328 0.40383 0.45622 0.39598 0.48408 0.42693 0.76965
LDOF 68 0.46893 0.41010 0.45691 0.39676 0.48408 0.42693 0.76965
LDOF 99 0.45763 0.39755 0.47426 0.41603 0.47619 0.41817 0.77160
ODIN 18 0.28870 0.20991 0.22939 0.14403 0.34835 0.27617 0.70259
ODIN 27 0.31916 0.24375 0.24142 0.15740 0.36028 0.28942 0.69924
ODIN 35 0.33046 0.25630 0.24087 0.15679 0.36494 0.29460 0.69915
ODIN 57 0.30621 0.22937 0.23618 0.15158 0.37643 0.30736 0.70168
FastABOD 24 0.45198 0.39128 0.41215 0.34703 0.47492 0.41676 0.77714
FastABOD 25 0.45763 0.39755 0.41456 0.34972 0.47869 0.42095 0.77695
FastABOD 28 0.44068 0.37873 0.41055 0.34526 0.47950 0.42184 0.77447
KDEOS 61 0.12994 0.03357 0.17784 0.08678 0.25161 0.16872 0.64241
KDEOS 66 0.20904 0.12143 0.15863 0.06543 0.27258 0.19201 0.65197
KDEOS 68 0.19774 0.10888 0.15973 0.06666 0.26738 0.18623 0.65638
LDF 99 0.32203 0.24694 0.17999 0.08916 0.36538 0.29509 0.66208
LDF 100 0.33333 0.25949 0.18204 0.09144 0.36538 0.29509 0.66420
INFLO 36 0.45763 0.39755 0.43417 0.37150 0.46499 0.40573 0.78246
INFLO 91 0.49718 0.44148 0.52607 0.47357 0.53821 0.48706 0.76613
INFLO 98 0.49153 0.43521 0.53209 0.48026 0.55814 0.50920 0.76293
INFLO 99 0.49153 0.43521 0.53320 0.48150 0.55629 0.50714 0.76359
COF 6 0.23729 0.15281 0.18048 0.08970 0.24490 0.16126 0.61462
COF 9 0.25989 0.17791 0.16997 0.07803 0.27204 0.19141 0.57609
COF 10 0.24294 0.15908 0.18716 0.09713 0.25137 0.16844 0.57633

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 3122 objects, 312 outliers (9.99%)

Download raw algorithm results (13.7 MB) Download raw algorithm evaluation table (74.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.46352 0.40396 0.49396 0.43778 0.47487 0.41657 0.84682
KNN 9 0.49293 0.43663 0.53693 0.48551 0.49587 0.43989 0.84091
KNN 10 0.46474 0.40531 0.54287 0.49212 0.50000 0.44448 0.82917
KNN 12 0.44765 0.38632 0.53082 0.47873 0.50901 0.45449 0.81059
KNNW 14 0.45833 0.39819 0.48822 0.43139 0.47792 0.41995 0.84344
KNNW 18 0.50000 0.44448 0.51842 0.46495 0.50394 0.44886 0.83912
KNNW 25 0.47115 0.41243 0.53008 0.47790 0.49068 0.43413 0.82555
LOF 11 0.13479 0.03873 0.14098 0.04560 0.26435 0.18267 0.65275
SimplifiedLOF 12 0.12589 0.02883 0.12833 0.03154 0.23422 0.14920 0.61733
SimplifiedLOF 21 0.12932 0.03265 0.12858 0.03182 0.23599 0.15116 0.61719
LoOP 1 0.21581 0.12874 0.14365 0.04856 0.23851 0.15396 0.58115
LoOP 79 0.17628 0.08482 0.16195 0.06890 0.28249 0.20282 0.66721
LoOP 95 0.16667 0.07414 0.15418 0.06027 0.28422 0.20474 0.66779
LoOP 96 0.17308 0.08126 0.15438 0.06049 0.28033 0.20043 0.66824
LDOF 77 0.20192 0.11331 0.16315 0.07023 0.28490 0.20550 0.65472
LDOF 78 0.20192 0.11331 0.16408 0.07127 0.28681 0.20762 0.65517
LDOF 100 0.18910 0.09907 0.16400 0.07118 0.29004 0.21121 0.66809
ODIN 23 0.26741 0.18607 0.20948 0.12171 0.35352 0.28174 0.70491
ODIN 79 0.36953 0.29953 0.22799 0.14227 0.37624 0.30698 0.68475
ODIN 87 0.36735 0.29710 0.23184 0.14655 0.37856 0.30956 0.68567
ODIN 89 0.36735 0.29710 0.23203 0.14676 0.37856 0.30956 0.68574
FastABOD 34 0.12500 0.02785 0.19040 0.10051 0.35794 0.28665 0.74128
FastABOD 36 0.11859 0.02072 0.19002 0.10009 0.36000 0.28894 0.73933
FastABOD 74 0.16026 0.06702 0.19490 0.10550 0.35424 0.28254 0.74095
FastABOD 75 0.16026 0.06702 0.19504 0.10567 0.35424 0.28254 0.74095
KDEOS 2 0.08333 -0.01845 0.11683 0.01877 0.25301 0.17007 0.57864
KDEOS 6 0.10577 0.00648 0.10641 0.00719 0.23043 0.14499 0.54287
KDEOS 12 0.08974 -0.01132 0.12000 0.02229 0.22877 0.14314 0.59811
LDF 1 0.22756 0.14180 0.12285 0.02546 0.23256 0.14735 0.41716
LDF 34 0.13707 0.04126 0.10731 0.00820 0.18299 0.09228 0.52836
INFLO 11 0.13479 0.03873 0.13330 0.03706 0.24852 0.16508 0.63240
INFLO 12 0.12589 0.02883 0.12842 0.03164 0.25063 0.16743 0.62354
COF 38 0.14103 0.04565 0.14146 0.04613 0.24318 0.15915 0.64912
COF 77 0.20192 0.11331 0.14959 0.05517 0.24384 0.15988 0.62890
COF 78 0.19551 0.10619 0.15103 0.05677 0.24231 0.15818 0.63044
COF 88 0.19231 0.10263 0.14783 0.05321 0.25798 0.17559 0.63343

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO