Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (5% of outliers version#01)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1682 objects, 84 outliers (4.99%)

Download raw algorithm results (10.5 MB) Download raw algorithm evaluation table (67.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.43590 0.40624 0.48150 0.45424 0.48333 0.45617 0.82511
KNN 5 0.47789 0.45045 0.50320 0.47709 0.52174 0.49660 0.81775
KNN 11 0.50085 0.47461 0.48034 0.45303 0.55319 0.52970 0.76870
KNN 16 0.49339 0.46676 0.48830 0.46141 0.58824 0.56659 0.75252
KNNW 6 0.44048 0.41106 0.47503 0.44743 0.46617 0.43810 0.82540
KNNW 21 0.51190 0.48625 0.52954 0.50481 0.57343 0.55100 0.79085
KNNW 22 0.52381 0.49878 0.52870 0.50393 0.57143 0.54890 0.78870
KNNW 25 0.52381 0.49878 0.52790 0.50309 0.58503 0.56322 0.78223
LOF 26 0.41667 0.38600 0.44458 0.41538 0.47407 0.44643 0.83782
LOF 52 0.52381 0.49878 0.53708 0.51274 0.57931 0.55720 0.80721
LOF 60 0.53571 0.51131 0.53698 0.51264 0.57343 0.55100 0.80252
LOF 100 0.51190 0.48625 0.53245 0.50787 0.59854 0.57744 0.78624
SimplifiedLOF 26 0.46429 0.43613 0.47240 0.44467 0.47337 0.44569 0.84598
SimplifiedLOF 56 0.53571 0.51131 0.55182 0.52826 0.56757 0.54484 0.81928
SimplifiedLOF 81 0.51190 0.48625 0.54502 0.52111 0.58647 0.56473 0.80831
LoOP 13 0.47619 0.44866 0.41001 0.37899 0.53521 0.51078 0.75214
LoOP 54 0.45238 0.42359 0.44606 0.41694 0.46067 0.43232 0.83925
LoOP 84 0.50000 0.47372 0.49363 0.46701 0.51136 0.48568 0.82930
LoOP 100 0.50000 0.47372 0.50670 0.48077 0.51701 0.49162 0.82412
LDOF 43 0.46429 0.43613 0.42098 0.39054 0.47337 0.44569 0.83584
LDOF 58 0.51190 0.48625 0.46560 0.43751 0.51190 0.48625 0.83317
LDOF 81 0.48810 0.46119 0.48379 0.45666 0.51462 0.48911 0.83379
LDOF 85 0.48810 0.46119 0.48755 0.46062 0.51462 0.48911 0.83555
ODIN 50 0.33521 0.30027 0.22540 0.18468 0.35060 0.31646 0.74277
ODIN 96 0.35635 0.32252 0.24570 0.20605 0.37668 0.34392 0.74108
ODIN 98 0.35635 0.32252 0.25040 0.21099 0.37838 0.34570 0.74141
FastABOD 12 0.45238 0.42359 0.42232 0.39196 0.47674 0.44924 0.80655
FastABOD 22 0.51190 0.48625 0.43975 0.41030 0.51220 0.48655 0.80011
FastABOD 24 0.48810 0.46119 0.44049 0.41107 0.49718 0.47074 0.80435
KDEOS 9 0.19048 0.14792 0.11615 0.06969 0.20513 0.16335 0.65750
KDEOS 11 0.17857 0.13539 0.11306 0.06644 0.21622 0.17502 0.65764
KDEOS 58 0.11905 0.07274 0.11645 0.07001 0.15615 0.11179 0.70221
KDEOS 74 0.11905 0.07274 0.10579 0.05879 0.18605 0.14326 0.71247
LDF 3 0.09907 0.05171 0.07501 0.02639 0.15753 0.11325 0.52612
LDF 100 0.05952 0.01009 0.09430 0.04669 0.23324 0.19293 0.67162
INFLO 28 0.45238 0.42359 0.43897 0.40948 0.46154 0.43323 0.84313
INFLO 85 0.55952 0.53637 0.54299 0.51896 0.56627 0.54347 0.81873
INFLO 91 0.55952 0.53637 0.54111 0.51699 0.56970 0.54708 0.81537
COF 4 0.23810 0.19805 0.13500 0.08953 0.24828 0.20876 0.63596

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2957 objects, 147 outliers (4.97%)

Download raw algorithm results (12.6 MB) Download raw algorithm evaluation table (71.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.49867 0.47245 0.51484 0.48946 0.50883 0.48314 0.87776
KNN 7 0.47522 0.44777 0.53859 0.51446 0.55111 0.52763 0.86696
KNN 13 0.45837 0.43004 0.49590 0.46953 0.56364 0.54081 0.80594
KNNW 9 0.43537 0.40584 0.48696 0.46012 0.46640 0.43849 0.87763
KNNW 12 0.47619 0.44879 0.51327 0.48781 0.49782 0.47155 0.87186
KNNW 18 0.46939 0.44163 0.53364 0.50924 0.55204 0.52860 0.85386
KNNW 20 0.47619 0.44879 0.53131 0.50679 0.56621 0.54352 0.84778
LOF 7 0.09642 0.04915 0.09552 0.04820 0.20241 0.16069 0.72552
LOF 8 0.09639 0.04911 0.09816 0.05098 0.21121 0.16994 0.73522
SimplifiedLOF 8 0.10165 0.05465 0.09098 0.04343 0.16914 0.12568 0.70161
LoOP 73 0.19728 0.15529 0.13254 0.08716 0.27110 0.23297 0.73738
LoOP 93 0.16327 0.11949 0.12272 0.07683 0.21647 0.17548 0.74221
LDOF 77 0.16327 0.11949 0.14790 0.10332 0.28434 0.24690 0.74967
LDOF 78 0.16327 0.11949 0.14845 0.10390 0.28571 0.24835 0.75092
LDOF 95 0.16327 0.11949 0.14059 0.09564 0.23881 0.19899 0.75310
ODIN 46 0.34966 0.31564 0.22869 0.18834 0.40749 0.37650 0.78634
ODIN 84 0.38758 0.35554 0.25749 0.21864 0.42932 0.39947 0.78401
ODIN 99 0.38590 0.35377 0.26143 0.22279 0.43968 0.41037 0.78277
ODIN 100 0.38590 0.35377 0.26149 0.22286 0.43968 0.41037 0.78286
FastABOD 30 0.04082 -0.00936 0.12129 0.07532 0.26451 0.22603 0.77717
FastABOD 73 0.11565 0.06938 0.12485 0.07906 0.27036 0.23219 0.77475
FastABOD 83 0.11565 0.06938 0.12551 0.07976 0.27642 0.23857 0.77573
FastABOD 94 0.11565 0.06938 0.12475 0.07896 0.27946 0.24177 0.77314
KDEOS 10 0.06803 0.01927 0.08014 0.03202 0.16343 0.11967 0.67351
KDEOS 76 0.08844 0.04075 0.07703 0.02874 0.14088 0.09593 0.64499
LDF 1 0.12472 0.07893 0.05505 0.00562 0.14545 0.10075 0.35928
LDF 3 0.10219 0.05522 0.05867 0.00942 0.15591 0.11176 0.44298
LDF 14 0.07519 0.02681 0.06150 0.01241 0.11050 0.06396 0.57675
LDF 16 0.08679 0.03902 0.06151 0.01241 0.11165 0.06518 0.56274
INFLO 7 0.09642 0.04915 0.08746 0.03972 0.19906 0.15716 0.68877
INFLO 8 0.09639 0.04911 0.09159 0.04407 0.20571 0.16416 0.71039
COF 36 0.10204 0.05507 0.10565 0.05886 0.21259 0.17140 0.71729
COF 75 0.19048 0.14813 0.11913 0.07305 0.22785 0.18745 0.70283
COF 78 0.19048 0.14813 0.12268 0.07679 0.24403 0.20448 0.70577

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO