Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (2% of outliers version#04)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1630 objects, 32 outliers (1.96%)

Download raw algorithm results (10.2 MB) Download raw algorithm evaluation table (55.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.57500 0.56649 0.56943 0.56081 0.64000 0.63279 0.89440
KNN 2 0.55859 0.54975 0.57229 0.56373 0.64000 0.63279 0.90642
KNN 3 0.56250 0.55374 0.58087 0.57247 0.66667 0.65999 0.88947
KNNW 2 0.62500 0.61749 0.52371 0.51417 0.65517 0.64827 0.90291
KNNW 4 0.62500 0.61749 0.57535 0.56685 0.64516 0.63806 0.91040
KNNW 11 0.56250 0.55374 0.59460 0.58648 0.65385 0.64691 0.86824
KNNW 18 0.56250 0.55374 0.58138 0.57300 0.67925 0.67282 0.84211
LOF 9 0.40625 0.39436 0.43107 0.41968 0.46809 0.45743 0.91343
LOF 18 0.56250 0.55374 0.54824 0.53919 0.58333 0.57499 0.90562
LOF 50 0.56250 0.55374 0.58129 0.57290 0.64000 0.63279 0.84659
LOF 52 0.56250 0.55374 0.58273 0.57438 0.64000 0.63279 0.84250
SimplifiedLOF 13 0.50000 0.48999 0.51126 0.50147 0.52459 0.51507 0.92438
SimplifiedLOF 36 0.59375 0.58561 0.59524 0.58714 0.59574 0.58765 0.87866
SimplifiedLOF 52 0.56250 0.55374 0.60954 0.60172 0.64286 0.63571 0.86213
LoOP 18 0.56250 0.55374 0.53584 0.52655 0.60377 0.59584 0.93727
LoOP 23 0.56250 0.55374 0.56038 0.55158 0.63158 0.62420 0.92858
LoOP 59 0.59375 0.58561 0.59592 0.58783 0.59375 0.58561 0.88750
LoOP 71 0.56250 0.55374 0.60948 0.60166 0.60377 0.59584 0.88040
LDOF 18 0.50000 0.48999 0.47951 0.46909 0.50000 0.48999 0.93130
LDOF 32 0.56250 0.55374 0.54586 0.53677 0.57143 0.56285 0.91110
LDOF 54 0.56250 0.55374 0.61863 0.61099 0.64151 0.63433 0.90128
ODIN 33 0.27051 0.25590 0.18404 0.16770 0.33735 0.32408 0.89577
ODIN 57 0.29375 0.27961 0.19881 0.18277 0.36538 0.35268 0.88044
ODIN 96 0.35326 0.34031 0.21239 0.19662 0.36364 0.35089 0.86092
ODIN 100 0.35326 0.34031 0.21257 0.19680 0.36364 0.35089 0.86142
FastABOD 23 0.46875 0.45811 0.43943 0.42820 0.50746 0.49760 0.88912
FastABOD 96 0.50000 0.48999 0.41883 0.40719 0.54286 0.53370 0.85292
FastABOD 98 0.53125 0.52186 0.42684 0.41536 0.53968 0.53046 0.85057
KDEOS 7 0.18750 0.17123 0.10327 0.08532 0.19048 0.17427 0.74087
KDEOS 8 0.18750 0.17123 0.12448 0.10695 0.20339 0.18744 0.75072
KDEOS 70 0.15625 0.13935 0.12979 0.11237 0.20290 0.18694 0.82996
LDF 3 0.03824 0.01898 0.03364 0.01429 0.07807 0.05961 0.62813
LDF 5 0.03292 0.01356 0.04272 0.02355 0.10891 0.09107 0.77628
LDF 100 0.00000 -0.02003 0.04787 0.02881 0.14346 0.12631 0.71132
INFLO 18 0.56250 0.55374 0.55246 0.54350 0.57971 0.57129 0.92610
INFLO 24 0.59375 0.58561 0.57733 0.56886 0.59375 0.58561 0.90979
INFLO 52 0.53125 0.52186 0.61178 0.60400 0.65385 0.64691 0.87240
COF 5 0.25000 0.23498 0.12251 0.10494 0.27273 0.25816 0.69058

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2867 objects, 57 outliers (1.99%)

Download raw algorithm results (12.0 MB) Download raw algorithm evaluation table (67.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.39145 0.37910 0.43085 0.41931 0.47059 0.45985 0.89749
KNN 3 0.47456 0.46390 0.48564 0.47521 0.54118 0.53187 0.89281
KNN 6 0.49123 0.48091 0.47839 0.46781 0.57143 0.56274 0.87076
KNN 9 0.47368 0.46301 0.44897 0.43779 0.58140 0.57290 0.83659
KNNW 5 0.43860 0.42721 0.46333 0.45244 0.50000 0.48986 0.89613
KNNW 9 0.49123 0.48091 0.50248 0.49239 0.56180 0.55291 0.88717
KNNW 11 0.49123 0.48091 0.50767 0.49769 0.59574 0.58754 0.87952
KNNW 15 0.49123 0.48091 0.49628 0.48606 0.60000 0.59189 0.86288
LOF 7 0.04902 0.02973 0.04967 0.03039 0.12052 0.10268 0.78312
LOF 10 0.04565 0.02629 0.04955 0.03027 0.11339 0.09541 0.80760
SimplifiedLOF 6 0.04661 0.02727 0.04466 0.02528 0.09237 0.07396 0.76072
SimplifiedLOF 9 0.04431 0.02493 0.04500 0.02562 0.10177 0.08355 0.77571
SimplifiedLOF 10 0.04313 0.02372 0.04491 0.02554 0.09919 0.08092 0.78373
LoOP 1 0.17333 0.15656 0.12277 0.10498 0.20225 0.18607 0.64702
LoOP 22 0.11298 0.09499 0.11517 0.09722 0.17964 0.16300 0.84426
LoOP 71 0.17544 0.15871 0.10902 0.09094 0.22857 0.21292 0.82396
LoOP 78 0.19298 0.17661 0.09972 0.08146 0.20870 0.19264 0.82038
LDOF 9 0.19298 0.17661 0.09197 0.07355 0.20370 0.18755 0.79147
LDOF 75 0.19298 0.17661 0.11603 0.09810 0.23313 0.21757 0.82231
ODIN 77 0.39583 0.38358 0.29137 0.27700 0.45882 0.44785 0.92333
ODIN 83 0.40912 0.39714 0.29801 0.28377 0.46061 0.44966 0.91820
ODIN 99 0.40418 0.39210 0.30022 0.28603 0.47205 0.46134 0.91472
ODIN 100 0.40418 0.39210 0.30022 0.28603 0.47205 0.46134 0.91474
FastABOD 36 0.04724 0.02792 0.07085 0.05201 0.17484 0.15810 0.82691
FastABOD 73 0.05172 0.03249 0.07661 0.05788 0.16931 0.15246 0.82796
FastABOD 97 0.05172 0.03249 0.07675 0.05803 0.17308 0.15630 0.82494
KDEOS 11 0.04918 0.02989 0.05107 0.03182 0.12214 0.10433 0.76719
KDEOS 73 0.12281 0.10501 0.05101 0.03176 0.13913 0.12167 0.71401
KDEOS 76 0.12281 0.10501 0.05423 0.03504 0.13913 0.12167 0.72339
LDF 1 0.07216 0.05334 0.02772 0.00800 0.10573 0.08759 0.36288
LDF 2 0.06061 0.04155 0.02862 0.00891 0.10879 0.09071 0.49270
LDF 8 0.00000 -0.02028 0.02846 0.00875 0.06575 0.04680 0.61450
INFLO 7 0.04910 0.02981 0.04830 0.02899 0.11994 0.10209 0.77247
INFLO 10 0.04565 0.02629 0.04847 0.02917 0.11506 0.09710 0.79764
COF 36 0.05263 0.03341 0.08147 0.06284 0.19077 0.17435 0.79624
COF 73 0.17544 0.15871 0.09384 0.07546 0.21698 0.20110 0.76722
COF 74 0.17544 0.15871 0.10106 0.08282 0.22979 0.21416 0.77279
COF 75 0.17544 0.15871 0.10087 0.08263 0.23529 0.21978 0.77405

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO