Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (5% of outliers version#09)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1682 objects, 84 outliers (4.99%)

Download raw algorithm results (10.4 MB) Download raw algorithm evaluation table (67.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.45040 0.42151 0.40148 0.37002 0.45399 0.42529 0.78782
KNN 14 0.47619 0.44866 0.47310 0.44540 0.54962 0.52594 0.72837
KNN 16 0.47090 0.44309 0.47744 0.44997 0.57778 0.55558 0.72421
KNNW 3 0.39286 0.36094 0.37226 0.33926 0.40559 0.37435 0.79175
KNNW 17 0.48810 0.46119 0.47125 0.44346 0.49351 0.46688 0.77263
KNNW 27 0.47619 0.44866 0.48597 0.45895 0.55224 0.52870 0.74968
KNNW 36 0.47619 0.44866 0.47980 0.45245 0.57364 0.55123 0.73449
LOF 26 0.40476 0.37347 0.40440 0.37310 0.45588 0.42728 0.79960
LOF 49 0.48810 0.46119 0.49192 0.46521 0.53147 0.50684 0.76144
LOF 78 0.46429 0.43613 0.48739 0.46044 0.56489 0.54201 0.75031
LOF 86 0.47619 0.44866 0.50112 0.47489 0.56489 0.54201 0.74947
SimplifiedLOF 28 0.45238 0.42359 0.44289 0.41361 0.47407 0.44643 0.81259
SimplifiedLOF 49 0.51190 0.48625 0.49610 0.46961 0.53061 0.50594 0.78411
SimplifiedLOF 80 0.48810 0.46119 0.49573 0.46922 0.55385 0.53039 0.76372
SimplifiedLOF 85 0.48810 0.46119 0.50738 0.48148 0.55385 0.53039 0.76493
LoOP 48 0.39286 0.36094 0.37174 0.33871 0.42553 0.39533 0.80324
LoOP 90 0.48810 0.46119 0.44551 0.41636 0.48837 0.46148 0.78348
LoOP 100 0.48810 0.46119 0.44964 0.42071 0.50955 0.48377 0.77980
LDOF 36 0.41667 0.38600 0.35835 0.32462 0.43796 0.40841 0.81023
LDOF 54 0.42857 0.39853 0.40330 0.37193 0.48529 0.45824 0.79285
LDOF 80 0.45238 0.42359 0.41967 0.38916 0.46667 0.43863 0.79159
LDOF 86 0.45238 0.42359 0.43310 0.40330 0.47682 0.44932 0.79254
ODIN 26 0.28700 0.24952 0.18216 0.13917 0.30901 0.27269 0.71446
ODIN 47 0.27694 0.23893 0.18646 0.14369 0.32407 0.28854 0.72473
ODIN 55 0.28333 0.24566 0.18434 0.14146 0.33654 0.30166 0.71848
FastABOD 15 0.20238 0.16045 0.17535 0.13200 0.39111 0.35910 0.79232
FastABOD 18 0.47619 0.44866 0.33036 0.29516 0.48837 0.46148 0.78747
FastABOD 20 0.46429 0.43613 0.33414 0.29913 0.49123 0.46448 0.79088
FastABOD 23 0.46429 0.43613 0.34587 0.31148 0.48315 0.45598 0.78805
KDEOS 62 0.20238 0.16045 0.11776 0.07139 0.22772 0.18713 0.69755
KDEOS 63 0.21429 0.17298 0.12762 0.08176 0.25137 0.21201 0.69577
KDEOS 64 0.23810 0.19805 0.13895 0.09368 0.24309 0.20331 0.69393
LDF 3 0.07941 0.03102 0.06913 0.02020 0.14311 0.09807 0.56447
LDF 100 0.00287 -0.04954 0.10797 0.06108 0.28571 0.24817 0.67485
INFLO 36 0.45238 0.42359 0.41637 0.38569 0.46250 0.43425 0.80686
INFLO 80 0.51190 0.48625 0.47888 0.45149 0.51190 0.48625 0.77245
INFLO 83 0.50000 0.47372 0.48584 0.45881 0.51701 0.49162 0.77571
INFLO 85 0.50000 0.47372 0.49601 0.46951 0.51701 0.49162 0.77443
COF 2 0.13095 0.08527 0.10722 0.06029 0.18972 0.14713 0.64356
COF 9 0.16667 0.12286 0.09391 0.04628 0.17021 0.12659 0.54029
COF 70 0.16667 0.12286 0.12194 0.07579 0.17391 0.13049 0.53527

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2957 objects, 147 outliers (4.97%)

Download raw algorithm results (12.6 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.50572 0.47986 0.52900 0.50436 0.51118 0.48561 0.89230
KNN 5 0.51884 0.49367 0.53938 0.51529 0.52448 0.49960 0.89056
KNN 8 0.49975 0.47358 0.53817 0.51401 0.53913 0.51502 0.87158
KNNW 9 0.48299 0.45595 0.50023 0.47408 0.49498 0.46856 0.88601
KNNW 10 0.48980 0.46311 0.51209 0.48657 0.50658 0.48077 0.88546
KNNW 16 0.46259 0.43447 0.52430 0.49942 0.49133 0.46472 0.87311
KNNW 37 0.44898 0.42015 0.50270 0.47668 0.51639 0.49109 0.82952
LOF 7 0.07962 0.03147 0.08874 0.04107 0.19907 0.15718 0.70682
LOF 9 0.09343 0.04601 0.09440 0.04702 0.19706 0.15506 0.72415
LOF 10 0.09413 0.04674 0.09216 0.04466 0.18999 0.14762 0.71121
SimplifiedLOF 9 0.09130 0.04376 0.08621 0.03840 0.16571 0.12207 0.69607
SimplifiedLOF 10 0.09032 0.04273 0.08575 0.03793 0.16768 0.12414 0.69514
LoOP 12 0.10204 0.05507 0.11608 0.06984 0.21001 0.16869 0.72022
LoOP 19 0.11565 0.06938 0.13479 0.08953 0.22562 0.18511 0.70995
LoOP 30 0.12925 0.08370 0.11479 0.06848 0.25205 0.21292 0.68830
LoOP 72 0.16327 0.11949 0.11056 0.06403 0.21190 0.17067 0.69651
LDOF 77 0.15646 0.11233 0.10853 0.06190 0.19880 0.15688 0.69345
LDOF 78 0.15646 0.11233 0.10891 0.06230 0.19880 0.15688 0.69485
LDOF 87 0.14966 0.10518 0.10770 0.06102 0.19335 0.15116 0.69938
ODIN 16 0.19558 0.15350 0.16783 0.12430 0.31429 0.27841 0.78391
ODIN 82 0.39703 0.36549 0.25558 0.21664 0.43702 0.40757 0.77546
ODIN 87 0.38819 0.35619 0.25952 0.22078 0.44503 0.41599 0.77618
ODIN 100 0.38600 0.35388 0.25830 0.21950 0.44562 0.41662 0.77263
FastABOD 34 0.03401 -0.01652 0.12373 0.07789 0.27566 0.23777 0.78412
FastABOD 73 0.12925 0.08370 0.12712 0.08146 0.25849 0.21969 0.78355
FastABOD 100 0.12925 0.08370 0.12824 0.08264 0.26917 0.23093 0.78401
KDEOS 2 0.05442 0.00496 0.06979 0.02112 0.17602 0.13291 0.60659
KDEOS 10 0.04762 -0.00220 0.07521 0.02684 0.15654 0.11241 0.67474
KDEOS 75 0.08163 0.03359 0.06227 0.01321 0.12673 0.08105 0.60015
LDF 22 0.09032 0.04273 0.06749 0.01871 0.16848 0.12498 0.59245
LDF 24 0.14907 0.10455 0.06912 0.02042 0.15584 0.11168 0.56569
INFLO 7 0.07974 0.03160 0.08350 0.03556 0.19773 0.15576 0.69867
INFLO 9 0.09357 0.04615 0.09098 0.04342 0.19541 0.15332 0.71567
INFLO 10 0.09413 0.04674 0.09069 0.04313 0.19153 0.14923 0.71630
COF 81 0.12925 0.08370 0.10680 0.06008 0.24878 0.20948 0.68819
COF 91 0.17007 0.12665 0.09443 0.04706 0.20366 0.16200 0.66162

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO