Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (5% of outliers version#08)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1682 objects, 84 outliers (4.99%)

Download raw algorithm results (10.5 MB) Download raw algorithm evaluation table (66.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.41837 0.38779 0.43216 0.40231 0.43750 0.40793 0.83599
KNN 3 0.43398 0.40423 0.44970 0.42078 0.48227 0.45505 0.82398
KNN 11 0.44048 0.41106 0.41535 0.38462 0.49612 0.46964 0.73423
KNN 12 0.43934 0.40987 0.42008 0.38959 0.51128 0.48559 0.72914
KNNW 4 0.41667 0.38600 0.40417 0.37285 0.44324 0.41398 0.84021
KNNW 6 0.47619 0.44866 0.44197 0.41264 0.48193 0.45469 0.83237
KNNW 12 0.45238 0.42359 0.45609 0.42750 0.50704 0.48113 0.79505
KNNW 20 0.46429 0.43613 0.45579 0.42718 0.51969 0.49444 0.76473
LOF 28 0.41667 0.38600 0.38704 0.35482 0.44295 0.41367 0.81672
LOF 65 0.46429 0.43613 0.45674 0.42818 0.51852 0.49321 0.77768
LOF 78 0.46429 0.43613 0.46379 0.43561 0.53333 0.50880 0.76567
SimplifiedLOF 28 0.45238 0.42359 0.42581 0.39562 0.47619 0.44866 0.83592
SimplifiedLOF 49 0.47619 0.44866 0.46476 0.43663 0.49275 0.46609 0.79668
SimplifiedLOF 65 0.46429 0.43613 0.46696 0.43894 0.52414 0.49912 0.79595
SimplifiedLOF 87 0.46429 0.43613 0.47369 0.44602 0.51128 0.48559 0.77872
LoOP 42 0.38095 0.34841 0.36529 0.33193 0.40523 0.37396 0.82351
LoOP 80 0.46429 0.43613 0.41869 0.38813 0.46429 0.43613 0.80975
LoOP 96 0.45238 0.42359 0.43139 0.40150 0.48649 0.45949 0.80274
LoOP 100 0.45238 0.42359 0.43508 0.40538 0.48649 0.45949 0.80165
LDOF 56 0.42857 0.39853 0.40824 0.37713 0.43636 0.40674 0.81842
LDOF 78 0.46429 0.43613 0.42416 0.39389 0.46980 0.44193 0.81571
LDOF 80 0.46429 0.43613 0.42488 0.39464 0.46980 0.44193 0.81636
ODIN 54 0.32614 0.29071 0.20049 0.15847 0.35122 0.31712 0.74918
FastABOD 16 0.46429 0.43613 0.33286 0.29779 0.46707 0.43905 0.80974
FastABOD 17 0.46429 0.43613 0.33948 0.30476 0.47399 0.44634 0.81221
FastABOD 20 0.45238 0.42359 0.34135 0.30673 0.47059 0.44276 0.81516
KDEOS 58 0.15476 0.11033 0.15838 0.11414 0.22069 0.17972 0.73987
KDEOS 60 0.16667 0.12286 0.13389 0.08836 0.24324 0.20346 0.73969
KDEOS 62 0.22619 0.18551 0.13343 0.08788 0.23457 0.19433 0.73634
KDEOS 72 0.22619 0.18551 0.14386 0.09886 0.23636 0.19622 0.74314
LDF 3 0.09264 0.04495 0.07773 0.02925 0.16066 0.11654 0.59075
LDF 99 0.05952 0.01009 0.08215 0.03390 0.19767 0.15550 0.64115
LDF 100 0.07143 0.02262 0.08307 0.03487 0.19767 0.15550 0.64175
INFLO 28 0.39286 0.36094 0.38750 0.35531 0.41135 0.38040 0.83178
INFLO 76 0.47619 0.44866 0.44424 0.41502 0.47887 0.45148 0.79515
INFLO 92 0.46429 0.43613 0.46330 0.43508 0.50704 0.48113 0.78814
COF 4 0.19048 0.14792 0.11350 0.06690 0.20305 0.16115 0.64149

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2957 objects, 147 outliers (4.97%)

Download raw algorithm results (12.6 MB) Download raw algorithm evaluation table (72.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.49915 0.47295 0.53217 0.50769 0.51301 0.48754 0.88382
KNN 5 0.53354 0.50914 0.55930 0.53624 0.56303 0.54017 0.87936
KNNW 9 0.48299 0.45595 0.51956 0.49442 0.53077 0.50622 0.87466
KNNW 14 0.49660 0.47026 0.53959 0.51550 0.52336 0.49843 0.86577
KNNW 18 0.51020 0.48458 0.53793 0.51376 0.53061 0.50606 0.85492
KNNW 26 0.49660 0.47026 0.52651 0.50174 0.54884 0.52524 0.83746
LOF 9 0.09197 0.04447 0.09728 0.05006 0.21413 0.17302 0.73664
SimplifiedLOF 9 0.08987 0.04226 0.08819 0.04049 0.17707 0.13402 0.71074
LoOP 2 0.17687 0.13381 0.13553 0.09031 0.19016 0.14780 0.67951
LoOP 11 0.10884 0.06222 0.12112 0.07514 0.23035 0.19009 0.75022
LoOP 73 0.12925 0.08370 0.12770 0.08207 0.25858 0.21979 0.74701
LDOF 77 0.17007 0.12665 0.12436 0.07856 0.24532 0.20584 0.75007
LDOF 78 0.17007 0.12665 0.12529 0.07953 0.24790 0.20855 0.75256
LDOF 80 0.17007 0.12665 0.12463 0.07884 0.24473 0.20521 0.75448
ODIN 85 0.41148 0.38069 0.27012 0.23193 0.42236 0.39214 0.78394
ODIN 99 0.41998 0.38964 0.27219 0.23412 0.42675 0.39676 0.78149
ODIN 100 0.41998 0.38964 0.27232 0.23426 0.42675 0.39676 0.78163
FastABOD 29 0.02721 -0.02368 0.11657 0.07035 0.27769 0.23990 0.75697
FastABOD 73 0.10884 0.06222 0.12019 0.07416 0.25455 0.21555 0.75308
FastABOD 99 0.10884 0.06222 0.12212 0.07619 0.26738 0.22905 0.75825
KDEOS 10 0.06122 0.01211 0.07718 0.02891 0.16715 0.12358 0.67229
KDEOS 75 0.06803 0.01927 0.07146 0.02289 0.14665 0.10201 0.64973
LDF 1 0.11565 0.06938 0.05196 0.00237 0.14907 0.10455 0.33414
LDF 15 0.06320 0.01419 0.05956 0.01036 0.11321 0.06682 0.57745
INFLO 9 0.09211 0.04461 0.09453 0.04717 0.21098 0.16970 0.73044
COF 74 0.12925 0.08370 0.11260 0.06618 0.25676 0.21788 0.70670
COF 78 0.12245 0.07654 0.11668 0.07047 0.26289 0.22433 0.70095
COF 80 0.10884 0.06222 0.11567 0.06941 0.26501 0.22656 0.70380
COF 88 0.13605 0.09086 0.10078 0.05374 0.23318 0.19307 0.68587

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO