Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (5% of outliers version#06)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1682 objects, 84 outliers (4.99%)

Download raw algorithm results (10.5 MB) Download raw algorithm evaluation table (67.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.43315 0.40335 0.49594 0.46945 0.51128 0.48559 0.87054
KNN 14 0.51190 0.48625 0.48555 0.45851 0.56000 0.53687 0.77520
KNN 17 0.51190 0.48625 0.49227 0.46558 0.56738 0.54463 0.76480
KNNW 5 0.45238 0.42359 0.45573 0.42712 0.46328 0.43506 0.86794
KNNW 20 0.48810 0.46119 0.50095 0.47472 0.51799 0.49265 0.81724
KNNW 22 0.47619 0.44866 0.50183 0.47564 0.52555 0.50061 0.81145
KNNW 32 0.46429 0.43613 0.49485 0.46829 0.55319 0.52970 0.78886
LOF 26 0.44048 0.41106 0.46943 0.44154 0.49624 0.46976 0.85289
LOF 58 0.52381 0.49878 0.52380 0.49877 0.57333 0.55091 0.82091
LOF 70 0.52381 0.49878 0.53437 0.50989 0.58571 0.56394 0.82005
LOF 84 0.51190 0.48625 0.53062 0.50594 0.60432 0.58352 0.81026
SimplifiedLOF 26 0.47619 0.44866 0.48874 0.46186 0.50000 0.47372 0.86937
SimplifiedLOF 48 0.52381 0.49878 0.52261 0.49751 0.53691 0.51257 0.84609
SimplifiedLOF 70 0.52381 0.49878 0.54051 0.51635 0.56944 0.54681 0.83533
SimplifiedLOF 84 0.52381 0.49878 0.53838 0.51411 0.58741 0.56572 0.82538
LoOP 11 0.54762 0.52384 0.44329 0.41402 0.55621 0.53289 0.80748
LoOP 12 0.53571 0.51131 0.44384 0.41461 0.55629 0.53297 0.79882
LoOP 53 0.44048 0.41106 0.44821 0.41920 0.44311 0.41384 0.85392
LoOP 100 0.48810 0.46119 0.49280 0.46614 0.50000 0.47372 0.84487
LDOF 35 0.42857 0.39853 0.41363 0.38281 0.44571 0.41658 0.85720
LDOF 64 0.46429 0.43613 0.46291 0.43467 0.47312 0.44542 0.83950
LDOF 70 0.44048 0.41106 0.47362 0.44596 0.47482 0.44721 0.84879
LDOF 87 0.45238 0.42359 0.47786 0.45042 0.47368 0.44602 0.85098
ODIN 7 0.21259 0.17119 0.15925 0.11505 0.27011 0.23175 0.78630
ODIN 56 0.33798 0.30318 0.22190 0.18100 0.34783 0.31354 0.77179
ODIN 96 0.32900 0.29373 0.22771 0.18712 0.36607 0.33275 0.76715
ODIN 100 0.32423 0.28871 0.23043 0.18997 0.36607 0.33275 0.76749
FastABOD 20 0.45238 0.42359 0.33700 0.30215 0.47500 0.44740 0.84997
FastABOD 24 0.46429 0.43613 0.33551 0.30058 0.47853 0.45112 0.85492
FastABOD 25 0.47619 0.44866 0.33540 0.30047 0.47619 0.44866 0.85321
FastABOD 28 0.45238 0.42359 0.33173 0.29660 0.48276 0.45557 0.85092
KDEOS 64 0.23810 0.19805 0.15350 0.10901 0.27083 0.23250 0.76252
KDEOS 65 0.26190 0.22311 0.15140 0.10679 0.26596 0.22737 0.76041
KDEOS 71 0.22619 0.18551 0.16289 0.11888 0.26016 0.22127 0.75784
LDF 3 0.09804 0.05063 0.07699 0.02847 0.15961 0.11543 0.56032
LDF 99 0.05952 0.01009 0.09327 0.04561 0.22807 0.18749 0.68179
LDF 100 0.05952 0.01009 0.09407 0.04645 0.22807 0.18749 0.68212
INFLO 35 0.42857 0.39853 0.42020 0.38972 0.43678 0.40718 0.85912
INFLO 87 0.52381 0.49878 0.51760 0.49225 0.54321 0.51920 0.83755
INFLO 89 0.52381 0.49878 0.51725 0.49187 0.54658 0.52275 0.83627
COF 5 0.21429 0.17298 0.13587 0.09045 0.21839 0.17730 0.64101
COF 8 0.20238 0.16045 0.12362 0.07755 0.23158 0.19119 0.60795

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2957 objects, 147 outliers (4.97%)

Download raw algorithm results (12.6 MB) Download raw algorithm evaluation table (72.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.49447 0.46803 0.51590 0.49058 0.51948 0.49434 0.88112
KNN 9 0.54304 0.51914 0.57656 0.55441 0.59361 0.57235 0.86458
KNN 10 0.55215 0.52873 0.57355 0.55124 0.60550 0.58487 0.85177
KNN 15 0.52773 0.50302 0.55395 0.53061 0.61062 0.59025 0.81863
KNNW 11 0.46939 0.44163 0.51186 0.48633 0.48889 0.46215 0.88382
KNNW 22 0.54422 0.52037 0.56669 0.54402 0.57658 0.55443 0.86338
KNNW 24 0.56463 0.54185 0.56631 0.54362 0.58150 0.55960 0.85916
KNNW 37 0.52381 0.49890 0.55092 0.52742 0.59690 0.57581 0.83351
LOF 8 0.06955 0.02088 0.08285 0.03487 0.20172 0.15996 0.68737
LOF 10 0.08230 0.03430 0.08923 0.04159 0.19335 0.15116 0.72216
LOF 11 0.08596 0.03814 0.08824 0.04055 0.18739 0.14488 0.71398
SimplifiedLOF 11 0.08565 0.03781 0.08328 0.03533 0.16827 0.12476 0.68849
LoOP 32 0.16327 0.11949 0.12641 0.08071 0.23826 0.19841 0.76234
LoOP 72 0.16327 0.11949 0.13403 0.08873 0.24474 0.20523 0.77644
LoOP 74 0.15646 0.11233 0.13303 0.08768 0.24561 0.20615 0.77757
LoOP 75 0.15646 0.11233 0.13282 0.08746 0.23938 0.19959 0.77787
LDOF 75 0.15646 0.11233 0.13149 0.08606 0.25150 0.21234 0.76913
LDOF 80 0.15646 0.11233 0.13321 0.08787 0.25726 0.21841 0.77321
LDOF 81 0.15646 0.11233 0.13399 0.08868 0.25726 0.21841 0.77514
LDOF 98 0.15646 0.11233 0.13342 0.08808 0.24635 0.20692 0.77867
ODIN 81 0.34680 0.31263 0.24019 0.20044 0.39687 0.36532 0.81597
ODIN 82 0.34736 0.31322 0.24108 0.20138 0.39687 0.36532 0.81573
ODIN 98 0.35828 0.32471 0.23929 0.19950 0.39660 0.36503 0.81031
FastABOD 34 0.03401 -0.01652 0.11942 0.07336 0.27597 0.23810 0.77777
FastABOD 36 0.03401 -0.01652 0.12121 0.07523 0.27488 0.23695 0.78216
FastABOD 74 0.10884 0.06222 0.12605 0.08033 0.26611 0.22771 0.78169
FastABOD 97 0.11565 0.06938 0.12460 0.07881 0.27010 0.23191 0.77752
KDEOS 2 0.03253 -0.01808 0.06301 0.01399 0.16051 0.11659 0.59748
KDEOS 23 0.08163 0.03359 0.07427 0.02584 0.14986 0.10539 0.67037
KDEOS 75 0.11565 0.06938 0.07565 0.02729 0.14103 0.09609 0.65616
KDEOS 86 0.11565 0.06938 0.07846 0.03025 0.14580 0.10112 0.66967
LDF 1 0.11454 0.06821 0.05345 0.00393 0.14590 0.10122 0.33342
LDF 12 0.06061 0.01146 0.05443 0.00496 0.10303 0.05611 0.52166
INFLO 9 0.07738 0.02912 0.08147 0.03342 0.19335 0.15115 0.69087
INFLO 11 0.08596 0.03814 0.08510 0.03724 0.18562 0.14302 0.70493
COF 85 0.10884 0.06222 0.12109 0.07512 0.23698 0.19707 0.76509
COF 87 0.12245 0.07654 0.12356 0.07771 0.24490 0.20540 0.76170
COF 89 0.13605 0.09086 0.12118 0.07521 0.24742 0.20805 0.75002

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO