Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (10% of outliers version#04)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1775 objects, 177 outliers (9.97%)

Download raw algorithm results (13.0 MB) Download raw algorithm evaluation table (73.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.48766 0.43091 0.52619 0.47371 0.49123 0.43487 0.82090
KNN 5 0.49840 0.44284 0.53300 0.48127 0.51444 0.46065 0.81317
KNN 7 0.49894 0.44344 0.52770 0.47539 0.51145 0.45734 0.78478
KNN 21 0.47522 0.41710 0.50016 0.44480 0.52698 0.47459 0.72953
KNNW 7 0.49718 0.44148 0.48999 0.43350 0.52535 0.47277 0.82718
KNNW 11 0.52542 0.47286 0.54601 0.49572 0.53710 0.48583 0.81149
KNNW 12 0.51977 0.46658 0.54998 0.50013 0.54355 0.49300 0.80743
KNNW 16 0.51412 0.46031 0.55208 0.50247 0.53285 0.48110 0.79390
LOF 53 0.48023 0.42265 0.53052 0.47852 0.51525 0.46156 0.78926
LOF 94 0.51977 0.46658 0.56411 0.51583 0.56429 0.51602 0.77558
LOF 96 0.51977 0.46658 0.56935 0.52165 0.57971 0.53316 0.77489
LOF 98 0.51977 0.46658 0.56884 0.52108 0.58156 0.53521 0.77343
SimplifiedLOF 55 0.51412 0.46031 0.53882 0.48774 0.51832 0.46497 0.80401
SimplifiedLOF 95 0.53107 0.47913 0.57181 0.52439 0.56875 0.52098 0.78832
SimplifiedLOF 96 0.53107 0.47913 0.57578 0.52879 0.57823 0.53151 0.78810
SimplifiedLOF 97 0.52542 0.47286 0.57600 0.52904 0.57823 0.53151 0.78732
LoOP 31 0.46893 0.41010 0.40129 0.33497 0.48094 0.42345 0.73353
LoOP 93 0.47458 0.41638 0.47703 0.41910 0.47458 0.41638 0.79369
LoOP 97 0.45763 0.39755 0.48143 0.42399 0.46567 0.40649 0.79391
LoOP 99 0.45763 0.39755 0.48307 0.42581 0.46567 0.40649 0.79370
LDOF 71 0.47458 0.41638 0.45393 0.39345 0.47863 0.42088 0.77527
LDOF 96 0.45198 0.39128 0.46581 0.40664 0.46575 0.40658 0.79307
ODIN 7 0.20334 0.11510 0.18050 0.08973 0.31978 0.24443 0.69824
ODIN 25 0.31751 0.24192 0.22475 0.13888 0.35354 0.28193 0.68644
ODIN 30 0.32839 0.25400 0.22248 0.13636 0.36031 0.28946 0.68309
ODIN 42 0.33226 0.25830 0.21396 0.12689 0.34521 0.27268 0.68472
FastABOD 18 0.46328 0.40383 0.41463 0.34980 0.50676 0.45212 0.78986
FastABOD 23 0.49718 0.44148 0.42127 0.35717 0.50286 0.44779 0.79241
FastABOD 25 0.49153 0.43521 0.42673 0.36323 0.50641 0.45174 0.79410
KDEOS 15 0.23729 0.15281 0.16692 0.07465 0.24625 0.16276 0.61374
KDEOS 18 0.22599 0.14026 0.16996 0.07802 0.24060 0.15649 0.63723
KDEOS 25 0.20339 0.11515 0.16923 0.07721 0.25239 0.16958 0.65322
KDEOS 66 0.18079 0.09005 0.16598 0.07360 0.25832 0.17617 0.64556
LDF 98 0.32203 0.24694 0.19176 0.10223 0.39059 0.32309 0.68634
LDF 100 0.34463 0.27204 0.19609 0.10704 0.39059 0.32309 0.68745
INFLO 55 0.45198 0.39128 0.49970 0.44428 0.46667 0.40759 0.80070
INFLO 96 0.50282 0.44776 0.55554 0.50631 0.53425 0.48266 0.79731
COF 4 0.20904 0.12143 0.16032 0.06731 0.23869 0.15437 0.61584
COF 9 0.25989 0.17791 0.16846 0.07635 0.26136 0.17955 0.58992
COF 52 0.22599 0.14026 0.18211 0.09151 0.24615 0.16266 0.57761

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 3122 objects, 312 outliers (9.99%)

Download raw algorithm results (13.7 MB) Download raw algorithm evaluation table (74.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.43989 0.37770 0.46889 0.40992 0.47633 0.41819 0.84976
KNN 9 0.47097 0.41223 0.52579 0.47313 0.47157 0.41290 0.83365
KNN 10 0.46096 0.40110 0.53577 0.48422 0.50895 0.45442 0.82235
KNNW 13 0.44231 0.38039 0.48794 0.43109 0.46578 0.40647 0.84718
KNNW 21 0.49038 0.43380 0.52880 0.47648 0.49196 0.43555 0.83287
KNNW 24 0.47756 0.41956 0.53243 0.48052 0.48734 0.43042 0.82672
LOF 10 0.12547 0.02836 0.13978 0.04426 0.26800 0.18673 0.65436
LOF 30 0.12886 0.03213 0.12344 0.02611 0.22967 0.14414 0.58321
SimplifiedLOF 3 0.06604 -0.03766 0.10532 0.00598 0.24025 0.15589 0.54054
SimplifiedLOF 19 0.12687 0.02992 0.12759 0.03072 0.23271 0.14752 0.61706
SimplifiedLOF 30 0.12886 0.03213 0.12495 0.02779 0.22833 0.14264 0.59521
LoOP 72 0.22756 0.14180 0.16369 0.07084 0.28020 0.20028 0.66373
LoOP 78 0.21474 0.12755 0.16780 0.07540 0.29042 0.21163 0.67345
LoOP 81 0.20833 0.12043 0.16689 0.07438 0.29879 0.22093 0.67300
LoOP 88 0.21154 0.12399 0.16730 0.07485 0.29307 0.21458 0.67570
LDOF 75 0.22115 0.13468 0.16168 0.06860 0.27193 0.19109 0.64947
LDOF 99 0.20833 0.12043 0.16675 0.07423 0.29576 0.21757 0.66983
LDOF 100 0.20833 0.12043 0.16696 0.07446 0.29576 0.21757 0.66983
ODIN 16 0.23810 0.15350 0.19325 0.10367 0.31156 0.23512 0.71177
ODIN 87 0.35686 0.28545 0.24096 0.15668 0.35932 0.28819 0.69016
FastABOD 33 0.13782 0.04209 0.18495 0.09446 0.34686 0.27434 0.73484
FastABOD 35 0.10897 0.01004 0.18309 0.09238 0.34985 0.27766 0.73227
FastABOD 74 0.15705 0.06346 0.18746 0.09724 0.34015 0.26688 0.73210
FastABOD 99 0.14744 0.05277 0.18905 0.09901 0.34167 0.26857 0.73429
KDEOS 2 0.12179 0.02429 0.11545 0.01723 0.24248 0.15837 0.57196
KDEOS 4 0.08120 -0.02082 0.10900 0.01008 0.25027 0.16703 0.55542
KDEOS 11 0.10256 0.00292 0.11981 0.02208 0.22782 0.14209 0.59140
KDEOS 20 0.10577 0.00648 0.11761 0.01964 0.23604 0.15122 0.59947
LDF 1 0.19872 0.10975 0.10729 0.00817 0.20189 0.11328 0.37700
LDF 2 0.19231 0.10263 0.11977 0.02203 0.22951 0.14396 0.44208
LDF 3 0.19872 0.10975 0.12197 0.02448 0.21405 0.12678 0.48153
LDF 35 0.11978 0.02204 0.10316 0.00358 0.18177 0.09091 0.51248
INFLO 9 0.12863 0.03188 0.12914 0.03245 0.24733 0.16376 0.62031
INFLO 10 0.12547 0.02836 0.12909 0.03239 0.24943 0.16610 0.62534
INFLO 30 0.12886 0.03213 0.12435 0.02713 0.23151 0.14618 0.59511
COF 84 0.19551 0.10619 0.15702 0.06343 0.26515 0.18355 0.65331
COF 87 0.22115 0.13468 0.15798 0.06449 0.27307 0.19236 0.64840

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO