Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (5% of outliers version#02)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1682 objects, 84 outliers (4.99%)

Download raw algorithm results (10.4 MB) Download raw algorithm evaluation table (66.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.49060 0.46382 0.49784 0.47144 0.49102 0.46426 0.85414
KNN 3 0.55238 0.52885 0.56848 0.54579 0.58741 0.56572 0.85112
KNN 12 0.50737 0.48147 0.49622 0.46974 0.59259 0.57118 0.77252
KNNW 4 0.51190 0.48625 0.52833 0.50354 0.52113 0.49595 0.85879
KNNW 7 0.58333 0.56143 0.57332 0.55089 0.60274 0.58186 0.85030
KNNW 10 0.57143 0.54890 0.57660 0.55435 0.60563 0.58490 0.83724
KNNW 16 0.54762 0.52384 0.56744 0.54470 0.62411 0.60435 0.81629
LOF 19 0.45238 0.42359 0.45272 0.42395 0.50000 0.47372 0.83386
LOF 57 0.51190 0.48625 0.52804 0.50323 0.57746 0.55525 0.80974
LOF 73 0.52381 0.49878 0.52031 0.49509 0.58571 0.56394 0.80136
SimplifiedLOF 24 0.52381 0.49878 0.50722 0.48131 0.53623 0.51185 0.84726
SimplifiedLOF 31 0.53571 0.51131 0.51252 0.48689 0.55319 0.52970 0.84040
SimplifiedLOF 54 0.51190 0.48625 0.54845 0.52472 0.57746 0.55525 0.82588
SimplifiedLOF 73 0.52381 0.49878 0.53443 0.50996 0.58571 0.56394 0.81333
LoOP 13 0.57143 0.54890 0.46343 0.43523 0.58974 0.56818 0.80507
LoOP 14 0.58333 0.56143 0.47095 0.44314 0.58442 0.56257 0.80956
LoOP 31 0.50000 0.47372 0.49246 0.46578 0.52500 0.50003 0.85404
LoOP 99 0.52381 0.49878 0.54245 0.51840 0.57143 0.54890 0.83185
LDOF 41 0.50000 0.47372 0.47042 0.44258 0.51389 0.48834 0.84525
LDOF 79 0.53571 0.51131 0.53077 0.50611 0.56164 0.53860 0.83711
LDOF 84 0.53571 0.51131 0.53185 0.50724 0.55782 0.53458 0.83705
ODIN 32 0.28571 0.24817 0.23764 0.19756 0.40693 0.37575 0.79721
ODIN 96 0.36996 0.33685 0.25822 0.21923 0.44898 0.42001 0.77451
ODIN 98 0.37287 0.33990 0.26144 0.22262 0.44898 0.42001 0.77519
ODIN 100 0.37287 0.33990 0.26161 0.22279 0.44898 0.42001 0.77550
FastABOD 13 0.52381 0.49878 0.46321 0.43499 0.52514 0.50018 0.84747
FastABOD 15 0.50000 0.47372 0.46320 0.43499 0.52222 0.49711 0.85013
FastABOD 16 0.50000 0.47372 0.46433 0.43618 0.53846 0.51420 0.84995
FastABOD 25 0.52381 0.49878 0.46501 0.43689 0.53179 0.50718 0.84052
KDEOS 66 0.25000 0.21058 0.15150 0.10690 0.26590 0.22731 0.73184
KDEOS 67 0.26190 0.22311 0.16992 0.12628 0.27160 0.23332 0.73144
KDEOS 71 0.25000 0.21058 0.17220 0.12868 0.25767 0.21865 0.72860
LDF 4 0.08683 0.03882 0.08105 0.03274 0.18721 0.14448 0.63180
LDF 100 0.00340 -0.04899 0.11364 0.06705 0.28571 0.24817 0.70303
INFLO 23 0.51190 0.48625 0.48967 0.46285 0.53086 0.50620 0.84938
INFLO 79 0.53571 0.51131 0.54426 0.52030 0.59155 0.57008 0.82183
INFLO 84 0.54762 0.52384 0.54794 0.52418 0.59155 0.57008 0.82066
COF 5 0.15476 0.11033 0.10273 0.05557 0.20690 0.16521 0.62074
COF 6 0.20238 0.16045 0.14298 0.09793 0.21192 0.17049 0.59384

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2957 objects, 147 outliers (4.97%)

Download raw algorithm results (12.6 MB) Download raw algorithm evaluation table (71.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.45632 0.42788 0.49847 0.47224 0.47909 0.45184 0.87257
KNN 5 0.45878 0.43047 0.50098 0.47488 0.49339 0.46689 0.86553
KNN 14 0.42063 0.39033 0.44449 0.41543 0.51185 0.48631 0.77659
KNNW 9 0.41497 0.38436 0.45117 0.42246 0.44358 0.41447 0.85574
KNNW 11 0.46259 0.43447 0.46958 0.44184 0.48327 0.45624 0.85377
KNNW 17 0.44898 0.42015 0.49084 0.46420 0.53153 0.50702 0.83663
KNNW 18 0.44898 0.42015 0.49004 0.46337 0.53211 0.50763 0.83340
LOF 8 0.09774 0.05054 0.10054 0.05349 0.21243 0.17123 0.74457
LOF 9 0.10245 0.05550 0.10185 0.05486 0.21645 0.17546 0.74048
SimplifiedLOF 9 0.10014 0.05307 0.09229 0.04481 0.17137 0.12802 0.71083
SimplifiedLOF 10 0.09669 0.04944 0.09023 0.04264 0.17270 0.12942 0.70648
LoOP 30 0.20408 0.16244 0.16119 0.11731 0.28509 0.24769 0.77118
LoOP 31 0.20408 0.16244 0.16076 0.11686 0.28761 0.25034 0.77136
LoOP 72 0.21769 0.17676 0.15339 0.10910 0.31073 0.27468 0.75633
LDOF 76 0.19048 0.14813 0.14895 0.10443 0.29064 0.25353 0.75718
LDOF 77 0.19728 0.15529 0.14824 0.10368 0.29146 0.25439 0.75577
ODIN 87 0.39229 0.36050 0.28961 0.25245 0.45055 0.42181 0.80669
ODIN 91 0.39512 0.36348 0.28895 0.25175 0.45378 0.42521 0.80512
ODIN 100 0.40136 0.37004 0.29044 0.25332 0.45378 0.42521 0.80406
FastABOD 32 0.02041 -0.03084 0.10853 0.06190 0.24456 0.20504 0.74374
FastABOD 38 0.02041 -0.03084 0.11037 0.06383 0.25240 0.21329 0.74338
FastABOD 73 0.13605 0.09086 0.11483 0.06852 0.23929 0.19950 0.74248
FastABOD 74 0.13605 0.09086 0.11530 0.06902 0.23964 0.19987 0.74302
KDEOS 10 0.09524 0.04791 0.08853 0.04084 0.18432 0.14165 0.70164
KDEOS 75 0.10884 0.06222 0.08321 0.03525 0.17621 0.13312 0.65828
LDF 1 0.10204 0.05507 0.04796 -0.00184 0.12698 0.08131 0.35387
LDF 5 0.11833 0.07221 0.06222 0.01316 0.12256 0.07666 0.49938
LDF 12 0.06996 0.02131 0.05775 0.00846 0.10526 0.05846 0.55166
INFLO 9 0.10245 0.05550 0.09944 0.05233 0.20640 0.16488 0.74179
COF 36 0.10884 0.06222 0.11488 0.06857 0.25558 0.21664 0.72323
COF 76 0.20408 0.16244 0.13918 0.09414 0.29032 0.25320 0.72254
COF 77 0.20408 0.16244 0.13797 0.09287 0.29282 0.25582 0.72107

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO