Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (20% of outliers version#08)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (28.3 MB) Download raw algorithm evaluation table (74.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.32753 0.15941 0.28916 0.11145 0.41636 0.27045 0.66718
KNN 6 0.32120 0.15150 0.28987 0.11233 0.41553 0.26941 0.66950
KNN 7 0.31804 0.14755 0.28844 0.11054 0.41306 0.26633 0.66981
KNN 10 0.33386 0.16733 0.28485 0.10607 0.40918 0.26148 0.66546
KNNW 12 0.32753 0.15941 0.28085 0.10107 0.40323 0.25404 0.65558
KNNW 13 0.32595 0.15744 0.28091 0.10113 0.40383 0.25479 0.65596
KNNW 17 0.31487 0.14359 0.27966 0.09958 0.40698 0.25873 0.65653
KNNW 20 0.31487 0.14359 0.27838 0.09797 0.40619 0.25773 0.65654
LOF 81 0.23734 0.04668 0.24053 0.05067 0.37518 0.21897 0.59625
LOF 99 0.26582 0.08228 0.24475 0.05594 0.37090 0.21363 0.60294
LOF 100 0.26266 0.07832 0.24498 0.05622 0.37155 0.21443 0.60308
SimplifiedLOF 2 0.22152 0.02690 0.20398 0.00498 0.33413 0.16766 0.47810
SimplifiedLOF 86 0.16297 -0.04628 0.21528 0.01911 0.36339 0.20424 0.54271
SimplifiedLOF 100 0.16772 -0.04035 0.21672 0.02091 0.36062 0.20078 0.54726
LoOP 92 0.22152 0.02690 0.22457 0.03072 0.36577 0.20721 0.56526
LoOP 94 0.21677 0.02097 0.22503 0.03129 0.36716 0.20895 0.56651
LoOP 100 0.21835 0.02294 0.22620 0.03275 0.36401 0.20502 0.56917
LDOF 47 0.21203 0.01503 0.20822 0.01028 0.35365 0.19206 0.53974
LDOF 94 0.20095 0.00119 0.21972 0.02464 0.36127 0.20159 0.54982
LDOF 100 0.20411 0.00514 0.22049 0.02561 0.36000 0.20000 0.55050
ODIN 46 0.23022 0.03778 0.22491 0.03114 0.35579 0.19473 0.56556
ODIN 99 0.23910 0.04887 0.23410 0.04263 0.34803 0.18504 0.57734
ODIN 100 0.23897 0.04871 0.23447 0.04309 0.34795 0.18493 0.57770
FastABOD 3 0.27532 0.09415 0.24479 0.05599 0.35426 0.19283 0.57824
FastABOD 93 0.28006 0.10008 0.24283 0.05354 0.34795 0.18493 0.57139
KDEOS 27 0.21994 0.02492 0.20306 0.00382 0.33455 0.16819 0.50083
KDEOS 30 0.20411 0.00514 0.20428 0.00535 0.33493 0.16867 0.50367
KDEOS 96 0.18987 -0.01266 0.20100 0.00125 0.34970 0.18713 0.52637
KDEOS 97 0.19304 -0.00870 0.20160 0.00200 0.34908 0.18635 0.52692
LDF 75 0.29114 0.11392 0.26289 0.07861 0.37898 0.22372 0.61654
LDF 99 0.28006 0.10008 0.26205 0.07756 0.39624 0.24530 0.62978
LDF 100 0.27532 0.09415 0.26157 0.07697 0.39702 0.24627 0.62949
INFLO 79 0.24209 0.05261 0.23357 0.04196 0.36456 0.20570 0.58237
INFLO 97 0.23892 0.04866 0.23847 0.04809 0.36532 0.20665 0.59373
INFLO 100 0.24209 0.05261 0.23912 0.04890 0.36384 0.20480 0.59456
COF 49 0.22310 0.02888 0.20897 0.01121 0.34765 0.18456 0.53764
COF 62 0.19937 -0.00079 0.21300 0.01625 0.35692 0.19614 0.54286
COF 83 0.18354 -0.02057 0.21079 0.01349 0.36454 0.20567 0.54804
COF 100 0.17722 -0.02848 0.21169 0.01461 0.35961 0.19952 0.55228

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (29.1 MB) Download raw algorithm evaluation table (74.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.32568 0.15710 0.29326 0.11657 0.41030 0.26287 0.66688
KNN 6 0.32281 0.15352 0.29616 0.12020 0.41052 0.26315 0.67310
KNN 7 0.32425 0.15531 0.29490 0.11862 0.41237 0.26546 0.67169
KNNW 14 0.30416 0.13020 0.28102 0.10127 0.39495 0.24368 0.65060
KNNW 21 0.28407 0.10509 0.27855 0.09819 0.40624 0.25780 0.65316
KNNW 33 0.28407 0.10509 0.27486 0.09358 0.40363 0.25454 0.65496
LOF 2 0.23673 0.04591 0.21007 0.01259 0.33461 0.16827 0.51332
LOF 9 0.23386 0.04232 0.22277 0.02846 0.35441 0.19301 0.56668
LOF 36 0.18364 -0.02044 0.20217 0.00271 0.36596 0.20745 0.54331
SimplifiedLOF 1 0.24103 0.05129 0.22433 0.03041 0.33341 0.16677 0.51939
SimplifiedLOF 2 0.24534 0.05667 0.21753 0.02191 0.33349 0.16687 0.51763
SimplifiedLOF 53 0.16930 -0.03838 0.19668 -0.00415 0.35335 0.19168 0.52838
SimplifiedLOF 100 0.14634 -0.06707 0.19473 -0.00658 0.35789 0.19737 0.51413
LoOP 1 0.23386 0.04232 0.23630 0.04538 0.33333 0.16667 0.51854
LoOP 2 0.23673 0.04591 0.22956 0.03695 0.33333 0.16667 0.53452
LoOP 16 0.21521 0.01901 0.21330 0.01662 0.34436 0.18045 0.54437
LoOP 88 0.17073 -0.03659 0.20702 0.00877 0.36180 0.20225 0.53868
LDOF 2 0.24247 0.05308 0.22460 0.03075 0.33333 0.16667 0.48124
LDOF 78 0.19082 -0.01148 0.20806 0.01008 0.35926 0.19907 0.54134
LDOF 100 0.18508 -0.01865 0.20713 0.00891 0.36074 0.20093 0.53603
ODIN 5 0.21169 0.01461 0.21644 0.02055 0.35893 0.19866 0.55123
ODIN 99 0.25251 0.06564 0.23320 0.04150 0.35078 0.18848 0.57728
ODIN 100 0.25138 0.06423 0.23337 0.04171 0.34963 0.18703 0.57700
FastABOD 20 0.24677 0.05846 0.21879 0.02348 0.34798 0.18497 0.54316
FastABOD 43 0.25395 0.06743 0.21958 0.02448 0.34776 0.18470 0.54512
FastABOD 93 0.24964 0.06205 0.22361 0.02951 0.34789 0.18486 0.54843
FastABOD 97 0.24964 0.06205 0.22360 0.02950 0.34789 0.18486 0.54851
KDEOS 3 0.21090 0.01363 0.21492 0.01865 0.33634 0.17042 0.50430
KDEOS 63 0.21377 0.01722 0.20849 0.01061 0.34151 0.17689 0.53672
KDEOS 97 0.19225 -0.00968 0.20486 0.00607 0.35261 0.19076 0.54476
KDEOS 100 0.18938 -0.01327 0.20388 0.00485 0.35337 0.19171 0.54342
LDF 5 0.25968 0.07461 0.23306 0.04132 0.33948 0.17434 0.56269
LDF 8 0.24534 0.05667 0.24294 0.05367 0.37902 0.22377 0.60560
LDF 9 0.22812 0.03515 0.23926 0.04908 0.37961 0.22452 0.60142
INFLO 9 0.22382 0.02977 0.21184 0.01480 0.35305 0.19131 0.54389
INFLO 12 0.21090 0.01363 0.21194 0.01493 0.35967 0.19959 0.55044
INFLO 19 0.20660 0.00825 0.20951 0.01189 0.36251 0.20314 0.55389
INFLO 96 0.16069 -0.04914 0.20328 0.00410 0.36958 0.21198 0.52698
COF 2 0.25968 0.07461 0.22740 0.03425 0.34635 0.18294 0.54086
COF 87 0.15638 -0.05452 0.18851 -0.01436 0.34954 0.18692 0.49420

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (27.4 MB) Download raw algorithm evaluation table (72.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.44462 0.30578 0.44318 0.30398 0.45983 0.32479 0.73138
KNN 10 0.44937 0.31171 0.44181 0.30226 0.46042 0.32553 0.74038
KNN 50 0.41930 0.27413 0.42156 0.27695 0.47390 0.34237 0.74126
KNN 98 0.41930 0.27413 0.41600 0.27000 0.47099 0.33873 0.74300
KNNW 13 0.44304 0.30380 0.44060 0.30075 0.46373 0.32966 0.73519
KNNW 14 0.44462 0.30578 0.44054 0.30068 0.46371 0.32963 0.73578
KNNW 99 0.41772 0.27215 0.42268 0.27835 0.47217 0.34021 0.74153
KNNW 100 0.41772 0.27215 0.42267 0.27833 0.47217 0.34021 0.74156
LOF 90 0.30063 0.12579 0.27520 0.09400 0.36553 0.20691 0.61072
LOF 99 0.30063 0.12579 0.28018 0.10023 0.37154 0.21443 0.61805
LOF 100 0.29905 0.12381 0.28064 0.10080 0.37107 0.21384 0.61876
SimplifiedLOF 53 0.24367 0.05459 0.24495 0.05619 0.35082 0.18853 0.56448
SimplifiedLOF 100 0.27215 0.09019 0.27083 0.08854 0.34966 0.18708 0.58484
LoOP 37 0.19620 -0.00475 0.21227 0.01534 0.34288 0.17860 0.53232
LoOP 98 0.25158 0.06448 0.23966 0.04957 0.33397 0.16746 0.55452
LoOP 100 0.24842 0.06052 0.24048 0.05060 0.33536 0.16920 0.55599
LDOF 2 0.22785 0.03481 0.21550 0.01938 0.33413 0.16766 0.44861
LDOF 69 0.17722 -0.02848 0.19297 -0.00878 0.33547 0.16934 0.47696
LDOF 100 0.19620 -0.00475 0.20337 0.00421 0.33423 0.16778 0.49250
ODIN 5 0.17955 -0.02556 0.20581 0.00726 0.34955 0.18694 0.52808
ODIN 22 0.14446 -0.06942 0.19840 -0.00200 0.36701 0.20876 0.52916
ODIN 88 0.19854 -0.00183 0.20517 0.00646 0.35874 0.19843 0.53762
ODIN 100 0.19993 -0.00009 0.20530 0.00662 0.35899 0.19874 0.53731
FastABOD 3 0.41139 0.26424 0.41161 0.26452 0.46095 0.32618 0.72824
FastABOD 4 0.41456 0.26820 0.41227 0.26533 0.45940 0.32425 0.72661
FastABOD 12 0.40190 0.25237 0.40987 0.26234 0.46345 0.32931 0.72156
KDEOS 93 0.21361 0.01701 0.21728 0.02160 0.35384 0.19229 0.55817
KDEOS 99 0.20886 0.01108 0.21961 0.02451 0.35709 0.19636 0.56064
KDEOS 100 0.20886 0.01108 0.21957 0.02446 0.35648 0.19561 0.56091
LDF 94 0.39557 0.24446 0.40353 0.25442 0.44005 0.30006 0.70852
LDF 100 0.38608 0.23259 0.40992 0.26240 0.44662 0.30828 0.71321
INFLO 96 0.26266 0.07832 0.25624 0.07030 0.43101 0.28876 0.60619
INFLO 97 0.26108 0.07634 0.25697 0.07122 0.43325 0.29156 0.60813
INFLO 99 0.26108 0.07634 0.25794 0.07242 0.43397 0.29247 0.60767
INFLO 100 0.26266 0.07832 0.25761 0.07202 0.43429 0.29286 0.60794
COF 92 0.31804 0.14755 0.33448 0.16810 0.37174 0.21468 0.61643
COF 93 0.31013 0.13766 0.33542 0.16928 0.37438 0.21797 0.61603
COF 99 0.31329 0.14161 0.34197 0.17746 0.37212 0.21515 0.62093
COF 100 0.31329 0.14161 0.34257 0.17821 0.37113 0.21392 0.62062

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (28.6 MB) Download raw algorithm evaluation table (72.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.42898 0.28623 0.43191 0.28988 0.44456 0.30570 0.73005
KNN 13 0.42324 0.27905 0.43368 0.29210 0.45370 0.31712 0.73979
KNN 88 0.41750 0.27188 0.41180 0.26475 0.46380 0.32975 0.74562
KNN 99 0.41463 0.26829 0.41036 0.26295 0.46578 0.33223 0.74417
KNNW 10 0.43902 0.29878 0.42609 0.28261 0.44150 0.30188 0.72089
KNNW 18 0.42898 0.28623 0.43169 0.28962 0.44855 0.31069 0.73239
KNNW 80 0.41463 0.26829 0.41890 0.27362 0.46145 0.32681 0.74214
KNNW 100 0.41463 0.26829 0.41632 0.27040 0.45969 0.32462 0.74309
LOF 100 0.28838 0.11047 0.26722 0.08403 0.37025 0.21281 0.61032
SimplifiedLOF 2 0.25108 0.06385 0.21385 0.01732 0.33333 0.16667 0.49582
SimplifiedLOF 99 0.21808 0.02260 0.22632 0.03290 0.35126 0.18907 0.56336
SimplifiedLOF 100 0.21951 0.02439 0.22927 0.03658 0.35115 0.18893 0.56457
LoOP 1 0.22812 0.03515 0.23799 0.04749 0.33333 0.16667 0.50378
LoOP 74 0.18364 -0.02044 0.20143 0.00178 0.33914 0.17392 0.52612
LoOP 100 0.20660 0.00825 0.21669 0.02086 0.33484 0.16854 0.54006
LDOF 2 0.24534 0.05667 0.22551 0.03189 0.33333 0.16667 0.45865
LDOF 76 0.16786 -0.04017 0.18104 -0.02369 0.34075 0.17593 0.47282
LDOF 100 0.15925 -0.05093 0.18861 -0.01424 0.34046 0.17557 0.48418
ODIN 1 0.20596 0.00744 0.21497 0.01872 0.35348 0.19185 0.54610
ODIN 13 0.17152 -0.03560 0.20670 0.00837 0.36324 0.20405 0.54651
ODIN 21 0.15973 -0.05033 0.20683 0.00854 0.37205 0.21507 0.54647
FastABOD 70 0.38594 0.23242 0.39123 0.23904 0.45243 0.31553 0.71653
FastABOD 91 0.38451 0.23063 0.39245 0.24056 0.45446 0.31807 0.71693
FastABOD 98 0.38451 0.23063 0.39254 0.24067 0.45433 0.31791 0.71696
FastABOD 100 0.38451 0.23063 0.39258 0.24073 0.45433 0.31791 0.71696
KDEOS 3 0.20516 0.00646 0.20758 0.00947 0.33408 0.16760 0.49184
KDEOS 100 0.19225 -0.00968 0.20932 0.01165 0.35220 0.19025 0.54151
LDF 99 0.35581 0.19476 0.37957 0.22447 0.40994 0.26242 0.68467
LDF 100 0.36729 0.20911 0.38081 0.22602 0.40891 0.26114 0.68527
INFLO 90 0.24247 0.05308 0.24374 0.05468 0.43750 0.29687 0.60919
INFLO 97 0.23529 0.04412 0.24648 0.05809 0.44054 0.30067 0.61266
INFLO 100 0.24103 0.05129 0.24782 0.05977 0.44017 0.30021 0.60804
COF 100 0.25825 0.07281 0.25167 0.06458 0.36424 0.20530 0.59954

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO