Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (20% of outliers version#07)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (28.2 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.30222 0.12777 0.27389 0.09236 0.37851 0.22314 0.61381
KNN 5 0.28797 0.10997 0.28413 0.10516 0.41310 0.26637 0.65621
KNN 8 0.27690 0.09612 0.28299 0.10374 0.41546 0.26933 0.66082
KNN 9 0.28006 0.10008 0.28278 0.10348 0.41322 0.26653 0.66168
KNNW 6 0.29905 0.12381 0.26950 0.08688 0.37292 0.21615 0.61741
KNNW 12 0.28639 0.10799 0.27582 0.09477 0.40016 0.25020 0.64260
KNNW 16 0.27215 0.09019 0.27490 0.09363 0.40735 0.25919 0.64596
KNNW 23 0.27532 0.09415 0.27364 0.09205 0.40530 0.25663 0.64821
LOF 55 0.18987 -0.01266 0.23013 0.03766 0.36898 0.21123 0.55895
LOF 97 0.24051 0.05063 0.23812 0.04765 0.36797 0.20996 0.58264
LOF 99 0.24525 0.05657 0.23797 0.04746 0.36717 0.20897 0.58252
SimplifiedLOF 2 0.25475 0.06843 0.22184 0.02729 0.33431 0.16789 0.50350
SimplifiedLOF 36 0.19304 -0.00870 0.20664 0.00830 0.35161 0.18951 0.53667
SimplifiedLOF 87 0.15981 -0.05024 0.21718 0.02147 0.36224 0.20280 0.52916
LoOP 2 0.22468 0.03085 0.21508 0.01886 0.33333 0.16667 0.50067
LoOP 95 0.21519 0.01899 0.23196 0.03994 0.36463 0.20579 0.55600
LoOP 97 0.21361 0.01701 0.23220 0.04025 0.36323 0.20404 0.55691
LoOP 100 0.21677 0.02097 0.23211 0.04014 0.36397 0.20497 0.55754
LDOF 4 0.22943 0.03679 0.20965 0.01206 0.33430 0.16788 0.49186
LDOF 57 0.20411 0.00514 0.21603 0.02004 0.35743 0.19679 0.55139
LDOF 93 0.18987 -0.01266 0.22732 0.03415 0.36352 0.20440 0.54576
LDOF 100 0.19462 -0.00672 0.22880 0.03600 0.36024 0.20030 0.54527
ODIN 28 0.22690 0.03362 0.23099 0.03874 0.36222 0.20277 0.57087
ODIN 96 0.24721 0.05902 0.24199 0.05249 0.36042 0.20053 0.58968
ODIN 97 0.24809 0.06012 0.24201 0.05251 0.36087 0.20108 0.58960
ODIN 100 0.25263 0.06578 0.24174 0.05218 0.35902 0.19878 0.58906
FastABOD 4 0.27690 0.09612 0.24258 0.05322 0.34106 0.17632 0.56311
FastABOD 5 0.27532 0.09415 0.24355 0.05443 0.34301 0.17876 0.56137
FastABOD 43 0.27848 0.09810 0.24448 0.05560 0.33910 0.17388 0.55785
FastABOD 82 0.26741 0.08426 0.24676 0.05844 0.33947 0.17434 0.56029
KDEOS 28 0.22627 0.03283 0.21048 0.01310 0.33377 0.16722 0.50809
KDEOS 42 0.22152 0.02690 0.21539 0.01924 0.33608 0.17010 0.52742
KDEOS 93 0.18987 -0.01266 0.20724 0.00905 0.35181 0.18976 0.53620
KDEOS 100 0.19778 -0.00277 0.20646 0.00808 0.35317 0.19146 0.53490
LDF 3 0.24209 0.05261 0.21684 0.02105 0.33619 0.17024 0.52897
LDF 100 0.23576 0.04470 0.24522 0.05653 0.38787 0.23484 0.60749
INFLO 60 0.21044 0.01305 0.23062 0.03827 0.36438 0.20547 0.55274
INFLO 97 0.24209 0.05261 0.23739 0.04674 0.36380 0.20476 0.58017
INFLO 100 0.24209 0.05261 0.23750 0.04688 0.36369 0.20461 0.58055
COF 2 0.24525 0.05657 0.21806 0.02258 0.33360 0.16700 0.50032
COF 33 0.21361 0.01701 0.20756 0.00945 0.34058 0.17573 0.52538
COF 96 0.17089 -0.03639 0.19876 -0.00155 0.36011 0.20014 0.50520

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (29.1 MB) Download raw algorithm evaluation table (74.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.30273 0.12841 0.28043 0.10054 0.40171 0.25214 0.64672
KNN 7 0.29555 0.11944 0.28753 0.10941 0.41433 0.26791 0.66618
KNN 9 0.28981 0.11227 0.28660 0.10825 0.41896 0.27370 0.67300
KNN 10 0.28551 0.10689 0.28332 0.10415 0.42160 0.27700 0.67296
KNNW 8 0.28838 0.11047 0.27228 0.09035 0.39284 0.24105 0.63557
KNNW 13 0.28694 0.10868 0.27721 0.09651 0.40108 0.25134 0.65143
KNNW 39 0.27260 0.09075 0.27487 0.09359 0.41398 0.26748 0.66389
KNNW 44 0.27403 0.09254 0.27365 0.09206 0.41602 0.27003 0.66256
LOF 2 0.24821 0.06026 0.22578 0.03223 0.33381 0.16727 0.53492
LOF 3 0.25108 0.06385 0.22416 0.03020 0.33511 0.16889 0.52693
LOF 13 0.21521 0.01901 0.21234 0.01543 0.34877 0.18596 0.54902
LOF 78 0.14778 -0.06528 0.19868 -0.00165 0.36482 0.20603 0.52603
SimplifiedLOF 3 0.25538 0.06923 0.22606 0.03258 0.33333 0.16667 0.53168
SimplifiedLOF 4 0.25825 0.07281 0.22338 0.02922 0.33405 0.16757 0.52010
SimplifiedLOF 100 0.14060 -0.07425 0.19160 -0.01050 0.35905 0.19881 0.50172
LoOP 1 0.23386 0.04232 0.23405 0.04257 0.33333 0.16667 0.52370
LoOP 3 0.25108 0.06385 0.22738 0.03422 0.33333 0.16667 0.53482
LoOP 16 0.22382 0.02977 0.21179 0.01474 0.34131 0.17664 0.53902
LoOP 100 0.15925 -0.05093 0.19542 -0.00573 0.36076 0.20095 0.51966
LDOF 2 0.24247 0.05308 0.23000 0.03750 0.33373 0.16717 0.49369
LDOF 3 0.24821 0.06026 0.21917 0.02396 0.33607 0.17009 0.50264
LDOF 80 0.19082 -0.01148 0.19694 -0.00382 0.35343 0.19178 0.52137
LDOF 100 0.17073 -0.03659 0.19550 -0.00562 0.35740 0.19676 0.51726
ODIN 56 0.22712 0.03390 0.23320 0.04150 0.36284 0.20355 0.57323
ODIN 95 0.24369 0.05461 0.23618 0.04522 0.35670 0.19588 0.58233
ODIN 99 0.24112 0.05140 0.23644 0.04555 0.35782 0.19727 0.58237
ODIN 100 0.24247 0.05308 0.23635 0.04544 0.35794 0.19742 0.58253
FastABOD 3 0.23960 0.04950 0.21205 0.01507 0.34618 0.18273 0.53983
FastABOD 86 0.23816 0.04770 0.22290 0.02863 0.34937 0.18671 0.55037
FastABOD 98 0.23960 0.04950 0.22344 0.02930 0.34937 0.18671 0.55078
FastABOD 100 0.23960 0.04950 0.22357 0.02946 0.34937 0.18671 0.55078
KDEOS 3 0.22382 0.02977 0.21912 0.02390 0.33400 0.16749 0.51006
KDEOS 64 0.23099 0.03874 0.21382 0.01727 0.33709 0.17136 0.53545
KDEOS 67 0.22382 0.02977 0.21405 0.01756 0.34010 0.17512 0.53862
KDEOS 100 0.18508 -0.01865 0.19936 -0.00080 0.34990 0.18737 0.52710
LDF 7 0.27547 0.09433 0.26287 0.07859 0.37869 0.22337 0.60445
INFLO 2 0.23816 0.04770 0.22046 0.02557 0.33572 0.16965 0.53545
INFLO 6 0.23960 0.04950 0.21878 0.02347 0.33552 0.16939 0.52875
INFLO 15 0.22525 0.03156 0.20668 0.00835 0.35557 0.19446 0.53986
INFLO 99 0.15208 -0.05990 0.19960 -0.00050 0.37025 0.21281 0.51407
COF 2 0.24964 0.06205 0.23112 0.03889 0.34443 0.18053 0.54655
COF 71 0.14491 -0.06887 0.18367 -0.02041 0.35152 0.18939 0.48812

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (27.5 MB) Download raw algorithm evaluation table (71.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.44620 0.30775 0.44456 0.30570 0.45322 0.31652 0.74362
KNN 9 0.44778 0.30973 0.44222 0.30277 0.45466 0.31832 0.74485
KNN 74 0.42089 0.27611 0.41787 0.27233 0.46082 0.32602 0.74844
KNN 98 0.42247 0.27809 0.41488 0.26860 0.46389 0.32986 0.74802
KNNW 14 0.44937 0.31171 0.44181 0.30226 0.44947 0.31183 0.73778
KNNW 16 0.44462 0.30578 0.44224 0.30280 0.45072 0.31340 0.73934
KNNW 56 0.42722 0.28402 0.42805 0.28506 0.46354 0.32943 0.74602
KNNW 100 0.42247 0.27809 0.42234 0.27793 0.46237 0.32796 0.74765
LOF 89 0.31329 0.14161 0.29024 0.11279 0.37661 0.22076 0.62593
LOF 100 0.30696 0.13370 0.29541 0.11926 0.38109 0.22637 0.63208
SimplifiedLOF 95 0.28323 0.10403 0.28467 0.10584 0.35335 0.19169 0.59470
SimplifiedLOF 100 0.28323 0.10403 0.28673 0.10841 0.35459 0.19324 0.59724
LoOP 70 0.25791 0.07239 0.24462 0.05578 0.34213 0.17767 0.55885
LoOP 86 0.26899 0.08623 0.25123 0.06403 0.33938 0.17422 0.56493
LoOP 87 0.26582 0.08228 0.25165 0.06457 0.34069 0.17586 0.56560
LoOP 100 0.26424 0.08030 0.25537 0.06921 0.33900 0.17375 0.56430
LDOF 2 0.23418 0.04272 0.22247 0.02809 0.33421 0.16777 0.46115
LDOF 39 0.16456 -0.04430 0.19135 -0.01082 0.33505 0.16882 0.46623
LDOF 100 0.21203 0.01503 0.21345 0.01681 0.33441 0.16801 0.50358
ODIN 23 0.15929 -0.05089 0.20718 0.00897 0.36973 0.21216 0.54648
ODIN 81 0.21586 0.01982 0.21354 0.01693 0.36353 0.20441 0.54995
ODIN 92 0.21123 0.01404 0.21424 0.01779 0.36212 0.20264 0.55062
ODIN 99 0.20904 0.01130 0.21459 0.01823 0.36196 0.20244 0.55055
FastABOD 3 0.40823 0.26028 0.41665 0.27081 0.45326 0.31658 0.72665
FastABOD 6 0.40032 0.25040 0.41642 0.27053 0.45711 0.32139 0.72755
FastABOD 20 0.40665 0.25831 0.41740 0.27175 0.45439 0.31799 0.72806
FastABOD 86 0.40506 0.25633 0.41880 0.27350 0.45449 0.31812 0.72775
KDEOS 86 0.21677 0.02097 0.22087 0.02608 0.35350 0.19187 0.55939
KDEOS 99 0.21203 0.01503 0.22489 0.03111 0.35946 0.19932 0.56690
KDEOS 100 0.21203 0.01503 0.22523 0.03154 0.35837 0.19796 0.56732
LDF 98 0.38608 0.23259 0.38904 0.23629 0.44197 0.30246 0.69822
LDF 100 0.39241 0.24051 0.39135 0.23918 0.44277 0.30347 0.69811
INFLO 99 0.28323 0.10403 0.27233 0.09042 0.43986 0.29982 0.61531
INFLO 100 0.28323 0.10403 0.27206 0.09007 0.43837 0.29796 0.61538
COF 95 0.30380 0.12975 0.34918 0.18647 0.37442 0.21803 0.63036
COF 100 0.32437 0.15546 0.35452 0.19315 0.37265 0.21582 0.63576

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (28.6 MB) Download raw algorithm evaluation table (73.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 10 0.44620 0.30775 0.45055 0.31318 0.46566 0.33208 0.75878
KNN 15 0.44620 0.30775 0.45208 0.31510 0.46961 0.33702 0.76035
KNN 33 0.43759 0.29699 0.44549 0.30686 0.46756 0.33445 0.76283
KNN 80 0.43759 0.29699 0.43370 0.29212 0.47337 0.34171 0.76166
KNNW 18 0.44189 0.30237 0.45159 0.31449 0.46799 0.33499 0.75642
KNNW 20 0.44333 0.30416 0.45122 0.31403 0.47024 0.33780 0.75754
KNNW 26 0.43472 0.29340 0.44985 0.31231 0.47206 0.34007 0.75891
KNNW 52 0.43902 0.29878 0.44651 0.30813 0.46971 0.33714 0.76207
LOF 95 0.30129 0.12661 0.27492 0.09365 0.38149 0.22686 0.62988
LOF 98 0.30703 0.13379 0.27730 0.09663 0.37941 0.22426 0.63192
LOF 100 0.30703 0.13379 0.27916 0.09895 0.38028 0.22535 0.63277
SimplifiedLOF 99 0.24247 0.05308 0.23865 0.04831 0.36223 0.20279 0.59257
SimplifiedLOF 100 0.24247 0.05308 0.23996 0.04995 0.36214 0.20267 0.59337
LoOP 1 0.23386 0.04232 0.24118 0.05147 0.33333 0.16667 0.51465
LoOP 100 0.22956 0.03694 0.22792 0.03490 0.35446 0.19307 0.56925
LDOF 2 0.22095 0.02618 0.22080 0.02600 0.33333 0.16667 0.45236
LDOF 98 0.16499 -0.04376 0.19289 -0.00889 0.34184 0.17730 0.50577
LDOF 100 0.16643 -0.04197 0.19394 -0.00757 0.34175 0.17718 0.50753
ODIN 1 0.20669 0.00837 0.21781 0.02226 0.35642 0.19553 0.55330
ODIN 21 0.17153 -0.03559 0.21542 0.01927 0.37525 0.21907 0.56837
ODIN 26 0.17088 -0.03640 0.21389 0.01736 0.37878 0.22347 0.56593
ODIN 100 0.22265 0.02832 0.21149 0.01436 0.36016 0.20020 0.55583
FastABOD 70 0.40316 0.25395 0.40679 0.25849 0.46747 0.33433 0.72284
FastABOD 74 0.40316 0.25395 0.40746 0.25932 0.46888 0.33610 0.72292
FastABOD 94 0.40316 0.25395 0.40805 0.26006 0.46858 0.33572 0.72319
FastABOD 98 0.40316 0.25395 0.40807 0.26008 0.46798 0.33497 0.72313
KDEOS 2 0.21112 0.01389 0.20913 0.01141 0.33500 0.16874 0.51784
KDEOS 94 0.19225 -0.00968 0.21501 0.01876 0.36239 0.20298 0.56307
KDEOS 100 0.19656 -0.00430 0.21821 0.02276 0.36183 0.20229 0.56741
LDF 97 0.38594 0.23242 0.40507 0.25634 0.41376 0.26721 0.70215
LDF 99 0.39598 0.24498 0.40805 0.26006 0.41301 0.26626 0.70436
LDF 100 0.39455 0.24319 0.40810 0.26012 0.41353 0.26691 0.70479
INFLO 98 0.26255 0.07819 0.25360 0.06699 0.45042 0.31302 0.62215
INFLO 100 0.25825 0.07281 0.25505 0.06881 0.45134 0.31418 0.62277
COF 92 0.28264 0.10330 0.25605 0.07006 0.37822 0.22278 0.61476
COF 97 0.28981 0.11227 0.26290 0.07863 0.37423 0.21779 0.61806
COF 100 0.28551 0.10689 0.26667 0.08334 0.37488 0.21860 0.62097

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO