Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (20% of outliers version#01)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (28.2 MB) Download raw algorithm evaluation table (73.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.28481 0.10601 0.27237 0.09046 0.38744 0.23430 0.62269
KNN 6 0.27532 0.09415 0.27983 0.09979 0.40191 0.25239 0.64429
KNN 12 0.28165 0.10206 0.27450 0.09313 0.40240 0.25300 0.64624
KNN 13 0.28006 0.10008 0.27376 0.09220 0.40538 0.25672 0.64570
KNNW 1 0.27373 0.09217 0.24558 0.05697 0.33593 0.16991 0.53461
KNNW 14 0.27215 0.09019 0.27206 0.09007 0.39743 0.24679 0.63340
KNNW 28 0.27215 0.09019 0.26932 0.08664 0.39811 0.24764 0.63834
KNNW 33 0.27373 0.09217 0.26830 0.08537 0.40067 0.25084 0.63812
LOF 2 0.24367 0.05459 0.23841 0.04802 0.33550 0.16937 0.53006
LOF 3 0.24684 0.05854 0.23470 0.04337 0.33713 0.17142 0.52498
LOF 87 0.20728 0.00910 0.22604 0.03255 0.36081 0.20101 0.55817
LOF 100 0.21835 0.02294 0.22669 0.03336 0.35834 0.19793 0.56224
SimplifiedLOF 2 0.25791 0.07239 0.23368 0.04210 0.33413 0.16766 0.52602
SimplifiedLOF 3 0.24842 0.06052 0.23605 0.04507 0.33360 0.16700 0.52064
SimplifiedLOF 93 0.15823 -0.05222 0.20489 0.00611 0.35252 0.19065 0.50126
LoOP 2 0.25000 0.06250 0.22296 0.02870 0.33333 0.16667 0.52502
LoOP 5 0.24525 0.05657 0.23479 0.04349 0.33333 0.16667 0.52497
LoOP 91 0.19462 -0.00672 0.22053 0.02566 0.35319 0.19149 0.52808
LoOP 100 0.21044 0.01305 0.22216 0.02770 0.35227 0.19034 0.53240
LDOF 5 0.24367 0.05459 0.23273 0.04091 0.33369 0.16711 0.52166
LDOF 6 0.24367 0.05459 0.23596 0.04495 0.33404 0.16755 0.52362
LDOF 14 0.21677 0.02097 0.21813 0.02266 0.33704 0.17130 0.53338
LDOF 96 0.18671 -0.01661 0.21598 0.01997 0.35089 0.18861 0.51574
ODIN 42 0.24416 0.05520 0.23350 0.04188 0.34911 0.18638 0.56689
ODIN 100 0.24146 0.05182 0.24141 0.05176 0.35195 0.18994 0.57719
FastABOD 4 0.27057 0.08821 0.24783 0.05979 0.34394 0.17993 0.57490
FastABOD 5 0.25949 0.07437 0.24803 0.06003 0.34217 0.17771 0.57183
FastABOD 69 0.24684 0.05854 0.24582 0.05727 0.35346 0.19182 0.56807
KDEOS 38 0.22943 0.03679 0.21020 0.01275 0.33369 0.16711 0.51417
KDEOS 40 0.22627 0.03283 0.21080 0.01350 0.33351 0.16689 0.51525
KDEOS 53 0.20886 0.01108 0.20597 0.00746 0.33617 0.17022 0.52051
KDEOS 98 0.20728 0.00910 0.20388 0.00485 0.34071 0.17589 0.51202
LDF 94 0.25158 0.06448 0.25010 0.06262 0.38547 0.23184 0.61872
LDF 97 0.25633 0.07041 0.25043 0.06304 0.38242 0.22802 0.62064
LDF 99 0.25949 0.07437 0.25029 0.06286 0.38441 0.23051 0.62042
LDF 100 0.25791 0.07239 0.25075 0.06343 0.38518 0.23147 0.62036
INFLO 3 0.24209 0.05261 0.23227 0.04034 0.33423 0.16779 0.52353
INFLO 6 0.24525 0.05657 0.21974 0.02467 0.33360 0.16700 0.52863
INFLO 97 0.21835 0.02294 0.22704 0.03379 0.36304 0.20380 0.55796
INFLO 99 0.21677 0.02097 0.22718 0.03397 0.36249 0.20312 0.55847
COF 2 0.25949 0.07437 0.22717 0.03396 0.33413 0.16766 0.52227
COF 5 0.23576 0.04470 0.23490 0.04362 0.33333 0.16667 0.51940
COF 96 0.15823 -0.05222 0.19219 -0.00976 0.34066 0.17582 0.48820

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (29.1 MB) Download raw algorithm evaluation table (74.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.32425 0.15531 0.29075 0.11344 0.41209 0.26511 0.67263
KNN 9 0.31851 0.14813 0.28739 0.10924 0.41461 0.26826 0.67374
KNNW 13 0.29699 0.12123 0.27710 0.09638 0.39835 0.24794 0.65589
KNNW 21 0.30129 0.12661 0.27461 0.09327 0.40481 0.25602 0.65751
KNNW 23 0.29986 0.12482 0.27397 0.09246 0.40671 0.25838 0.65785
KNNW 31 0.29699 0.12123 0.27280 0.09100 0.40543 0.25679 0.65836
LOF 12 0.23960 0.04950 0.21940 0.02424 0.35004 0.18755 0.55311
LOF 19 0.21521 0.01901 0.21601 0.02002 0.35387 0.19234 0.55915
LOF 37 0.18938 -0.01327 0.20703 0.00879 0.36344 0.20430 0.54842
SimplifiedLOF 1 0.22382 0.02977 0.22166 0.02707 0.33341 0.16677 0.50859
SimplifiedLOF 5 0.24677 0.05846 0.21549 0.01937 0.33349 0.16687 0.51534
SimplifiedLOF 37 0.19512 -0.00610 0.20324 0.00406 0.34494 0.18118 0.53385
SimplifiedLOF 91 0.15065 -0.06169 0.20196 0.00245 0.35942 0.19928 0.51845
LoOP 2 0.23386 0.04232 0.23348 0.04184 0.33333 0.16667 0.52986
LoOP 5 0.24247 0.05308 0.21785 0.02231 0.33333 0.16667 0.53235
LoOP 43 0.19799 -0.00251 0.21380 0.01725 0.35542 0.19428 0.54939
LoOP 97 0.17934 -0.02582 0.21051 0.01314 0.36203 0.20254 0.54174
LDOF 3 0.22812 0.03515 0.21863 0.02329 0.33333 0.16667 0.48810
LDOF 4 0.24103 0.05129 0.21409 0.01761 0.33461 0.16827 0.49906
LDOF 78 0.19943 -0.00072 0.21356 0.01695 0.35874 0.19843 0.54860
LDOF 99 0.18221 -0.02224 0.21189 0.01486 0.35983 0.19979 0.54459
ODIN 49 0.25095 0.06368 0.23941 0.04926 0.36152 0.20190 0.58182
ODIN 100 0.26777 0.08471 0.23967 0.04959 0.35479 0.19349 0.58279
FastABOD 30 0.24103 0.05129 0.21755 0.02194 0.34835 0.18544 0.54887
FastABOD 99 0.24247 0.05308 0.22148 0.02686 0.34817 0.18522 0.55182
FastABOD 100 0.24390 0.05488 0.22153 0.02692 0.34817 0.18522 0.55179
KDEOS 3 0.22095 0.02618 0.21887 0.02359 0.33511 0.16888 0.50235
KDEOS 33 0.23099 0.03874 0.20673 0.00842 0.33502 0.16877 0.51707
KDEOS 69 0.21808 0.02260 0.21839 0.02299 0.34180 0.17724 0.54643
KDEOS 99 0.19225 -0.00968 0.20484 0.00605 0.35134 0.18917 0.53736
LDF 5 0.27690 0.09613 0.23825 0.04782 0.34463 0.18079 0.56298
LDF 9 0.20516 0.00646 0.22569 0.03211 0.36240 0.20300 0.57701
LDF 16 0.18508 -0.01865 0.21220 0.01525 0.36921 0.21152 0.55012
INFLO 11 0.23242 0.04053 0.21219 0.01524 0.34638 0.18297 0.54007
INFLO 13 0.22956 0.03694 0.21727 0.02159 0.35237 0.19047 0.54891
INFLO 18 0.23099 0.03874 0.21489 0.01861 0.35245 0.19057 0.55238
INFLO 91 0.15495 -0.05631 0.21052 0.01315 0.36841 0.21051 0.53314
COF 1 0.22525 0.03156 0.22578 0.03223 0.34158 0.17697 0.52221
COF 7 0.25108 0.06385 0.21671 0.02089 0.33341 0.16677 0.51887
COF 28 0.22812 0.03515 0.21312 0.01640 0.33836 0.17295 0.53007
COF 85 0.14204 -0.07245 0.18437 -0.01954 0.35363 0.19204 0.48146

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (27.5 MB) Download raw algorithm evaluation table (71.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.44620 0.30775 0.44401 0.30501 0.45896 0.32370 0.73830
KNN 8 0.43987 0.29984 0.44473 0.30591 0.46459 0.33073 0.73852
KNN 51 0.41930 0.27413 0.41448 0.26810 0.46502 0.33128 0.73727
KNN 98 0.41772 0.27215 0.40951 0.26188 0.45977 0.32471 0.73915
KNNW 14 0.43987 0.29984 0.44174 0.30218 0.45851 0.32313 0.73450
KNNW 16 0.43987 0.29984 0.44211 0.30264 0.46007 0.32509 0.73489
KNNW 35 0.41930 0.27413 0.43047 0.28809 0.46323 0.32903 0.73570
KNNW 100 0.41930 0.27413 0.41698 0.27123 0.46185 0.32731 0.73825
LOF 94 0.30380 0.12975 0.28728 0.10910 0.37397 0.21746 0.61889
LOF 100 0.30222 0.12777 0.28978 0.11223 0.37622 0.22028 0.62308
SimplifiedLOF 72 0.28323 0.10403 0.26915 0.08644 0.34733 0.18416 0.57818
SimplifiedLOF 100 0.28006 0.10008 0.28551 0.10688 0.35485 0.19357 0.59142
LoOP 1 0.25949 0.07437 0.25732 0.07165 0.33333 0.16667 0.51707
LoOP 77 0.26108 0.07634 0.24322 0.05403 0.34231 0.17789 0.55852
LoOP 86 0.26424 0.08030 0.24730 0.05913 0.33794 0.17242 0.56269
LoOP 99 0.25633 0.07041 0.25210 0.06512 0.33761 0.17201 0.56756
LDOF 2 0.24842 0.06052 0.24619 0.05774 0.33377 0.16722 0.47901
LDOF 100 0.21203 0.01503 0.21383 0.01729 0.33351 0.16689 0.50404
ODIN 25 0.17743 -0.02821 0.21240 0.01550 0.36508 0.20635 0.54685
ODIN 99 0.22231 0.02789 0.21688 0.02109 0.35795 0.19744 0.55368
ODIN 100 0.22127 0.02659 0.21687 0.02108 0.35748 0.19685 0.55390
FastABOD 3 0.40981 0.26226 0.40967 0.26209 0.44965 0.31206 0.72339
FastABOD 6 0.39873 0.24842 0.41141 0.26426 0.45724 0.32154 0.72468
FastABOD 9 0.39399 0.24248 0.41119 0.26399 0.46078 0.32597 0.72362
FastABOD 100 0.39873 0.24842 0.41192 0.26490 0.45612 0.32015 0.72094
KDEOS 80 0.22468 0.03085 0.22826 0.03532 0.34885 0.18606 0.56072
KDEOS 99 0.21677 0.02097 0.23389 0.04236 0.35258 0.19073 0.56778
KDEOS 100 0.21677 0.02097 0.23404 0.04255 0.35222 0.19027 0.56818
LDF 93 0.39715 0.24644 0.39367 0.24208 0.42629 0.28286 0.69420
LDF 97 0.39715 0.24644 0.39812 0.24764 0.43028 0.28785 0.69707
LDF 98 0.39715 0.24644 0.39947 0.24934 0.42825 0.28531 0.69779
LDF 100 0.39399 0.24248 0.39929 0.24912 0.42832 0.28541 0.69852
INFLO 96 0.27848 0.09810 0.26806 0.08507 0.44137 0.30172 0.61635
INFLO 100 0.27690 0.09612 0.27039 0.08798 0.44626 0.30783 0.62089
COF 100 0.31962 0.14953 0.34296 0.17869 0.37049 0.21311 0.61884

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (28.6 MB) Download raw algorithm evaluation table (73.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.46485 0.33106 0.46619 0.33274 0.47382 0.34227 0.76355
KNN 83 0.44476 0.30595 0.44266 0.30332 0.48039 0.35049 0.76610
KNN 89 0.44620 0.30775 0.44199 0.30249 0.47922 0.34902 0.76650
KNNW 16 0.46198 0.32747 0.46405 0.33006 0.46637 0.33296 0.75910
KNNW 82 0.45050 0.31313 0.45053 0.31317 0.48033 0.35041 0.76503
KNNW 100 0.44620 0.30775 0.44899 0.31124 0.47920 0.34900 0.76556
LOF 96 0.31277 0.14096 0.28458 0.10572 0.37657 0.22071 0.62688
LOF 100 0.31133 0.13917 0.28901 0.11126 0.37861 0.22326 0.62967
SimplifiedLOF 2 0.26112 0.07640 0.22153 0.02691 0.33357 0.16697 0.49870
SimplifiedLOF 93 0.24677 0.05846 0.24022 0.05028 0.35752 0.19690 0.58466
SimplifiedLOF 100 0.25108 0.06385 0.24801 0.06001 0.35528 0.19410 0.58941
LoOP 3 0.24534 0.05667 0.22433 0.03042 0.33333 0.16667 0.52494
LoOP 97 0.23529 0.04412 0.23225 0.04031 0.34868 0.18585 0.56756
LoOP 100 0.23960 0.04950 0.23449 0.04312 0.34804 0.18505 0.56867
LDOF 2 0.23816 0.04770 0.22625 0.03281 0.33349 0.16687 0.45579
LDOF 100 0.18508 -0.01865 0.20013 0.00016 0.34139 0.17674 0.50876
ODIN 6 0.20033 0.00042 0.21651 0.02064 0.36463 0.20579 0.55845
ODIN 13 0.17197 -0.03504 0.21530 0.01912 0.36546 0.20682 0.56114
ODIN 24 0.16243 -0.04696 0.21104 0.01380 0.37734 0.22167 0.55741
ODIN 99 0.21951 0.02439 0.21314 0.01643 0.35900 0.19875 0.55563
FastABOD 76 0.41033 0.26291 0.41899 0.27374 0.46362 0.32952 0.73070
FastABOD 96 0.41320 0.26650 0.41912 0.27390 0.46274 0.32842 0.73078
FastABOD 100 0.41320 0.26650 0.41915 0.27393 0.46274 0.32842 0.73081
KDEOS 11 0.21234 0.01542 0.19242 -0.00947 0.33577 0.16971 0.48107
KDEOS 100 0.21090 0.01363 0.22244 0.02805 0.35873 0.19841 0.56771
LDF 98 0.38020 0.22525 0.40054 0.25067 0.40987 0.26233 0.68809
LDF 100 0.37877 0.22346 0.40238 0.25298 0.41050 0.26312 0.68972
INFLO 87 0.25251 0.06564 0.25733 0.07167 0.44635 0.30794 0.62355
INFLO 91 0.26255 0.07819 0.26069 0.07586 0.44525 0.30656 0.62643
INFLO 100 0.26973 0.08716 0.26367 0.07959 0.44301 0.30376 0.62280
COF 86 0.28838 0.11047 0.25616 0.07020 0.36812 0.21016 0.61338
COF 96 0.29555 0.11944 0.26964 0.08705 0.36683 0.20854 0.61470
COF 97 0.29268 0.11585 0.27182 0.08977 0.36563 0.20703 0.61750
COF 100 0.28551 0.10689 0.27388 0.09235 0.36545 0.20682 0.61748

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO