Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (20% of outliers version#06)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (28.2 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.28639 0.10799 0.26529 0.08161 0.39098 0.23873 0.62694
KNN 9 0.28481 0.10601 0.26282 0.07852 0.39538 0.24423 0.63570
KNN 12 0.27532 0.09415 0.25677 0.07096 0.39907 0.24884 0.63062
KNNW 13 0.25791 0.07239 0.25603 0.07004 0.37985 0.22482 0.61794
KNNW 25 0.26424 0.08030 0.25391 0.06739 0.39094 0.23867 0.62361
KNNW 49 0.26741 0.08426 0.24847 0.06059 0.38874 0.23593 0.61881
LOF 87 0.21677 0.02097 0.22987 0.03734 0.36177 0.20221 0.57478
LOF 97 0.22468 0.03085 0.22916 0.03645 0.36483 0.20604 0.57537
LOF 98 0.22627 0.03283 0.22922 0.03652 0.36376 0.20469 0.57551
LOF 100 0.21994 0.02492 0.22982 0.03728 0.36469 0.20586 0.57746
SimplifiedLOF 1 0.25633 0.07041 0.24953 0.06192 0.33351 0.16689 0.51315
SimplifiedLOF 2 0.25791 0.07239 0.23607 0.04509 0.33395 0.16744 0.50640
SimplifiedLOF 34 0.20095 0.00119 0.20812 0.01014 0.34366 0.17958 0.53426
SimplifiedLOF 99 0.15823 -0.05222 0.21106 0.01383 0.35440 0.19300 0.52386
LoOP 1 0.25633 0.07041 0.25032 0.06290 0.33333 0.16667 0.51392
LoOP 95 0.21361 0.01701 0.22545 0.03182 0.35700 0.19625 0.54936
LoOP 100 0.21519 0.01899 0.22527 0.03159 0.35667 0.19584 0.55147
LDOF 2 0.24842 0.06052 0.24418 0.05522 0.33448 0.16810 0.49816
LDOF 50 0.20253 0.00316 0.21571 0.01964 0.34692 0.18364 0.54377
LDOF 98 0.19146 -0.01068 0.22317 0.02896 0.35373 0.19216 0.53588
ODIN 33 0.23443 0.04304 0.22736 0.03420 0.35602 0.19502 0.56521
ODIN 87 0.25753 0.07191 0.23893 0.04867 0.35044 0.18805 0.58020
ODIN 100 0.24927 0.06159 0.23999 0.04999 0.35036 0.18796 0.58223
FastABOD 3 0.23259 0.04074 0.22157 0.02696 0.33748 0.17185 0.53003
FastABOD 12 0.22943 0.03679 0.21967 0.02459 0.33767 0.17209 0.52473
FastABOD 23 0.23418 0.04272 0.22000 0.02500 0.33758 0.17197 0.52568
FastABOD 82 0.22943 0.03679 0.22350 0.02937 0.33753 0.17191 0.52782
KDEOS 3 0.19620 -0.00475 0.21715 0.02144 0.33589 0.16987 0.46922
KDEOS 84 0.17722 -0.02848 0.20312 0.00389 0.34708 0.18385 0.52329
KDEOS 99 0.21994 0.02492 0.20725 0.00906 0.34450 0.18063 0.53166
KDEOS 100 0.22468 0.03085 0.20706 0.00883 0.34395 0.17993 0.53147
LDF 3 0.26582 0.08228 0.24489 0.05611 0.34350 0.17938 0.55539
LDF 100 0.21203 0.01503 0.23113 0.03892 0.38788 0.23485 0.59671
INFLO 68 0.21677 0.02097 0.22799 0.03499 0.36220 0.20274 0.55660
INFLO 77 0.22943 0.03679 0.22766 0.03457 0.36072 0.20090 0.56087
INFLO 100 0.22468 0.03085 0.22994 0.03742 0.36193 0.20242 0.57575
COF 1 0.25633 0.07041 0.24934 0.06167 0.33377 0.16722 0.51319
COF 30 0.22627 0.03283 0.21166 0.01457 0.33445 0.16806 0.52436
COF 95 0.18354 -0.02057 0.19759 -0.00301 0.35062 0.18828 0.50769

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (29.0 MB) Download raw algorithm evaluation table (74.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.29412 0.11765 0.28891 0.11114 0.41783 0.27229 0.66965
KNNW 13 0.27260 0.09075 0.27100 0.08874 0.39913 0.24891 0.64480
KNNW 16 0.26686 0.08357 0.27162 0.08952 0.40227 0.25283 0.64803
KNNW 32 0.26686 0.08357 0.27007 0.08759 0.41109 0.26386 0.65382
KNNW 37 0.26829 0.08537 0.26931 0.08663 0.40985 0.26231 0.65399
LOF 2 0.24534 0.05667 0.22521 0.03151 0.33349 0.16687 0.52542
LOF 10 0.20516 0.00646 0.21053 0.01316 0.35423 0.19279 0.54995
LOF 87 0.14060 -0.07425 0.20217 0.00271 0.36492 0.20616 0.52702
SimplifiedLOF 2 0.24677 0.05846 0.22822 0.03527 0.33373 0.16717 0.52330
SimplifiedLOF 3 0.25251 0.06564 0.22328 0.02910 0.33333 0.16667 0.50695
SimplifiedLOF 98 0.14060 -0.07425 0.19356 -0.00805 0.36099 0.20124 0.50320
LoOP 2 0.23960 0.04950 0.23864 0.04830 0.33333 0.16667 0.53665
LoOP 4 0.24677 0.05846 0.23043 0.03804 0.33333 0.16667 0.51344
LoOP 100 0.16499 -0.04376 0.19999 -0.00001 0.36420 0.20526 0.52294
LDOF 2 0.25251 0.06564 0.22736 0.03420 0.33349 0.16687 0.50123
LDOF 79 0.17217 -0.03479 0.19995 -0.00006 0.35693 0.19616 0.52232
LDOF 98 0.17360 -0.03300 0.19992 -0.00009 0.35800 0.19750 0.51951
ODIN 79 0.21303 0.01629 0.22710 0.03387 0.36164 0.20204 0.57202
ODIN 99 0.23529 0.04412 0.23095 0.03868 0.35992 0.19990 0.57708
ODIN 100 0.23642 0.04552 0.23067 0.03834 0.35926 0.19908 0.57659
FastABOD 40 0.23386 0.04232 0.21515 0.01894 0.34885 0.18606 0.53369
FastABOD 61 0.23099 0.03874 0.21519 0.01899 0.34911 0.18639 0.53412
FastABOD 99 0.23099 0.03874 0.22080 0.02600 0.34911 0.18639 0.53607
FastABOD 100 0.23099 0.03874 0.22079 0.02599 0.34911 0.18639 0.53608
KDEOS 2 0.19886 -0.00143 0.21255 0.01568 0.33413 0.16767 0.50643
KDEOS 6 0.22669 0.03336 0.20616 0.00770 0.33721 0.17152 0.48265
KDEOS 94 0.19369 -0.00789 0.20410 0.00512 0.35269 0.19087 0.53709
KDEOS 98 0.19225 -0.00968 0.20296 0.00370 0.35347 0.19183 0.53549
LDF 7 0.26829 0.08537 0.25538 0.06922 0.38377 0.22971 0.61269
LDF 8 0.26542 0.08178 0.24861 0.06076 0.38810 0.23513 0.61013
INFLO 2 0.24534 0.05667 0.22065 0.02581 0.33365 0.16707 0.52333
INFLO 17 0.18795 -0.01506 0.20065 0.00081 0.35812 0.19765 0.53496
INFLO 99 0.14778 -0.06528 0.20118 0.00147 0.36860 0.21076 0.51549
COF 2 0.24534 0.05667 0.23276 0.04095 0.34721 0.18401 0.53577
COF 3 0.25825 0.07281 0.22989 0.03737 0.33349 0.16687 0.51775
COF 80 0.14347 -0.07066 0.18179 -0.02276 0.35095 0.18869 0.47592

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (27.4 MB) Download raw algorithm evaluation table (72.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 8 0.43829 0.29786 0.42305 0.27882 0.44307 0.30383 0.72889
KNN 14 0.43196 0.28995 0.42616 0.28270 0.44867 0.31084 0.72982
KNN 85 0.40981 0.26226 0.39938 0.24923 0.45880 0.32350 0.73480
KNN 95 0.40823 0.26028 0.39857 0.24822 0.45722 0.32152 0.73528
KNNW 11 0.43987 0.29984 0.42088 0.27611 0.44057 0.30071 0.71958
KNNW 19 0.43513 0.29391 0.42467 0.28084 0.44498 0.30622 0.72546
KNNW 100 0.41139 0.26424 0.40588 0.25735 0.45718 0.32148 0.73335
LOF 95 0.28323 0.10403 0.27254 0.09068 0.36579 0.20724 0.60715
LOF 100 0.28797 0.10997 0.27398 0.09248 0.36438 0.20548 0.60944
SimplifiedLOF 58 0.26582 0.08228 0.25243 0.06553 0.35324 0.19155 0.57185
SimplifiedLOF 73 0.27532 0.09415 0.26184 0.07730 0.34805 0.18506 0.57690
SimplifiedLOF 100 0.27373 0.09217 0.27189 0.08986 0.35120 0.18901 0.58787
LoOP 33 0.21519 0.01899 0.21127 0.01408 0.33852 0.17314 0.53647
LoOP 100 0.26899 0.08623 0.24279 0.05349 0.33562 0.16952 0.56053
LDOF 2 0.23418 0.04272 0.23348 0.04186 0.33430 0.16788 0.45917
LDOF 54 0.17722 -0.02848 0.18805 -0.01494 0.33600 0.16999 0.47765
LDOF 100 0.21361 0.01701 0.20570 0.00713 0.33440 0.16800 0.50155
ODIN 17 0.14551 -0.06811 0.19989 -0.00014 0.36837 0.21046 0.53268
ODIN 92 0.21930 0.02413 0.21010 0.01262 0.35882 0.19852 0.54144
ODIN 97 0.21907 0.02383 0.21032 0.01290 0.35602 0.19503 0.54272
ODIN 98 0.21875 0.02344 0.21040 0.01299 0.35605 0.19507 0.54231
FastABOD 3 0.39082 0.23853 0.38753 0.23442 0.43929 0.29912 0.70311
FastABOD 4 0.38924 0.23655 0.38593 0.23241 0.43289 0.29111 0.70383
FastABOD 74 0.38924 0.23655 0.38230 0.22788 0.44091 0.30114 0.70124
KDEOS 97 0.21994 0.02492 0.22205 0.02756 0.35421 0.19276 0.56637
KDEOS 98 0.21835 0.02294 0.22220 0.02775 0.35491 0.19364 0.56699
KDEOS 100 0.21677 0.02097 0.22302 0.02877 0.35455 0.19319 0.56783
LDF 95 0.37342 0.21677 0.38851 0.23564 0.42044 0.27555 0.68612
LDF 98 0.36867 0.21084 0.39098 0.23872 0.42284 0.27856 0.69009
LDF 100 0.37025 0.21282 0.39054 0.23817 0.42496 0.28120 0.69234
INFLO 93 0.27373 0.09217 0.25712 0.07140 0.43179 0.28974 0.61198
INFLO 100 0.27215 0.09019 0.25992 0.07490 0.43850 0.29812 0.61702
COF 96 0.33070 0.16337 0.33650 0.17063 0.35703 0.19629 0.60535
COF 98 0.32911 0.16139 0.33700 0.17125 0.35993 0.19992 0.60598

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (28.6 MB) Download raw algorithm evaluation table (73.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.44476 0.30595 0.45245 0.31556 0.46860 0.33576 0.75394
KNN 8 0.45768 0.32209 0.45216 0.31520 0.46921 0.33651 0.75349
KNN 60 0.43472 0.29340 0.43291 0.29113 0.48507 0.35634 0.75920
KNN 89 0.44189 0.30237 0.42922 0.28652 0.48222 0.35277 0.76155
KNNW 18 0.45337 0.31671 0.44967 0.31208 0.46907 0.33633 0.75033
KNNW 91 0.43759 0.29699 0.43613 0.29516 0.48451 0.35564 0.75899
KNNW 100 0.43759 0.29699 0.43533 0.29416 0.48421 0.35526 0.75948
LOF 97 0.28694 0.10868 0.26498 0.08122 0.37549 0.21936 0.61426
LOF 100 0.28694 0.10868 0.26727 0.08409 0.37739 0.22173 0.61699
SimplifiedLOF 1 0.24677 0.05846 0.22105 0.02631 0.33333 0.16667 0.50416
SimplifiedLOF 100 0.22238 0.02798 0.23006 0.03757 0.35127 0.18909 0.56713
LoOP 1 0.24534 0.05667 0.23823 0.04778 0.33333 0.16667 0.50480
LoOP 71 0.18077 -0.02403 0.20116 0.00144 0.34362 0.17953 0.52961
LoOP 99 0.20230 0.00287 0.21678 0.02097 0.33858 0.17322 0.54539
LDOF 2 0.23386 0.04232 0.21804 0.02255 0.33349 0.16687 0.44976
LDOF 72 0.15352 -0.05811 0.17605 -0.02994 0.34283 0.17854 0.46541
LDOF 100 0.16930 -0.03838 0.18686 -0.01642 0.34008 0.17510 0.48692
ODIN 1 0.18834 -0.01457 0.20786 0.00982 0.35418 0.19273 0.52824
ODIN 11 0.16504 -0.04370 0.20503 0.00629 0.36466 0.20582 0.54486
ODIN 25 0.14598 -0.06752 0.20269 0.00336 0.37867 0.22334 0.53985
ODIN 99 0.20660 0.00825 0.20441 0.00552 0.35782 0.19727 0.54281
FastABOD 70 0.40316 0.25395 0.40448 0.25560 0.45875 0.32343 0.71993
FastABOD 85 0.40603 0.25753 0.40591 0.25739 0.45694 0.32118 0.72031
FastABOD 100 0.40603 0.25753 0.40611 0.25763 0.45775 0.32218 0.72054
KDEOS 3 0.20373 0.00466 0.19852 -0.00184 0.33373 0.16717 0.47416
KDEOS 99 0.18938 -0.01327 0.21436 0.01795 0.35160 0.18950 0.54889
KDEOS 100 0.19082 -0.01148 0.21455 0.01819 0.35092 0.18865 0.54920
LDF 100 0.38451 0.23063 0.39419 0.24274 0.41099 0.26374 0.69038
INFLO 99 0.24247 0.05308 0.24107 0.05134 0.42517 0.28147 0.59706
INFLO 100 0.24103 0.05129 0.24365 0.05457 0.43110 0.28887 0.60345
COF 64 0.27260 0.09075 0.22633 0.03291 0.33999 0.17498 0.55656
COF 97 0.25538 0.06923 0.25294 0.06617 0.35834 0.19793 0.59168
COF 99 0.25825 0.07281 0.25736 0.07170 0.35780 0.19725 0.59631
COF 100 0.25681 0.07102 0.25911 0.07389 0.35743 0.19679 0.59572

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO