Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (20% of outliers version#02)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (28.2 MB) Download raw algorithm evaluation table (74.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.30380 0.12975 0.26915 0.08644 0.38275 0.22843 0.61406
KNN 6 0.29430 0.11788 0.28223 0.10278 0.41702 0.27128 0.66581
KNN 15 0.29272 0.11590 0.27401 0.09251 0.41788 0.27236 0.66106
KNNW 14 0.30063 0.12579 0.27305 0.09132 0.40312 0.25390 0.65012
KNNW 15 0.29747 0.12184 0.27326 0.09157 0.40490 0.25613 0.65096
KNNW 27 0.29114 0.11392 0.27274 0.09092 0.41680 0.27099 0.65644
KNNW 31 0.28956 0.11195 0.27240 0.09050 0.41593 0.26991 0.65686
LOF 2 0.24051 0.05063 0.21973 0.02466 0.33906 0.17383 0.52618
LOF 93 0.22310 0.02888 0.23339 0.04173 0.37139 0.21424 0.57822
LOF 98 0.23259 0.04074 0.23422 0.04278 0.37091 0.21364 0.58000
LOF 100 0.23259 0.04074 0.23403 0.04254 0.37000 0.21250 0.58021
SimplifiedLOF 1 0.24684 0.05854 0.23784 0.04729 0.33333 0.16667 0.52416
SimplifiedLOF 2 0.25791 0.07239 0.23697 0.04621 0.33395 0.16744 0.53089
SimplifiedLOF 3 0.25949 0.07437 0.22743 0.03429 0.33342 0.16678 0.52794
SimplifiedLOF 93 0.15190 -0.06013 0.20955 0.01194 0.36341 0.20427 0.51855
LoOP 1 0.24684 0.05854 0.23822 0.04777 0.33333 0.16667 0.52462
LoOP 99 0.20570 0.00712 0.22237 0.02796 0.36600 0.20750 0.54611
LoOP 100 0.20411 0.00514 0.22244 0.02805 0.36461 0.20576 0.54631
LDOF 2 0.25316 0.06646 0.23658 0.04573 0.33529 0.16912 0.51444
LDOF 49 0.19937 -0.00079 0.20611 0.00764 0.34876 0.18595 0.53074
LDOF 99 0.19304 -0.00870 0.21557 0.01946 0.35897 0.19872 0.52716
ODIN 45 0.23059 0.03824 0.23569 0.04461 0.36345 0.20431 0.58208
ODIN 98 0.25199 0.06499 0.24279 0.05348 0.35885 0.19856 0.59392
ODIN 99 0.25126 0.06407 0.24298 0.05372 0.35830 0.19787 0.59395
ODIN 100 0.24784 0.05980 0.24290 0.05363 0.35855 0.19818 0.59402
FastABOD 3 0.27690 0.09612 0.24740 0.05925 0.34341 0.17926 0.56824
FastABOD 52 0.27848 0.09810 0.23973 0.04967 0.34200 0.17751 0.56126
FastABOD 98 0.27057 0.08821 0.24074 0.05092 0.34716 0.18395 0.56534
KDEOS 3 0.23101 0.03877 0.22311 0.02889 0.33859 0.17324 0.50641
KDEOS 99 0.21677 0.02097 0.20597 0.00746 0.34771 0.18463 0.52279
KDEOS 100 0.21994 0.02492 0.20647 0.00809 0.34761 0.18451 0.52433
LDF 99 0.25316 0.06646 0.25186 0.06483 0.39529 0.24412 0.62534
LDF 100 0.24842 0.06052 0.25327 0.06658 0.39552 0.24439 0.62661
INFLO 81 0.23734 0.04668 0.22855 0.03568 0.36558 0.20698 0.56142
INFLO 92 0.23259 0.04074 0.23179 0.03974 0.36663 0.20828 0.57150
INFLO 99 0.23418 0.04272 0.23298 0.04122 0.36584 0.20730 0.57422
INFLO 100 0.23576 0.04470 0.23299 0.04124 0.36496 0.20620 0.57418
COF 2 0.26582 0.08228 0.24395 0.05493 0.33395 0.16744 0.53076
COF 79 0.16930 -0.03837 0.19726 -0.00342 0.34998 0.18748 0.51148

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (29.1 MB) Download raw algorithm evaluation table (74.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.29986 0.12482 0.28090 0.10113 0.41435 0.26794 0.66659
KNN 8 0.29555 0.11944 0.28093 0.10117 0.41963 0.27454 0.67131
KNN 9 0.29268 0.11585 0.28112 0.10140 0.41917 0.27396 0.67201
KNNW 14 0.29268 0.11585 0.27015 0.08769 0.40044 0.25055 0.64994
KNNW 37 0.27977 0.09971 0.26517 0.08146 0.41274 0.26592 0.65510
KNNW 44 0.28264 0.10330 0.26386 0.07983 0.41377 0.26721 0.65292
LOF 5 0.24103 0.05129 0.21782 0.02228 0.34097 0.17621 0.53432
LOF 15 0.23386 0.04232 0.21567 0.01959 0.35112 0.18890 0.54894
LOF 87 0.14060 -0.07425 0.20587 0.00734 0.36963 0.21203 0.53610
SimplifiedLOF 1 0.24247 0.05308 0.22678 0.03347 0.33365 0.16707 0.52171
SimplifiedLOF 2 0.25251 0.06564 0.22414 0.03017 0.33373 0.16717 0.51089
SimplifiedLOF 6 0.23242 0.04053 0.21644 0.02055 0.33349 0.16687 0.52916
SimplifiedLOF 100 0.14347 -0.07066 0.19684 -0.00395 0.36315 0.20394 0.51313
LoOP 2 0.24677 0.05846 0.23813 0.04767 0.33333 0.16667 0.52953
LoOP 32 0.22669 0.03336 0.21211 0.01514 0.34815 0.18519 0.54162
LoOP 100 0.15782 -0.05273 0.20365 0.00456 0.36417 0.20521 0.53452
LDOF 2 0.23099 0.03874 0.22280 0.02849 0.33341 0.16677 0.47890
LDOF 78 0.18077 -0.02403 0.20524 0.00654 0.35878 0.19848 0.53652
LDOF 81 0.17791 -0.02762 0.20478 0.00598 0.35977 0.19971 0.53586
ODIN 25 0.25711 0.07139 0.23149 0.03937 0.36084 0.20105 0.57804
ODIN 30 0.25125 0.06406 0.23140 0.03925 0.36651 0.20814 0.57672
ODIN 100 0.25263 0.06578 0.23963 0.04954 0.35609 0.19511 0.58577
FastABOD 30 0.23673 0.04591 0.21395 0.01744 0.34828 0.18535 0.54133
FastABOD 72 0.24390 0.05488 0.21650 0.02063 0.34820 0.18525 0.54456
FastABOD 92 0.24247 0.05308 0.21675 0.02093 0.34828 0.18535 0.54484
FastABOD 99 0.23960 0.04950 0.21673 0.02092 0.34828 0.18535 0.54493
KDEOS 67 0.23099 0.03874 0.21961 0.02451 0.34104 0.17630 0.54582
KDEOS 69 0.22525 0.03156 0.22064 0.02580 0.34165 0.17706 0.54715
KDEOS 94 0.21951 0.02439 0.20962 0.01202 0.35221 0.19027 0.54344
LDF 5 0.26973 0.08716 0.24153 0.05191 0.33854 0.17317 0.55742
LDF 6 0.25538 0.06923 0.24210 0.05262 0.34691 0.18364 0.56359
LDF 9 0.20947 0.01184 0.22414 0.03017 0.37580 0.21975 0.58175
LDF 10 0.22095 0.02618 0.22738 0.03422 0.36729 0.20911 0.58532
INFLO 1 0.21808 0.02260 0.21436 0.01795 0.33577 0.16971 0.50076
INFLO 2 0.24103 0.05129 0.21226 0.01533 0.33365 0.16707 0.51109
INFLO 15 0.23816 0.04770 0.21229 0.01536 0.35494 0.19367 0.54500
INFLO 99 0.15208 -0.05990 0.20760 0.00951 0.36950 0.21187 0.52526
COF 1 0.24103 0.05129 0.23084 0.03855 0.34872 0.18589 0.53719
COF 2 0.26542 0.08178 0.23082 0.03853 0.34328 0.17909 0.53536
COF 74 0.17360 -0.03300 0.18887 -0.01391 0.35200 0.19001 0.49312

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (27.4 MB) Download raw algorithm evaluation table (72.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.43038 0.28797 0.43962 0.29953 0.44415 0.30519 0.72815
KNN 6 0.43829 0.29786 0.43587 0.29484 0.44398 0.30498 0.73321
KNN 74 0.41456 0.26820 0.41794 0.27243 0.45558 0.31947 0.74207
KNN 100 0.40981 0.26226 0.41406 0.26757 0.45859 0.32324 0.74188
KNNW 12 0.44146 0.30182 0.43430 0.29287 0.44399 0.30499 0.72691
KNNW 21 0.42880 0.28600 0.43688 0.29610 0.44665 0.30832 0.73291
KNNW 57 0.41772 0.27215 0.42649 0.28311 0.45685 0.32106 0.73786
KNNW 100 0.41297 0.26622 0.42158 0.27698 0.45595 0.31994 0.74041
LOF 98 0.31646 0.14557 0.29949 0.12436 0.37309 0.21637 0.62897
LOF 100 0.31329 0.14161 0.30009 0.12511 0.37379 0.21724 0.62966
SimplifiedLOF 83 0.28006 0.10008 0.27939 0.09923 0.35580 0.19475 0.59181
SimplifiedLOF 92 0.29114 0.11392 0.28659 0.10824 0.35482 0.19353 0.59762
SimplifiedLOF 100 0.29114 0.11392 0.29136 0.11420 0.35271 0.19089 0.60095
LoOP 85 0.28006 0.10008 0.24746 0.05933 0.34741 0.18427 0.57204
LoOP 93 0.28006 0.10008 0.25110 0.06388 0.34855 0.18569 0.57452
LoOP 99 0.27690 0.09612 0.25417 0.06771 0.34786 0.18483 0.57559
LoOP 100 0.27690 0.09612 0.25446 0.06808 0.34614 0.18267 0.57550
LDOF 2 0.24209 0.05261 0.22801 0.03501 0.33369 0.16711 0.46353
LDOF 53 0.16456 -0.04430 0.18879 -0.01401 0.33843 0.17303 0.47932
LDOF 99 0.22152 0.02690 0.21022 0.01277 0.33555 0.16944 0.51090
ODIN 23 0.15427 -0.05716 0.20115 0.00144 0.36910 0.21137 0.53614
ODIN 100 0.22822 0.03528 0.21283 0.01604 0.36303 0.20379 0.55473
FastABOD 3 0.40032 0.25040 0.39474 0.24343 0.44980 0.31225 0.71325
FastABOD 6 0.39873 0.24842 0.40067 0.25084 0.44508 0.30635 0.71466
FastABOD 7 0.39399 0.24248 0.40044 0.25055 0.44585 0.30731 0.71486
KDEOS 78 0.21677 0.02097 0.21261 0.01576 0.34960 0.18699 0.55011
KDEOS 100 0.21044 0.01305 0.21683 0.02103 0.35438 0.19298 0.56014
LDF 94 0.38766 0.23457 0.41356 0.26695 0.43396 0.29245 0.69900
LDF 97 0.39557 0.24446 0.41112 0.26390 0.43376 0.29220 0.70010
LDF 98 0.39557 0.24446 0.41130 0.26413 0.43624 0.29530 0.70112
LDF 100 0.39241 0.24051 0.41084 0.26355 0.43763 0.29704 0.70032
INFLO 94 0.28006 0.10008 0.27481 0.09351 0.45148 0.31435 0.63116
INFLO 99 0.27848 0.09810 0.27828 0.09784 0.45594 0.31993 0.63595
INFLO 100 0.27848 0.09810 0.27843 0.09804 0.45524 0.31905 0.63367
COF 100 0.31487 0.14359 0.36052 0.20065 0.37537 0.21921 0.63005

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (28.6 MB) Download raw algorithm evaluation table (74.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.45337 0.31671 0.45295 0.31619 0.45774 0.32217 0.75705
KNN 8 0.45624 0.32030 0.45256 0.31570 0.45785 0.32231 0.75678
KNN 33 0.43472 0.29340 0.44125 0.30156 0.46498 0.33122 0.75952
KNN 100 0.42755 0.28443 0.42576 0.28221 0.47240 0.34049 0.75540
KNNW 9 0.45481 0.31851 0.44651 0.30814 0.45740 0.32175 0.74315
KNNW 21 0.45337 0.31671 0.45206 0.31507 0.46118 0.32648 0.75500
KNNW 51 0.43902 0.29878 0.44272 0.30340 0.46601 0.33251 0.75825
KNNW 63 0.43902 0.29878 0.44021 0.30027 0.46784 0.33480 0.75807
LOF 98 0.31133 0.13917 0.28205 0.10257 0.38233 0.22791 0.63831
LOF 100 0.31133 0.13917 0.28386 0.10482 0.38541 0.23176 0.64003
SimplifiedLOF 2 0.26112 0.07640 0.22778 0.03472 0.33341 0.16677 0.51332
SimplifiedLOF 94 0.23386 0.04232 0.23567 0.04459 0.36303 0.20379 0.58849
SimplifiedLOF 100 0.23960 0.04950 0.24148 0.05184 0.36194 0.20242 0.59286
LoOP 1 0.24247 0.05308 0.24277 0.05346 0.33333 0.16667 0.52000
LoOP 2 0.24390 0.05488 0.22878 0.03597 0.33333 0.16667 0.53178
LoOP 98 0.21808 0.02260 0.22438 0.03048 0.35790 0.19738 0.56754
LoOP 99 0.22095 0.02618 0.22518 0.03148 0.35754 0.19692 0.56821
LDOF 2 0.24390 0.05488 0.22374 0.02967 0.33365 0.16707 0.46736
LDOF 72 0.14634 -0.06707 0.17997 -0.02504 0.34205 0.17756 0.47843
LDOF 100 0.16786 -0.04017 0.19256 -0.00929 0.34044 0.17554 0.50100
ODIN 1 0.21131 0.01414 0.21745 0.02182 0.35718 0.19648 0.55211
ODIN 21 0.16258 -0.04677 0.20750 0.00938 0.38403 0.23004 0.55469
ODIN 99 0.21693 0.02116 0.21332 0.01665 0.35952 0.19940 0.56269
ODIN 100 0.21602 0.02002 0.21386 0.01733 0.35977 0.19972 0.56391
FastABOD 20 0.39311 0.24139 0.38987 0.23734 0.45270 0.31587 0.71254
FastABOD 78 0.40459 0.25574 0.40727 0.25909 0.44951 0.31189 0.71446
FastABOD 100 0.40459 0.25574 0.40797 0.25996 0.44917 0.31146 0.71478
KDEOS 3 0.21951 0.02439 0.20976 0.01220 0.33701 0.17126 0.49769
KDEOS 98 0.20230 0.00287 0.21787 0.02234 0.36231 0.20288 0.56816
KDEOS 100 0.19656 -0.00430 0.21930 0.02413 0.36155 0.20193 0.56968
LDF 98 0.38737 0.23422 0.40173 0.25216 0.42498 0.28122 0.70308
LDF 99 0.39024 0.23780 0.40157 0.25196 0.42596 0.28245 0.70408
LDF 100 0.39168 0.23960 0.40116 0.25145 0.42343 0.27929 0.70352
INFLO 87 0.24390 0.05488 0.24885 0.06106 0.45584 0.31979 0.61814
INFLO 99 0.25395 0.06743 0.25460 0.06826 0.45501 0.31876 0.62094
INFLO 100 0.25825 0.07281 0.25457 0.06822 0.45264 0.31580 0.61903
COF 88 0.26112 0.07640 0.25633 0.07042 0.38295 0.22868 0.61799
COF 98 0.26542 0.08178 0.26950 0.08687 0.37907 0.22383 0.62542
COF 100 0.27834 0.09792 0.27180 0.08975 0.38120 0.22650 0.62483

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO