Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (20% of outliers version#10)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (28.3 MB) Download raw algorithm evaluation table (73.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.29589 0.11986 0.24949 0.06187 0.34786 0.18483 0.56543
KNN 7 0.27848 0.09810 0.27536 0.09420 0.41539 0.26924 0.66256
KNNW 9 0.29430 0.11788 0.25809 0.07261 0.38099 0.22623 0.61913
KNNW 24 0.26582 0.08228 0.26393 0.07991 0.40014 0.25018 0.64657
KNNW 28 0.27690 0.09612 0.26357 0.07947 0.40101 0.25127 0.64693
KNNW 31 0.27848 0.09810 0.26298 0.07873 0.40216 0.25270 0.64662
LOF 2 0.23418 0.04272 0.23079 0.03849 0.33843 0.17303 0.52001
LOF 85 0.24209 0.05261 0.22899 0.03624 0.37134 0.21418 0.57474
LOF 96 0.24051 0.05063 0.23065 0.03832 0.37276 0.21595 0.57851
LOF 98 0.24051 0.05063 0.23052 0.03815 0.37367 0.21709 0.57827
SimplifiedLOF 2 0.27848 0.09810 0.24830 0.06037 0.33404 0.16755 0.53240
SimplifiedLOF 99 0.15823 -0.05222 0.21486 0.01858 0.36178 0.20223 0.52286
LoOP 1 0.25791 0.07239 0.24099 0.05123 0.33333 0.16667 0.51957
LoOP 98 0.22152 0.02690 0.21765 0.02207 0.36257 0.20321 0.54780
LoOP 100 0.21994 0.02492 0.21799 0.02249 0.36209 0.20261 0.54860
LDOF 3 0.26741 0.08426 0.23797 0.04746 0.33333 0.16667 0.53381
LDOF 4 0.27373 0.09217 0.23731 0.04664 0.33413 0.16766 0.52870
LDOF 99 0.19304 -0.00870 0.21083 0.01354 0.35733 0.19666 0.52894
ODIN 44 0.20149 0.00187 0.22023 0.02529 0.35884 0.19855 0.56069
ODIN 89 0.23751 0.04689 0.22694 0.03368 0.35493 0.19366 0.57194
ODIN 97 0.22824 0.03530 0.22774 0.03467 0.35494 0.19367 0.57337
ODIN 98 0.22872 0.03589 0.22766 0.03457 0.35539 0.19424 0.57356
FastABOD 3 0.24525 0.05657 0.23335 0.04169 0.33734 0.17167 0.55092
FastABOD 4 0.25791 0.07239 0.23229 0.04036 0.33725 0.17156 0.54886
FastABOD 72 0.24842 0.06052 0.22945 0.03682 0.33857 0.17322 0.54442
KDEOS 3 0.20570 0.00712 0.21513 0.01892 0.33649 0.17062 0.49151
KDEOS 96 0.22468 0.03085 0.20491 0.00614 0.35231 0.19039 0.52727
KDEOS 100 0.21994 0.02492 0.20631 0.00789 0.35019 0.18774 0.52902
LDF 98 0.24842 0.06052 0.24868 0.06085 0.40488 0.25610 0.62274
LDF 99 0.24842 0.06052 0.24863 0.06079 0.40631 0.25789 0.62245
LDF 100 0.24842 0.06052 0.24910 0.06138 0.40550 0.25688 0.62295
INFLO 2 0.24367 0.05459 0.22906 0.03633 0.33484 0.16855 0.52007
INFLO 84 0.22627 0.03283 0.22611 0.03264 0.36691 0.20864 0.56663
INFLO 99 0.23418 0.04272 0.22776 0.03470 0.36606 0.20758 0.57167
COF 2 0.27373 0.09217 0.24407 0.05509 0.33360 0.16700 0.53424
COF 3 0.27690 0.09612 0.23742 0.04677 0.33342 0.16678 0.52270
COF 91 0.16772 -0.04035 0.20251 0.00313 0.35712 0.19640 0.50670

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (29.0 MB) Download raw algorithm evaluation table (74.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.32999 0.16248 0.29470 0.11837 0.40707 0.25883 0.66487
KNNW 13 0.29699 0.12123 0.28005 0.10006 0.39674 0.24593 0.64780
KNNW 15 0.29986 0.12482 0.27979 0.09974 0.39720 0.24651 0.64819
KNNW 24 0.29412 0.11765 0.27618 0.09523 0.39868 0.24835 0.64898
KNNW 100 0.27260 0.09075 0.25801 0.07251 0.40159 0.25198 0.63409
LOF 4 0.24103 0.05129 0.23331 0.04164 0.34892 0.18615 0.56212
LOF 9 0.23816 0.04770 0.23080 0.03851 0.36242 0.20302 0.57877
LOF 37 0.18795 -0.01506 0.20451 0.00564 0.37095 0.21369 0.55232
SimplifiedLOF 2 0.25108 0.06385 0.22301 0.02876 0.33365 0.16707 0.51719
SimplifiedLOF 4 0.24534 0.05667 0.22729 0.03411 0.33381 0.16727 0.52788
SimplifiedLOF 10 0.23099 0.03874 0.22022 0.02528 0.34353 0.17941 0.54969
SimplifiedLOF 87 0.14204 -0.07245 0.19285 -0.00894 0.36064 0.20080 0.51887
LoOP 1 0.22956 0.03694 0.24208 0.05260 0.33333 0.16667 0.51706
LoOP 6 0.23816 0.04770 0.22563 0.03204 0.34403 0.18004 0.55032
LoOP 10 0.23242 0.04053 0.22752 0.03440 0.35070 0.18837 0.56007
LoOP 76 0.16786 -0.04017 0.20656 0.00821 0.36402 0.20503 0.54197
LDOF 2 0.24534 0.05667 0.23529 0.04411 0.33357 0.16697 0.48651
LDOF 79 0.17647 -0.02941 0.21045 0.01306 0.36099 0.20123 0.54773
LDOF 85 0.17647 -0.02941 0.20949 0.01186 0.36306 0.20382 0.54609
ODIN 17 0.22231 0.02788 0.22308 0.02885 0.36054 0.20067 0.56650
ODIN 99 0.25275 0.06594 0.23273 0.04091 0.34988 0.18735 0.57040
ODIN 100 0.25462 0.06828 0.23264 0.04080 0.34993 0.18742 0.57042
FastABOD 11 0.24103 0.05129 0.21106 0.01383 0.34655 0.18318 0.53345
FastABOD 70 0.23673 0.04591 0.21722 0.02153 0.34872 0.18590 0.53966
FastABOD 82 0.23816 0.04770 0.21742 0.02177 0.34863 0.18579 0.53999
FastABOD 88 0.23816 0.04770 0.21758 0.02197 0.34863 0.18579 0.53997
KDEOS 3 0.22382 0.02977 0.21892 0.02365 0.33519 0.16898 0.50335
KDEOS 39 0.23099 0.03874 0.21110 0.01388 0.34128 0.17660 0.52380
KDEOS 66 0.20516 0.00646 0.20645 0.00807 0.35308 0.19136 0.53700
KDEOS 68 0.20660 0.00825 0.20675 0.00844 0.35200 0.19000 0.53848
LDF 5 0.27547 0.09433 0.26247 0.07808 0.36243 0.20304 0.59235
LDF 7 0.24964 0.06205 0.25242 0.06552 0.37786 0.22232 0.60790
LDF 8 0.25108 0.06385 0.25404 0.06755 0.37715 0.22144 0.61094
INFLO 4 0.21664 0.02080 0.22285 0.02857 0.34824 0.18530 0.54814
INFLO 8 0.22956 0.03694 0.21885 0.02357 0.36011 0.20014 0.55973
INFLO 13 0.22238 0.02798 0.21793 0.02241 0.36199 0.20249 0.56366
INFLO 17 0.20373 0.00466 0.21127 0.01409 0.36666 0.20832 0.55879
COF 1 0.22812 0.03515 0.22887 0.03609 0.34541 0.18176 0.53222
COF 2 0.26255 0.07819 0.22550 0.03187 0.34128 0.17661 0.53148
COF 6 0.23816 0.04770 0.22780 0.03476 0.33390 0.16737 0.53832
COF 72 0.18795 -0.01506 0.19482 -0.00648 0.35350 0.19187 0.51074

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (27.4 MB) Download raw algorithm evaluation table (71.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 8 0.45095 0.31369 0.43026 0.28782 0.45763 0.32203 0.73883
KNN 18 0.43987 0.29984 0.43545 0.29431 0.46019 0.32524 0.74342
KNN 99 0.42563 0.28204 0.41231 0.26538 0.46650 0.33312 0.75283
KNN 100 0.42089 0.27611 0.41194 0.26492 0.46697 0.33372 0.75279
KNNW 15 0.45411 0.31764 0.42892 0.28615 0.45462 0.31827 0.73303
KNNW 21 0.44620 0.30775 0.43188 0.28985 0.45389 0.31736 0.73726
KNNW 100 0.42563 0.28204 0.41804 0.27256 0.46224 0.32780 0.74981
LOF 94 0.31646 0.14557 0.28365 0.10457 0.37552 0.21940 0.63115
LOF 100 0.31487 0.14359 0.28648 0.10810 0.37923 0.22403 0.63536
SimplifiedLOF 97 0.27057 0.08821 0.27304 0.09130 0.35391 0.19239 0.59062
SimplifiedLOF 100 0.27373 0.09217 0.27462 0.09328 0.35342 0.19177 0.59267
LoOP 63 0.24525 0.05657 0.22922 0.03652 0.33785 0.17231 0.55045
LoOP 100 0.24051 0.05063 0.24237 0.05296 0.34670 0.18337 0.56259
LDOF 2 0.23892 0.04866 0.21837 0.02296 0.33369 0.16711 0.45160
LDOF 42 0.15190 -0.06013 0.18129 -0.02338 0.33369 0.16712 0.45708
LDOF 100 0.18987 -0.01266 0.20215 0.00269 0.33360 0.16700 0.49182
ODIN 23 0.13845 -0.07694 0.19879 -0.00152 0.36825 0.21032 0.53728
ODIN 98 0.21742 0.02178 0.21233 0.01541 0.36266 0.20332 0.55422
ODIN 100 0.21700 0.02125 0.21281 0.01601 0.36304 0.20380 0.55523
FastABOD 5 0.40823 0.26028 0.39845 0.24806 0.45450 0.31813 0.72496
FastABOD 14 0.39715 0.24644 0.39912 0.24889 0.46276 0.32845 0.72517
FastABOD 16 0.39873 0.24842 0.39924 0.24905 0.46244 0.32805 0.72522
KDEOS 89 0.21519 0.01899 0.21664 0.02081 0.34991 0.18738 0.55109
KDEOS 99 0.20728 0.00910 0.21895 0.02369 0.35496 0.19371 0.55700
KDEOS 100 0.20728 0.00910 0.21904 0.02380 0.35481 0.19351 0.55751
LDF 99 0.39241 0.24051 0.39826 0.24783 0.44433 0.30541 0.71438
LDF 100 0.39715 0.24644 0.39722 0.24653 0.44517 0.30646 0.71437
INFLO 78 0.25475 0.06843 0.25712 0.07140 0.44444 0.30556 0.61851
INFLO 96 0.26424 0.08030 0.25997 0.07497 0.44490 0.30613 0.61490
INFLO 97 0.26582 0.08228 0.26057 0.07571 0.44410 0.30512 0.61434
COF 97 0.31013 0.13766 0.33008 0.16260 0.36260 0.20326 0.61224
COF 100 0.30538 0.13172 0.33185 0.16481 0.36925 0.21157 0.61525

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (28.6 MB) Download raw algorithm evaluation table (73.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 9 0.46055 0.32568 0.43859 0.29823 0.46747 0.33434 0.74886
KNN 89 0.43329 0.29161 0.41954 0.27442 0.47913 0.34891 0.75987
KNN 100 0.43615 0.29519 0.41887 0.27359 0.48189 0.35237 0.75912
KNNW 16 0.44907 0.31133 0.43425 0.29281 0.46303 0.32878 0.74527
KNNW 23 0.43902 0.29878 0.43624 0.29530 0.46829 0.33536 0.74876
KNNW 37 0.44046 0.30057 0.43324 0.29155 0.47746 0.34683 0.75234
KNNW 99 0.43759 0.29699 0.42530 0.28163 0.47470 0.34337 0.75784
LOF 90 0.28694 0.10868 0.26083 0.07604 0.36364 0.20455 0.60470
LOF 100 0.28694 0.10868 0.27139 0.08924 0.37500 0.21875 0.61761
SimplifiedLOF 2 0.25108 0.06385 0.22607 0.03259 0.33333 0.16667 0.49962
SimplifiedLOF 100 0.23386 0.04232 0.23462 0.04328 0.35516 0.19395 0.57349
LoOP 2 0.22238 0.02798 0.24268 0.05335 0.33333 0.16667 0.51936
LoOP 3 0.24534 0.05667 0.22917 0.03646 0.33333 0.16667 0.51370
LoOP 79 0.20230 0.00287 0.20832 0.01040 0.34856 0.18570 0.53956
LoOP 100 0.22669 0.03336 0.22104 0.02630 0.34275 0.17844 0.55070
LDOF 2 0.23673 0.04591 0.22521 0.03151 0.33349 0.16687 0.45423
LDOF 70 0.16356 -0.04555 0.17937 -0.02578 0.34136 0.17669 0.46999
LDOF 100 0.17647 -0.02941 0.19166 -0.01043 0.34062 0.17578 0.48801
ODIN 2 0.20409 0.00511 0.21355 0.01694 0.35651 0.19564 0.54433
ODIN 16 0.16234 -0.04708 0.21125 0.01406 0.36659 0.20823 0.55401
ODIN 22 0.15270 -0.05913 0.20921 0.01151 0.37749 0.22186 0.55324
ODIN 97 0.21418 0.01773 0.20882 0.01103 0.36333 0.20417 0.55210
FastABOD 29 0.39455 0.24319 0.37412 0.21764 0.46774 0.33468 0.71621
FastABOD 70 0.40029 0.25036 0.38636 0.23294 0.46670 0.33338 0.71717
FastABOD 92 0.39885 0.24857 0.38759 0.23449 0.46701 0.33376 0.71757
FastABOD 100 0.39885 0.24857 0.38753 0.23442 0.46629 0.33286 0.71763
KDEOS 3 0.22238 0.02798 0.20856 0.01070 0.33429 0.16787 0.48700
KDEOS 93 0.19082 -0.01148 0.21487 0.01858 0.35758 0.19697 0.54809
KDEOS 100 0.19082 -0.01148 0.21921 0.02401 0.35610 0.19512 0.55421
LDF 99 0.37303 0.21628 0.38917 0.23646 0.41069 0.26336 0.69467
LDF 100 0.37446 0.21808 0.38893 0.23617 0.41184 0.26479 0.69502
INFLO 99 0.24390 0.05488 0.24924 0.06155 0.44112 0.30140 0.61114
INFLO 100 0.24390 0.05488 0.25075 0.06344 0.44413 0.30516 0.61411
COF 98 0.26112 0.07640 0.26183 0.07729 0.38206 0.22758 0.60960
COF 99 0.26255 0.07819 0.26228 0.07785 0.37849 0.22311 0.60698
COF 100 0.25968 0.07461 0.26409 0.08011 0.37726 0.22157 0.60863

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO