Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (5% of outliers version#06)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (43.5 MB) Download raw algorithm evaluation table (70.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.46875 0.44090 0.43934 0.40995 0.47525 0.44774 0.90435
KNN 30 0.45312 0.42445 0.47915 0.45184 0.46263 0.43446 0.91077
KNN 52 0.44531 0.41623 0.47693 0.44951 0.49732 0.47096 0.91017
KNNW 19 0.46094 0.43268 0.45226 0.42354 0.46903 0.44119 0.90696
KNNW 60 0.44141 0.41212 0.47472 0.44718 0.45884 0.43047 0.91198
KNNW 93 0.44531 0.41623 0.47947 0.45218 0.49129 0.46462 0.91103
KNNW 99 0.44922 0.42034 0.47915 0.45185 0.49554 0.46910 0.91071
LOF 32 0.44141 0.41212 0.38588 0.35368 0.45536 0.42680 0.82939
LOF 36 0.43750 0.40801 0.39536 0.36366 0.46187 0.43366 0.83754
LOF 100 0.40625 0.37512 0.43073 0.40088 0.42341 0.39318 0.92214
SimplifiedLOF 41 0.45312 0.42445 0.39624 0.36459 0.45312 0.42445 0.82739
SimplifiedLOF 97 0.44141 0.41212 0.42335 0.39312 0.46122 0.43297 0.87329
SimplifiedLOF 100 0.44141 0.41212 0.42528 0.39515 0.45738 0.42893 0.87763
LoOP 48 0.43359 0.40390 0.35925 0.32566 0.44764 0.41868 0.82501
LoOP 94 0.44141 0.41212 0.39200 0.36013 0.44269 0.41347 0.84581
LoOP 100 0.43750 0.40801 0.39438 0.36263 0.44094 0.41164 0.85103
LDOF 63 0.46094 0.43268 0.40037 0.36893 0.46494 0.43689 0.88980
LDOF 88 0.44922 0.42034 0.42473 0.39457 0.47803 0.45067 0.89922
LDOF 100 0.44141 0.41212 0.43043 0.40056 0.47222 0.44455 0.90458
ODIN 62 0.43774 0.40827 0.31821 0.28246 0.44815 0.41922 0.79253
ODIN 67 0.44283 0.41362 0.32274 0.28724 0.44660 0.41759 0.79594
ODIN 100 0.43510 0.40548 0.33874 0.30407 0.43687 0.40735 0.83407
FastABOD 32 0.38672 0.35457 0.37191 0.33898 0.39932 0.36783 0.81302
FastABOD 78 0.39062 0.35868 0.37861 0.34603 0.40678 0.37568 0.81173
FastABOD 85 0.39453 0.36279 0.37922 0.34668 0.40454 0.37332 0.81162
FastABOD 99 0.39062 0.35868 0.37982 0.34731 0.40530 0.37412 0.81130
KDEOS 75 0.11719 0.07090 0.10956 0.06288 0.21742 0.17639 0.74034
KDEOS 99 0.10547 0.05857 0.11771 0.07146 0.23745 0.19747 0.75765
KDEOS 100 0.10547 0.05857 0.11756 0.07129 0.23876 0.19885 0.75847
LDF 99 0.46484 0.43679 0.49282 0.46623 0.54412 0.52022 0.93728
LDF 100 0.46875 0.44090 0.49381 0.46728 0.54386 0.51995 0.93728
INFLO 68 0.42578 0.39568 0.36493 0.33163 0.42683 0.39678 0.76432
INFLO 93 0.41797 0.38745 0.36124 0.32775 0.43333 0.40362 0.76214
INFLO 97 0.41406 0.38334 0.36387 0.33052 0.43096 0.40113 0.77155
COF 43 0.44922 0.42034 0.40886 0.37787 0.46085 0.43258 0.79975
COF 44 0.44531 0.41623 0.41193 0.38110 0.46799 0.44010 0.79882
COF 59 0.43750 0.40801 0.42393 0.39372 0.45940 0.43105 0.78353

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (43.6 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 9 0.46899 0.44111 0.43568 0.40604 0.47505 0.44748 0.85008
KNN 75 0.44961 0.42071 0.47446 0.44686 0.46667 0.43866 0.91889
KNN 91 0.42248 0.39215 0.47234 0.44463 0.49245 0.46579 0.91913
KNN 100 0.40698 0.37583 0.47398 0.44635 0.48967 0.46287 0.91913
KNNW 16 0.47287 0.44519 0.43945 0.41001 0.48165 0.45443 0.84772
KNNW 25 0.46124 0.43295 0.45083 0.42199 0.48736 0.46044 0.86135
KNNW 100 0.45349 0.42479 0.47614 0.44863 0.46580 0.43775 0.91778
LOF 23 0.45736 0.42887 0.40459 0.37332 0.47558 0.44804 0.85487
LOF 25 0.46512 0.43703 0.41029 0.37932 0.48055 0.45327 0.85017
LOF 34 0.43798 0.40847 0.42088 0.39047 0.45135 0.42253 0.83442
SimplifiedLOF 29 0.48062 0.45335 0.41975 0.38928 0.49351 0.46691 0.82937
SimplifiedLOF 34 0.46899 0.44111 0.43294 0.40316 0.49778 0.47140 0.83424
SimplifiedLOF 37 0.47674 0.44927 0.43796 0.40844 0.49676 0.47033 0.83595
SimplifiedLOF 41 0.46512 0.43703 0.44000 0.41060 0.48908 0.46225 0.83400
LoOP 41 0.44186 0.41255 0.37712 0.34441 0.44726 0.41823 0.81805
LoOP 42 0.44186 0.41255 0.37825 0.34560 0.44820 0.41923 0.81872
LoOP 46 0.43023 0.40031 0.37919 0.34659 0.45089 0.42206 0.81757
LoOP 58 0.43023 0.40031 0.38352 0.35114 0.43340 0.40365 0.81375
LDOF 79 0.47287 0.44519 0.42575 0.39560 0.49534 0.46884 0.89053
LDOF 81 0.48062 0.45335 0.42570 0.39554 0.49722 0.47081 0.89128
LDOF 82 0.48837 0.46150 0.42566 0.39550 0.49907 0.47276 0.89050
LDOF 83 0.49225 0.46558 0.42539 0.39522 0.49811 0.47176 0.88996
ODIN 53 0.42894 0.39895 0.29830 0.26145 0.43137 0.40151 0.76290
ODIN 71 0.41008 0.37910 0.31240 0.27629 0.44009 0.41068 0.77501
ODIN 98 0.40181 0.37040 0.32219 0.28659 0.42601 0.39587 0.77918
ODIN 100 0.40310 0.37176 0.32210 0.28650 0.42601 0.39587 0.78098
FastABOD 3 0.41473 0.38399 0.36140 0.32786 0.41767 0.38709 0.79055
FastABOD 68 0.42248 0.39215 0.38032 0.34778 0.42389 0.39364 0.77869
FastABOD 99 0.41860 0.38807 0.38206 0.34961 0.42692 0.39683 0.77735
FastABOD 100 0.41860 0.38807 0.38213 0.34968 0.42610 0.39597 0.77731
KDEOS 71 0.10465 0.05763 0.10425 0.05721 0.19641 0.15421 0.72316
KDEOS 81 0.10853 0.06171 0.10559 0.05862 0.20475 0.16299 0.72105
KDEOS 96 0.12791 0.08211 0.10778 0.06092 0.20039 0.15840 0.71895
KDEOS 100 0.12016 0.07395 0.10857 0.06176 0.20212 0.16022 0.71816
LDF 22 0.46512 0.43703 0.46615 0.43811 0.48148 0.45425 0.86808
LDF 23 0.45349 0.42479 0.46956 0.44171 0.47826 0.45086 0.86837
LDF 28 0.47287 0.44519 0.45677 0.42825 0.47600 0.44848 0.85730
LDF 100 0.41860 0.38807 0.40572 0.37451 0.42458 0.39436 0.90466
INFLO 35 0.43411 0.40439 0.35857 0.32489 0.44444 0.41527 0.75054
INFLO 36 0.43798 0.40847 0.36019 0.32659 0.44351 0.41429 0.75223
INFLO 45 0.43411 0.40439 0.37070 0.33766 0.43922 0.40977 0.76545
INFLO 72 0.42636 0.39623 0.36339 0.32996 0.43160 0.40175 0.77312
COF 41 0.44574 0.41663 0.42778 0.39773 0.45724 0.42874 0.81703
COF 59 0.48837 0.46150 0.43193 0.40209 0.49398 0.46740 0.78370
COF 60 0.48837 0.46150 0.42891 0.39891 0.49704 0.47063 0.77698

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (44.6 MB) Download raw algorithm evaluation table (71.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.17578 0.13257 0.16300 0.11912 0.20245 0.16064 0.57229
KNN 4 0.17188 0.12846 0.17484 0.13158 0.19289 0.15058 0.59843
KNN 7 0.18750 0.14490 0.17387 0.13055 0.19658 0.15446 0.60459
KNN 17 0.16016 0.11613 0.17059 0.12710 0.19805 0.15601 0.61402
KNNW 1 0.17578 0.13257 0.15988 0.11584 0.20178 0.15993 0.55355
KNNW 2 0.16797 0.12435 0.16158 0.11762 0.20772 0.16618 0.56505
KNNW 16 0.16797 0.12435 0.17183 0.12842 0.18229 0.13942 0.60563
KNNW 64 0.14844 0.10379 0.16414 0.12032 0.17808 0.13499 0.61259
LOF 85 0.50781 0.48201 0.52438 0.49944 0.53420 0.50978 0.92616
LOF 86 0.50391 0.47790 0.52458 0.49965 0.53952 0.51538 0.92599
LOF 90 0.51953 0.49434 0.52339 0.49840 0.55126 0.52773 0.92603
LOF 99 0.51172 0.48612 0.51955 0.49436 0.56128 0.53828 0.92517
SimplifiedLOF 96 0.51953 0.49434 0.50927 0.48354 0.52055 0.49541 0.92597
SimplifiedLOF 100 0.51953 0.49434 0.51082 0.48517 0.52290 0.49789 0.92669
LoOP 88 0.47266 0.44501 0.48973 0.46298 0.47401 0.44644 0.90816
LoOP 94 0.46875 0.44090 0.49630 0.46989 0.48233 0.45519 0.91255
LoOP 100 0.46875 0.44090 0.50370 0.47768 0.48033 0.45309 0.91551
LDOF 88 0.42969 0.39979 0.45772 0.42929 0.46083 0.43256 0.89752
LDOF 100 0.42578 0.39568 0.46957 0.44176 0.45434 0.42574 0.91072
ODIN 99 0.39983 0.36837 0.36847 0.33536 0.40671 0.37560 0.88513
ODIN 100 0.40151 0.37013 0.37012 0.33709 0.40426 0.37302 0.88617
FastABOD 3 0.14453 0.09968 0.13884 0.09369 0.17308 0.12972 0.47539
FastABOD 7 0.13672 0.09146 0.14313 0.09821 0.17647 0.13330 0.47579
FastABOD 15 0.13281 0.08735 0.14285 0.09791 0.18239 0.13953 0.47625
FastABOD 99 0.13281 0.08735 0.14194 0.09695 0.18012 0.13714 0.47682
KDEOS 99 0.10156 0.05446 0.09807 0.05078 0.17203 0.12862 0.73972
KDEOS 100 0.09766 0.05035 0.09883 0.05159 0.17132 0.12787 0.74059
LDF 63 0.52344 0.49845 0.52101 0.49590 0.54867 0.52501 0.91591
LDF 70 0.51172 0.48612 0.52647 0.50165 0.55611 0.53284 0.91668
LDF 78 0.52344 0.49845 0.52354 0.49856 0.55268 0.52923 0.91706
INFLO 82 0.49219 0.46556 0.47622 0.44876 0.49315 0.46658 0.86488
INFLO 98 0.48438 0.45734 0.48781 0.46096 0.51040 0.48473 0.88674
INFLO 100 0.48828 0.46145 0.48902 0.46223 0.50847 0.48271 0.88760
COF 89 0.49609 0.46968 0.48804 0.46120 0.50497 0.47902 0.87196
COF 96 0.51172 0.48612 0.49514 0.46867 0.51351 0.48801 0.86897
COF 100 0.51172 0.48612 0.49836 0.47206 0.52128 0.49618 0.86755

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (44.7 MB) Download raw algorithm evaluation table (72.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.18992 0.14738 0.15545 0.11110 0.19958 0.15754 0.55829
KNN 2 0.18217 0.13922 0.15674 0.11246 0.20274 0.16087 0.56716
KNN 4 0.17442 0.13106 0.15805 0.11383 0.19370 0.15136 0.58103
KNN 15 0.17054 0.12698 0.15430 0.10989 0.19893 0.15687 0.60428
KNNW 1 0.17829 0.13514 0.16165 0.11762 0.21106 0.16962 0.53945
KNNW 3 0.18217 0.13922 0.15814 0.11393 0.20163 0.15971 0.55762
KNNW 57 0.15116 0.10659 0.14607 0.10123 0.17548 0.13218 0.59967
LOF 74 0.51550 0.49006 0.51696 0.49159 0.51953 0.49430 0.90435
LOF 95 0.52713 0.50230 0.50807 0.48224 0.56954 0.54693 0.90685
LOF 97 0.52713 0.50230 0.50692 0.48102 0.57000 0.54742 0.90689
LOF 100 0.52713 0.50230 0.50495 0.47895 0.56951 0.54691 0.90736
SimplifiedLOF 93 0.50775 0.48190 0.50831 0.48248 0.52778 0.50298 0.91213
SimplifiedLOF 95 0.51550 0.49006 0.50859 0.48279 0.52688 0.50204 0.91201
SimplifiedLOF 98 0.51938 0.49414 0.50733 0.48146 0.52773 0.50293 0.91182
SimplifiedLOF 100 0.51550 0.49006 0.50745 0.48159 0.53577 0.51140 0.91196
LoOP 84 0.48837 0.46150 0.48769 0.46079 0.49315 0.46653 0.89943
LoOP 92 0.48450 0.45743 0.49674 0.47031 0.49603 0.46957 0.90470
LoOP 100 0.48450 0.45743 0.50248 0.47635 0.49600 0.46953 0.90834
LDOF 78 0.44186 0.41255 0.44454 0.41537 0.45521 0.42660 0.88972
LDOF 95 0.43798 0.40847 0.46522 0.43713 0.46818 0.44025 0.90699
LDOF 100 0.43798 0.40847 0.46706 0.43908 0.46083 0.43252 0.91122
ODIN 95 0.40310 0.37176 0.39687 0.36520 0.41453 0.38378 0.88213
ODIN 100 0.40116 0.36972 0.40625 0.37507 0.42637 0.39625 0.88674
FastABOD 3 0.16667 0.12291 0.13228 0.08671 0.20225 0.16035 0.49019
FastABOD 8 0.13953 0.09435 0.13272 0.08718 0.17367 0.13028 0.46830
KDEOS 7 0.08915 0.04132 0.07272 0.02403 0.12580 0.07989 0.58754
KDEOS 100 0.08915 0.04132 0.09359 0.04600 0.17034 0.12677 0.72658
LDF 46 0.46899 0.44111 0.52232 0.49724 0.54407 0.52013 0.89981
LDF 47 0.46899 0.44111 0.52147 0.49634 0.54600 0.52216 0.89990
LDF 67 0.52713 0.50230 0.51568 0.49025 0.55829 0.53510 0.88988
LDF 97 0.53876 0.51454 0.48225 0.45506 0.55113 0.52755 0.88802
INFLO 99 0.50388 0.47782 0.48526 0.45823 0.51812 0.49281 0.87713
INFLO 100 0.50388 0.47782 0.48620 0.45922 0.51930 0.49405 0.87895
COF 71 0.48837 0.46150 0.47407 0.44645 0.50000 0.47374 0.86995
COF 88 0.53101 0.50638 0.49220 0.46554 0.53543 0.51104 0.86421
COF 100 0.53101 0.50638 0.50121 0.47501 0.54017 0.51602 0.85091

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO