Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (5% of outliers version#02)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (43.4 MB) Download raw algorithm evaluation table (71.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 8 0.46094 0.43268 0.44103 0.41173 0.46214 0.43394 0.90288
KNN 42 0.43359 0.40390 0.46612 0.43813 0.46287 0.43471 0.91147
KNN 47 0.43750 0.40801 0.46648 0.43851 0.47285 0.44522 0.91120
KNN 48 0.44141 0.41212 0.46657 0.43861 0.47285 0.44522 0.91105
KNNW 5 0.45312 0.42445 0.40510 0.37391 0.46212 0.43392 0.87221
KNNW 65 0.42188 0.39157 0.46660 0.43863 0.45283 0.42414 0.91262
KNNW 83 0.43359 0.40390 0.46889 0.44104 0.46701 0.43906 0.91188
KNNW 84 0.43359 0.40390 0.46879 0.44094 0.46780 0.43989 0.91183
LOF 28 0.46094 0.43268 0.37440 0.34160 0.46799 0.44010 0.82572
LOF 32 0.45312 0.42445 0.38816 0.35609 0.48578 0.45882 0.83453
LOF 100 0.42969 0.39979 0.41015 0.37923 0.43275 0.40301 0.92374
SimplifiedLOF 47 0.47266 0.44501 0.40141 0.37003 0.47674 0.44931 0.82289
SimplifiedLOF 48 0.47266 0.44501 0.40368 0.37242 0.47767 0.45029 0.82368
SimplifiedLOF 66 0.46094 0.43268 0.40759 0.37654 0.47347 0.44587 0.83952
SimplifiedLOF 100 0.44531 0.41623 0.40654 0.37543 0.46501 0.43696 0.87521
LoOP 45 0.43750 0.40801 0.36126 0.32777 0.45824 0.42984 0.81380
LoOP 51 0.45312 0.42445 0.36604 0.33280 0.45635 0.42785 0.81592
LoOP 98 0.44141 0.41212 0.38114 0.34869 0.45148 0.42272 0.84954
LoOP 100 0.44141 0.41212 0.37744 0.34480 0.45339 0.42473 0.85149
LDOF 86 0.49219 0.46556 0.42230 0.39202 0.49310 0.46652 0.89269
LDOF 87 0.48828 0.46145 0.42292 0.39266 0.49320 0.46663 0.89350
LDOF 99 0.48047 0.45323 0.42850 0.39854 0.48954 0.46278 0.89961
LDOF 100 0.47656 0.44912 0.42800 0.39801 0.48671 0.45980 0.90007
ODIN 66 0.45052 0.42171 0.32732 0.29206 0.45187 0.42313 0.78229
ODIN 80 0.43490 0.40527 0.33542 0.30058 0.45243 0.42372 0.78476
ODIN 87 0.44336 0.41418 0.33631 0.30151 0.44776 0.41881 0.79134
ODIN 100 0.43450 0.40485 0.33467 0.29979 0.44776 0.41881 0.80543
FastABOD 22 0.40234 0.37101 0.36219 0.32875 0.41046 0.37956 0.80225
FastABOD 50 0.39844 0.36690 0.36763 0.33448 0.41929 0.38884 0.80103
FastABOD 73 0.41016 0.37923 0.36999 0.33696 0.41631 0.38571 0.80018
FastABOD 97 0.40625 0.37512 0.37275 0.33986 0.41781 0.38729 0.79918
KDEOS 69 0.16406 0.12024 0.12118 0.07511 0.23159 0.19131 0.74171
KDEOS 82 0.14062 0.09557 0.12450 0.07860 0.23686 0.19685 0.74887
KDEOS 100 0.11328 0.06679 0.12441 0.07851 0.24441 0.20480 0.75896
LDF 100 0.46094 0.43268 0.47312 0.44550 0.54354 0.51961 0.93755
INFLO 36 0.45703 0.42857 0.35045 0.31640 0.46123 0.43299 0.76246
INFLO 37 0.46094 0.43268 0.34968 0.31559 0.46825 0.44038 0.75764
COF 40 0.44922 0.42034 0.38664 0.35449 0.45700 0.42853 0.78119
COF 71 0.44141 0.41212 0.40164 0.37027 0.46620 0.43822 0.77119
COF 78 0.43359 0.40390 0.40080 0.36938 0.48000 0.45274 0.77086
COF 100 0.44141 0.41212 0.39082 0.35888 0.45833 0.42994 0.79031

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (43.5 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 8 0.39922 0.36768 0.36742 0.33420 0.42762 0.39756 0.78834
KNN 99 0.33721 0.30240 0.40406 0.37277 0.44516 0.41602 0.89112
KNN 100 0.33721 0.30240 0.40449 0.37321 0.44408 0.41489 0.89126
KNNW 13 0.40310 0.37176 0.36573 0.33242 0.41575 0.38507 0.78435
KNNW 28 0.39147 0.35952 0.38141 0.34893 0.42206 0.39171 0.79466
KNNW 100 0.37597 0.34320 0.39197 0.36004 0.38983 0.35779 0.88198
LOF 27 0.47674 0.44927 0.43236 0.40255 0.50463 0.47862 0.82972
LOF 28 0.47674 0.44927 0.43681 0.40723 0.50926 0.48349 0.82879
LOF 30 0.47674 0.44927 0.43858 0.40910 0.50575 0.47979 0.82727
SimplifiedLOF 26 0.50388 0.47782 0.42785 0.39780 0.50657 0.48065 0.82480
SimplifiedLOF 27 0.50388 0.47782 0.42895 0.39897 0.51341 0.48786 0.82052
SimplifiedLOF 28 0.51163 0.48598 0.43284 0.40306 0.51243 0.48682 0.82067
SimplifiedLOF 38 0.48450 0.45743 0.44912 0.42019 0.49293 0.46630 0.81870
LoOP 32 0.44961 0.42071 0.37029 0.33722 0.45049 0.42163 0.80982
LoOP 33 0.43798 0.40847 0.37377 0.34088 0.45055 0.42170 0.81113
LoOP 34 0.44186 0.41255 0.37739 0.34470 0.44737 0.41835 0.81297
LDOF 47 0.44961 0.42071 0.40392 0.37261 0.48464 0.45758 0.84090
LDOF 68 0.46124 0.43295 0.41092 0.37999 0.47253 0.44483 0.83835
LDOF 98 0.44186 0.41255 0.41669 0.38606 0.44677 0.41772 0.87063
LDOF 99 0.43411 0.40439 0.41696 0.38634 0.44764 0.41863 0.87002
ODIN 39 0.38490 0.35259 0.25483 0.21570 0.39781 0.36619 0.72387
ODIN 43 0.39147 0.35952 0.26064 0.22182 0.39394 0.36211 0.73184
ODIN 99 0.37689 0.34417 0.32343 0.28790 0.39198 0.36005 0.76498
ODIN 100 0.37745 0.34475 0.32332 0.28779 0.38877 0.35667 0.76626
FastABOD 3 0.35659 0.32280 0.30150 0.26482 0.36765 0.33444 0.75393
FastABOD 4 0.35659 0.32280 0.33933 0.30463 0.38636 0.35414 0.75323
KDEOS 60 0.10853 0.06171 0.10993 0.06319 0.21680 0.17567 0.71716
KDEOS 72 0.10853 0.06171 0.10691 0.06001 0.22111 0.18020 0.71352
KDEOS 91 0.13953 0.09435 0.10900 0.06221 0.20729 0.16566 0.70981
KDEOS 100 0.13566 0.09027 0.11002 0.06329 0.20667 0.16501 0.70751
LDF 26 0.49612 0.46966 0.48119 0.45394 0.53333 0.50883 0.83810
LDF 29 0.50388 0.47782 0.47400 0.44638 0.52026 0.49506 0.83925
LDF 100 0.36434 0.33096 0.35799 0.32428 0.38543 0.35316 0.85740
INFLO 28 0.44574 0.41663 0.35982 0.32620 0.45957 0.43119 0.74262
INFLO 31 0.43411 0.40439 0.37084 0.33780 0.44743 0.41841 0.74798
COF 35 0.47287 0.44519 0.42827 0.39824 0.48361 0.45649 0.79049
COF 39 0.48062 0.45335 0.43056 0.40065 0.49458 0.46804 0.77049
COF 43 0.48062 0.45335 0.43178 0.40194 0.50455 0.47853 0.77178
COF 44 0.46512 0.43703 0.43259 0.40279 0.50337 0.47729 0.76953

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (44.6 MB) Download raw algorithm evaluation table (71.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.17578 0.13257 0.13776 0.09255 0.17787 0.13477 0.56396
KNN 2 0.16797 0.12435 0.13962 0.09451 0.18130 0.13838 0.57734
KNN 10 0.16016 0.11613 0.13745 0.09223 0.18394 0.14116 0.60513
KNN 44 0.12891 0.08324 0.12746 0.08172 0.16479 0.12101 0.61154
KNNW 3 0.16797 0.12435 0.13724 0.09201 0.17790 0.13480 0.56831
KNNW 11 0.14844 0.10379 0.13932 0.09420 0.17486 0.13160 0.59413
KNNW 19 0.14062 0.09557 0.13615 0.09086 0.18625 0.14358 0.60123
KNNW 72 0.12891 0.08324 0.12850 0.08281 0.16728 0.12362 0.60886
LOF 94 0.50391 0.47790 0.48500 0.45800 0.52430 0.49936 0.92406
LOF 95 0.50000 0.47379 0.48530 0.45832 0.52613 0.50129 0.92431
LOF 100 0.49609 0.46968 0.48561 0.45864 0.54243 0.51844 0.92382
SimplifiedLOF 95 0.50000 0.47379 0.46366 0.43554 0.50099 0.47483 0.91886
SimplifiedLOF 98 0.49609 0.46968 0.46386 0.43575 0.50299 0.47694 0.92017
SimplifiedLOF 100 0.49609 0.46968 0.46593 0.43793 0.50198 0.47587 0.92110
LoOP 94 0.47266 0.44501 0.46573 0.43772 0.48936 0.46259 0.90207
LoOP 100 0.48047 0.45323 0.46839 0.44052 0.48771 0.46086 0.90755
LDOF 94 0.41797 0.38745 0.43182 0.40203 0.45089 0.42210 0.88530
LDOF 98 0.43359 0.40390 0.43421 0.40455 0.44860 0.41969 0.89010
LDOF 100 0.43359 0.40390 0.43441 0.40476 0.44907 0.42019 0.89290
ODIN 97 0.40000 0.36854 0.36861 0.33551 0.40260 0.37128 0.88128
ODIN 100 0.39499 0.36327 0.37445 0.34166 0.40789 0.37685 0.88466
FastABOD 4 0.13672 0.09146 0.10774 0.06096 0.13725 0.09202 0.46710
FastABOD 8 0.12891 0.08324 0.10928 0.06259 0.13661 0.09135 0.46918
FastABOD 29 0.12891 0.08324 0.10901 0.06230 0.14224 0.09727 0.46984
FastABOD 98 0.12891 0.08324 0.10895 0.06223 0.13779 0.09258 0.47121
KDEOS 86 0.10156 0.05446 0.09482 0.04736 0.16027 0.11625 0.72055
KDEOS 98 0.09375 0.04624 0.09830 0.05103 0.17158 0.12815 0.73874
KDEOS 99 0.09375 0.04624 0.09846 0.05119 0.17108 0.12762 0.73970
KDEOS 100 0.08594 0.03802 0.09843 0.05117 0.17115 0.12770 0.74086
LDF 63 0.49219 0.46556 0.49077 0.46408 0.53165 0.50709 0.92100
LDF 70 0.51172 0.48612 0.49419 0.46768 0.55862 0.53548 0.92096
LDF 78 0.51562 0.49023 0.48942 0.46265 0.56153 0.53854 0.92004
LDF 93 0.52344 0.49845 0.48400 0.45695 0.55379 0.53040 0.91719
INFLO 92 0.48438 0.45734 0.43910 0.40969 0.49524 0.46878 0.86636
INFLO 94 0.49219 0.46556 0.43936 0.40997 0.49516 0.46870 0.86701
INFLO 100 0.48828 0.46145 0.44638 0.41735 0.49156 0.46490 0.87941
COF 93 0.50781 0.48201 0.48017 0.45292 0.52008 0.49492 0.85115
COF 100 0.52734 0.50256 0.48814 0.46131 0.53414 0.50971 0.84927

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (44.7 MB) Download raw algorithm evaluation table (71.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.20930 0.16778 0.15805 0.11383 0.21497 0.17375 0.51291
KNN 3 0.18992 0.14738 0.15934 0.11520 0.20645 0.16478 0.53151
KNN 72 0.13178 0.08619 0.13608 0.09071 0.15536 0.11101 0.56546
KNNW 1 0.19380 0.15146 0.15168 0.10713 0.22118 0.18028 0.50579
KNNW 3 0.19767 0.15554 0.15787 0.11365 0.20417 0.16237 0.51628
KNNW 5 0.18992 0.14738 0.15830 0.11410 0.20285 0.16099 0.52622
KNNW 100 0.13953 0.09435 0.13930 0.09410 0.16535 0.12152 0.56125
LOF 89 0.46512 0.43703 0.46609 0.43805 0.47564 0.44811 0.87764
LOF 100 0.48062 0.45335 0.46461 0.43650 0.50842 0.48260 0.87997
SimplifiedLOF 80 0.47287 0.44519 0.44784 0.41885 0.47431 0.44670 0.86478
SimplifiedLOF 99 0.46899 0.44111 0.45184 0.42305 0.47732 0.44987 0.88386
SimplifiedLOF 100 0.46899 0.44111 0.45193 0.42315 0.47732 0.44987 0.88420
LoOP 84 0.43798 0.40847 0.44226 0.41297 0.47475 0.44716 0.85403
LoOP 100 0.45349 0.42479 0.45410 0.42543 0.45940 0.43101 0.87375
LDOF 76 0.41473 0.38399 0.41472 0.38399 0.44068 0.41131 0.84839
LDOF 93 0.42636 0.39623 0.42706 0.39697 0.43269 0.40290 0.86561
LDOF 100 0.41473 0.38399 0.43130 0.40143 0.43796 0.40844 0.87330
ODIN 100 0.37099 0.33795 0.35634 0.32254 0.38278 0.35036 0.85183
FastABOD 3 0.14729 0.10251 0.11178 0.06513 0.18329 0.14040 0.44567
FastABOD 4 0.14341 0.09843 0.12161 0.07548 0.18391 0.14105 0.44087
FastABOD 15 0.13566 0.09027 0.12859 0.08283 0.17109 0.12756 0.43333
KDEOS 89 0.07752 0.02908 0.08238 0.03420 0.15632 0.11201 0.68626
KDEOS 100 0.07752 0.02908 0.08598 0.03798 0.16357 0.11964 0.70069
LDF 55 0.44961 0.42071 0.47028 0.44247 0.48750 0.46059 0.87674
LDF 63 0.46124 0.43295 0.47280 0.44511 0.51176 0.48613 0.87051
LDF 79 0.50000 0.47374 0.46500 0.43691 0.53445 0.51001 0.85896
LDF 80 0.50775 0.48190 0.46477 0.43667 0.53109 0.50647 0.85781
INFLO 82 0.44961 0.42071 0.42265 0.39233 0.47205 0.44433 0.81338
INFLO 100 0.45736 0.42887 0.42991 0.39997 0.45965 0.43127 0.83730
COF 92 0.48062 0.45335 0.43322 0.40346 0.48249 0.45531 0.81149
COF 96 0.46512 0.43703 0.43733 0.40778 0.48178 0.45457 0.82367
COF 97 0.47287 0.44519 0.43831 0.40881 0.48760 0.46070 0.82200
COF 100 0.47674 0.44927 0.44065 0.41127 0.48621 0.45923 0.81921

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO