Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (5% of outliers version#04)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (43.4 MB) Download raw algorithm evaluation table (71.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 24 0.44531 0.41623 0.45306 0.42438 0.45752 0.42908 0.91182
KNN 30 0.43750 0.40801 0.47346 0.44586 0.46575 0.43774 0.91067
KNN 63 0.48047 0.45323 0.46763 0.43971 0.49722 0.47086 0.90578
KNNW 57 0.42578 0.39568 0.46190 0.43369 0.45605 0.42753 0.91168
KNNW 82 0.45703 0.42857 0.47020 0.44243 0.47604 0.44857 0.91048
KNNW 98 0.44922 0.42034 0.47122 0.44350 0.48288 0.45577 0.90922
KNNW 99 0.44922 0.42034 0.47091 0.44318 0.48454 0.45751 0.90917
LOF 32 0.44141 0.41212 0.36787 0.33473 0.45135 0.42258 0.83620
LOF 38 0.43750 0.40801 0.37627 0.34357 0.45872 0.43034 0.83770
LOF 100 0.38672 0.35457 0.40837 0.37736 0.42424 0.39406 0.92134
SimplifiedLOF 42 0.44531 0.41623 0.38705 0.35491 0.46593 0.43793 0.81088
SimplifiedLOF 80 0.45703 0.42857 0.40024 0.36880 0.45703 0.42857 0.84290
SimplifiedLOF 100 0.44141 0.41212 0.40700 0.37591 0.44898 0.42009 0.87459
LoOP 98 0.42188 0.39157 0.37640 0.34371 0.43058 0.40073 0.85860
LoOP 100 0.42969 0.39979 0.37727 0.34462 0.42969 0.39979 0.85999
LDOF 76 0.45703 0.42857 0.40029 0.36885 0.47434 0.44678 0.89700
LDOF 95 0.44531 0.41623 0.41564 0.38501 0.48263 0.45551 0.90563
LDOF 100 0.44531 0.41623 0.41974 0.38932 0.48059 0.45336 0.90747
ODIN 81 0.42773 0.39773 0.33919 0.30454 0.42885 0.39891 0.81590
ODIN 88 0.42097 0.39062 0.34096 0.30641 0.42910 0.39917 0.82347
ODIN 99 0.41374 0.38300 0.34448 0.31011 0.41975 0.38933 0.84046
ODIN 100 0.41211 0.38129 0.34357 0.30915 0.41935 0.38891 0.84197
FastABOD 6 0.39453 0.36279 0.37214 0.33922 0.40568 0.37452 0.79155
FastABOD 12 0.38672 0.35457 0.36778 0.33464 0.40074 0.36932 0.79332
FastABOD 72 0.40234 0.37101 0.36960 0.33655 0.41113 0.38026 0.78888
FastABOD 88 0.40625 0.37512 0.36976 0.33672 0.41026 0.37934 0.78815
KDEOS 82 0.16797 0.12435 0.13319 0.08775 0.22006 0.17918 0.74130
KDEOS 100 0.14062 0.09557 0.13773 0.09252 0.23875 0.19884 0.75855
LDF 96 0.42578 0.39568 0.46544 0.43741 0.54494 0.52109 0.93343
LDF 98 0.44141 0.41212 0.46812 0.44023 0.54266 0.51868 0.93360
LDF 100 0.44922 0.42034 0.47013 0.44235 0.54266 0.51868 0.93356
INFLO 60 0.41016 0.37923 0.33375 0.29882 0.41257 0.38178 0.75608
INFLO 72 0.40234 0.37101 0.33698 0.30222 0.41649 0.38590 0.75873
INFLO 99 0.40234 0.37101 0.33927 0.30463 0.40964 0.37869 0.76743
COF 45 0.43750 0.40801 0.38159 0.34917 0.44094 0.41164 0.80183
COF 64 0.42188 0.39157 0.39402 0.36225 0.45411 0.42549 0.78537
COF 81 0.43359 0.40390 0.39842 0.36688 0.44856 0.41965 0.78407
COF 90 0.44922 0.42034 0.39431 0.36256 0.45174 0.42299 0.77971

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (43.5 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 31 0.41473 0.38399 0.41621 0.38555 0.41473 0.38399 0.82901
KNN 100 0.35659 0.32280 0.45288 0.42415 0.47695 0.44948 0.90312
KNNW 23 0.40698 0.37583 0.40285 0.37150 0.44186 0.41255 0.80449
KNNW 24 0.41085 0.37991 0.40372 0.37241 0.44084 0.41147 0.80483
KNNW 100 0.39535 0.36360 0.43527 0.40561 0.41723 0.38663 0.89319
LOF 27 0.46899 0.44111 0.42180 0.39144 0.48276 0.45560 0.80601
LOF 28 0.47674 0.44927 0.41811 0.38755 0.49024 0.46347 0.80365
LOF 31 0.47287 0.44519 0.41970 0.38922 0.49670 0.47027 0.80281
LOF 39 0.43798 0.40847 0.40884 0.37780 0.44351 0.41429 0.81029
SimplifiedLOF 26 0.47287 0.44519 0.41367 0.38288 0.48065 0.45338 0.80556
SimplifiedLOF 28 0.48450 0.45743 0.42195 0.39160 0.49310 0.46648 0.80296
SimplifiedLOF 31 0.48062 0.45335 0.42430 0.39406 0.49661 0.47018 0.79913
SimplifiedLOF 39 0.46124 0.43295 0.43902 0.40956 0.48085 0.45359 0.79838
LoOP 27 0.39922 0.36768 0.33635 0.30150 0.40137 0.36994 0.79019
LoOP 34 0.44574 0.41663 0.36480 0.33144 0.44800 0.41901 0.78707
LoOP 40 0.43411 0.40439 0.36971 0.33661 0.44167 0.41235 0.78601
LDOF 41 0.45349 0.42479 0.39273 0.36085 0.47653 0.44905 0.85225
LDOF 43 0.46124 0.43295 0.39441 0.36261 0.47568 0.44814 0.84841
LDOF 100 0.44574 0.41663 0.41187 0.38098 0.45545 0.42685 0.86082
ODIN 52 0.39535 0.36360 0.28857 0.25121 0.39535 0.36360 0.72684
ODIN 73 0.37955 0.34696 0.30971 0.27346 0.39913 0.36758 0.75235
ODIN 90 0.39147 0.35952 0.32182 0.28621 0.40909 0.37806 0.74816
ODIN 100 0.38984 0.35780 0.32414 0.28864 0.40605 0.37486 0.75061
FastABOD 4 0.39147 0.35952 0.33780 0.30302 0.40930 0.37828 0.76164
FastABOD 100 0.38372 0.35136 0.34833 0.31411 0.38929 0.35721 0.70448
KDEOS 49 0.11240 0.06579 0.10627 0.05933 0.18593 0.14318 0.69534
KDEOS 59 0.12403 0.07803 0.10992 0.06318 0.19150 0.14905 0.69355
KDEOS 64 0.13566 0.09027 0.10971 0.06296 0.19482 0.15254 0.69279
KDEOS 67 0.12791 0.08211 0.10751 0.06064 0.20029 0.15830 0.69218
LDF 23 0.49612 0.46966 0.46403 0.43588 0.52915 0.50442 0.82092
LDF 30 0.46899 0.44111 0.46686 0.43887 0.50450 0.47848 0.82324
LDF 100 0.36822 0.33504 0.36286 0.32940 0.39597 0.36425 0.85148
INFLO 26 0.39922 0.36768 0.33718 0.30237 0.40773 0.37662 0.73597
INFLO 34 0.42636 0.39623 0.35111 0.31704 0.44397 0.41478 0.72152
INFLO 35 0.43411 0.40439 0.35187 0.31783 0.43564 0.40601 0.72077
INFLO 38 0.41085 0.37991 0.35319 0.31922 0.42630 0.39618 0.71790
COF 36 0.48062 0.45335 0.41341 0.38261 0.49794 0.47158 0.78100
COF 38 0.49225 0.46558 0.41114 0.38022 0.50916 0.48339 0.75104
COF 45 0.46899 0.44111 0.41260 0.38176 0.51739 0.49205 0.74364
COF 49 0.44961 0.42071 0.41803 0.38747 0.50226 0.47612 0.73600

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (44.6 MB) Download raw algorithm evaluation table (71.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.17578 0.13257 0.15074 0.10622 0.20202 0.16018 0.59462
KNN 8 0.19531 0.15313 0.14852 0.10388 0.20202 0.16018 0.60993
KNN 20 0.16016 0.11613 0.14616 0.10140 0.21346 0.17223 0.62020
KNN 21 0.16016 0.11613 0.14568 0.10089 0.21346 0.17223 0.62050
KNNW 1 0.17578 0.13257 0.15222 0.10778 0.20305 0.16126 0.55512
KNNW 3 0.18750 0.14490 0.15064 0.10611 0.20755 0.16600 0.57266
KNNW 56 0.16016 0.11613 0.14063 0.09557 0.19565 0.15348 0.61818
LOF 58 0.51562 0.49023 0.50679 0.48093 0.51562 0.49023 0.91811
LOF 84 0.50391 0.47790 0.50370 0.47768 0.54388 0.51997 0.92666
LOF 98 0.51953 0.49434 0.49749 0.47114 0.56766 0.54499 0.92568
LOF 100 0.51953 0.49434 0.49553 0.46908 0.57002 0.54747 0.92545
SimplifiedLOF 88 0.50781 0.48201 0.49211 0.46548 0.52174 0.49667 0.92544
SimplifiedLOF 98 0.52344 0.49845 0.49152 0.46486 0.52574 0.50087 0.92805
SimplifiedLOF 100 0.52344 0.49845 0.49136 0.46470 0.52783 0.50307 0.92823
LoOP 83 0.50000 0.47379 0.47849 0.45115 0.50617 0.48028 0.90769
LoOP 99 0.50000 0.47379 0.48850 0.46168 0.51613 0.49076 0.91822
LoOP 100 0.50000 0.47379 0.48806 0.46123 0.51880 0.49357 0.91847
LDOF 99 0.45312 0.42445 0.45516 0.42660 0.48115 0.45395 0.91049
LDOF 100 0.44531 0.41623 0.45573 0.42720 0.48336 0.45628 0.91149
ODIN 100 0.44043 0.41109 0.39399 0.36222 0.44402 0.41487 0.89366
FastABOD 3 0.14062 0.09557 0.12348 0.07752 0.15730 0.11312 0.48035
FastABOD 4 0.13672 0.09146 0.12322 0.07725 0.15864 0.11453 0.47964
FastABOD 5 0.13281 0.08735 0.12242 0.07641 0.15205 0.10759 0.48064
KDEOS 73 0.10156 0.05446 0.08768 0.03985 0.16275 0.11885 0.70839
KDEOS 100 0.09375 0.04624 0.10521 0.05830 0.19053 0.14809 0.75006
LDF 33 0.48438 0.45734 0.52232 0.49728 0.50663 0.48076 0.90076
LDF 65 0.51172 0.48612 0.50133 0.47518 0.56000 0.53693 0.91830
LDF 70 0.50781 0.48201 0.50032 0.47413 0.57241 0.55000 0.91781
LDF 89 0.52344 0.49845 0.48467 0.45765 0.56616 0.54342 0.91485
INFLO 80 0.50391 0.47790 0.45740 0.42896 0.50652 0.48065 0.86653
INFLO 98 0.48438 0.45734 0.47121 0.44349 0.52340 0.49841 0.89157
INFLO 99 0.48438 0.45734 0.47072 0.44297 0.52613 0.50129 0.89158
COF 88 0.49609 0.46968 0.46214 0.43394 0.50000 0.47379 0.88690
COF 94 0.50000 0.47379 0.46618 0.43819 0.51838 0.49313 0.88588
COF 99 0.51172 0.48612 0.46721 0.43928 0.51462 0.48917 0.88241

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (44.7 MB) Download raw algorithm evaluation table (71.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.19380 0.15146 0.17938 0.13629 0.21364 0.17234 0.56651
KNN 14 0.20155 0.15962 0.17755 0.13436 0.23571 0.19558 0.59014
KNN 50 0.15116 0.10659 0.16456 0.12069 0.19236 0.14995 0.59134
KNNW 1 0.18992 0.14738 0.17582 0.13254 0.22287 0.18206 0.51781
KNNW 3 0.19767 0.15554 0.17401 0.13063 0.21530 0.17409 0.54012
KNNW 8 0.18992 0.14738 0.17744 0.13424 0.21429 0.17302 0.56249
KNNW 97 0.15116 0.10659 0.16410 0.12020 0.19220 0.14978 0.58780
LOF 70 0.50388 0.47782 0.50704 0.48115 0.50958 0.48382 0.89675
LOF 96 0.51163 0.48598 0.49260 0.46595 0.53217 0.50761 0.89767
LOF 100 0.50388 0.47782 0.48901 0.46218 0.53608 0.51172 0.89687
SimplifiedLOF 81 0.50000 0.47374 0.50020 0.47395 0.50783 0.48198 0.89643
SimplifiedLOF 92 0.51163 0.48598 0.49779 0.47142 0.51677 0.49139 0.90130
SimplifiedLOF 97 0.51163 0.48598 0.49687 0.47045 0.51881 0.49354 0.90273
SimplifiedLOF 100 0.51163 0.48598 0.49714 0.47074 0.51923 0.49398 0.90265
LoOP 93 0.47674 0.44927 0.48588 0.45889 0.50111 0.47491 0.89707
LoOP 99 0.48837 0.46150 0.49026 0.46349 0.50000 0.47374 0.90037
LoOP 100 0.48837 0.46150 0.49066 0.46391 0.50000 0.47374 0.90035
LDOF 63 0.45349 0.42479 0.40994 0.37896 0.45525 0.42665 0.86266
LDOF 96 0.43798 0.40847 0.46179 0.43353 0.47912 0.45177 0.90482
LDOF 99 0.45349 0.42479 0.46494 0.43684 0.47534 0.44778 0.90794
LDOF 100 0.44574 0.41663 0.46454 0.43642 0.47511 0.44755 0.90840
ODIN 98 0.40698 0.37583 0.35692 0.32315 0.41237 0.38151 0.87303
ODIN 100 0.40698 0.37583 0.36051 0.32692 0.41434 0.38359 0.87559
FastABOD 3 0.18992 0.14738 0.15450 0.11010 0.22961 0.18915 0.48856
FastABOD 4 0.18992 0.14738 0.16911 0.12548 0.23313 0.19286 0.48934
KDEOS 10 0.09690 0.04947 0.06612 0.01708 0.11845 0.07216 0.57629
KDEOS 95 0.07752 0.02908 0.08686 0.03891 0.16697 0.12322 0.70613
KDEOS 100 0.08915 0.04132 0.08923 0.04140 0.16611 0.12232 0.71147
LDF 46 0.48450 0.45743 0.51908 0.49383 0.52282 0.49776 0.89896
LDF 55 0.48062 0.45335 0.52010 0.49490 0.52480 0.49985 0.89107
LDF 62 0.50000 0.47374 0.51223 0.48661 0.54419 0.52026 0.88611
LDF 64 0.50775 0.48190 0.50887 0.48308 0.53872 0.51450 0.88329
INFLO 93 0.49612 0.46966 0.47255 0.44485 0.50098 0.47477 0.86803
INFLO 99 0.50000 0.47374 0.47132 0.44356 0.50297 0.47687 0.86834
INFLO 100 0.50000 0.47374 0.47061 0.44281 0.50688 0.48098 0.86811
COF 73 0.49225 0.46558 0.46023 0.43188 0.49910 0.47279 0.84920
COF 93 0.50775 0.48190 0.46138 0.43310 0.51064 0.48494 0.83176
COF 96 0.50388 0.47782 0.46261 0.43438 0.51190 0.48627 0.83156
COF 100 0.49612 0.46966 0.46473 0.43662 0.50763 0.48178 0.82756

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO