Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (5% of outliers version#03)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (43.4 MB) Download raw algorithm evaluation table (71.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 36 0.39453 0.36279 0.43782 0.40835 0.43553 0.40594 0.89372
KNN 59 0.42969 0.39979 0.44343 0.41425 0.46690 0.43895 0.89285
KNN 76 0.46094 0.43268 0.44592 0.41687 0.46184 0.43363 0.89193
KNN 99 0.45312 0.42445 0.44694 0.41795 0.45509 0.42652 0.89079
KNNW 9 0.40625 0.37512 0.38824 0.35617 0.41379 0.38306 0.82130
KNNW 92 0.38672 0.35457 0.44290 0.41369 0.45310 0.42443 0.89479
KNNW 99 0.37891 0.34634 0.44438 0.41525 0.46204 0.43383 0.89466
KNNW 100 0.38281 0.35046 0.44461 0.41549 0.46204 0.43383 0.89467
LOF 20 0.43750 0.40801 0.35989 0.32633 0.44278 0.41356 0.81593
LOF 24 0.42578 0.39568 0.37759 0.34496 0.45089 0.42210 0.81802
LOF 100 0.39844 0.36690 0.39397 0.36220 0.40514 0.37396 0.90586
SimplifiedLOF 25 0.46094 0.43268 0.37976 0.34725 0.47200 0.44432 0.80671
SimplifiedLOF 27 0.46484 0.43679 0.38755 0.35544 0.46718 0.43925 0.80737
SimplifiedLOF 39 0.44141 0.41212 0.40967 0.37872 0.45370 0.42506 0.80382
SimplifiedLOF 100 0.41016 0.37923 0.38868 0.35663 0.42182 0.39151 0.83239
LoOP 36 0.44922 0.42034 0.36020 0.32666 0.45149 0.42273 0.79521
LoOP 42 0.44531 0.41623 0.36592 0.33268 0.45322 0.42456 0.79256
LoOP 52 0.42969 0.39979 0.37395 0.34113 0.44676 0.41776 0.79040
LoOP 100 0.41016 0.37923 0.37227 0.33936 0.41667 0.38608 0.83225
LDOF 55 0.45703 0.42857 0.40247 0.37114 0.45882 0.43045 0.88831
LDOF 60 0.44922 0.42034 0.40760 0.37654 0.46694 0.43900 0.89338
LDOF 100 0.44141 0.41212 0.42010 0.38970 0.44841 0.41949 0.89832
ODIN 45 0.44832 0.41939 0.31234 0.27629 0.44970 0.42085 0.77038
ODIN 46 0.44575 0.41669 0.31476 0.27883 0.45020 0.42137 0.77007
ODIN 100 0.41406 0.38334 0.34720 0.31298 0.41953 0.38910 0.81990
FastABOD 20 0.31250 0.27646 0.30933 0.27312 0.32453 0.28912 0.73650
FastABOD 33 0.32422 0.28879 0.30963 0.27343 0.32422 0.28879 0.73626
FastABOD 85 0.32422 0.28879 0.31105 0.27493 0.34167 0.30715 0.73402
FastABOD 100 0.32422 0.28879 0.31173 0.27565 0.34167 0.30715 0.73344
KDEOS 63 0.15625 0.11201 0.12989 0.08427 0.22984 0.18947 0.72937
KDEOS 72 0.16797 0.12435 0.13073 0.08516 0.23146 0.19117 0.72797
KDEOS 95 0.15625 0.11201 0.13538 0.09005 0.24375 0.20410 0.72465
KDEOS 100 0.14844 0.10379 0.13121 0.08566 0.24558 0.20602 0.72751
LDF 100 0.46875 0.44090 0.46631 0.43833 0.53398 0.50955 0.92300
INFLO 30 0.43750 0.40801 0.34489 0.31055 0.43883 0.40941 0.72712
INFLO 37 0.43359 0.40390 0.35388 0.32001 0.44628 0.41725 0.72883
INFLO 40 0.43359 0.40390 0.35606 0.32231 0.43960 0.41022 0.73417
INFLO 48 0.41016 0.37923 0.35368 0.31980 0.41975 0.38933 0.74107
COF 27 0.38672 0.35457 0.35087 0.31684 0.40541 0.37423 0.78186
COF 33 0.42578 0.39568 0.37863 0.34605 0.43373 0.40405 0.77190
COF 43 0.41016 0.37923 0.40169 0.37033 0.43437 0.40471 0.77868
COF 54 0.41016 0.37923 0.38956 0.35756 0.44244 0.41321 0.75579

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (43.5 MB) Download raw algorithm evaluation table (72.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.40698 0.37583 0.35103 0.31695 0.40876 0.37771 0.77681
KNN 75 0.39147 0.35952 0.41330 0.38249 0.44893 0.41999 0.90281
KNN 93 0.34496 0.31056 0.40964 0.37864 0.46906 0.44117 0.90171
KNNW 10 0.41085 0.37991 0.36399 0.33059 0.41681 0.38619 0.79801
KNNW 23 0.38760 0.35544 0.38237 0.34994 0.42686 0.39676 0.82305
KNNW 100 0.39535 0.36360 0.40723 0.37610 0.39535 0.36360 0.90043
LOF 21 0.45736 0.42887 0.38934 0.35727 0.46094 0.43263 0.83465
LOF 31 0.44961 0.42071 0.41270 0.38186 0.47619 0.44868 0.83150
LOF 100 0.35659 0.32280 0.33340 0.29839 0.37908 0.34648 0.83997
SimplifiedLOF 26 0.47287 0.44519 0.40097 0.36951 0.47740 0.44995 0.82654
SimplifiedLOF 31 0.47287 0.44519 0.41944 0.38895 0.47818 0.45078 0.83246
SimplifiedLOF 35 0.46512 0.43703 0.42568 0.39552 0.48087 0.45361 0.83069
SimplifiedLOF 36 0.46899 0.44111 0.42447 0.39425 0.48347 0.45635 0.82911
LoOP 33 0.41860 0.38807 0.36457 0.33120 0.43691 0.40734 0.81978
LoOP 37 0.42248 0.39215 0.37104 0.33801 0.44280 0.41354 0.81526
LoOP 39 0.43023 0.40031 0.37020 0.33713 0.44571 0.41661 0.80946
LoOP 40 0.43798 0.40847 0.37025 0.33718 0.44530 0.41617 0.80897
LDOF 34 0.42636 0.39623 0.38534 0.35307 0.47059 0.44279 0.89093
LDOF 35 0.44186 0.41255 0.39107 0.35909 0.47772 0.45029 0.88986
LDOF 48 0.46899 0.44111 0.40579 0.37458 0.47280 0.44511 0.85051
LDOF 54 0.45349 0.42479 0.40689 0.37574 0.46586 0.43781 0.84162
ODIN 41 0.41344 0.38263 0.26948 0.23112 0.41379 0.38301 0.73476
ODIN 42 0.40698 0.37583 0.27179 0.23355 0.41393 0.38316 0.73629
ODIN 100 0.38090 0.34839 0.31159 0.27544 0.38889 0.35680 0.78504
FastABOD 4 0.36822 0.33504 0.32859 0.29333 0.38136 0.34887 0.73495
KDEOS 41 0.10078 0.05355 0.10838 0.06155 0.20434 0.16256 0.72115
KDEOS 63 0.11240 0.06579 0.11750 0.07116 0.21321 0.17189 0.71663
KDEOS 92 0.13953 0.09435 0.11202 0.06539 0.19554 0.15329 0.70945
LDF 22 0.45349 0.42479 0.42329 0.39301 0.48758 0.46068 0.83849
LDF 27 0.46899 0.44111 0.43485 0.40517 0.47658 0.44909 0.84009
LDF 30 0.45349 0.42479 0.44101 0.41166 0.46437 0.43624 0.83809
LDF 100 0.37209 0.33912 0.36515 0.33181 0.40057 0.36909 0.90357
INFLO 27 0.42248 0.39215 0.34955 0.31539 0.42610 0.39597 0.75469
INFLO 30 0.43411 0.40439 0.35724 0.32349 0.44776 0.41876 0.74687
INFLO 31 0.43798 0.40847 0.35926 0.32561 0.44238 0.41310 0.74988
INFLO 33 0.44186 0.41255 0.35666 0.32288 0.44398 0.41478 0.75086
COF 30 0.43023 0.40031 0.37567 0.34289 0.45041 0.42155 0.78208
COF 40 0.44961 0.42071 0.39107 0.35909 0.47679 0.44932 0.75693
COF 45 0.44961 0.42071 0.39686 0.36518 0.49327 0.46666 0.75760
COF 46 0.44961 0.42071 0.39534 0.36359 0.49453 0.46799 0.75944

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (44.6 MB) Download raw algorithm evaluation table (70.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.20312 0.16135 0.19137 0.14897 0.23529 0.19520 0.59464
KNN 11 0.24219 0.20246 0.18931 0.14681 0.24750 0.20805 0.61210
KNN 44 0.17578 0.13257 0.17306 0.12971 0.19413 0.15188 0.61365
KNNW 1 0.21484 0.17368 0.18402 0.14124 0.24138 0.20161 0.55726
KNNW 6 0.20312 0.16135 0.19024 0.14779 0.24731 0.20785 0.58679
KNNW 7 0.20312 0.16135 0.19049 0.14805 0.24403 0.20440 0.58968
KNNW 65 0.17969 0.13668 0.17625 0.13306 0.19907 0.15708 0.61229
LOF 85 0.48828 0.46145 0.49896 0.47269 0.51057 0.48491 0.92280
LOF 89 0.48438 0.45734 0.50095 0.47479 0.51826 0.49300 0.92265
LOF 98 0.51172 0.48612 0.49883 0.47255 0.54639 0.52261 0.92149
LOF 99 0.50781 0.48201 0.49831 0.47201 0.55258 0.52913 0.92134
SimplifiedLOF 96 0.48828 0.46145 0.49380 0.46726 0.49383 0.46729 0.92380
SimplifiedLOF 100 0.48828 0.46145 0.49542 0.46897 0.49796 0.47164 0.92495
LoOP 62 0.47266 0.44501 0.45955 0.43121 0.48900 0.46221 0.87079
LoOP 69 0.46484 0.43679 0.46757 0.43966 0.49756 0.47122 0.87983
LoOP 100 0.46094 0.43268 0.49183 0.46519 0.49327 0.46671 0.91279
LDOF 70 0.46484 0.43679 0.45542 0.42687 0.46985 0.44206 0.87379
LDOF 95 0.45703 0.42857 0.48492 0.45791 0.49327 0.46671 0.89823
LDOF 97 0.46094 0.43268 0.48656 0.45965 0.48899 0.46220 0.90053
LDOF 100 0.46484 0.43679 0.48652 0.45960 0.49107 0.46439 0.90357
ODIN 94 0.41650 0.38591 0.37330 0.34045 0.42083 0.39047 0.87885
ODIN 98 0.41016 0.37923 0.38183 0.34942 0.42373 0.39352 0.88308
ODIN 100 0.41055 0.37964 0.38487 0.35262 0.42038 0.38999 0.88489
FastABOD 3 0.16797 0.12435 0.16003 0.11600 0.20649 0.16489 0.48302
FastABOD 5 0.17969 0.13668 0.16126 0.11728 0.20408 0.16235 0.48187
FastABOD 7 0.17969 0.13668 0.16230 0.11839 0.20710 0.16553 0.48174
KDEOS 95 0.10156 0.05446 0.10201 0.05494 0.18460 0.14185 0.74603
KDEOS 99 0.09375 0.04624 0.10375 0.05676 0.18875 0.14622 0.74944
KDEOS 100 0.10156 0.05446 0.10441 0.05746 0.18827 0.14571 0.75045
LDF 66 0.50781 0.48201 0.49749 0.47115 0.54017 0.51606 0.91610
LDF 72 0.51562 0.49023 0.50171 0.47558 0.55688 0.53364 0.91535
LDF 75 0.52344 0.49845 0.49851 0.47222 0.56068 0.53765 0.91515
LDF 86 0.51172 0.48612 0.49608 0.46967 0.56271 0.53979 0.91303
INFLO 72 0.45312 0.42445 0.44927 0.42040 0.48998 0.46324 0.83790
INFLO 96 0.48047 0.45323 0.46317 0.43502 0.48249 0.45536 0.87375
INFLO 99 0.48047 0.45323 0.46547 0.43745 0.48619 0.45925 0.87892
INFLO 100 0.48047 0.45323 0.46561 0.43759 0.48699 0.46009 0.87890
COF 89 0.45312 0.42445 0.45200 0.42327 0.48155 0.45437 0.86838
COF 93 0.48047 0.45323 0.45642 0.42793 0.49215 0.46552 0.86683
COF 100 0.47656 0.44912 0.46496 0.43691 0.51282 0.48728 0.86434

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (44.7 MB) Download raw algorithm evaluation table (71.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.20155 0.15962 0.14342 0.09844 0.20896 0.16741 0.52496
KNN 4 0.18992 0.14738 0.14661 0.10180 0.19847 0.15638 0.55670
KNN 14 0.19380 0.15146 0.14053 0.09539 0.20455 0.16277 0.57830
KNNW 1 0.18605 0.14330 0.14367 0.09870 0.21154 0.17013 0.51871
KNNW 3 0.20155 0.15962 0.14455 0.09963 0.20463 0.16287 0.53227
KNNW 5 0.20155 0.15962 0.14655 0.10173 0.20278 0.16092 0.54353
KNNW 36 0.15504 0.11067 0.13633 0.09098 0.18182 0.13885 0.57299
LOF 47 0.46512 0.43703 0.48870 0.46185 0.48908 0.46225 0.89488
LOF 48 0.46512 0.43703 0.48922 0.46240 0.48188 0.45467 0.89486
LOF 89 0.49225 0.46558 0.46743 0.43946 0.53807 0.51381 0.88774
LOF 97 0.50775 0.48190 0.45989 0.43153 0.53807 0.51381 0.88910
SimplifiedLOF 57 0.48062 0.45335 0.47923 0.45188 0.48598 0.45899 0.88072
SimplifiedLOF 88 0.48450 0.45743 0.47127 0.44351 0.49660 0.47016 0.89749
SimplifiedLOF 95 0.49612 0.46966 0.46929 0.44142 0.49730 0.47090 0.89701
SimplifiedLOF 100 0.49612 0.46966 0.46874 0.44084 0.51339 0.48783 0.89674
LoOP 83 0.48450 0.45743 0.48391 0.45680 0.49899 0.47268 0.89062
LoOP 84 0.49225 0.46558 0.48369 0.45658 0.49597 0.46950 0.89111
LoOP 97 0.49225 0.46558 0.48493 0.45788 0.49485 0.46832 0.89621
LoOP 98 0.48837 0.46150 0.48467 0.45761 0.49462 0.46808 0.89632
LDOF 96 0.45736 0.42887 0.46480 0.43669 0.47722 0.44977 0.90417
LDOF 97 0.45736 0.42887 0.46640 0.43838 0.47391 0.44629 0.90527
LDOF 98 0.46124 0.43295 0.46531 0.43723 0.47289 0.44520 0.90584
LDOF 100 0.46124 0.43295 0.46455 0.43644 0.47391 0.44629 0.90670
ODIN 86 0.43023 0.40031 0.38054 0.34801 0.43023 0.40031 0.87054
ODIN 100 0.42636 0.39623 0.40548 0.37426 0.44033 0.41094 0.88225
FastABOD 3 0.15116 0.10659 0.11359 0.06704 0.17647 0.13322 0.46630
FastABOD 4 0.15116 0.10659 0.12006 0.07385 0.17127 0.12775 0.46332
KDEOS 10 0.08527 0.03724 0.06662 0.01761 0.12285 0.07679 0.59107
KDEOS 100 0.07364 0.02500 0.09678 0.04935 0.18761 0.14495 0.73358
LDF 30 0.46512 0.43703 0.51441 0.48891 0.48523 0.45819 0.89389
LDF 34 0.45736 0.42887 0.50707 0.48118 0.48411 0.45702 0.89589
LDF 70 0.49225 0.46558 0.45929 0.43090 0.53403 0.50956 0.87070
LDF 72 0.50388 0.47782 0.45665 0.42812 0.53310 0.50858 0.86871
INFLO 57 0.48450 0.45743 0.45861 0.43018 0.48638 0.45941 0.83397
INFLO 59 0.48837 0.46150 0.45695 0.42843 0.49174 0.46504 0.83475
INFLO 69 0.48837 0.46150 0.45279 0.42406 0.50312 0.47703 0.85050
INFLO 94 0.48450 0.45743 0.45335 0.42464 0.48948 0.46267 0.86783
COF 66 0.45349 0.42479 0.45372 0.42503 0.47479 0.44721 0.84974
COF 78 0.46512 0.43703 0.45067 0.42182 0.48536 0.45833 0.85880
COF 93 0.49612 0.46966 0.45157 0.42276 0.52095 0.49579 0.84685
COF 98 0.51163 0.48598 0.45235 0.42360 0.51557 0.49013 0.83653

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO