Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (5% of outliers version#01)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (43.4 MB) Download raw algorithm evaluation table (71.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 47 0.42969 0.39979 0.43559 0.40600 0.47350 0.44590 0.90448
KNN 59 0.46094 0.43268 0.43759 0.40811 0.48619 0.45925 0.90353
KNN 60 0.46094 0.43268 0.43705 0.40754 0.48987 0.46313 0.90337
KNN 61 0.46875 0.44090 0.43745 0.40796 0.48816 0.46133 0.90338
KNNW 10 0.42188 0.39157 0.38097 0.34852 0.42424 0.39406 0.85587
KNNW 79 0.41797 0.38745 0.43332 0.40361 0.45750 0.42905 0.90444
KNNW 94 0.41797 0.38745 0.43491 0.40528 0.46589 0.43789 0.90424
KNNW 100 0.41797 0.38745 0.43450 0.40485 0.47179 0.44410 0.90414
LOF 28 0.40625 0.37512 0.33093 0.29586 0.42857 0.39861 0.80761
LOF 72 0.42188 0.39157 0.36847 0.33536 0.42270 0.39243 0.89159
LOF 100 0.39453 0.36279 0.38423 0.35195 0.41288 0.38210 0.91740
SimplifiedLOF 32 0.44141 0.41212 0.34451 0.31015 0.45403 0.42541 0.79635
SimplifiedLOF 100 0.43359 0.40390 0.38117 0.34873 0.43359 0.40390 0.86226
LoOP 98 0.43359 0.40390 0.35552 0.32173 0.43495 0.40533 0.84362
LoOP 100 0.43359 0.40390 0.35665 0.32292 0.43714 0.40763 0.84629
LDOF 64 0.44141 0.41212 0.38980 0.35781 0.47207 0.44439 0.88364
LDOF 81 0.45703 0.42857 0.39691 0.36529 0.46092 0.43266 0.89117
LDOF 100 0.45312 0.42445 0.40557 0.37440 0.46435 0.43627 0.90129
ODIN 55 0.43043 0.40057 0.30951 0.27331 0.43053 0.40067 0.78542
ODIN 71 0.41242 0.38161 0.32120 0.28562 0.43234 0.40258 0.79181
ODIN 81 0.41843 0.38794 0.31856 0.28283 0.43636 0.40681 0.79241
ODIN 100 0.41589 0.38526 0.32033 0.28469 0.41941 0.38897 0.81481
FastABOD 30 0.35547 0.32168 0.32769 0.29244 0.36320 0.32981 0.76243
FastABOD 43 0.36328 0.32990 0.32988 0.29474 0.36408 0.33074 0.76194
FastABOD 95 0.35938 0.32579 0.33333 0.29837 0.37121 0.33825 0.75913
FastABOD 100 0.36328 0.32990 0.33386 0.29894 0.37121 0.33825 0.75898
KDEOS 93 0.16016 0.11613 0.12927 0.08362 0.23116 0.19086 0.73539
KDEOS 98 0.17188 0.12846 0.13097 0.08541 0.22488 0.18424 0.73799
KDEOS 100 0.16406 0.12024 0.13010 0.08450 0.22626 0.18570 0.73891
LDF 26 0.41797 0.38745 0.35979 0.32622 0.42623 0.39615 0.80772
LDF 100 0.40234 0.37101 0.45211 0.42339 0.53571 0.51137 0.93583
INFLO 66 0.42188 0.39157 0.32235 0.28682 0.42270 0.39243 0.74447
INFLO 69 0.41016 0.37923 0.32440 0.28898 0.41584 0.38522 0.74934
INFLO 100 0.39453 0.36279 0.32110 0.28551 0.40385 0.37259 0.75375
COF 32 0.37500 0.34223 0.33273 0.29775 0.38665 0.35449 0.77014
COF 48 0.45312 0.42445 0.37899 0.34643 0.45312 0.42445 0.76454
COF 51 0.43359 0.40390 0.37738 0.34473 0.45322 0.42456 0.76228

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (43.5 MB) Download raw algorithm evaluation table (72.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.45736 0.42887 0.40764 0.37653 0.46365 0.43549 0.80618
KNN 100 0.43023 0.40031 0.44656 0.41750 0.45588 0.42731 0.90466
KNNW 6 0.44574 0.41663 0.40986 0.37887 0.45657 0.42803 0.80930
KNNW 26 0.41473 0.38399 0.42644 0.39632 0.45833 0.42989 0.83368
KNNW 99 0.41473 0.38399 0.43719 0.40764 0.41879 0.38827 0.90101
KNNW 100 0.41473 0.38399 0.43719 0.40764 0.41879 0.38827 0.90109
LOF 25 0.50388 0.47782 0.43321 0.40344 0.51368 0.48815 0.83786
LOF 26 0.49225 0.46558 0.43395 0.40422 0.50980 0.48406 0.83888
LOF 100 0.39922 0.36768 0.36355 0.33013 0.40685 0.37570 0.85723
SimplifiedLOF 28 0.49612 0.46966 0.42511 0.39492 0.50086 0.47465 0.83083
SimplifiedLOF 29 0.49612 0.46966 0.42938 0.39941 0.50177 0.47560 0.83140
SimplifiedLOF 32 0.48450 0.45743 0.43770 0.40817 0.50554 0.47958 0.82615
SimplifiedLOF 37 0.47674 0.44927 0.44139 0.41206 0.50215 0.47600 0.82443
LoOP 30 0.42248 0.39215 0.35401 0.32009 0.43590 0.40627 0.81647
LoOP 42 0.46124 0.43295 0.37402 0.34115 0.46422 0.43608 0.81159
LoOP 52 0.45349 0.42479 0.37922 0.34662 0.45992 0.43155 0.80549
LDOF 83 0.48837 0.46150 0.42512 0.39493 0.48837 0.46150 0.88061
LDOF 85 0.48062 0.45335 0.42673 0.39662 0.49293 0.46630 0.88333
LDOF 92 0.48450 0.45743 0.42879 0.39880 0.49336 0.46675 0.87664
LDOF 96 0.48837 0.46150 0.42767 0.39762 0.49615 0.46970 0.87381
ODIN 68 0.42918 0.39920 0.32112 0.28547 0.42940 0.39944 0.76568
ODIN 86 0.41783 0.38726 0.34019 0.30554 0.44161 0.41229 0.77126
ODIN 98 0.41809 0.38753 0.34239 0.30785 0.43187 0.40203 0.78321
ODIN 100 0.41085 0.37991 0.34177 0.30720 0.42947 0.39951 0.78610
FastABOD 4 0.39535 0.36360 0.32651 0.29115 0.40605 0.37486 0.74840
FastABOD 13 0.41085 0.37991 0.36263 0.32916 0.43818 0.40867 0.73451
FastABOD 90 0.41860 0.38807 0.37188 0.33890 0.42572 0.39556 0.73076
FastABOD 99 0.42248 0.39215 0.37169 0.33869 0.42581 0.39565 0.73014
KDEOS 58 0.13178 0.08619 0.10846 0.06164 0.19643 0.15423 0.72072
KDEOS 63 0.15116 0.10659 0.10958 0.06282 0.19482 0.15253 0.72022
KDEOS 100 0.14729 0.10251 0.11567 0.06923 0.22698 0.18638 0.71400
LDF 21 0.48837 0.46150 0.46933 0.44146 0.51152 0.48587 0.84264
LDF 24 0.47287 0.44519 0.47332 0.44566 0.50360 0.47753 0.84128
LDF 100 0.40698 0.37583 0.40482 0.37357 0.40856 0.37750 0.91616
INFLO 26 0.44961 0.42071 0.35927 0.32562 0.45049 0.42163 0.75293
INFLO 30 0.44961 0.42071 0.37095 0.33791 0.46502 0.43693 0.74583
INFLO 33 0.46124 0.43295 0.36778 0.33458 0.46781 0.43986 0.72781
INFLO 38 0.45349 0.42479 0.36734 0.33411 0.46894 0.44105 0.72999
COF 28 0.45736 0.42887 0.40148 0.37005 0.46065 0.43233 0.80415
COF 33 0.47674 0.44927 0.41993 0.38946 0.52370 0.49869 0.79004
COF 54 0.45349 0.42479 0.43684 0.40727 0.49524 0.46873 0.75882

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (44.6 MB) Download raw algorithm evaluation table (71.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.16016 0.11613 0.14101 0.09597 0.16993 0.12642 0.56130
KNN 2 0.15234 0.10790 0.14283 0.09789 0.16092 0.11693 0.57598
KNN 15 0.13672 0.09146 0.14086 0.09582 0.18062 0.13767 0.61240
KNN 17 0.13281 0.08735 0.14022 0.09514 0.18182 0.13892 0.61206
KNNW 2 0.16016 0.11613 0.14014 0.09506 0.16637 0.12266 0.55413
KNNW 8 0.13281 0.08735 0.14240 0.09744 0.16138 0.11742 0.58892
KNNW 22 0.13281 0.08735 0.14087 0.09583 0.18359 0.14079 0.60642
KNNW 59 0.12891 0.08324 0.13391 0.08850 0.16909 0.12553 0.61028
LOF 62 0.51953 0.49434 0.53268 0.50818 0.52692 0.50212 0.92763
LOF 87 0.53125 0.50667 0.52590 0.50104 0.55903 0.53591 0.93406
LOF 93 0.53125 0.50667 0.52195 0.49689 0.56530 0.54250 0.93359
LOF 100 0.54297 0.51901 0.51449 0.48903 0.56176 0.53878 0.93223
SimplifiedLOF 91 0.52734 0.50256 0.51752 0.49222 0.53476 0.51037 0.93009
SimplifiedLOF 94 0.53125 0.50667 0.51662 0.49128 0.53986 0.51573 0.93062
SimplifiedLOF 100 0.52734 0.50256 0.51393 0.48845 0.55098 0.52744 0.93146
LoOP 69 0.51172 0.48612 0.47443 0.44687 0.51272 0.48717 0.89881
LoOP 80 0.49219 0.46556 0.49404 0.46751 0.52079 0.49566 0.91263
LoOP 100 0.50781 0.48201 0.51595 0.49057 0.51321 0.48769 0.92532
LDOF 97 0.47656 0.44912 0.47645 0.44901 0.49569 0.46925 0.91662
LDOF 100 0.47266 0.44501 0.47989 0.45262 0.50220 0.47610 0.91968
ODIN 97 0.43880 0.40938 0.41381 0.38307 0.45299 0.42431 0.90522
ODIN 100 0.43652 0.40698 0.42046 0.39007 0.45570 0.42716 0.90781
FastABOD 3 0.11719 0.07090 0.11661 0.07030 0.14861 0.10397 0.46868
FastABOD 8 0.10547 0.05857 0.11817 0.07193 0.14650 0.10175 0.46882
FastABOD 100 0.10547 0.05857 0.11600 0.06965 0.14286 0.09792 0.47006
KDEOS 94 0.08203 0.03391 0.09624 0.04885 0.18061 0.13765 0.74431
KDEOS 98 0.08203 0.03391 0.09771 0.05041 0.18374 0.14094 0.74783
KDEOS 100 0.08203 0.03391 0.09896 0.05172 0.18364 0.14084 0.74956
LDF 48 0.49219 0.46556 0.53794 0.51372 0.54038 0.51629 0.92139
LDF 67 0.52734 0.50256 0.52167 0.49660 0.56093 0.53792 0.92393
LDF 72 0.52344 0.49845 0.51665 0.49131 0.56164 0.53866 0.92486
LDF 82 0.51172 0.48612 0.50818 0.48240 0.56401 0.54116 0.92429
INFLO 82 0.51562 0.49023 0.49036 0.46364 0.51562 0.49023 0.88927
INFLO 98 0.49609 0.46968 0.49613 0.46971 0.53791 0.51368 0.89998
INFLO 99 0.50000 0.47379 0.49585 0.46942 0.53996 0.51585 0.90004
INFLO 100 0.49609 0.46968 0.49514 0.46867 0.54255 0.51857 0.89983
COF 75 0.48828 0.46145 0.48661 0.45970 0.51765 0.49236 0.87337
COF 89 0.52734 0.50256 0.49392 0.46739 0.52874 0.50403 0.87189
COF 96 0.52734 0.50256 0.50104 0.47488 0.54090 0.51683 0.86989
COF 100 0.52734 0.50256 0.50194 0.47583 0.53501 0.51063 0.86673

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (44.7 MB) Download raw algorithm evaluation table (71.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.19380 0.15146 0.14876 0.10406 0.19417 0.15186 0.54597
KNN 2 0.17829 0.13514 0.15046 0.10584 0.18219 0.13924 0.55064
KNN 14 0.16667 0.12291 0.14462 0.09970 0.18382 0.14096 0.57598
KNNW 1 0.16667 0.12291 0.14481 0.09990 0.18594 0.14319 0.51969
KNNW 3 0.17829 0.13514 0.14852 0.10381 0.18008 0.13702 0.54265
KNNW 5 0.17442 0.13106 0.14953 0.10487 0.17954 0.13646 0.54838
KNNW 67 0.12791 0.08211 0.13566 0.09027 0.16237 0.11838 0.57163
LOF 52 0.47287 0.44519 0.50116 0.47497 0.48131 0.45407 0.89494
LOF 62 0.48837 0.46150 0.49279 0.46616 0.49160 0.46490 0.89843
LOF 99 0.51163 0.48598 0.47550 0.44796 0.53952 0.51534 0.89303
SimplifiedLOF 54 0.48062 0.45335 0.49122 0.46450 0.49389 0.46731 0.87689
SimplifiedLOF 90 0.49612 0.46966 0.48296 0.45581 0.49821 0.47186 0.90745
SimplifiedLOF 93 0.50388 0.47782 0.48323 0.45609 0.50388 0.47782 0.90693
SimplifiedLOF 95 0.50388 0.47782 0.48280 0.45564 0.50485 0.47885 0.90713
LoOP 91 0.49612 0.46966 0.50439 0.47837 0.50110 0.47490 0.90014
LoOP 93 0.49225 0.46558 0.50550 0.47953 0.50000 0.47374 0.90126
LoOP 96 0.49225 0.46558 0.50480 0.47880 0.50644 0.48052 0.90187
LoOP 100 0.49225 0.46558 0.50477 0.47877 0.50638 0.48046 0.90289
LDOF 93 0.46512 0.43703 0.48124 0.45400 0.48035 0.45306 0.90588
LDOF 99 0.47287 0.44519 0.48312 0.45598 0.48035 0.45306 0.91191
LDOF 100 0.46512 0.43703 0.48254 0.45536 0.47930 0.45196 0.91253
ODIN 94 0.43023 0.40031 0.41334 0.38253 0.43348 0.40373 0.88395
ODIN 100 0.42506 0.39487 0.42112 0.39073 0.44017 0.41077 0.88883
FastABOD 3 0.12016 0.07395 0.10530 0.05831 0.15427 0.10986 0.45829
FastABOD 7 0.12016 0.07395 0.12300 0.07695 0.14535 0.10047 0.45049
KDEOS 7 0.09302 0.04539 0.06749 0.01852 0.12194 0.07583 0.58701
KDEOS 100 0.08140 0.03316 0.10289 0.05578 0.19171 0.14926 0.74860
LDF 28 0.45736 0.42887 0.50282 0.47671 0.47635 0.44885 0.88158
LDF 48 0.46124 0.43295 0.49293 0.46630 0.50230 0.47616 0.89465
LDF 64 0.50775 0.48190 0.48307 0.45592 0.52870 0.50395 0.88170
LDF 66 0.50775 0.48190 0.48037 0.45309 0.53147 0.50686 0.88068
INFLO 74 0.48450 0.45743 0.46344 0.43527 0.51073 0.48504 0.84010
INFLO 94 0.49612 0.46966 0.46986 0.44202 0.50101 0.47481 0.86761
INFLO 99 0.49612 0.46966 0.46762 0.43966 0.50662 0.48071 0.87063
INFLO 100 0.50000 0.47374 0.46697 0.43898 0.51032 0.48460 0.87031
COF 71 0.46124 0.43295 0.46847 0.44056 0.48889 0.46205 0.84929
COF 95 0.50388 0.47782 0.48071 0.45344 0.53425 0.50979 0.83194
COF 97 0.50775 0.48190 0.47730 0.44985 0.53898 0.51477 0.82197
COF 98 0.51938 0.49414 0.47655 0.44906 0.52941 0.50470 0.82567

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO