Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (5% of outliers version#07)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (43.4 MB) Download raw algorithm evaluation table (70.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 53 0.42578 0.39568 0.46885 0.44101 0.48168 0.45450 0.90512
KNN 56 0.44922 0.42034 0.46868 0.44082 0.48592 0.45896 0.90447
KNN 58 0.44922 0.42034 0.46909 0.44125 0.48336 0.45628 0.90422
KNN 69 0.46094 0.43268 0.46775 0.43985 0.46880 0.44095 0.90348
KNNW 53 0.41797 0.38745 0.44450 0.41537 0.41879 0.38832 0.90284
KNNW 81 0.41406 0.38334 0.46001 0.43170 0.45342 0.42476 0.90504
KNNW 93 0.40625 0.37512 0.46356 0.43544 0.47682 0.44939 0.90487
KNNW 100 0.40625 0.37512 0.46441 0.43633 0.47463 0.44709 0.90473
LOF 33 0.39844 0.36690 0.34741 0.31320 0.41304 0.38227 0.79486
LOF 47 0.40625 0.37512 0.36103 0.32753 0.40705 0.37596 0.81288
LOF 100 0.39453 0.36279 0.38030 0.34781 0.40354 0.37227 0.88497
SimplifiedLOF 30 0.41406 0.38334 0.33285 0.29788 0.42798 0.39799 0.76043
SimplifiedLOF 41 0.42188 0.39157 0.36368 0.33032 0.42353 0.39331 0.76806
SimplifiedLOF 100 0.40625 0.37512 0.37923 0.34668 0.41966 0.38923 0.82234
LoOP 38 0.40625 0.37512 0.30713 0.27080 0.40945 0.37849 0.74891
LoOP 98 0.39844 0.36690 0.35177 0.31779 0.41309 0.38232 0.81113
LoOP 100 0.39844 0.36690 0.35349 0.31959 0.41224 0.38143 0.81313
LDOF 76 0.42969 0.39979 0.39005 0.35807 0.43545 0.40586 0.87658
LDOF 98 0.42188 0.39157 0.40026 0.36881 0.44156 0.41228 0.88234
LDOF 100 0.42188 0.39157 0.40182 0.37046 0.43636 0.40681 0.88302
ODIN 99 0.40137 0.36998 0.31657 0.28074 0.40232 0.37099 0.78356
ODIN 100 0.39779 0.36621 0.31668 0.28085 0.40000 0.36854 0.78439
FastABOD 7 0.32422 0.28879 0.31680 0.28099 0.33712 0.30236 0.77693
FastABOD 8 0.32422 0.28879 0.31622 0.28037 0.33504 0.30018 0.77709
FastABOD 14 0.34766 0.31346 0.31566 0.27978 0.34766 0.31346 0.77622
KDEOS 69 0.17188 0.12846 0.11775 0.07149 0.20599 0.16437 0.70707
KDEOS 87 0.16016 0.11613 0.11527 0.06888 0.22645 0.18590 0.71226
KDEOS 100 0.14844 0.10379 0.11674 0.07044 0.22365 0.18295 0.71989
LDF 100 0.41797 0.38745 0.44825 0.41932 0.48991 0.46317 0.92158
INFLO 80 0.39453 0.36279 0.32090 0.28530 0.39925 0.36775 0.70686
INFLO 100 0.39453 0.36279 0.32254 0.28702 0.39600 0.36433 0.71840
COF 34 0.38672 0.35457 0.30695 0.27062 0.39382 0.36204 0.74563
COF 50 0.33984 0.30523 0.33415 0.29924 0.37736 0.34472 0.76164
COF 95 0.37109 0.33812 0.34373 0.30933 0.37524 0.34249 0.70630

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (43.6 MB) Download raw algorithm evaluation table (72.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.47674 0.44927 0.45068 0.42183 0.47674 0.44927 0.85062
KNN 98 0.41550 0.38481 0.49066 0.46391 0.49180 0.46512 0.91835
KNN 100 0.42636 0.39623 0.49118 0.46447 0.48968 0.46288 0.91826
KNNW 14 0.47674 0.44927 0.46083 0.43252 0.48583 0.45883 0.84835
KNNW 16 0.46899 0.44111 0.46554 0.43747 0.48636 0.45939 0.85169
KNNW 100 0.42636 0.39623 0.48338 0.45625 0.43575 0.40612 0.91651
LOF 20 0.49225 0.46558 0.40966 0.37866 0.49516 0.46865 0.83566
LOF 23 0.48837 0.46150 0.42548 0.39531 0.50862 0.48282 0.84680
LOF 26 0.48837 0.46150 0.43297 0.40319 0.49661 0.47018 0.84902
LOF 100 0.46124 0.43295 0.39553 0.36379 0.46400 0.43585 0.86028
SimplifiedLOF 36 0.48062 0.45335 0.44606 0.41697 0.49750 0.47111 0.84322
SimplifiedLOF 49 0.50000 0.47374 0.46384 0.43568 0.52610 0.50121 0.83647
SimplifiedLOF 53 0.50388 0.47782 0.46219 0.43395 0.53138 0.50677 0.83511
SimplifiedLOF 61 0.51163 0.48598 0.45267 0.42393 0.51586 0.49043 0.82319
LoOP 46 0.44574 0.41663 0.38594 0.35370 0.45747 0.42898 0.83040
LoOP 64 0.49612 0.46966 0.40012 0.36862 0.49709 0.47068 0.81521
LoOP 80 0.48062 0.45335 0.40430 0.37301 0.49269 0.46605 0.80825
LDOF 88 0.49612 0.46966 0.43978 0.41036 0.50758 0.48172 0.88764
LDOF 89 0.49612 0.46966 0.44071 0.41134 0.51024 0.48452 0.88713
LDOF 95 0.50388 0.47782 0.44112 0.41177 0.50829 0.48247 0.88115
LDOF 100 0.50000 0.47374 0.44196 0.41266 0.50638 0.48045 0.88003
ODIN 94 0.42878 0.39878 0.35004 0.31591 0.44610 0.41701 0.78966
ODIN 100 0.44365 0.41443 0.35250 0.31850 0.44402 0.41483 0.79814
FastABOD 3 0.46124 0.43295 0.37432 0.34147 0.46421 0.43607 0.79305
FastABOD 4 0.45349 0.42479 0.39313 0.36126 0.47191 0.44418 0.78585
FastABOD 5 0.43798 0.40847 0.40666 0.37550 0.46296 0.43476 0.78027
KDEOS 10 0.11628 0.06987 0.08373 0.03561 0.14565 0.10078 0.64528
KDEOS 68 0.08527 0.03724 0.10632 0.05939 0.19478 0.15250 0.72511
KDEOS 99 0.11240 0.06579 0.11010 0.06337 0.21929 0.17829 0.71957
LDF 17 0.50388 0.47782 0.44974 0.42084 0.50881 0.48301 0.81945
LDF 21 0.49612 0.46966 0.46260 0.43437 0.52101 0.49585 0.85382
LDF 24 0.49225 0.46558 0.46986 0.44202 0.51417 0.48866 0.85698
LDF 100 0.44961 0.42071 0.44303 0.41378 0.45541 0.42681 0.92283
INFLO 52 0.49612 0.46966 0.39486 0.36308 0.49805 0.47170 0.76380
INFLO 60 0.46899 0.44111 0.39291 0.36103 0.48000 0.45269 0.77046
COF 39 0.47287 0.44519 0.43621 0.40660 0.48207 0.45487 0.79934
COF 45 0.49612 0.46966 0.45767 0.42919 0.53034 0.50567 0.79437
COF 49 0.50388 0.47782 0.45943 0.43104 0.50711 0.48123 0.78258
COF 51 0.50000 0.47374 0.46721 0.43923 0.51887 0.49360 0.78332

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (44.6 MB) Download raw algorithm evaluation table (71.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.19531 0.15313 0.18004 0.13705 0.20822 0.16671 0.59279
KNN 4 0.18359 0.14079 0.18059 0.13763 0.20155 0.15969 0.60978
KNN 16 0.17188 0.12846 0.17428 0.13099 0.20939 0.16794 0.62989
KNN 38 0.14453 0.09968 0.16484 0.12106 0.18726 0.14465 0.63421
KNNW 3 0.19141 0.14901 0.17639 0.13321 0.20231 0.16049 0.58397
KNNW 5 0.17969 0.13668 0.17943 0.13641 0.20613 0.16451 0.59599
KNNW 6 0.17969 0.13668 0.17975 0.13675 0.20386 0.16212 0.60044
KNNW 70 0.14453 0.09968 0.16497 0.12119 0.18611 0.14344 0.63121
LOF 88 0.51562 0.49023 0.53473 0.51034 0.55000 0.52641 0.93300
LOF 96 0.54688 0.52312 0.53562 0.51128 0.57095 0.54846 0.93198
LOF 97 0.55469 0.53134 0.53533 0.51097 0.56998 0.54744 0.93191
SimplifiedLOF 91 0.52734 0.50256 0.51278 0.48723 0.52734 0.50256 0.92883
SimplifiedLOF 98 0.52344 0.49845 0.51765 0.49236 0.53077 0.50617 0.93139
SimplifiedLOF 100 0.52734 0.50256 0.51965 0.49446 0.53077 0.50617 0.93179
LoOP 97 0.50000 0.47379 0.51550 0.49010 0.50633 0.48045 0.91778
LoOP 99 0.50000 0.47379 0.51722 0.49190 0.51125 0.48562 0.91912
LoOP 100 0.50000 0.47379 0.51824 0.49299 0.51125 0.48562 0.91942
LDOF 80 0.44531 0.41623 0.45998 0.43167 0.48276 0.45564 0.88420
LDOF 99 0.45703 0.42857 0.47858 0.45124 0.47202 0.44434 0.90848
LDOF 100 0.46094 0.43268 0.47851 0.45117 0.47111 0.44338 0.90936
ODIN 97 0.39551 0.36382 0.38950 0.35749 0.40435 0.37312 0.88855
ODIN 100 0.39551 0.36382 0.39602 0.36435 0.40899 0.37800 0.89197
FastABOD 4 0.15234 0.10790 0.14882 0.10419 0.16794 0.12432 0.49092
FastABOD 5 0.15234 0.10790 0.14823 0.10358 0.16613 0.12242 0.49127
FastABOD 15 0.14453 0.09968 0.14725 0.10254 0.17035 0.12685 0.48986
KDEOS 87 0.08984 0.04213 0.08708 0.03922 0.16332 0.11946 0.71376
KDEOS 100 0.08984 0.04213 0.09350 0.04597 0.17145 0.12801 0.73161
LDF 71 0.53906 0.51490 0.54070 0.51662 0.58018 0.55817 0.92881
LDF 74 0.53906 0.51490 0.54010 0.51599 0.58423 0.56243 0.92898
LDF 79 0.54688 0.52312 0.53873 0.51455 0.58467 0.56290 0.92819
LDF 88 0.56641 0.54367 0.53410 0.50967 0.57848 0.55638 0.92577
INFLO 91 0.52344 0.49845 0.49431 0.46780 0.52860 0.50389 0.88241
INFLO 94 0.52734 0.50256 0.49735 0.47100 0.52734 0.50256 0.88800
INFLO 100 0.51953 0.49434 0.50114 0.47499 0.52751 0.50274 0.89163
COF 89 0.52734 0.50256 0.52446 0.49952 0.53532 0.51095 0.86214
COF 99 0.54297 0.51901 0.53562 0.51128 0.55535 0.53204 0.85813
COF 100 0.54297 0.51901 0.53606 0.51174 0.55957 0.53648 0.85822

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (44.7 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.17054 0.12698 0.15040 0.10578 0.20231 0.16042 0.55207
KNN 4 0.16667 0.12291 0.15577 0.11143 0.18767 0.14501 0.57353
KNN 17 0.14729 0.10251 0.14896 0.10427 0.18095 0.13794 0.59110
KNNW 2 0.17442 0.13106 0.14723 0.10245 0.19760 0.15547 0.54561
KNNW 5 0.16279 0.11883 0.15339 0.10893 0.20225 0.16035 0.56291
KNNW 63 0.13566 0.09027 0.14407 0.09912 0.16506 0.12122 0.58833
LOF 56 0.51550 0.49006 0.50069 0.47447 0.51550 0.49006 0.88928
LOF 59 0.51163 0.48598 0.50134 0.47516 0.51538 0.48994 0.89121
LOF 90 0.49612 0.46966 0.48621 0.45923 0.53577 0.51140 0.89486
LOF 100 0.51550 0.49006 0.47613 0.44862 0.54637 0.52255 0.89458
SimplifiedLOF 58 0.51550 0.49006 0.49061 0.46386 0.52459 0.49962 0.86949
SimplifiedLOF 95 0.51938 0.49414 0.48261 0.45544 0.51938 0.49414 0.90348
SimplifiedLOF 97 0.51163 0.48598 0.48270 0.45554 0.51823 0.49293 0.90381
LoOP 80 0.50388 0.47782 0.48600 0.45901 0.50813 0.48230 0.89010
LoOP 88 0.49612 0.46966 0.49011 0.46334 0.51467 0.48919 0.89698
LoOP 100 0.48837 0.46150 0.49386 0.46728 0.50924 0.48347 0.90222
LDOF 98 0.48062 0.45335 0.46792 0.43998 0.48532 0.45830 0.90847
LDOF 100 0.48062 0.45335 0.46914 0.44126 0.49194 0.46526 0.91036
ODIN 95 0.42991 0.39997 0.38703 0.35484 0.43548 0.40584 0.88110
ODIN 97 0.43101 0.40113 0.38950 0.35744 0.43177 0.40193 0.88302
ODIN 100 0.42725 0.39717 0.39361 0.36176 0.42934 0.39937 0.88571
FastABOD 3 0.16279 0.11883 0.12204 0.07593 0.19747 0.15532 0.48977
FastABOD 15 0.14341 0.09843 0.12703 0.08119 0.17192 0.12843 0.46695
KDEOS 9 0.09690 0.04947 0.07301 0.02433 0.13497 0.08954 0.60282
KDEOS 99 0.08915 0.04132 0.09515 0.04763 0.17386 0.13048 0.72357
KDEOS 100 0.09690 0.04947 0.09583 0.04835 0.17339 0.12998 0.72484
LDF 40 0.48837 0.46150 0.51287 0.48729 0.49801 0.47165 0.88751
LDF 46 0.48837 0.46150 0.51101 0.48533 0.50352 0.47745 0.89305
LDF 71 0.51163 0.48598 0.48062 0.45335 0.54861 0.52491 0.87705
LDF 74 0.51550 0.49006 0.47878 0.45141 0.54419 0.52026 0.87473
INFLO 54 0.50388 0.47782 0.45716 0.42866 0.51383 0.48830 0.81010
INFLO 59 0.49612 0.46966 0.46060 0.43227 0.51731 0.49196 0.81675
INFLO 62 0.49225 0.46558 0.46482 0.43672 0.51639 0.49100 0.82924
INFLO 97 0.48837 0.46150 0.46369 0.43553 0.50091 0.47470 0.87115
COF 74 0.48062 0.45335 0.48053 0.45325 0.50644 0.48052 0.86124
COF 96 0.50388 0.47782 0.49015 0.46338 0.52797 0.50318 0.84868
COF 100 0.51938 0.49414 0.49080 0.46407 0.52710 0.50227 0.83988

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO