Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (5% of outliers version#09)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (43.4 MB) Download raw algorithm evaluation table (70.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 47 0.42969 0.39979 0.46488 0.43683 0.47500 0.44748 0.91451
KNN 54 0.43359 0.40390 0.46593 0.43793 0.47782 0.45044 0.91413
KNN 60 0.43750 0.40801 0.46503 0.43698 0.48772 0.46086 0.91336
KNN 66 0.45312 0.42445 0.46554 0.43752 0.48325 0.45615 0.91295
KNNW 12 0.44141 0.41212 0.39553 0.36384 0.44567 0.41661 0.87796
KNNW 74 0.42188 0.39157 0.46250 0.43432 0.46319 0.43505 0.91496
KNNW 99 0.42969 0.39979 0.46496 0.43691 0.48105 0.45385 0.91412
KNNW 100 0.42578 0.39568 0.46490 0.43685 0.48264 0.45552 0.91406
LOF 29 0.43750 0.40801 0.36316 0.32977 0.44533 0.41625 0.81071
LOF 100 0.41406 0.38334 0.39990 0.36844 0.42836 0.39839 0.92003
SimplifiedLOF 38 0.44141 0.41212 0.37341 0.34057 0.44718 0.41820 0.79751
SimplifiedLOF 47 0.43750 0.40801 0.38582 0.35362 0.45652 0.42803 0.80641
SimplifiedLOF 99 0.42969 0.39979 0.39322 0.36140 0.43606 0.40649 0.85833
SimplifiedLOF 100 0.42969 0.39979 0.39304 0.36122 0.43253 0.40278 0.86110
LoOP 58 0.44141 0.41212 0.35000 0.31592 0.44358 0.41441 0.81351
LoOP 59 0.44141 0.41212 0.35204 0.31807 0.44622 0.41718 0.81859
LoOP 100 0.41797 0.38745 0.36374 0.33038 0.42963 0.39973 0.84746
LDOF 56 0.41016 0.37923 0.36621 0.33298 0.46801 0.44012 0.87477
LDOF 64 0.43359 0.40390 0.37281 0.33993 0.45392 0.42530 0.87914
LDOF 100 0.43359 0.40390 0.39978 0.36831 0.45814 0.42973 0.89414
ODIN 54 0.43806 0.40860 0.32224 0.28670 0.43836 0.40891 0.78960
ODIN 96 0.41113 0.38026 0.34501 0.31067 0.41453 0.38384 0.82153
ODIN 100 0.40681 0.37571 0.34483 0.31048 0.42105 0.39070 0.82710
FastABOD 5 0.33203 0.29701 0.30351 0.26700 0.33465 0.29976 0.75324
FastABOD 8 0.32031 0.28468 0.31141 0.27531 0.33853 0.30385 0.75842
FastABOD 34 0.32031 0.28468 0.31134 0.27524 0.32998 0.29485 0.76395
FastABOD 100 0.32812 0.29290 0.31427 0.27832 0.33621 0.30141 0.76147
KDEOS 74 0.14453 0.09968 0.11491 0.06851 0.20670 0.16511 0.72064
KDEOS 91 0.12500 0.07913 0.12137 0.07531 0.22508 0.18445 0.73236
KDEOS 100 0.12500 0.07913 0.12259 0.07659 0.22442 0.18376 0.73725
LDF 99 0.44141 0.41212 0.47191 0.44422 0.55474 0.53140 0.93825
LDF 100 0.44141 0.41212 0.47318 0.44556 0.56012 0.53706 0.93849
INFLO 40 0.43750 0.40801 0.33433 0.29943 0.44269 0.41347 0.74421
INFLO 49 0.42578 0.39568 0.34299 0.30855 0.43177 0.40198 0.76368
INFLO 50 0.42188 0.39157 0.34336 0.30894 0.42857 0.39861 0.76335
COF 48 0.40234 0.37101 0.36534 0.33207 0.40619 0.37506 0.78056
COF 49 0.41016 0.37923 0.36529 0.33201 0.41420 0.38349 0.77940
COF 54 0.39844 0.36690 0.36666 0.33346 0.40879 0.37780 0.77308
COF 91 0.37500 0.34223 0.35383 0.31996 0.41709 0.38653 0.73215

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (43.5 MB) Download raw algorithm evaluation table (72.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.44767 0.41867 0.40047 0.36898 0.45473 0.42609 0.77115
KNN 93 0.35271 0.31872 0.43050 0.40059 0.47235 0.44464 0.90431
KNN 100 0.34884 0.31464 0.43367 0.40393 0.46822 0.44029 0.90511
KNNW 15 0.43411 0.40439 0.40960 0.37860 0.45433 0.42568 0.78613
KNNW 19 0.42248 0.39215 0.41122 0.38031 0.46262 0.43440 0.78892
KNNW 100 0.40310 0.37176 0.41926 0.38876 0.40876 0.37771 0.89190
LOF 22 0.47287 0.44519 0.43036 0.40044 0.48000 0.45269 0.84169
LOF 26 0.48450 0.45743 0.44697 0.41793 0.49497 0.46845 0.83821
LOF 29 0.46512 0.43703 0.45069 0.42185 0.49770 0.47132 0.83853
LOF 32 0.46512 0.43703 0.45688 0.42836 0.48931 0.46249 0.83431
SimplifiedLOF 29 0.50000 0.47374 0.46722 0.43924 0.51200 0.48637 0.83863
SimplifiedLOF 31 0.50388 0.47782 0.47346 0.44581 0.50682 0.48092 0.83878
SimplifiedLOF 36 0.50000 0.47374 0.48027 0.45297 0.50693 0.48104 0.84204
SimplifiedLOF 41 0.49225 0.46558 0.48483 0.45777 0.50893 0.48314 0.83459
LoOP 38 0.45736 0.42887 0.40845 0.37738 0.47500 0.44743 0.82504
LoOP 39 0.45736 0.42887 0.41036 0.37940 0.47917 0.45182 0.82339
LoOP 40 0.46124 0.43295 0.41154 0.38064 0.47599 0.44847 0.82132
LoOP 41 0.45736 0.42887 0.41269 0.38185 0.47280 0.44512 0.82008
LDOF 40 0.47674 0.44927 0.44398 0.41478 0.49831 0.47196 0.89236
LDOF 46 0.49225 0.46558 0.44720 0.41817 0.50916 0.48338 0.87491
LDOF 49 0.47674 0.44927 0.45057 0.42171 0.49728 0.47088 0.86582
ODIN 49 0.41182 0.38093 0.28772 0.25031 0.41326 0.38244 0.72635
ODIN 99 0.39028 0.35826 0.32223 0.28664 0.39208 0.36016 0.74980
ODIN 100 0.38984 0.35780 0.32169 0.28607 0.39511 0.36335 0.75026
FastABOD 3 0.40310 0.37176 0.33658 0.30174 0.41517 0.38446 0.74826
FastABOD 6 0.36047 0.32688 0.36026 0.32667 0.41429 0.38353 0.72366
KDEOS 41 0.09302 0.04539 0.10352 0.05644 0.19398 0.15165 0.72187
KDEOS 65 0.11240 0.06579 0.11330 0.06674 0.21324 0.17192 0.71067
KDEOS 95 0.14729 0.10251 0.11018 0.06346 0.19388 0.15155 0.69942
LDF 20 0.48450 0.45743 0.49225 0.46558 0.51802 0.49271 0.85595
LDF 23 0.49225 0.46558 0.48699 0.46005 0.51481 0.48933 0.84491
INFLO 31 0.46899 0.44111 0.39797 0.36635 0.47266 0.44496 0.77157
INFLO 34 0.46899 0.44111 0.40013 0.36863 0.47431 0.44670 0.76830
INFLO 36 0.45349 0.42479 0.40175 0.37033 0.46067 0.43235 0.76431
COF 38 0.48062 0.45335 0.46061 0.43229 0.53207 0.50749 0.79941
COF 40 0.48062 0.45335 0.46585 0.43780 0.52278 0.49772 0.81380
COF 45 0.48450 0.45743 0.46937 0.44151 0.53023 0.50556 0.77836
COF 46 0.49225 0.46558 0.46769 0.43974 0.51316 0.48759 0.77719

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (44.6 MB) Download raw algorithm evaluation table (70.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.20703 0.16546 0.16265 0.11875 0.21074 0.16936 0.58351
KNN 3 0.19531 0.15313 0.16531 0.12155 0.20957 0.16813 0.58893
KNN 14 0.16406 0.12024 0.15616 0.11192 0.20443 0.16272 0.60873
KNNW 1 0.19141 0.14901 0.15975 0.11569 0.21395 0.17274 0.56164
KNNW 5 0.19922 0.15724 0.16332 0.11946 0.20370 0.16196 0.58327
KNNW 6 0.20312 0.16135 0.16304 0.11916 0.20513 0.16346 0.58666
KNNW 53 0.15234 0.10790 0.14848 0.10384 0.17987 0.13687 0.60605
LOF 58 0.51562 0.49023 0.52469 0.49977 0.51765 0.49236 0.92172
LOF 81 0.51953 0.49434 0.52056 0.49543 0.53621 0.51189 0.92846
LOF 94 0.54297 0.51901 0.51417 0.48870 0.56328 0.54038 0.92687
LOF 100 0.53906 0.51490 0.50966 0.48396 0.56696 0.54425 0.92529
SimplifiedLOF 85 0.51562 0.49023 0.51218 0.48660 0.52586 0.50100 0.92366
SimplifiedLOF 95 0.52734 0.50256 0.51136 0.48574 0.53358 0.50913 0.92658
SimplifiedLOF 100 0.52734 0.50256 0.51175 0.48615 0.54448 0.52060 0.92713
LoOP 87 0.51172 0.48612 0.51651 0.49116 0.53159 0.50703 0.91162
LoOP 96 0.52344 0.49845 0.52184 0.49677 0.52747 0.50270 0.91686
LoOP 100 0.52344 0.49845 0.52461 0.49969 0.52860 0.50389 0.91845
LDOF 71 0.48047 0.45323 0.46993 0.44214 0.48606 0.45911 0.88122
LDOF 97 0.47656 0.44912 0.50537 0.47944 0.51196 0.48638 0.91339
LDOF 100 0.47656 0.44912 0.50610 0.48021 0.50718 0.48134 0.91651
ODIN 92 0.43047 0.40061 0.40356 0.37229 0.43478 0.40515 0.88998
ODIN 100 0.41776 0.38724 0.42157 0.39124 0.43621 0.40666 0.89726
FastABOD 4 0.14062 0.09557 0.13362 0.08820 0.17009 0.12658 0.47512
FastABOD 7 0.14062 0.09557 0.13430 0.08892 0.16817 0.12456 0.47375
KDEOS 8 0.09375 0.04624 0.07087 0.02216 0.12513 0.07926 0.59656
KDEOS 100 0.07422 0.02568 0.09953 0.05232 0.18928 0.14678 0.75090
LDF 41 0.48047 0.45323 0.54128 0.51723 0.50368 0.47766 0.91999
LDF 59 0.50781 0.48201 0.51800 0.49273 0.55348 0.53007 0.92200
LDF 68 0.53125 0.50667 0.51392 0.48843 0.56811 0.54546 0.92156
LDF 73 0.54297 0.51901 0.50882 0.48307 0.56471 0.54188 0.92124
INFLO 79 0.51953 0.49434 0.48983 0.46309 0.52137 0.49627 0.88122
INFLO 94 0.50391 0.47790 0.49440 0.46789 0.51724 0.49193 0.89157
INFLO 96 0.51562 0.49023 0.49464 0.46815 0.52039 0.49524 0.89023
INFLO 100 0.51562 0.49023 0.49289 0.46631 0.53125 0.50667 0.88751
COF 88 0.52344 0.49845 0.50086 0.47470 0.53308 0.50860 0.87781
COF 96 0.54688 0.52312 0.50316 0.47711 0.55106 0.52752 0.87525
COF 97 0.54688 0.52312 0.50409 0.47809 0.55535 0.53204 0.87467

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (44.7 MB) Download raw algorithm evaluation table (71.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.20155 0.15962 0.17647 0.13322 0.22599 0.18534 0.52262
KNN 2 0.21318 0.17186 0.17789 0.13472 0.22099 0.18009 0.53490
KNN 51 0.15116 0.10659 0.15452 0.11012 0.17974 0.13667 0.56277
KNNW 2 0.20155 0.15962 0.17642 0.13317 0.23324 0.19297 0.51553
KNNW 3 0.20155 0.15962 0.17861 0.13547 0.23034 0.18992 0.52487
KNNW 6 0.20543 0.16370 0.17702 0.13380 0.22520 0.18451 0.53548
KNNW 63 0.16667 0.12291 0.15892 0.11475 0.18497 0.14217 0.56037
LOF 58 0.46512 0.43703 0.47303 0.44536 0.47082 0.44304 0.87447
LOF 78 0.47674 0.44927 0.47496 0.44739 0.49662 0.47019 0.87088
LOF 83 0.49612 0.46966 0.47143 0.44367 0.51087 0.48518 0.86830
LOF 95 0.49225 0.46558 0.46232 0.43409 0.52065 0.49547 0.86493
SimplifiedLOF 81 0.47287 0.44519 0.47156 0.44381 0.48065 0.45338 0.87170
SimplifiedLOF 94 0.48062 0.45335 0.46897 0.44108 0.49446 0.46792 0.87305
SimplifiedLOF 98 0.49612 0.46966 0.46895 0.44106 0.49810 0.47174 0.87247
SimplifiedLOF 100 0.49225 0.46558 0.46854 0.44063 0.50355 0.47748 0.87218
LoOP 62 0.45736 0.42887 0.43848 0.40899 0.48687 0.45993 0.83648
LoOP 81 0.46899 0.44111 0.46136 0.43308 0.48223 0.45504 0.86130
LoOP 99 0.46124 0.43295 0.46800 0.44006 0.47894 0.45157 0.86794
LoOP 100 0.46512 0.43703 0.46911 0.44123 0.47473 0.44714 0.86742
LDOF 65 0.45736 0.42887 0.42517 0.39499 0.45736 0.42887 0.85510
LDOF 84 0.45736 0.42887 0.44910 0.42017 0.47534 0.44778 0.87536
LDOF 100 0.45736 0.42887 0.45895 0.43054 0.46882 0.44092 0.89208
ODIN 94 0.41473 0.38399 0.38175 0.34928 0.43368 0.40394 0.86185
ODIN 100 0.41085 0.37991 0.39291 0.36103 0.43805 0.40854 0.86801
FastABOD 4 0.17054 0.12698 0.14715 0.10237 0.20800 0.16641 0.45686
FastABOD 6 0.16279 0.11883 0.15974 0.11561 0.20168 0.15976 0.45249
KDEOS 8 0.08527 0.03724 0.07254 0.02383 0.11952 0.07328 0.57996
KDEOS 100 0.06589 0.01684 0.08798 0.04009 0.17155 0.12805 0.70307
LDF 17 0.49225 0.46558 0.45800 0.42954 0.49225 0.46558 0.83048
LDF 43 0.43798 0.40847 0.48188 0.45467 0.46329 0.43511 0.87416
LDF 50 0.45349 0.42479 0.48554 0.45853 0.49789 0.47152 0.87229
LDF 67 0.48837 0.46150 0.47007 0.44224 0.51458 0.48909 0.85975
INFLO 89 0.46124 0.43295 0.45049 0.42163 0.47415 0.44654 0.84995
INFLO 98 0.48062 0.45335 0.44934 0.42042 0.48872 0.46187 0.85066
INFLO 100 0.47674 0.44927 0.44893 0.42000 0.49387 0.46729 0.85026
COF 80 0.47287 0.44519 0.43567 0.40604 0.47287 0.44519 0.80181
COF 98 0.50000 0.47374 0.44173 0.41242 0.50097 0.47476 0.78546

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO