Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (5% of outliers version#08)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (43.4 MB) Download raw algorithm evaluation table (71.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 48 0.45312 0.42445 0.47517 0.44766 0.46422 0.43613 0.90336
KNN 62 0.46094 0.43268 0.47853 0.45119 0.47104 0.44331 0.90259
KNN 69 0.47266 0.44501 0.47720 0.44980 0.47727 0.44987 0.90221
KNN 76 0.47266 0.44501 0.47757 0.45018 0.47732 0.44991 0.90182
KNNW 92 0.43359 0.40390 0.47158 0.44388 0.45574 0.42720 0.90317
KNNW 98 0.43359 0.40390 0.47305 0.44542 0.46261 0.43443 0.90323
KNNW 99 0.43359 0.40390 0.47310 0.44547 0.46341 0.43528 0.90319
KNNW 100 0.43359 0.40390 0.47349 0.44589 0.46341 0.43528 0.90319
LOF 35 0.42188 0.39157 0.34491 0.31056 0.45351 0.42486 0.78792
LOF 36 0.42969 0.39979 0.34946 0.31535 0.44898 0.42009 0.79034
LOF 100 0.41016 0.37923 0.40710 0.37602 0.43234 0.40258 0.91543
SimplifiedLOF 48 0.42969 0.39979 0.36749 0.33433 0.46532 0.43729 0.77147
SimplifiedLOF 55 0.44141 0.41212 0.37496 0.34219 0.45133 0.42256 0.78515
SimplifiedLOF 97 0.43359 0.40390 0.39424 0.36248 0.44344 0.41426 0.84901
SimplifiedLOF 100 0.43750 0.40801 0.39406 0.36229 0.44269 0.41347 0.85427
LoOP 99 0.41797 0.38745 0.35816 0.32451 0.41879 0.38832 0.83835
LoOP 100 0.41406 0.38334 0.35850 0.32487 0.41732 0.38677 0.83907
LDOF 85 0.46094 0.43268 0.38831 0.35624 0.46494 0.43689 0.88378
LDOF 92 0.45703 0.42857 0.39437 0.36262 0.46923 0.44140 0.88795
LDOF 100 0.45312 0.42445 0.40124 0.36985 0.46565 0.43763 0.89453
ODIN 97 0.40964 0.37868 0.31342 0.27742 0.42127 0.39093 0.81144
ODIN 99 0.40625 0.37512 0.31433 0.27838 0.42500 0.39485 0.81376
ODIN 100 0.40469 0.37348 0.31452 0.27858 0.42171 0.39139 0.81497
FastABOD 6 0.33594 0.30112 0.30261 0.26605 0.35556 0.32177 0.72629
FastABOD 11 0.34766 0.31346 0.30644 0.27008 0.35203 0.31806 0.72721
FastABOD 16 0.34375 0.30934 0.30696 0.27062 0.34940 0.31529 0.72781
KDEOS 72 0.13672 0.09146 0.10330 0.05629 0.19417 0.15193 0.69364
KDEOS 100 0.12109 0.07502 0.11241 0.06588 0.22836 0.18791 0.72432
LDF 98 0.45703 0.42857 0.47158 0.44388 0.54740 0.52367 0.93250
LDF 100 0.46484 0.43679 0.47328 0.44567 0.54545 0.52162 0.93293
INFLO 88 0.40234 0.37101 0.32009 0.28444 0.40486 0.37366 0.72372
INFLO 90 0.40234 0.37101 0.31948 0.28380 0.40900 0.37801 0.72296
INFLO 94 0.40234 0.37101 0.32288 0.28738 0.40741 0.37634 0.73101
COF 54 0.41016 0.37923 0.33201 0.29699 0.41339 0.38263 0.74837
COF 56 0.40625 0.37512 0.33271 0.29772 0.41210 0.38128 0.75017
COF 58 0.40234 0.37101 0.33288 0.29790 0.41509 0.38443 0.74923
COF 82 0.39844 0.36690 0.32806 0.29284 0.43128 0.40146 0.74234

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (43.5 MB) Download raw algorithm evaluation table (72.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 57 0.37984 0.34728 0.39411 0.36229 0.38004 0.34748 0.88733
KNN 76 0.34496 0.31056 0.40332 0.37198 0.42254 0.39221 0.89058
KNN 99 0.31008 0.27385 0.40023 0.36874 0.43942 0.40998 0.88986
KNNW 11 0.37209 0.33912 0.34223 0.30769 0.38004 0.34748 0.77920
KNNW 22 0.37209 0.33912 0.36180 0.32828 0.40000 0.36849 0.78639
KNNW 100 0.36434 0.33096 0.39033 0.35831 0.37288 0.33995 0.88528
LOF 26 0.46512 0.43703 0.37567 0.34288 0.47158 0.44383 0.81778
LOF 29 0.45736 0.42887 0.38562 0.35336 0.47379 0.44616 0.81944
LOF 33 0.44574 0.41663 0.39419 0.36238 0.46053 0.43220 0.81886
LOF 100 0.38760 0.35544 0.35281 0.31882 0.39364 0.36180 0.84354
SimplifiedLOF 27 0.45736 0.42887 0.37547 0.34267 0.47300 0.44532 0.81921
SimplifiedLOF 38 0.47674 0.44927 0.41389 0.38311 0.48703 0.46009 0.81034
SimplifiedLOF 43 0.46124 0.43295 0.41771 0.38714 0.47205 0.44433 0.80263
LoOP 33 0.43411 0.40439 0.34932 0.31516 0.43629 0.40669 0.80451
LoOP 35 0.42636 0.39623 0.35252 0.31852 0.43077 0.40088 0.80701
LoOP 39 0.41860 0.38807 0.35442 0.32052 0.42739 0.39732 0.80524
LDOF 32 0.41860 0.38807 0.35725 0.32350 0.42692 0.39683 0.86650
LDOF 46 0.44186 0.41255 0.36237 0.32888 0.45989 0.43153 0.82286
LDOF 47 0.43023 0.40031 0.36188 0.32837 0.46235 0.43411 0.81845
LDOF 97 0.41860 0.38807 0.38690 0.35470 0.42804 0.39801 0.86259
ODIN 47 0.38513 0.35284 0.27048 0.23216 0.38697 0.35478 0.74544
ODIN 99 0.36047 0.32688 0.29590 0.25893 0.37676 0.34403 0.78697
ODIN 100 0.36693 0.33368 0.29823 0.26138 0.37743 0.34473 0.78679
FastABOD 3 0.30233 0.26569 0.28574 0.24823 0.34243 0.30790 0.73918
FastABOD 4 0.32558 0.29017 0.29276 0.25562 0.33884 0.30412 0.74878
KDEOS 43 0.13953 0.09435 0.12045 0.07426 0.21386 0.17258 0.71629
KDEOS 59 0.15891 0.11475 0.11538 0.06893 0.20627 0.16459 0.70395
KDEOS 88 0.13178 0.08619 0.11440 0.06789 0.22372 0.18296 0.70406
LDF 22 0.46124 0.43295 0.40379 0.37248 0.48073 0.45346 0.81154
LDF 27 0.46124 0.43295 0.41635 0.38570 0.49652 0.47008 0.82128
LDF 33 0.45349 0.42479 0.43831 0.40882 0.46847 0.44056 0.82502
LDF 100 0.36434 0.33096 0.38359 0.35122 0.41463 0.38389 0.89925
INFLO 27 0.41085 0.37991 0.32363 0.28812 0.42128 0.39089 0.73900
INFLO 30 0.43798 0.40847 0.33566 0.30077 0.43969 0.41026 0.73648
INFLO 32 0.41860 0.38807 0.33727 0.30246 0.44364 0.41442 0.72925
COF 29 0.39922 0.36768 0.36059 0.32701 0.41026 0.37929 0.77345
COF 37 0.44961 0.42071 0.37949 0.34691 0.45601 0.42745 0.75914
COF 40 0.44186 0.41255 0.38928 0.35721 0.44860 0.41964 0.75205
COF 45 0.43798 0.40847 0.37631 0.34356 0.45952 0.43114 0.72452

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (44.6 MB) Download raw algorithm evaluation table (71.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.20312 0.16135 0.16393 0.12009 0.21095 0.16959 0.57056
KNN 4 0.17969 0.13668 0.16878 0.12520 0.20652 0.16492 0.58977
KNN 23 0.17578 0.13257 0.16429 0.12048 0.22465 0.18400 0.61817
KNN 40 0.15234 0.10790 0.15792 0.11378 0.21164 0.17031 0.62001
KNNW 1 0.20703 0.16546 0.17200 0.12860 0.22912 0.18870 0.56064
KNNW 74 0.15234 0.10790 0.15816 0.11403 0.21067 0.16928 0.61799
LOF 83 0.49219 0.46556 0.48840 0.46158 0.50095 0.47479 0.92169
LOF 88 0.48828 0.46145 0.48989 0.46314 0.51863 0.49339 0.92153
LOF 93 0.50000 0.47379 0.48805 0.46122 0.53061 0.50600 0.92067
LOF 99 0.49609 0.46968 0.48600 0.45905 0.54949 0.52587 0.91903
SimplifiedLOF 98 0.50391 0.47790 0.47631 0.44885 0.50588 0.47998 0.92258
SimplifiedLOF 100 0.50000 0.47379 0.47675 0.44932 0.50579 0.47988 0.92310
LoOP 98 0.49219 0.46556 0.47350 0.44590 0.49372 0.46718 0.91023
LoOP 99 0.49219 0.46556 0.47313 0.44550 0.49497 0.46849 0.91085
LoOP 100 0.49219 0.46556 0.47317 0.44555 0.49398 0.46745 0.91151
LDOF 100 0.43750 0.40801 0.44445 0.41532 0.45018 0.42136 0.90183
ODIN 100 0.42480 0.39465 0.36095 0.32745 0.42718 0.39715 0.88608
FastABOD 4 0.15625 0.11201 0.14082 0.09578 0.18384 0.14106 0.48813
FastABOD 5 0.15625 0.11201 0.14203 0.09705 0.18592 0.14324 0.48814
FastABOD 6 0.15234 0.10790 0.14250 0.09754 0.18333 0.14052 0.48818
FastABOD 7 0.15234 0.10790 0.14277 0.09782 0.18539 0.14269 0.48771
KDEOS 82 0.08594 0.03802 0.09068 0.04301 0.16813 0.12452 0.71985
KDEOS 100 0.08203 0.03391 0.09942 0.05221 0.18059 0.13763 0.74514
LDF 55 0.48047 0.45323 0.49623 0.46982 0.51829 0.49304 0.91737
LDF 69 0.51562 0.49023 0.49795 0.47163 0.54733 0.52360 0.91584
LDF 80 0.51953 0.49434 0.49144 0.46478 0.54758 0.52386 0.91255
LDF 81 0.51562 0.49023 0.49194 0.46530 0.55000 0.52641 0.91236
INFLO 97 0.48438 0.45734 0.44593 0.41688 0.48438 0.45734 0.86804
INFLO 100 0.48438 0.45734 0.44823 0.41930 0.48864 0.46183 0.87501
COF 90 0.46484 0.43679 0.44307 0.41387 0.50089 0.47473 0.85132
COF 93 0.47266 0.44501 0.44431 0.41518 0.50277 0.47670 0.84929
COF 99 0.46484 0.43679 0.44889 0.42000 0.50751 0.48169 0.84632
COF 100 0.45703 0.42857 0.45011 0.42128 0.50423 0.47824 0.84573

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (44.7 MB) Download raw algorithm evaluation table (71.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.18605 0.14330 0.16119 0.11714 0.19672 0.15454 0.54322
KNN 14 0.20155 0.15962 0.15685 0.11257 0.21374 0.17245 0.55822
KNN 77 0.12791 0.08211 0.13949 0.09430 0.16258 0.11860 0.56339
KNNW 2 0.16667 0.12291 0.15606 0.11174 0.20359 0.16177 0.51742
KNNW 4 0.17054 0.12698 0.15939 0.11524 0.19830 0.15620 0.53037
KNNW 10 0.18217 0.13922 0.15885 0.11468 0.19355 0.15120 0.54339
KNNW 100 0.13566 0.09027 0.14304 0.09804 0.16981 0.12622 0.56072
LOF 93 0.48837 0.46150 0.46718 0.43919 0.50070 0.47448 0.88200
LOF 98 0.48450 0.45743 0.46893 0.44104 0.51840 0.49311 0.88328
LOF 100 0.48450 0.45743 0.46913 0.44125 0.51792 0.49260 0.88366
SimplifiedLOF 79 0.46899 0.44111 0.45276 0.42402 0.48858 0.46173 0.86960
SimplifiedLOF 97 0.47674 0.44927 0.45777 0.42930 0.48497 0.45792 0.88656
SimplifiedLOF 100 0.47674 0.44927 0.45858 0.43015 0.48713 0.46020 0.88782
LoOP 87 0.46512 0.43703 0.45605 0.42749 0.49038 0.46362 0.86385
LoOP 94 0.47674 0.44927 0.46018 0.43183 0.48812 0.46124 0.87341
LoOP 98 0.47674 0.44927 0.46270 0.43448 0.48539 0.45837 0.87735
LoOP 100 0.47287 0.44519 0.46251 0.43428 0.48539 0.45837 0.87911
LDOF 77 0.42636 0.39623 0.42784 0.39779 0.46602 0.43798 0.85412
LDOF 95 0.44186 0.41255 0.43984 0.41043 0.45909 0.43069 0.87371
LDOF 100 0.44186 0.41255 0.44106 0.41170 0.45647 0.42793 0.87942
ODIN 100 0.39406 0.36224 0.35647 0.32267 0.40252 0.37114 0.85702
FastABOD 3 0.17829 0.13514 0.14274 0.09773 0.22741 0.18683 0.48117
FastABOD 4 0.17829 0.13514 0.14964 0.10498 0.22492 0.18422 0.47817
KDEOS 10 0.08140 0.03316 0.06590 0.01684 0.12543 0.07951 0.58871
KDEOS 100 0.07752 0.02908 0.08796 0.04007 0.16021 0.11611 0.70709
LDF 30 0.45349 0.42479 0.47727 0.44982 0.46626 0.43823 0.86711
LDF 45 0.45349 0.42479 0.47518 0.44762 0.47647 0.44898 0.87826
LDF 73 0.50388 0.47782 0.47531 0.44776 0.54545 0.52158 0.86521
LDF 86 0.51163 0.48598 0.46597 0.43793 0.54082 0.51670 0.85888
INFLO 86 0.48062 0.45335 0.42318 0.39289 0.48062 0.45335 0.81601
INFLO 100 0.47674 0.44927 0.43328 0.40352 0.47969 0.45237 0.84094
COF 78 0.45349 0.42479 0.44549 0.41637 0.46680 0.43880 0.81862
COF 91 0.47674 0.44927 0.44629 0.41722 0.48141 0.45418 0.81346
COF 100 0.46899 0.44111 0.45302 0.42430 0.49281 0.46617 0.80876

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO