Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (5% of outliers version#05)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (43.4 MB) Download raw algorithm evaluation table (71.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 41 0.44141 0.41212 0.47950 0.45221 0.47712 0.44971 0.90603
KNN 59 0.46875 0.44090 0.48227 0.45513 0.50435 0.47836 0.90498
KNN 62 0.46875 0.44090 0.48172 0.45454 0.50690 0.48104 0.90475
KNN 66 0.47656 0.44912 0.48122 0.45403 0.50166 0.47553 0.90456
KNNW 21 0.46484 0.43679 0.42457 0.39440 0.46614 0.43815 0.87947
KNNW 82 0.44141 0.41212 0.47740 0.45000 0.47761 0.45022 0.90699
KNNW 95 0.42969 0.39979 0.47922 0.45192 0.49271 0.46611 0.90670
KNNW 100 0.42969 0.39979 0.48003 0.45277 0.49266 0.46606 0.90651
LOF 34 0.44531 0.41623 0.38357 0.35125 0.44681 0.41781 0.81372
LOF 36 0.43750 0.40801 0.38721 0.35508 0.45815 0.42974 0.81520
LOF 100 0.41406 0.38334 0.41794 0.38743 0.43574 0.40615 0.91233
SimplifiedLOF 38 0.44141 0.41212 0.39061 0.35866 0.45474 0.42615 0.80774
SimplifiedLOF 97 0.43750 0.40801 0.41117 0.38030 0.44399 0.41484 0.85294
SimplifiedLOF 100 0.42969 0.39979 0.41108 0.38021 0.44399 0.41484 0.85795
LoOP 45 0.41406 0.38334 0.35282 0.31889 0.44255 0.41333 0.79986
LoOP 74 0.43359 0.40390 0.37606 0.34335 0.43529 0.40569 0.82756
LoOP 100 0.42188 0.39157 0.38520 0.35297 0.43552 0.40592 0.84278
LDOF 97 0.44141 0.41212 0.42454 0.39437 0.45961 0.43127 0.89911
LDOF 99 0.44922 0.42034 0.42625 0.39617 0.45640 0.42790 0.89972
LDOF 100 0.44141 0.41212 0.42682 0.39677 0.45489 0.42631 0.90014
ODIN 74 0.43080 0.40096 0.33183 0.29680 0.43137 0.40156 0.79440
ODIN 98 0.42188 0.39157 0.33693 0.30217 0.43568 0.40610 0.81404
ODIN 100 0.42188 0.39157 0.33665 0.30187 0.43033 0.40046 0.81572
FastABOD 7 0.34766 0.31346 0.33903 0.30438 0.36401 0.33067 0.77966
FastABOD 15 0.34375 0.30934 0.33943 0.30480 0.35343 0.31953 0.78196
FastABOD 22 0.35156 0.31757 0.33836 0.30367 0.35260 0.31866 0.78115
FastABOD 99 0.34766 0.31346 0.34199 0.30749 0.36134 0.32786 0.77577
KDEOS 69 0.16016 0.11613 0.12133 0.07526 0.21186 0.17054 0.72697
KDEOS 93 0.15234 0.10790 0.12647 0.08067 0.22668 0.18613 0.73633
KDEOS 100 0.13281 0.08735 0.12563 0.07979 0.23208 0.19182 0.74145
LDF 100 0.48047 0.45323 0.48037 0.45313 0.54519 0.52135 0.93028
INFLO 42 0.41406 0.38334 0.34085 0.30629 0.42702 0.39698 0.73441
INFLO 44 0.41406 0.38334 0.34246 0.30799 0.43182 0.40203 0.73802
INFLO 74 0.41016 0.37923 0.35460 0.32076 0.42735 0.39733 0.75683
INFLO 75 0.41406 0.38334 0.35484 0.32101 0.42918 0.39926 0.75495
COF 44 0.41406 0.38334 0.37657 0.34388 0.42510 0.39496 0.78248
COF 50 0.39062 0.35868 0.38213 0.34974 0.43662 0.40708 0.77675
COF 92 0.42969 0.39979 0.36415 0.33082 0.44444 0.41532 0.73542
COF 100 0.42578 0.39568 0.35791 0.32425 0.45676 0.42828 0.73710

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (43.5 MB) Download raw algorithm evaluation table (72.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.46899 0.44111 0.40327 0.37194 0.47485 0.44727 0.78939
KNN 100 0.36434 0.33096 0.43083 0.40094 0.46535 0.43727 0.89996
KNNW 4 0.45736 0.42887 0.40025 0.36875 0.46275 0.43453 0.78445
KNNW 5 0.45736 0.42887 0.40405 0.37275 0.46809 0.44015 0.78882
KNNW 99 0.40698 0.37583 0.42237 0.39204 0.41502 0.38430 0.89182
KNNW 100 0.40698 0.37583 0.42228 0.39194 0.41420 0.38344 0.89212
LOF 24 0.49612 0.46966 0.43478 0.40509 0.49801 0.47165 0.84070
LOF 25 0.48837 0.46150 0.43592 0.40630 0.49587 0.46939 0.84086
LOF 29 0.48837 0.46150 0.42988 0.39994 0.51271 0.48712 0.82871
SimplifiedLOF 25 0.48062 0.45335 0.44024 0.41084 0.48540 0.45838 0.83119
SimplifiedLOF 32 0.48837 0.46150 0.44995 0.42106 0.50279 0.47668 0.82388
SimplifiedLOF 40 0.48837 0.46150 0.45550 0.42690 0.50101 0.47481 0.82506
SimplifiedLOF 41 0.49225 0.46558 0.45308 0.42436 0.49590 0.46943 0.82370
LoOP 27 0.44186 0.41255 0.37236 0.33940 0.45714 0.42864 0.81034
LoOP 28 0.44961 0.42071 0.37411 0.34125 0.45136 0.42255 0.81082
LoOP 35 0.43411 0.40439 0.39427 0.36246 0.44493 0.41579 0.81912
LDOF 34 0.46512 0.43703 0.42178 0.39142 0.47300 0.44532 0.88299
LDOF 35 0.46124 0.43295 0.42562 0.39546 0.47339 0.44574 0.88652
LDOF 40 0.44961 0.42071 0.42527 0.39509 0.48166 0.45444 0.87253
LDOF 65 0.43798 0.40847 0.42759 0.39753 0.46099 0.43269 0.84780
ODIN 35 0.37582 0.34304 0.27981 0.24199 0.39556 0.36381 0.71608
ODIN 45 0.38443 0.35210 0.29262 0.25548 0.38956 0.35750 0.72789
ODIN 100 0.38217 0.34973 0.33069 0.29555 0.38878 0.35668 0.76961
FastABOD 3 0.43023 0.40031 0.35872 0.32505 0.43444 0.40474 0.78364
FastABOD 4 0.43798 0.40847 0.36053 0.32694 0.43798 0.40847 0.75910
FastABOD 5 0.42248 0.39215 0.36466 0.33130 0.44295 0.41370 0.76055
FastABOD 99 0.39535 0.36360 0.37151 0.33851 0.43049 0.40059 0.73559
KDEOS 40 0.15116 0.10659 0.11857 0.07228 0.19623 0.15402 0.72106
KDEOS 60 0.12791 0.08211 0.12567 0.07975 0.21717 0.17606 0.72340
KDEOS 63 0.13566 0.09027 0.12534 0.07941 0.22068 0.17975 0.72263
LDF 22 0.49225 0.46558 0.47147 0.44371 0.51464 0.48916 0.84761
LDF 23 0.49225 0.46558 0.47457 0.44698 0.51789 0.49258 0.84855
LDF 100 0.39922 0.36768 0.39473 0.36294 0.41240 0.38155 0.89576
INFLO 23 0.44186 0.41255 0.37083 0.33779 0.46771 0.43975 0.75173
INFLO 27 0.43411 0.40439 0.37412 0.34125 0.44889 0.41995 0.75455
INFLO 30 0.42636 0.39623 0.37474 0.34191 0.44206 0.41276 0.74814
COF 30 0.48837 0.46150 0.43059 0.40068 0.49802 0.47166 0.78229
COF 36 0.48062 0.45335 0.43916 0.40971 0.50431 0.47828 0.79400
COF 37 0.47674 0.44927 0.43805 0.40854 0.50685 0.48095 0.79875
COF 41 0.46899 0.44111 0.43689 0.40732 0.51537 0.48992 0.78410

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 5139 objects, 256 outliers (4.98%)

Download raw algorithm results (44.6 MB) Download raw algorithm evaluation table (71.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.19922 0.15724 0.17321 0.12987 0.20870 0.16721 0.60134
KNN 4 0.19141 0.14901 0.17813 0.13504 0.20935 0.16790 0.62545
KNN 5 0.19141 0.14901 0.17627 0.13309 0.21905 0.17810 0.62895
KNN 20 0.16797 0.12435 0.17020 0.12669 0.20840 0.16690 0.64080
KNNW 1 0.19922 0.15724 0.17738 0.13426 0.22275 0.18200 0.58210
KNNW 49 0.16797 0.12435 0.16601 0.12229 0.19140 0.14901 0.63809
LOF 81 0.53906 0.51490 0.57593 0.55370 0.55556 0.53225 0.93748
LOF 85 0.54688 0.52312 0.57524 0.55298 0.56376 0.54089 0.93774
LOF 93 0.56641 0.54367 0.56954 0.54697 0.58333 0.56149 0.93744
LOF 100 0.56250 0.53956 0.56357 0.54069 0.58966 0.56814 0.93653
SimplifiedLOF 82 0.55078 0.52723 0.55965 0.53656 0.56015 0.53709 0.93429
SimplifiedLOF 88 0.55078 0.52723 0.56106 0.53805 0.55666 0.53342 0.93732
SimplifiedLOF 100 0.55078 0.52723 0.56076 0.53773 0.56762 0.54495 0.94004
LoOP 93 0.52734 0.50256 0.54630 0.52251 0.53892 0.51475 0.92713
LoOP 98 0.53125 0.50667 0.55061 0.52705 0.53707 0.51280 0.92976
LoOP 100 0.53125 0.50667 0.55217 0.52870 0.53707 0.51280 0.93058
LDOF 83 0.48438 0.45734 0.49350 0.46695 0.49072 0.46402 0.90442
LDOF 95 0.48438 0.45734 0.51131 0.48569 0.51982 0.49465 0.91836
LDOF 100 0.48438 0.45734 0.51526 0.48985 0.51556 0.49016 0.92306
ODIN 97 0.41797 0.38745 0.40358 0.37231 0.42259 0.39232 0.89439
ODIN 100 0.41235 0.38155 0.41036 0.37945 0.43070 0.40086 0.89744
FastABOD 3 0.14453 0.09968 0.14688 0.10215 0.19335 0.15106 0.50158
FastABOD 4 0.15234 0.10790 0.14792 0.10325 0.19162 0.14924 0.50113
FastABOD 99 0.14844 0.10379 0.14520 0.10039 0.18462 0.14187 0.50161
KDEOS 92 0.07812 0.02979 0.09327 0.04574 0.17433 0.13105 0.73931
KDEOS 96 0.08203 0.03391 0.09410 0.04661 0.17106 0.12760 0.74261
KDEOS 100 0.08203 0.03391 0.09627 0.04889 0.17314 0.12980 0.74615
LDF 46 0.50781 0.48201 0.58074 0.55876 0.54192 0.51790 0.92728
LDF 62 0.56250 0.53956 0.57474 0.55245 0.58031 0.55831 0.93145
LDF 75 0.57422 0.55190 0.56246 0.53952 0.60000 0.57903 0.93077
LDF 80 0.57812 0.55601 0.55930 0.53619 0.59469 0.57344 0.93017
INFLO 95 0.53906 0.51490 0.53749 0.51324 0.55516 0.53184 0.90047
INFLO 98 0.53125 0.50667 0.53939 0.51524 0.55856 0.53542 0.90500
INFLO 99 0.53125 0.50667 0.53811 0.51390 0.56115 0.53814 0.90310
COF 83 0.53906 0.51490 0.53206 0.50753 0.55645 0.53320 0.89743
COF 98 0.57422 0.55190 0.54425 0.52036 0.57905 0.55698 0.89376
COF 99 0.57422 0.55190 0.54551 0.52168 0.58509 0.56333 0.89257

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5171 objects, 258 outliers (4.99%)

Download raw algorithm results (44.7 MB) Download raw algorithm evaluation table (71.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.21705 0.17594 0.18478 0.14196 0.23587 0.19575 0.56128
KNN 3 0.20543 0.16370 0.18433 0.14149 0.24227 0.20248 0.56647
KNN 11 0.22868 0.18818 0.17373 0.13033 0.22868 0.18818 0.58703
KNN 13 0.22093 0.18002 0.17299 0.12957 0.22351 0.18273 0.58835
KNNW 1 0.20930 0.16778 0.18029 0.13725 0.24570 0.20609 0.54520
KNNW 4 0.21318 0.17186 0.18409 0.14124 0.23398 0.19376 0.55845
KNNW 57 0.18217 0.13922 0.16541 0.12159 0.19535 0.15309 0.58612
LOF 61 0.48837 0.46150 0.50715 0.48127 0.49219 0.46552 0.89009
LOF 63 0.49225 0.46558 0.50729 0.48142 0.49706 0.47065 0.88997
LOF 92 0.51163 0.48598 0.49286 0.46622 0.53047 0.50581 0.88452
LOF 100 0.51163 0.48598 0.48711 0.46017 0.53793 0.51367 0.88463
SimplifiedLOF 81 0.50388 0.47782 0.50376 0.47770 0.50579 0.47984 0.89346
SimplifiedLOF 93 0.49612 0.46966 0.49995 0.47369 0.51115 0.48548 0.89524
SimplifiedLOF 99 0.50775 0.48190 0.49769 0.47131 0.51712 0.49177 0.89461
SimplifiedLOF 100 0.50775 0.48190 0.49794 0.47158 0.51966 0.49443 0.89463
LoOP 97 0.48062 0.45335 0.49946 0.47317 0.50471 0.47870 0.89284
LoOP 100 0.50000 0.47374 0.50161 0.47543 0.50370 0.47764 0.89336
LDOF 57 0.40698 0.37583 0.42248 0.39215 0.46970 0.44185 0.85847
LDOF 69 0.44574 0.41663 0.44353 0.41431 0.46465 0.43653 0.87096
LDOF 100 0.43411 0.40439 0.47458 0.44699 0.46925 0.44138 0.90480
ODIN 95 0.39750 0.36586 0.37292 0.33999 0.39768 0.36605 0.87096
ODIN 99 0.39605 0.36434 0.37971 0.34713 0.40326 0.37192 0.87459
ODIN 100 0.39664 0.36496 0.38123 0.34874 0.40316 0.37182 0.87562
FastABOD 3 0.18605 0.14330 0.13357 0.08807 0.22280 0.18198 0.48576
FastABOD 15 0.16667 0.12291 0.15279 0.10830 0.20225 0.16035 0.46968
KDEOS 8 0.09690 0.04947 0.07436 0.02575 0.13077 0.08512 0.59449
KDEOS 100 0.08915 0.04132 0.09422 0.04665 0.16523 0.12139 0.71592
LDF 40 0.47287 0.44519 0.50746 0.48160 0.48496 0.45791 0.89238
LDF 43 0.46512 0.43703 0.51215 0.48654 0.49209 0.46542 0.89137
LDF 66 0.51550 0.49006 0.49740 0.47101 0.53584 0.51146 0.87347
LDF 71 0.51938 0.49414 0.49020 0.46343 0.53242 0.50787 0.86841
INFLO 83 0.47674 0.44927 0.48483 0.45778 0.49467 0.46813 0.86701
INFLO 100 0.49612 0.46966 0.47853 0.45114 0.50746 0.48160 0.85966
COF 33 0.43023 0.40031 0.41347 0.38266 0.44681 0.41776 0.82240
COF 73 0.48450 0.45743 0.48593 0.45893 0.49072 0.46398 0.81464
COF 91 0.51550 0.49006 0.48541 0.45839 0.52143 0.49630 0.81216
COF 92 0.50000 0.47374 0.48524 0.45820 0.52416 0.49918 0.81289

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO