Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#02)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.1 MB) Download raw algorithm evaluation table (66.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.54545 0.53624 0.51629 0.50648 0.55738 0.54840 0.94164
KNN 26 0.49495 0.48471 0.46455 0.45370 0.50000 0.48986 0.94516
KNNW 1 0.54545 0.53624 0.46113 0.45021 0.55610 0.54710 0.88747
KNNW 2 0.53535 0.52593 0.48491 0.47447 0.56287 0.55401 0.92154
KNNW 7 0.54545 0.53624 0.51095 0.50103 0.55556 0.54654 0.94129
KNNW 56 0.49495 0.48471 0.47587 0.46524 0.50777 0.49779 0.94493
LOF 26 0.45455 0.44349 0.39426 0.38198 0.48087 0.47035 0.92596
LOF 38 0.46465 0.45379 0.39139 0.37905 0.46939 0.45863 0.93943
LOF 52 0.44444 0.43318 0.39779 0.38558 0.46073 0.44980 0.95046
LOF 100 0.43434 0.42288 0.39646 0.38422 0.45161 0.44049 0.96335
SimplifiedLOF 51 0.48485 0.47440 0.41652 0.40469 0.49038 0.48005 0.93987
SimplifiedLOF 93 0.47475 0.46410 0.42078 0.40903 0.49451 0.48426 0.95657
SimplifiedLOF 100 0.46465 0.45379 0.42497 0.41331 0.48677 0.47637 0.95822
LoOP 45 0.45455 0.44349 0.38678 0.37435 0.48864 0.47827 0.92148
LoOP 78 0.47475 0.46410 0.38941 0.37704 0.47475 0.46410 0.94303
LoOP 96 0.46465 0.45379 0.39623 0.38399 0.48387 0.47341 0.95003
LoOP 100 0.47475 0.46410 0.39569 0.38344 0.48677 0.47637 0.95042
LDOF 62 0.45455 0.44349 0.39763 0.38541 0.46377 0.45290 0.95365
LDOF 96 0.44444 0.43318 0.42867 0.41709 0.47887 0.46831 0.96301
LDOF 97 0.43434 0.42288 0.42766 0.41606 0.48148 0.47097 0.96286
ODIN 75 0.36869 0.35589 0.29913 0.28492 0.43290 0.42140 0.90036
ODIN 100 0.40741 0.39539 0.34304 0.32972 0.43172 0.42020 0.92319
FastABOD 6 0.51515 0.50532 0.47815 0.46757 0.52041 0.51068 0.89397
FastABOD 25 0.53535 0.52593 0.47738 0.46679 0.54545 0.53624 0.89340
FastABOD 40 0.53535 0.52593 0.47967 0.46912 0.55249 0.54341 0.89276
FastABOD 94 0.53535 0.52593 0.48231 0.47182 0.54945 0.54032 0.89173
KDEOS 11 0.06061 0.04156 0.03302 0.01341 0.06893 0.05005 0.66352
KDEOS 66 0.01010 -0.00997 0.05199 0.03277 0.12545 0.10772 0.81150
KDEOS 98 0.02020 0.00034 0.05293 0.03373 0.11314 0.09516 0.81900
LDF 15 0.46465 0.45379 0.40282 0.39072 0.46939 0.45863 0.90406
LDF 26 0.45455 0.44349 0.41783 0.40603 0.48276 0.47227 0.93622
LDF 53 0.41414 0.40226 0.42365 0.41197 0.44324 0.43196 0.95574
LDF 100 0.43434 0.42288 0.41947 0.40770 0.45643 0.44541 0.96058
INFLO 32 0.46465 0.45379 0.37796 0.36535 0.47191 0.46120 0.80579
INFLO 47 0.45455 0.44349 0.37364 0.36094 0.47887 0.46831 0.82018
INFLO 97 0.43434 0.42288 0.39476 0.38249 0.46809 0.45730 0.91903
COF 27 0.52525 0.51563 0.43580 0.42436 0.52525 0.51563 0.88407
COF 33 0.50505 0.49502 0.45175 0.44063 0.54237 0.53309 0.89586
COF 34 0.50505 0.49502 0.45182 0.44070 0.52222 0.51254 0.89900
COF 99 0.51515 0.50532 0.44257 0.43127 0.52850 0.51894 0.93507

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.4 MB) Download raw algorithm evaluation table (63.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.37000 0.35718 0.24135 0.22591 0.38043 0.36782 0.78590
KNN 14 0.34000 0.32657 0.26894 0.25406 0.35789 0.34483 0.88830
KNN 47 0.27000 0.25514 0.25031 0.23505 0.32208 0.30828 0.92764
KNNW 2 0.37000 0.35718 0.24402 0.22863 0.37374 0.36099 0.76059
KNNW 5 0.37000 0.35718 0.25497 0.23980 0.38835 0.37590 0.78366
KNNW 85 0.29000 0.27555 0.25669 0.24156 0.32500 0.31126 0.92834
KNNW 96 0.29000 0.27555 0.25694 0.24181 0.32323 0.30946 0.92820
LOF 20 0.44000 0.42860 0.36197 0.34899 0.44878 0.43756 0.81983
LOF 22 0.43000 0.41840 0.36818 0.35532 0.45411 0.44300 0.80831
LOF 100 0.30000 0.28575 0.28385 0.26928 0.32812 0.31445 0.95697
SimplifiedLOF 22 0.46000 0.44901 0.37493 0.36221 0.46809 0.45726 0.83019
SimplifiedLOF 28 0.42000 0.40819 0.38042 0.36781 0.44444 0.43314 0.81246
SimplifiedLOF 100 0.35000 0.33677 0.29976 0.28551 0.36967 0.35684 0.92512
LoOP 45 0.40000 0.38779 0.32566 0.31194 0.41593 0.40404 0.78584
LoOP 49 0.41000 0.39799 0.32384 0.31007 0.42453 0.41282 0.78195
LoOP 52 0.42000 0.40819 0.32089 0.30707 0.42211 0.41035 0.78386
LoOP 100 0.36000 0.34697 0.30726 0.29316 0.37209 0.35931 0.91710
LDOF 56 0.41000 0.39799 0.32986 0.31622 0.45810 0.44707 0.89300
LDOF 71 0.44000 0.42860 0.33442 0.32087 0.44944 0.43823 0.89750
LDOF 100 0.39000 0.37758 0.33632 0.32281 0.41111 0.39912 0.93097
ODIN 61 0.36077 0.34776 0.19937 0.18307 0.36967 0.35684 0.80163
ODIN 77 0.33385 0.32029 0.22416 0.20837 0.38053 0.36792 0.84884
ODIN 99 0.33333 0.31976 0.25370 0.23851 0.35135 0.33815 0.89092
ODIN 100 0.33200 0.31840 0.25223 0.23701 0.35455 0.34141 0.89266
FastABOD 3 0.34000 0.32657 0.25119 0.23595 0.37931 0.36668 0.83009
FastABOD 4 0.36000 0.34697 0.23171 0.21607 0.36548 0.35257 0.80256
KDEOS 18 0.04000 0.02046 0.03306 0.01338 0.06639 0.04739 0.64667
KDEOS 84 0.01000 -0.01015 0.04848 0.02911 0.13589 0.11831 0.70723
KDEOS 96 0.02000 0.00005 0.05059 0.03127 0.13133 0.11365 0.72369
KDEOS 100 0.02000 0.00005 0.05037 0.03104 0.12824 0.11050 0.72933
LDF 15 0.46000 0.44901 0.35194 0.33875 0.46000 0.44901 0.81583
LDF 19 0.43000 0.41840 0.38355 0.37101 0.44944 0.43823 0.81243
LDF 85 0.28000 0.26535 0.30785 0.29376 0.39474 0.38242 0.96251
INFLO 20 0.40000 0.38779 0.32155 0.30774 0.41237 0.40041 0.77215
INFLO 33 0.38000 0.36738 0.32793 0.31425 0.40678 0.39471 0.77615
INFLO 38 0.38000 0.36738 0.32252 0.30873 0.42254 0.41078 0.75640
INFLO 100 0.32000 0.30616 0.27847 0.26379 0.34821 0.33495 0.87784
COF 19 0.45000 0.43881 0.34600 0.33269 0.46995 0.45916 0.77457
COF 20 0.45000 0.43881 0.34762 0.33434 0.47619 0.46553 0.78610
COF 23 0.44000 0.42860 0.37020 0.35738 0.45405 0.44294 0.82418
COF 30 0.45000 0.43881 0.38556 0.37305 0.46078 0.44981 0.73731

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.2 MB) Download raw algorithm evaluation table (64.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.17172 0.15492 0.14706 0.12977 0.21477 0.19884 0.67952
KNN 2 0.18182 0.16523 0.15113 0.13392 0.20606 0.18996 0.67984
KNN 27 0.13131 0.11370 0.12774 0.11006 0.16552 0.14860 0.68084
KNNW 1 0.19192 0.17554 0.14313 0.12576 0.22422 0.20849 0.65791
KNNW 7 0.17172 0.15492 0.15155 0.13435 0.20000 0.18378 0.67851
KNNW 38 0.13131 0.11370 0.12962 0.11197 0.16901 0.15217 0.67974
LOF 53 0.46465 0.45379 0.46705 0.45625 0.51163 0.50173 0.95477
LOF 55 0.46465 0.45379 0.46290 0.45201 0.49724 0.48704 0.95493
LOF 63 0.48485 0.47440 0.44723 0.43602 0.48731 0.47692 0.95434
SimplifiedLOF 49 0.48485 0.47440 0.46041 0.44947 0.49711 0.48691 0.95065
SimplifiedLOF 52 0.48485 0.47440 0.46515 0.45430 0.50292 0.49285 0.95449
SimplifiedLOF 66 0.47475 0.46410 0.45951 0.44855 0.51087 0.50095 0.95921
SimplifiedLOF 67 0.47475 0.46410 0.45862 0.44765 0.50000 0.48986 0.95921
LoOP 73 0.46465 0.45379 0.45522 0.44418 0.51190 0.50201 0.95251
LoOP 77 0.47475 0.46410 0.46208 0.45117 0.50602 0.49601 0.95356
LoOP 79 0.47475 0.46410 0.46345 0.45257 0.50299 0.49292 0.95404
LoOP 86 0.47475 0.46410 0.46334 0.45245 0.50000 0.48986 0.95423
LDOF 64 0.44444 0.43318 0.42738 0.41577 0.48780 0.47742 0.93849
LDOF 72 0.45455 0.44349 0.43690 0.42549 0.48521 0.47477 0.94640
LDOF 95 0.45455 0.44349 0.45145 0.44033 0.47826 0.46768 0.96057
LDOF 100 0.45455 0.44349 0.44935 0.43819 0.47120 0.46048 0.96151
ODIN 93 0.39513 0.38286 0.40576 0.39371 0.44444 0.43318 0.93425
ODIN 98 0.40016 0.38799 0.41142 0.39949 0.43750 0.42610 0.93610
ODIN 100 0.40312 0.39102 0.41281 0.40090 0.43750 0.42610 0.93597
FastABOD 3 0.14141 0.12401 0.11292 0.09494 0.18182 0.16523 0.56876
FastABOD 6 0.13131 0.11370 0.11848 0.10060 0.19118 0.17478 0.56580
FastABOD 14 0.14141 0.12401 0.11925 0.10139 0.18310 0.16654 0.56243
KDEOS 12 0.07071 0.05187 0.03733 0.01782 0.07602 0.05728 0.69908
KDEOS 93 0.05051 0.03125 0.05374 0.03455 0.11429 0.09633 0.79340
KDEOS 100 0.06061 0.04156 0.05692 0.03780 0.11126 0.09324 0.80057
LDF 38 0.44444 0.43318 0.45496 0.44390 0.47727 0.46667 0.94424
LDF 39 0.45455 0.44349 0.45464 0.44359 0.46995 0.45920 0.94428
LDF 56 0.39394 0.38165 0.41608 0.40424 0.44053 0.42919 0.94507
INFLO 49 0.47475 0.46410 0.42269 0.41098 0.49162 0.48131 0.88382
INFLO 64 0.46465 0.45379 0.44284 0.43154 0.49451 0.48426 0.93437
INFLO 66 0.46465 0.45379 0.44252 0.43122 0.50273 0.49265 0.93428
INFLO 74 0.47475 0.46410 0.43888 0.42751 0.48958 0.47923 0.93477
COF 36 0.48485 0.47440 0.43750 0.42609 0.49231 0.48201 0.90668
COF 41 0.48485 0.47440 0.44529 0.43404 0.50847 0.49851 0.91771
COF 44 0.46465 0.45379 0.45029 0.43915 0.50602 0.49601 0.92443
COF 70 0.42424 0.41257 0.42996 0.41840 0.45136 0.44024 0.93539

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (61.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.12000 0.10209 0.06592 0.04691 0.12987 0.11216 0.49098
KNN 2 0.12000 0.10209 0.07126 0.05236 0.12903 0.11130 0.50033
KNN 7 0.10000 0.08168 0.06816 0.04919 0.13333 0.11569 0.51082
KNN 45 0.08000 0.06127 0.05318 0.03391 0.09091 0.07241 0.51990
KNNW 1 0.14000 0.12250 0.07276 0.05389 0.15596 0.13878 0.48258
KNNW 100 0.08000 0.06127 0.05292 0.03364 0.09091 0.07241 0.51690
LOF 34 0.34000 0.32657 0.28651 0.27199 0.36170 0.34871 0.85321
LOF 85 0.32000 0.30616 0.23412 0.21853 0.37705 0.36437 0.85816
LOF 100 0.32000 0.30616 0.22399 0.20819 0.36735 0.35447 0.86619
SimplifiedLOF 27 0.36000 0.34697 0.31107 0.29704 0.40000 0.38779 0.84228
SimplifiedLOF 31 0.37000 0.35718 0.30062 0.28639 0.40678 0.39471 0.83948
SimplifiedLOF 34 0.38000 0.36738 0.30289 0.28870 0.40000 0.38779 0.84000
SimplifiedLOF 63 0.33000 0.31636 0.26553 0.25058 0.34320 0.32983 0.87914
LoOP 34 0.36000 0.34697 0.25739 0.24227 0.36458 0.35165 0.82178
LoOP 38 0.35000 0.33677 0.26165 0.24662 0.37870 0.36605 0.82616
LoOP 54 0.34000 0.32657 0.26985 0.25499 0.37209 0.35931 0.85706
LoOP 84 0.32000 0.30616 0.25657 0.24144 0.36585 0.35295 0.87509
LDOF 26 0.32000 0.30616 0.30391 0.28974 0.36863 0.35578 0.85525
LDOF 30 0.35000 0.33677 0.29260 0.27820 0.35233 0.33915 0.85336
LDOF 34 0.35000 0.33677 0.29545 0.28111 0.39053 0.37813 0.85064
LDOF 100 0.32000 0.30616 0.27992 0.26526 0.35556 0.34244 0.91496
ODIN 65 0.27235 0.25754 0.20402 0.18782 0.29762 0.28332 0.83443
ODIN 99 0.27000 0.25514 0.23079 0.21514 0.32911 0.31546 0.86952
ODIN 100 0.27000 0.25514 0.23228 0.21665 0.32911 0.31546 0.87077
FastABOD 3 0.16000 0.14290 0.09186 0.07337 0.20000 0.18372 0.45999
KDEOS 3 0.07000 0.05107 0.07152 0.05262 0.09259 0.07412 0.57513
KDEOS 100 0.05000 0.03066 0.04927 0.02992 0.09698 0.07860 0.73868
LDF 12 0.34000 0.32657 0.30171 0.28750 0.37500 0.36228 0.84686
LDF 13 0.33200 0.31840 0.30828 0.29421 0.36158 0.34859 0.86627
LDF 30 0.31000 0.29596 0.28477 0.27021 0.35928 0.34624 0.86735
LDF 49 0.35000 0.33677 0.26517 0.25021 0.35407 0.34092 0.84714
INFLO 34 0.36000 0.34697 0.26209 0.24707 0.37037 0.35755 0.74370
INFLO 38 0.34000 0.32657 0.26665 0.25172 0.36686 0.35398 0.77623
INFLO 45 0.34000 0.32657 0.25106 0.23582 0.38202 0.36944 0.78362
INFLO 77 0.33000 0.31636 0.24240 0.22698 0.34884 0.33558 0.83152
COF 22 0.35000 0.33677 0.25830 0.24321 0.36047 0.34745 0.77191
COF 39 0.32000 0.30616 0.25481 0.23964 0.36709 0.35421 0.75796
COF 71 0.30000 0.28575 0.22909 0.21339 0.30303 0.28884 0.81920

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO