Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#05)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.2 MB) Download raw algorithm evaluation table (66.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.47475 0.46410 0.40215 0.39003 0.49515 0.48491 0.89045
KNN 7 0.43434 0.42288 0.42864 0.41705 0.46018 0.44923 0.94622
KNN 10 0.44444 0.43318 0.43281 0.42131 0.45378 0.44271 0.94477
KNNW 1 0.45455 0.44349 0.37286 0.36014 0.49704 0.48684 0.85100
KNNW 6 0.47475 0.46410 0.41689 0.40507 0.48193 0.47142 0.93044
KNNW 19 0.44444 0.43318 0.43113 0.41960 0.46561 0.45477 0.94386
KNNW 27 0.44444 0.43318 0.43386 0.42238 0.46739 0.45659 0.94350
LOF 18 0.44444 0.43318 0.33302 0.31950 0.45128 0.44016 0.87737
LOF 100 0.37374 0.36104 0.37284 0.36012 0.40719 0.39517 0.95982
SimplifiedLOF 30 0.43434 0.42288 0.34392 0.33062 0.43750 0.42610 0.86352
SimplifiedLOF 45 0.41414 0.40226 0.35234 0.33921 0.45455 0.44349 0.90401
SimplifiedLOF 100 0.39394 0.38165 0.37084 0.35808 0.41808 0.40628 0.95452
LoOP 49 0.43434 0.42288 0.34959 0.33641 0.43878 0.42740 0.89367
LoOP 60 0.42424 0.41257 0.34828 0.33507 0.46061 0.44967 0.90671
LoOP 74 0.41414 0.40226 0.35810 0.34509 0.45238 0.44128 0.92409
LoOP 100 0.40404 0.39196 0.35284 0.33971 0.43182 0.42030 0.94327
LDOF 57 0.43434 0.42288 0.38698 0.37455 0.43721 0.42580 0.93340
LDOF 66 0.42424 0.41257 0.39478 0.38251 0.45267 0.44158 0.93932
LDOF 96 0.41414 0.40226 0.41169 0.39977 0.43269 0.42119 0.95284
LDOF 100 0.40404 0.39196 0.41140 0.39946 0.43210 0.42058 0.95439
ODIN 82 0.42088 0.40913 0.29461 0.28030 0.43158 0.42005 0.89285
ODIN 89 0.40783 0.39582 0.30062 0.28644 0.43275 0.42125 0.90204
ODIN 99 0.40404 0.39196 0.29821 0.28398 0.43678 0.42536 0.91396
ODIN 100 0.40404 0.39196 0.29966 0.28547 0.43678 0.42536 0.91518
FastABOD 20 0.45455 0.44349 0.39509 0.38283 0.47568 0.46505 0.88386
FastABOD 25 0.46465 0.45379 0.39632 0.38408 0.46995 0.45920 0.88426
FastABOD 34 0.46465 0.45379 0.39843 0.38624 0.46465 0.45379 0.88465
FastABOD 87 0.45455 0.44349 0.40596 0.39392 0.46067 0.44974 0.88348
KDEOS 17 0.04040 0.02095 0.03399 0.01441 0.07661 0.05789 0.68829
KDEOS 53 0.04040 0.02095 0.05755 0.03844 0.13069 0.11307 0.78300
KDEOS 54 0.04040 0.02095 0.05633 0.03719 0.13283 0.11525 0.78197
KDEOS 96 0.03030 0.01064 0.04998 0.03072 0.11198 0.09398 0.79824
LDF 18 0.43434 0.42288 0.33966 0.32627 0.44330 0.43201 0.90881
LDF 73 0.37374 0.36104 0.39070 0.37835 0.45775 0.44675 0.95754
LDF 91 0.37374 0.36104 0.41403 0.40215 0.47899 0.46843 0.95635
LDF 100 0.37374 0.36104 0.41731 0.40550 0.47414 0.46348 0.95511
INFLO 45 0.43434 0.42288 0.34009 0.32671 0.43878 0.42740 0.81199
INFLO 59 0.41414 0.40226 0.33529 0.32181 0.45304 0.44195 0.80975
INFLO 100 0.39394 0.38165 0.36164 0.34870 0.42424 0.41257 0.89887
COF 27 0.43434 0.42288 0.33979 0.32640 0.45556 0.44452 0.80345
COF 98 0.38384 0.37135 0.37837 0.36576 0.41667 0.40484 0.93942
COF 100 0.38384 0.37135 0.38062 0.36806 0.41989 0.40813 0.93863

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.4 MB) Download raw algorithm evaluation table (62.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 6 0.43000 0.41840 0.36932 0.35648 0.43216 0.42060 0.91524
KNN 10 0.42000 0.40819 0.37943 0.36680 0.42857 0.41694 0.93410
KNN 14 0.43000 0.41840 0.37604 0.36334 0.43434 0.42283 0.93648
KNN 18 0.41000 0.39799 0.37249 0.35971 0.42781 0.41616 0.93974
KNNW 17 0.42000 0.40819 0.37284 0.36007 0.42188 0.41011 0.92944
KNNW 20 0.42000 0.40819 0.37303 0.36027 0.43299 0.42145 0.93334
KNNW 62 0.40000 0.38779 0.36793 0.35506 0.41026 0.39825 0.93976
LOF 13 0.35000 0.33677 0.22644 0.21070 0.38057 0.36796 0.83908
LOF 18 0.32000 0.30616 0.25643 0.24129 0.39535 0.38304 0.83521
LOF 100 0.33000 0.31636 0.28104 0.26640 0.34978 0.33654 0.95435
SimplifiedLOF 25 0.39000 0.37758 0.27447 0.25970 0.40212 0.38995 0.83765
SimplifiedLOF 47 0.32000 0.30616 0.29179 0.27738 0.37126 0.35846 0.84616
SimplifiedLOF 100 0.33000 0.31636 0.28549 0.27094 0.34462 0.33128 0.93316
LoOP 22 0.37000 0.35718 0.21638 0.20043 0.37563 0.36293 0.81679
LoOP 23 0.37000 0.35718 0.22451 0.20872 0.38542 0.37291 0.81698
LoOP 100 0.32000 0.30616 0.26853 0.25364 0.33636 0.32286 0.91769
LDOF 49 0.40000 0.38779 0.27774 0.26304 0.42623 0.41455 0.90316
LDOF 69 0.42000 0.40819 0.30296 0.28877 0.42211 0.41035 0.91134
LDOF 100 0.35000 0.33677 0.31378 0.29982 0.37874 0.36609 0.93257
ODIN 88 0.34417 0.33082 0.24469 0.22932 0.34872 0.33546 0.88434
ODIN 98 0.33778 0.32430 0.25464 0.23947 0.35294 0.33977 0.89726
ODIN 100 0.34000 0.32657 0.25604 0.24090 0.35294 0.33977 0.89897
FastABOD 4 0.37000 0.35718 0.22028 0.20441 0.37864 0.36599 0.82511
FastABOD 9 0.37000 0.35718 0.30667 0.29256 0.38596 0.37347 0.81998
FastABOD 11 0.37000 0.35718 0.30339 0.28921 0.39053 0.37813 0.82072
FastABOD 16 0.39000 0.37758 0.30178 0.28757 0.39000 0.37758 0.82197
KDEOS 4 0.04000 0.02046 0.02890 0.00913 0.06431 0.04526 0.57555
KDEOS 68 0.04000 0.02046 0.04772 0.02834 0.10236 0.08409 0.74546
KDEOS 85 0.01000 -0.01015 0.04577 0.02635 0.10757 0.08940 0.76219
KDEOS 100 0.01000 -0.01015 0.04587 0.02645 0.10526 0.08705 0.77099
LDF 8 0.37000 0.35718 0.22526 0.20949 0.40000 0.38779 0.81744
LDF 87 0.30000 0.28575 0.32126 0.30745 0.44521 0.43391 0.96026
LDF 89 0.30000 0.28575 0.32488 0.31114 0.43919 0.42777 0.96041
LDF 98 0.31000 0.29596 0.32880 0.31514 0.43262 0.42108 0.96023
INFLO 15 0.35000 0.33677 0.21417 0.19818 0.35354 0.34038 0.75284
INFLO 20 0.33000 0.31636 0.23221 0.21658 0.37273 0.35996 0.76213
INFLO 55 0.30000 0.28575 0.25354 0.23834 0.33929 0.32584 0.78868
INFLO 100 0.33000 0.31636 0.25192 0.23670 0.34783 0.33455 0.81999
COF 22 0.39000 0.37758 0.26129 0.24626 0.40385 0.39171 0.79268
COF 24 0.39000 0.37758 0.27946 0.26479 0.41475 0.40283 0.78894
COF 31 0.38000 0.36738 0.30643 0.29232 0.39462 0.38230 0.77797
COF 100 0.34000 0.32657 0.24697 0.23164 0.35122 0.33801 0.86962

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.2 MB) Download raw algorithm evaluation table (66.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.19192 0.17554 0.13835 0.12088 0.21333 0.19738 0.62269
KNN 5 0.17172 0.15492 0.13177 0.11417 0.19048 0.17406 0.63161
KNNW 2 0.20202 0.18584 0.13549 0.11797 0.21250 0.19653 0.61241
KNNW 4 0.19192 0.17554 0.13879 0.12132 0.20513 0.18901 0.61992
KNNW 14 0.16162 0.14462 0.12854 0.11088 0.16915 0.15231 0.62781
LOF 21 0.44444 0.43318 0.40490 0.39283 0.46927 0.45851 0.90825
LOF 23 0.46465 0.45379 0.40675 0.39472 0.46632 0.45550 0.91295
LOF 36 0.43434 0.42288 0.42053 0.40878 0.45745 0.44645 0.94121
LOF 56 0.38384 0.37135 0.39831 0.38611 0.40223 0.39012 0.95488
SimplifiedLOF 33 0.48485 0.47440 0.43110 0.41957 0.49735 0.48716 0.91777
SimplifiedLOF 38 0.46465 0.45379 0.43967 0.42831 0.48869 0.47832 0.92563
SimplifiedLOF 78 0.39394 0.38165 0.39515 0.38289 0.40548 0.39343 0.96140
LoOP 54 0.45455 0.44349 0.41871 0.40692 0.47191 0.46120 0.94002
LoOP 56 0.45455 0.44349 0.41747 0.40566 0.47727 0.46667 0.94343
LoOP 78 0.43434 0.42288 0.42609 0.41445 0.44920 0.43803 0.95719
LoOP 80 0.42424 0.41257 0.42409 0.41242 0.43523 0.42378 0.95721
LDOF 37 0.45455 0.44349 0.40554 0.39349 0.46602 0.45519 0.90937
LDOF 61 0.44444 0.43318 0.42126 0.40952 0.46927 0.45851 0.93885
LDOF 78 0.43434 0.42288 0.43711 0.42570 0.46067 0.44974 0.95417
LDOF 100 0.44444 0.43318 0.42882 0.41724 0.45226 0.44116 0.96161
ODIN 75 0.38889 0.37650 0.31666 0.30281 0.41379 0.40191 0.91834
ODIN 94 0.40404 0.39196 0.33809 0.32467 0.40506 0.39300 0.93027
ODIN 95 0.39731 0.38509 0.33848 0.32506 0.40764 0.39563 0.92977
ODIN 100 0.38961 0.37724 0.33380 0.32030 0.39759 0.38538 0.93075
FastABOD 3 0.16162 0.14462 0.11451 0.09656 0.17391 0.15716 0.49092
FastABOD 15 0.15152 0.13431 0.12024 0.10241 0.17391 0.15716 0.48399
FastABOD 23 0.15152 0.13431 0.11855 0.10068 0.17647 0.15977 0.48336
KDEOS 79 0.09091 0.07248 0.07822 0.05953 0.13020 0.11256 0.80479
KDEOS 96 0.13131 0.11370 0.07587 0.05713 0.14353 0.12616 0.81642
KDEOS 98 0.13131 0.11370 0.07750 0.05880 0.14529 0.12796 0.81855
KDEOS 100 0.12121 0.10340 0.07785 0.05915 0.14498 0.12764 0.82006
LDF 13 0.46465 0.45379 0.40340 0.39131 0.46939 0.45863 0.90752
LDF 16 0.45455 0.44349 0.41581 0.40397 0.47778 0.46719 0.91320
LDF 22 0.43434 0.42288 0.42447 0.41280 0.44828 0.43709 0.91082
LDF 37 0.37374 0.36104 0.39703 0.38480 0.39779 0.38558 0.94920
INFLO 25 0.44444 0.43318 0.38016 0.36760 0.44444 0.43318 0.84763
INFLO 34 0.44444 0.43318 0.39577 0.38352 0.46667 0.45585 0.86275
INFLO 39 0.43434 0.42288 0.40621 0.39418 0.45455 0.44349 0.87553
INFLO 73 0.37374 0.36104 0.38605 0.37360 0.40000 0.38784 0.93101
COF 32 0.47475 0.46410 0.41060 0.39865 0.48039 0.46986 0.89565
COF 44 0.42424 0.41257 0.41979 0.40803 0.45399 0.44292 0.91793
COF 61 0.41414 0.40226 0.40192 0.38979 0.43956 0.42820 0.93655

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (61.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.12000 0.10209 0.08696 0.06838 0.14615 0.12877 0.57761
KNN 4 0.12000 0.10209 0.09427 0.07584 0.15385 0.13662 0.61969
KNNW 1 0.13000 0.11229 0.09336 0.07491 0.15596 0.13878 0.57553
KNNW 16 0.11000 0.09188 0.08697 0.06838 0.13793 0.12038 0.61427
LOF 31 0.41000 0.39799 0.38016 0.36754 0.45349 0.44236 0.90833
LOF 33 0.42000 0.40819 0.38731 0.37484 0.45198 0.44082 0.91331
LOF 35 0.42000 0.40819 0.38775 0.37529 0.45349 0.44236 0.91386
LOF 90 0.36000 0.34697 0.30906 0.29500 0.39755 0.38529 0.91818
SimplifiedLOF 33 0.42000 0.40819 0.38653 0.37405 0.46154 0.45058 0.91166
SimplifiedLOF 46 0.42000 0.40819 0.39301 0.38065 0.45161 0.44045 0.92036
SimplifiedLOF 51 0.44000 0.42860 0.38759 0.37512 0.45304 0.44191 0.92497
SimplifiedLOF 62 0.42000 0.40819 0.37548 0.36277 0.42857 0.41694 0.92830
LoOP 43 0.45000 0.43881 0.36501 0.35209 0.45226 0.44111 0.90827
LoOP 67 0.43000 0.41840 0.38954 0.37711 0.46591 0.45504 0.92098
LoOP 75 0.44000 0.42860 0.39392 0.38159 0.45128 0.44011 0.92129
LoOP 80 0.43000 0.41840 0.39026 0.37785 0.45263 0.44149 0.92260
LDOF 74 0.43000 0.41840 0.38462 0.37209 0.43216 0.42060 0.92779
LDOF 85 0.42000 0.40819 0.39239 0.38003 0.43011 0.41851 0.93375
LDOF 92 0.42000 0.40819 0.38642 0.37393 0.43979 0.42839 0.93603
LDOF 100 0.40000 0.38779 0.38263 0.37006 0.42063 0.40884 0.93883
ODIN 99 0.38133 0.36874 0.34722 0.33394 0.39269 0.38033 0.91198
ODIN 100 0.38250 0.36993 0.34596 0.33265 0.39450 0.38217 0.91261
FastABOD 8 0.10000 0.08168 0.07591 0.05710 0.13793 0.12038 0.48458
FastABOD 24 0.11000 0.09188 0.07720 0.05842 0.14085 0.12336 0.48263
FastABOD 28 0.11000 0.09188 0.07729 0.05851 0.14286 0.12541 0.48278
FastABOD 29 0.11000 0.09188 0.07731 0.05853 0.14286 0.12541 0.48287
KDEOS 72 0.06000 0.04087 0.04516 0.02572 0.09320 0.07474 0.77433
KDEOS 99 0.05000 0.03066 0.05667 0.03747 0.12500 0.10719 0.79807
KDEOS 100 0.05000 0.03066 0.05657 0.03737 0.12392 0.10609 0.79833
LDF 12 0.41000 0.39799 0.37270 0.35994 0.42391 0.41219 0.91113
LDF 14 0.38000 0.36738 0.36776 0.35489 0.40659 0.39452 0.92015
LDF 26 0.40000 0.38779 0.38577 0.37327 0.43678 0.42532 0.90802
LDF 27 0.40000 0.38779 0.39128 0.37889 0.42938 0.41776 0.91037
INFLO 39 0.43000 0.41840 0.36773 0.35486 0.45087 0.43969 0.83551
INFLO 43 0.44000 0.42860 0.36506 0.35213 0.45810 0.44707 0.84349
INFLO 44 0.44000 0.42860 0.36470 0.35177 0.46067 0.44970 0.84108
INFLO 71 0.39000 0.37758 0.34885 0.33559 0.40394 0.39181 0.88374
COF 16 0.43000 0.41840 0.36163 0.34863 0.43000 0.41840 0.89032
COF 21 0.41000 0.39799 0.38653 0.37405 0.42169 0.40992 0.90744
COF 37 0.42000 0.40819 0.40141 0.38923 0.42640 0.41472 0.90363
COF 57 0.42000 0.40819 0.38187 0.36929 0.43716 0.42570 0.90410

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO