Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#04)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.1 MB) Download raw algorithm evaluation table (66.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.46465 0.45379 0.45135 0.44023 0.50588 0.49586 0.93467
KNN 6 0.44444 0.43318 0.44915 0.43798 0.48045 0.46991 0.93876
KNNW 5 0.47475 0.46410 0.42406 0.41238 0.51190 0.50201 0.93278
KNNW 6 0.47475 0.46410 0.42865 0.41706 0.51497 0.50514 0.93485
KNNW 11 0.45455 0.44349 0.44834 0.43715 0.49123 0.48091 0.93957
LOF 15 0.44444 0.43318 0.38248 0.36996 0.44776 0.43656 0.87287
LOF 19 0.44444 0.43318 0.39653 0.38429 0.46667 0.45585 0.88776
LOF 21 0.43434 0.42288 0.40009 0.38793 0.46409 0.45322 0.90228
LOF 87 0.38384 0.37135 0.35993 0.34695 0.40110 0.38896 0.95002
SimplifiedLOF 24 0.45455 0.44349 0.41072 0.39877 0.46784 0.45705 0.86814
SimplifiedLOF 26 0.46465 0.45379 0.40642 0.39438 0.46857 0.45780 0.87440
SimplifiedLOF 27 0.46465 0.45379 0.40406 0.39198 0.47059 0.45985 0.87779
SimplifiedLOF 100 0.40404 0.39196 0.39471 0.38244 0.42667 0.41504 0.95457
LoOP 31 0.41414 0.40226 0.39022 0.37786 0.45122 0.44009 0.88661
LoOP 38 0.44444 0.43318 0.39015 0.37779 0.44681 0.43559 0.89804
LoOP 47 0.41414 0.40226 0.38602 0.37358 0.45556 0.44452 0.90806
LoOP 100 0.39394 0.38165 0.37127 0.35852 0.42593 0.41429 0.94581
LDOF 38 0.46465 0.45379 0.41433 0.40245 0.47000 0.45925 0.92779
LDOF 72 0.44444 0.43318 0.44695 0.43574 0.48447 0.47402 0.95288
LDOF 90 0.44444 0.43318 0.44751 0.43631 0.46857 0.45780 0.95719
LDOF 99 0.44444 0.43318 0.44494 0.43369 0.46948 0.45873 0.95835
ODIN 54 0.39646 0.38423 0.29676 0.28250 0.40000 0.38784 0.86063
ODIN 71 0.38384 0.37135 0.34231 0.32898 0.41975 0.40799 0.88850
ODIN 100 0.38889 0.37650 0.35847 0.34547 0.40000 0.38784 0.91901
FastABOD 5 0.45455 0.44349 0.39887 0.38668 0.45771 0.44672 0.87598
FastABOD 6 0.46465 0.45379 0.40689 0.39487 0.46701 0.45620 0.87213
FastABOD 15 0.46465 0.45379 0.39668 0.38445 0.47668 0.46607 0.86608
FastABOD 45 0.47475 0.46410 0.39335 0.38105 0.47475 0.46410 0.86155
KDEOS 32 0.10101 0.08278 0.05555 0.03641 0.10557 0.08744 0.75802
KDEOS 54 0.06061 0.04156 0.07205 0.05323 0.14220 0.12481 0.81724
KDEOS 84 0.07071 0.05187 0.07482 0.05606 0.13421 0.11665 0.83244
KDEOS 88 0.08081 0.06217 0.07414 0.05537 0.13235 0.11476 0.83264
LDF 10 0.45455 0.44349 0.34423 0.33094 0.46305 0.45217 0.85053
LDF 11 0.44444 0.43318 0.37799 0.36538 0.46535 0.45451 0.86709
LDF 14 0.43434 0.42288 0.39365 0.38135 0.46409 0.45322 0.88738
LDF 54 0.38384 0.37135 0.38015 0.36758 0.42042 0.40867 0.94480
INFLO 26 0.41414 0.40226 0.37898 0.36639 0.44706 0.43585 0.79896
INFLO 27 0.41414 0.40226 0.37417 0.36149 0.44970 0.43855 0.78941
INFLO 28 0.42424 0.41257 0.37416 0.36147 0.44186 0.43054 0.79346
INFLO 97 0.39394 0.38165 0.36563 0.35276 0.41000 0.39804 0.90807
COF 23 0.44444 0.43318 0.36666 0.35382 0.51807 0.50830 0.81856
COF 27 0.47475 0.46410 0.37989 0.36732 0.49383 0.48356 0.83082
COF 30 0.46465 0.45379 0.39090 0.37855 0.50000 0.48986 0.84248
COF 96 0.44444 0.43318 0.36209 0.34915 0.44565 0.43441 0.91879

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.3 MB) Download raw algorithm evaluation table (62.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 19 0.43000 0.41840 0.38705 0.37457 0.43434 0.42283 0.94051
KNN 57 0.37000 0.35718 0.39289 0.38054 0.42525 0.41355 0.94896
KNN 60 0.37000 0.35718 0.39440 0.38207 0.42282 0.41107 0.94872
KNN 94 0.36000 0.34697 0.39371 0.38137 0.46840 0.45758 0.94780
KNNW 5 0.42000 0.40819 0.34674 0.33345 0.42254 0.41078 0.85127
KNNW 28 0.42000 0.40819 0.38202 0.36944 0.44444 0.43314 0.92961
KNNW 95 0.39000 0.37758 0.39615 0.38386 0.40351 0.39137 0.94805
KNNW 100 0.39000 0.37758 0.39661 0.38433 0.40179 0.38961 0.94804
LOF 22 0.35000 0.33677 0.26824 0.25335 0.36269 0.34972 0.80571
LOF 24 0.34000 0.32657 0.27508 0.26033 0.36872 0.35587 0.80622
LOF 28 0.34000 0.32657 0.28092 0.26628 0.35862 0.34557 0.80097
LOF 100 0.27000 0.25514 0.27452 0.25976 0.30882 0.29476 0.94344
SimplifiedLOF 26 0.38000 0.36738 0.28291 0.26831 0.38788 0.37542 0.82628
SimplifiedLOF 27 0.37000 0.35718 0.28978 0.27533 0.39773 0.38547 0.82441
SimplifiedLOF 39 0.36000 0.34697 0.31038 0.29634 0.36649 0.35360 0.81240
SimplifiedLOF 100 0.32000 0.30616 0.28451 0.26995 0.35294 0.33977 0.90918
LoOP 40 0.35000 0.33677 0.27032 0.25547 0.37805 0.36539 0.80379
LoOP 51 0.35000 0.33677 0.27111 0.25628 0.37778 0.36511 0.80248
LoOP 72 0.36000 0.34697 0.26099 0.24595 0.36000 0.34697 0.82904
LoOP 100 0.32000 0.30616 0.26209 0.24707 0.34783 0.33455 0.88577
LDOF 52 0.38000 0.36738 0.30085 0.28662 0.38000 0.36738 0.89906
LDOF 83 0.37000 0.35718 0.31385 0.29989 0.39216 0.37978 0.90900
LDOF 95 0.37000 0.35718 0.32013 0.30629 0.39024 0.37783 0.91944
LDOF 100 0.37000 0.35718 0.31521 0.30128 0.38278 0.37021 0.91983
ODIN 81 0.35000 0.33677 0.22359 0.20779 0.35000 0.33677 0.82774
ODIN 100 0.33714 0.32365 0.23237 0.21674 0.34314 0.32977 0.86496
FastABOD 4 0.33000 0.31636 0.22522 0.20944 0.36538 0.35247 0.81041
FastABOD 83 0.41000 0.39799 0.33377 0.32021 0.42718 0.41553 0.79385
FastABOD 94 0.41000 0.39799 0.33517 0.32164 0.42718 0.41553 0.79336
FastABOD 99 0.41000 0.39799 0.33504 0.32150 0.42927 0.41765 0.79314
KDEOS 10 0.06000 0.04087 0.03413 0.01447 0.06322 0.04415 0.64099
KDEOS 78 0.02000 0.00005 0.04840 0.02903 0.11979 0.10188 0.72783
KDEOS 89 0.00000 -0.02035 0.04928 0.02993 0.11782 0.09986 0.73706
KDEOS 95 0.00000 -0.02035 0.04853 0.02916 0.11765 0.09969 0.74143
LDF 15 0.38000 0.36738 0.30138 0.28716 0.38000 0.36738 0.85097
LDF 100 0.30000 0.28575 0.35379 0.34063 0.45775 0.44671 0.95964
INFLO 26 0.35000 0.33677 0.25859 0.24350 0.38554 0.37304 0.74449
INFLO 50 0.35000 0.33677 0.27264 0.25784 0.35484 0.34171 0.76705
INFLO 54 0.36000 0.34697 0.26515 0.25020 0.36000 0.34697 0.76238
INFLO 98 0.30000 0.28575 0.24798 0.23267 0.32287 0.30909 0.77160
COF 15 0.28000 0.26535 0.22853 0.21283 0.31111 0.29709 0.80503
COF 24 0.39000 0.37758 0.29580 0.28146 0.39196 0.37958 0.77921
COF 36 0.37000 0.35718 0.29358 0.27920 0.40223 0.39007 0.74798
COF 58 0.33000 0.31636 0.29889 0.28462 0.38150 0.36891 0.78130

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.2 MB) Download raw algorithm evaluation table (65.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.20202 0.18584 0.14536 0.12803 0.22819 0.21254 0.58661
KNN 3 0.21212 0.19615 0.15494 0.13781 0.21978 0.20396 0.62261
KNN 4 0.21212 0.19615 0.15512 0.13799 0.21429 0.19836 0.62550
KNN 6 0.19192 0.17554 0.15648 0.13938 0.22318 0.20743 0.62440
KNNW 2 0.19192 0.17554 0.14774 0.13046 0.23841 0.22297 0.58289
KNNW 7 0.20202 0.18584 0.15365 0.13649 0.21622 0.20033 0.61960
KNNW 8 0.21212 0.19615 0.15341 0.13624 0.21739 0.20152 0.62022
LOF 44 0.44444 0.43318 0.40725 0.39523 0.47826 0.46768 0.91667
LOF 49 0.44444 0.43318 0.41419 0.40231 0.46739 0.45659 0.91918
LOF 51 0.45455 0.44349 0.41202 0.40010 0.46632 0.45550 0.92025
LOF 68 0.43434 0.42288 0.39634 0.38411 0.43689 0.42548 0.92507
SimplifiedLOF 44 0.47475 0.46410 0.41806 0.40626 0.47475 0.46410 0.89129
SimplifiedLOF 45 0.47475 0.46410 0.42246 0.41075 0.48454 0.47409 0.89411
SimplifiedLOF 47 0.46465 0.45379 0.42322 0.41153 0.47312 0.46244 0.90035
SimplifiedLOF 82 0.39394 0.38165 0.39834 0.38614 0.43689 0.42548 0.92861
LoOP 47 0.46465 0.45379 0.37677 0.36413 0.46465 0.45379 0.88807
LoOP 52 0.44444 0.43318 0.38562 0.37317 0.47191 0.46120 0.89817
LoOP 82 0.45455 0.44349 0.40774 0.39574 0.47120 0.46048 0.92529
LDOF 74 0.44444 0.43318 0.40773 0.39572 0.44898 0.43781 0.92449
LDOF 87 0.42424 0.41257 0.41056 0.39861 0.45652 0.44550 0.93604
LDOF 95 0.42424 0.41257 0.41362 0.40174 0.44199 0.43068 0.93896
LDOF 100 0.42424 0.41257 0.40905 0.39707 0.43716 0.42575 0.94010
ODIN 96 0.33838 0.32497 0.31340 0.29948 0.35754 0.34452 0.90801
ODIN 97 0.33838 0.32497 0.31279 0.29885 0.35955 0.34657 0.90773
ODIN 99 0.34055 0.32718 0.31388 0.29997 0.35106 0.33791 0.90735
FastABOD 3 0.18182 0.16523 0.12873 0.11107 0.21250 0.19653 0.49529
FastABOD 11 0.17172 0.15492 0.13366 0.11609 0.21769 0.20183 0.48760
FastABOD 32 0.17172 0.15492 0.13221 0.11462 0.21918 0.20335 0.48339
KDEOS 93 0.10101 0.08278 0.06168 0.04266 0.12381 0.10605 0.76420
KDEOS 96 0.11111 0.09309 0.06106 0.04202 0.12727 0.10958 0.76750
KDEOS 100 0.12121 0.10340 0.06148 0.04245 0.13270 0.11512 0.76750
LDF 20 0.42424 0.41257 0.38797 0.37556 0.42857 0.41699 0.87362
LDF 33 0.40404 0.39196 0.40084 0.38869 0.41885 0.40707 0.91103
LDF 36 0.42424 0.41257 0.41749 0.40568 0.43564 0.42420 0.90819
LDF 47 0.41414 0.40226 0.39822 0.38602 0.45038 0.43924 0.90581
INFLO 50 0.46465 0.45379 0.37846 0.36585 0.47872 0.46815 0.85274
INFLO 51 0.45455 0.44349 0.37638 0.36374 0.48387 0.47341 0.84762
INFLO 72 0.41414 0.40226 0.38801 0.37560 0.43275 0.42125 0.89446
INFLO 85 0.41414 0.40226 0.38130 0.36876 0.43636 0.42494 0.89472
COF 36 0.44444 0.43318 0.37166 0.35892 0.45596 0.44493 0.86965
COF 50 0.40404 0.39196 0.40219 0.39007 0.45161 0.44049 0.89373
COF 55 0.40404 0.39196 0.40049 0.38834 0.46154 0.45062 0.89328
COF 56 0.38384 0.37135 0.40062 0.38846 0.46154 0.45062 0.89428

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (60.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.14000 0.12250 0.11859 0.10065 0.14925 0.13194 0.57652
KNN 5 0.12000 0.10209 0.11422 0.09619 0.16154 0.14447 0.60010
KNNW 1 0.14000 0.12250 0.10706 0.08889 0.15827 0.14114 0.53788
KNNW 7 0.12000 0.10209 0.11820 0.10025 0.14876 0.13143 0.58789
KNNW 38 0.12000 0.10209 0.10491 0.08669 0.14286 0.12541 0.59488
LOF 41 0.44000 0.42860 0.40176 0.38959 0.46512 0.45423 0.90066
LOF 46 0.44000 0.42860 0.40679 0.39471 0.46995 0.45916 0.90367
LOF 48 0.43000 0.41840 0.40465 0.39254 0.47514 0.46445 0.90499
LOF 98 0.39000 0.37758 0.35198 0.33879 0.42675 0.41508 0.91382
SimplifiedLOF 45 0.43000 0.41840 0.40600 0.39391 0.45614 0.44507 0.90192
SimplifiedLOF 48 0.44000 0.42860 0.40392 0.39179 0.45614 0.44507 0.90557
SimplifiedLOF 54 0.44000 0.42860 0.40236 0.39020 0.46784 0.45700 0.91355
SimplifiedLOF 92 0.39000 0.37758 0.37550 0.36279 0.43624 0.42477 0.92284
LoOP 61 0.41000 0.39799 0.39312 0.38076 0.45977 0.44877 0.91321
LoOP 76 0.43000 0.41840 0.39532 0.38302 0.44828 0.43705 0.92204
LoOP 85 0.43000 0.41840 0.40907 0.39705 0.44944 0.43823 0.92418
LoOP 92 0.42000 0.40819 0.40269 0.39053 0.43429 0.42277 0.92583
LDOF 24 0.40000 0.38779 0.31129 0.29727 0.40816 0.39612 0.87865
LDOF 85 0.40000 0.38779 0.39679 0.38451 0.42697 0.41530 0.93912
LDOF 86 0.40000 0.38779 0.39767 0.38541 0.42697 0.41530 0.94002
LDOF 100 0.40000 0.38779 0.39401 0.38167 0.41808 0.40623 0.94660
ODIN 92 0.37789 0.36523 0.34179 0.32839 0.41667 0.40479 0.90204
ODIN 98 0.39062 0.37822 0.34715 0.33386 0.41176 0.39979 0.90579
ODIN 100 0.40000 0.38779 0.34700 0.33371 0.40964 0.39762 0.90684
FastABOD 3 0.13000 0.11229 0.07325 0.05439 0.16327 0.14623 0.47424
FastABOD 12 0.11000 0.09188 0.10125 0.08296 0.15278 0.13553 0.45157
KDEOS 10 0.04000 0.02046 0.02972 0.00997 0.06869 0.04973 0.63563
KDEOS 94 0.04000 0.02046 0.04436 0.02491 0.09665 0.07827 0.74495
KDEOS 100 0.04000 0.02046 0.04652 0.02712 0.09662 0.07823 0.75118
LDF 14 0.38000 0.36738 0.35691 0.34382 0.41808 0.40623 0.89609
LDF 35 0.41000 0.39799 0.40623 0.39415 0.42286 0.41111 0.89222
LDF 46 0.40000 0.38779 0.40407 0.39194 0.44195 0.43059 0.88966
LDF 58 0.42000 0.40819 0.37595 0.36325 0.42857 0.41694 0.88419
INFLO 47 0.43000 0.41840 0.37825 0.36559 0.44444 0.43314 0.83895
INFLO 51 0.44000 0.42860 0.37482 0.36209 0.45405 0.44294 0.84543
INFLO 55 0.44000 0.42860 0.37618 0.36348 0.45810 0.44707 0.85041
INFLO 75 0.41000 0.39799 0.37559 0.36288 0.42105 0.40927 0.88331
COF 60 0.41000 0.39799 0.43100 0.41942 0.45783 0.44680 0.87829
COF 70 0.43000 0.41840 0.42337 0.41163 0.46927 0.45847 0.87846
COF 93 0.48000 0.46942 0.40336 0.39122 0.48000 0.46942 0.84450
COF 94 0.48000 0.46942 0.40422 0.39209 0.48485 0.47436 0.84775

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO