Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#03)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.3 MB) Download raw algorithm evaluation table (69.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.34343 0.33012 0.28932 0.27492 0.40164 0.38951 0.92599
KNN 8 0.32323 0.30951 0.29430 0.27999 0.38735 0.37493 0.93341
KNN 10 0.35354 0.34043 0.31858 0.30477 0.37736 0.36473 0.93241
KNN 13 0.33333 0.31982 0.32002 0.30623 0.36749 0.35467 0.93207
KNNW 5 0.35354 0.34043 0.27200 0.25724 0.37903 0.36644 0.91437
KNNW 15 0.32323 0.30951 0.29251 0.27817 0.40000 0.38784 0.93241
KNNW 16 0.33333 0.31982 0.29388 0.27957 0.39526 0.38300 0.93252
KNNW 29 0.34343 0.33012 0.30603 0.29196 0.37321 0.36050 0.93149
LOF 31 0.32323 0.30951 0.26931 0.25450 0.33846 0.32505 0.92452
LOF 43 0.29293 0.27859 0.28415 0.26963 0.31837 0.30455 0.94064
LOF 94 0.27273 0.25798 0.26058 0.24559 0.38854 0.37614 0.95641
LOF 100 0.27273 0.25798 0.26328 0.24834 0.38662 0.37418 0.95691
SimplifiedLOF 59 0.32323 0.30951 0.29320 0.27887 0.33702 0.32357 0.94438
SimplifiedLOF 70 0.29293 0.27859 0.29807 0.28384 0.35542 0.34235 0.94881
SimplifiedLOF 100 0.29293 0.27859 0.28992 0.27552 0.40491 0.39284 0.95638
LoOP 36 0.31313 0.29921 0.21903 0.20319 0.31313 0.29921 0.88810
LoOP 97 0.28283 0.26829 0.26309 0.24815 0.34568 0.33241 0.94740
LoOP 100 0.28283 0.26829 0.26170 0.24673 0.34884 0.33564 0.94875
LDOF 65 0.35354 0.34043 0.29465 0.28035 0.37460 0.36192 0.94933
LDOF 100 0.32323 0.30951 0.32039 0.30661 0.41472 0.40285 0.95817
ODIN 83 0.28171 0.26714 0.20703 0.19095 0.33333 0.31982 0.90874
ODIN 97 0.30909 0.29508 0.21947 0.20365 0.32579 0.31212 0.92050
ODIN 100 0.30415 0.29004 0.22878 0.21315 0.32331 0.30959 0.92417
FastABOD 7 0.28283 0.26829 0.20846 0.19241 0.31111 0.29714 0.84470
FastABOD 24 0.31313 0.29921 0.21303 0.19708 0.33498 0.32149 0.84202
FastABOD 29 0.32323 0.30951 0.21600 0.20011 0.33000 0.31642 0.84166
FastABOD 100 0.32323 0.30951 0.22462 0.20890 0.32653 0.31288 0.83953
KDEOS 46 0.05051 0.03125 0.04646 0.02713 0.10051 0.08227 0.77462
KDEOS 62 0.04040 0.02095 0.04813 0.02883 0.10950 0.09144 0.79237
KDEOS 100 0.03030 0.01064 0.05260 0.03339 0.10375 0.08558 0.81452
LDF 24 0.32323 0.30951 0.28216 0.26760 0.34746 0.33423 0.93076
LDF 48 0.27273 0.25798 0.29481 0.28051 0.37245 0.35973 0.95182
LDF 62 0.25253 0.23737 0.27971 0.26511 0.38057 0.36801 0.95452
LDF 100 0.30303 0.28890 0.28756 0.27312 0.39355 0.38125 0.95156
INFLO 32 0.32323 0.30951 0.23043 0.21483 0.32323 0.30951 0.78671
INFLO 80 0.30303 0.28890 0.24466 0.22934 0.36364 0.35073 0.91695
INFLO 100 0.30303 0.28890 0.25657 0.24149 0.38671 0.37427 0.90440
COF 73 0.35354 0.34043 0.26312 0.24818 0.37190 0.35917 0.93446
COF 83 0.33333 0.31982 0.26609 0.25121 0.38053 0.36797 0.93822
COF 92 0.35354 0.34043 0.27034 0.25555 0.36000 0.34702 0.94025
COF 99 0.35354 0.34043 0.26669 0.25182 0.36000 0.34702 0.94205

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.3 MB) Download raw algorithm evaluation table (62.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 9 0.40000 0.38779 0.35805 0.34498 0.41176 0.39979 0.90702
KNN 10 0.41000 0.39799 0.35688 0.34379 0.41860 0.40677 0.90597
KNN 31 0.36000 0.34697 0.34956 0.33632 0.36364 0.35068 0.93397
KNNW 21 0.38000 0.36738 0.35646 0.34337 0.41706 0.40520 0.90401
KNNW 23 0.40000 0.38779 0.35567 0.34255 0.41509 0.40319 0.91116
KNNW 76 0.35000 0.33677 0.35936 0.34632 0.38400 0.37146 0.93528
KNNW 83 0.36000 0.34697 0.36023 0.34720 0.38764 0.37518 0.93514
LOF 15 0.38000 0.36738 0.30614 0.29201 0.39858 0.38634 0.83247
LOF 16 0.37000 0.35718 0.29895 0.28468 0.41509 0.40319 0.83631
LOF 100 0.31000 0.29596 0.31203 0.29802 0.34839 0.33512 0.95592
SimplifiedLOF 22 0.42000 0.40819 0.33375 0.32019 0.43192 0.42036 0.82687
SimplifiedLOF 34 0.39000 0.37758 0.33996 0.32652 0.39827 0.38602 0.82396
SimplifiedLOF 100 0.33000 0.31636 0.31778 0.30389 0.35798 0.34491 0.93734
LoOP 33 0.37000 0.35718 0.28492 0.27036 0.38596 0.37347 0.80964
LoOP 39 0.36000 0.34697 0.28890 0.27443 0.39316 0.38081 0.81312
LoOP 97 0.34000 0.32657 0.31212 0.29812 0.35521 0.34209 0.92079
LoOP 100 0.33000 0.31636 0.31201 0.29801 0.35878 0.34573 0.92470
LDOF 44 0.37000 0.35718 0.30824 0.29416 0.39823 0.38598 0.89718
LDOF 73 0.37000 0.35718 0.32735 0.31366 0.42063 0.40884 0.91644
LDOF 89 0.35000 0.33677 0.34393 0.33058 0.40000 0.38779 0.93201
LDOF 100 0.36000 0.34697 0.34285 0.32948 0.39676 0.38448 0.94059
ODIN 60 0.38385 0.37130 0.25659 0.24146 0.38974 0.37732 0.83099
ODIN 100 0.36000 0.34697 0.30031 0.28607 0.36437 0.35143 0.90388
FastABOD 3 0.33000 0.31636 0.20517 0.18899 0.34146 0.32806 0.80519
FastABOD 6 0.31000 0.29596 0.27226 0.25744 0.34437 0.33103 0.77628
FastABOD 9 0.31000 0.29596 0.27802 0.26333 0.33557 0.32205 0.77828
KDEOS 68 0.05000 0.03066 0.05511 0.03587 0.13614 0.11856 0.74182
KDEOS 83 0.04000 0.02046 0.05809 0.03892 0.14060 0.12311 0.76048
KDEOS 85 0.03000 0.01026 0.05797 0.03880 0.14359 0.12616 0.76380
KDEOS 100 0.04000 0.02046 0.05600 0.03678 0.13208 0.11441 0.77459
LDF 15 0.41000 0.39799 0.34041 0.32699 0.42623 0.41455 0.84115
LDF 98 0.31000 0.29596 0.36588 0.35297 0.45865 0.44763 0.95845
LDF 99 0.31000 0.29596 0.36676 0.35387 0.45522 0.44414 0.95856
LDF 100 0.31000 0.29596 0.36736 0.35448 0.45588 0.44481 0.95851
INFLO 24 0.37000 0.35718 0.27911 0.26444 0.37000 0.35718 0.78531
INFLO 43 0.35000 0.33677 0.29516 0.28081 0.38333 0.37078 0.76412
INFLO 47 0.35000 0.33677 0.29781 0.28351 0.36800 0.35514 0.77504
INFLO 99 0.34000 0.32657 0.27308 0.25828 0.34826 0.33499 0.81366
COF 15 0.33000 0.31636 0.28185 0.26724 0.36444 0.35151 0.81799
COF 23 0.46000 0.44901 0.33054 0.31691 0.48421 0.47371 0.80460
COF 32 0.43000 0.41840 0.36893 0.35608 0.44681 0.43555 0.77790

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.1 MB) Download raw algorithm evaluation table (67.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.13131 0.11370 0.06185 0.04283 0.13472 0.11717 0.56414
KNN 3 0.13131 0.11370 0.06250 0.04350 0.14218 0.12479 0.57467
KNN 7 0.08081 0.06217 0.06212 0.04311 0.15079 0.13358 0.58005
KNN 26 0.06061 0.04156 0.05249 0.03328 0.09800 0.07971 0.58264
KNNW 2 0.12121 0.10340 0.05949 0.04042 0.12222 0.10443 0.54382
KNNW 4 0.12121 0.10340 0.06092 0.04188 0.12766 0.10997 0.56175
KNNW 6 0.12121 0.10340 0.06128 0.04224 0.12500 0.10726 0.56937
KNNW 49 0.06061 0.04156 0.05291 0.03370 0.09843 0.08016 0.58075
LOF 43 0.38384 0.37135 0.32033 0.30655 0.38776 0.37534 0.91812
LOF 44 0.38384 0.37135 0.32143 0.30767 0.39560 0.38335 0.91970
LOF 65 0.34343 0.33012 0.30854 0.29452 0.38168 0.36914 0.92708
LOF 71 0.34343 0.33012 0.29913 0.28492 0.40310 0.39100 0.92653
SimplifiedLOF 46 0.40404 0.39196 0.32618 0.31252 0.41026 0.39830 0.89759
SimplifiedLOF 53 0.41414 0.40226 0.31764 0.30381 0.41414 0.40226 0.91960
SimplifiedLOF 78 0.34343 0.33012 0.30503 0.29094 0.38911 0.37672 0.93295
SimplifiedLOF 86 0.34343 0.33012 0.29466 0.28036 0.41593 0.40409 0.93240
LoOP 60 0.39394 0.38165 0.31712 0.30327 0.43575 0.42431 0.91677
LoOP 62 0.39394 0.38165 0.31085 0.29688 0.44068 0.42934 0.91765
LoOP 73 0.41414 0.40226 0.30974 0.29575 0.43182 0.42030 0.92524
LoOP 81 0.40404 0.39196 0.31224 0.29830 0.42162 0.40990 0.92735
LDOF 70 0.39394 0.38165 0.30165 0.28749 0.39801 0.38580 0.91710
LDOF 87 0.38384 0.37135 0.30980 0.29581 0.40426 0.39218 0.93805
LDOF 89 0.38384 0.37135 0.31196 0.29801 0.40426 0.39218 0.93941
LDOF 100 0.38384 0.37135 0.30992 0.29593 0.39378 0.38149 0.94348
ODIN 63 0.33333 0.31982 0.24721 0.23195 0.35503 0.34195 0.88235
ODIN 78 0.32323 0.30951 0.27730 0.26265 0.37500 0.36233 0.89985
ODIN 91 0.33242 0.31888 0.28510 0.27061 0.36943 0.35664 0.90967
ODIN 100 0.32966 0.31607 0.28191 0.26735 0.35065 0.33748 0.91415
FastABOD 3 0.06061 0.04156 0.04439 0.02501 0.06977 0.05091 0.44226
FastABOD 4 0.06061 0.04156 0.04490 0.02553 0.06977 0.05091 0.44298
FastABOD 55 0.06061 0.04156 0.04503 0.02567 0.07519 0.05644 0.43892
FastABOD 87 0.06061 0.04156 0.04510 0.02574 0.07519 0.05644 0.43913
KDEOS 11 0.07071 0.05187 0.03847 0.01897 0.08191 0.06330 0.67058
KDEOS 100 0.07071 0.05187 0.05870 0.03962 0.12468 0.10694 0.80181
LDF 11 0.37374 0.36104 0.30728 0.29324 0.37864 0.36604 0.84405
LDF 17 0.36364 0.35073 0.33492 0.32143 0.39535 0.38309 0.86757
LDF 50 0.32323 0.30951 0.30755 0.29351 0.40727 0.39526 0.91219
LDF 72 0.35354 0.34043 0.27057 0.25578 0.38356 0.37106 0.91353
INFLO 36 0.37374 0.36104 0.30114 0.28697 0.38835 0.37595 0.83084
INFLO 47 0.40404 0.39196 0.29694 0.28269 0.40609 0.39405 0.83374
INFLO 55 0.40404 0.39196 0.28432 0.26981 0.41451 0.40264 0.83008
INFLO 67 0.37374 0.36104 0.29375 0.27943 0.38710 0.37467 0.89180
COF 46 0.37374 0.36104 0.29690 0.28265 0.39241 0.38009 0.86830
COF 48 0.37374 0.36104 0.30802 0.29399 0.41111 0.39917 0.87517
COF 50 0.37374 0.36104 0.30842 0.29440 0.39362 0.38132 0.87492
COF 58 0.35354 0.34043 0.29016 0.27577 0.36257 0.34965 0.87894

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (61.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.15000 0.13270 0.10444 0.08621 0.18405 0.16744 0.58957
KNN 2 0.16000 0.14290 0.10192 0.08364 0.17877 0.16206 0.60600
KNN 9 0.15000 0.13270 0.09531 0.07690 0.15917 0.14206 0.61229
KNNW 3 0.16000 0.14290 0.10178 0.08350 0.18286 0.16622 0.59506
KNNW 4 0.16000 0.14290 0.10456 0.08633 0.18391 0.16730 0.60032
KNNW 17 0.15000 0.13270 0.09503 0.07661 0.15228 0.13503 0.60717
LOF 42 0.42000 0.40819 0.39245 0.38009 0.45055 0.43937 0.91173
LOF 45 0.45000 0.43881 0.38937 0.37694 0.45000 0.43881 0.90821
LOF 46 0.44000 0.42860 0.38894 0.37650 0.46486 0.45397 0.90916
SimplifiedLOF 36 0.46000 0.44901 0.39813 0.38588 0.46465 0.45375 0.90240
SimplifiedLOF 39 0.46000 0.44901 0.40314 0.39099 0.46701 0.45616 0.90451
SimplifiedLOF 43 0.44000 0.42860 0.40089 0.38870 0.47191 0.46116 0.91155
SimplifiedLOF 56 0.44000 0.42860 0.39350 0.38116 0.45320 0.44207 0.92545
LoOP 52 0.44000 0.42860 0.39272 0.38036 0.47727 0.46663 0.91457
LoOP 66 0.45000 0.43881 0.39345 0.38110 0.46512 0.45423 0.92173
LoOP 85 0.44000 0.42860 0.40403 0.39190 0.45411 0.44300 0.92378
LoOP 99 0.43000 0.41840 0.38690 0.37443 0.45415 0.44304 0.92477
LDOF 22 0.47000 0.45921 0.35781 0.34474 0.47805 0.46742 0.89118
LDOF 24 0.47000 0.45921 0.36418 0.35124 0.47959 0.46900 0.89450
LDOF 49 0.43000 0.41840 0.39543 0.38313 0.46452 0.45362 0.91186
LDOF 99 0.42000 0.40819 0.38750 0.37503 0.43023 0.41864 0.94762
ODIN 59 0.37000 0.35718 0.28606 0.27153 0.37374 0.36099 0.87923
ODIN 98 0.34923 0.33598 0.33238 0.31880 0.40000 0.38779 0.90635
ODIN 99 0.34941 0.33617 0.33189 0.31830 0.40252 0.39035 0.90647
ODIN 100 0.35200 0.33881 0.33098 0.31736 0.39241 0.38004 0.90695
FastABOD 3 0.13000 0.11229 0.06429 0.04524 0.15172 0.13446 0.50290
FastABOD 8 0.12000 0.10209 0.08153 0.06284 0.14706 0.12970 0.48720
KDEOS 67 0.07000 0.05107 0.04684 0.02744 0.08786 0.06929 0.75229
KDEOS 94 0.07000 0.05107 0.05669 0.03749 0.11048 0.09237 0.78001
KDEOS 100 0.07000 0.05107 0.06034 0.04122 0.10942 0.09130 0.78543
LDF 27 0.38000 0.36738 0.37690 0.36422 0.40000 0.38779 0.90887
LDF 38 0.39000 0.37758 0.38196 0.36938 0.41975 0.40794 0.89835
LDF 48 0.44000 0.42860 0.36802 0.35516 0.44221 0.43086 0.88239
INFLO 37 0.44000 0.42860 0.38809 0.37563 0.46061 0.44963 0.83699
INFLO 53 0.45000 0.43881 0.37393 0.36118 0.47778 0.46715 0.85170
INFLO 55 0.46000 0.44901 0.37600 0.36330 0.46486 0.45397 0.85767
INFLO 67 0.44000 0.42860 0.37399 0.36124 0.46602 0.45515 0.87768
COF 16 0.40000 0.38779 0.34626 0.33295 0.42553 0.41384 0.86435
COF 56 0.43000 0.41840 0.37268 0.35991 0.44792 0.43668 0.85200
COF 87 0.45000 0.43881 0.33532 0.32179 0.45098 0.43981 0.81159
COF 92 0.45000 0.43881 0.33970 0.32627 0.46535 0.45446 0.80426

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO