Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#06)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.1 MB) Download raw algorithm evaluation table (67.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.45455 0.44349 0.42718 0.41557 0.47273 0.46204 0.94614
KNN 8 0.42424 0.41257 0.42862 0.41703 0.43979 0.42843 0.94762
KNN 9 0.42424 0.41257 0.43016 0.41861 0.43299 0.42149 0.94722
KNNW 4 0.44444 0.43318 0.39530 0.38304 0.45679 0.44578 0.93279
KNNW 5 0.44444 0.43318 0.40101 0.38887 0.47273 0.46204 0.93900
KNNW 13 0.42424 0.41257 0.43163 0.42010 0.44920 0.43803 0.94773
KNNW 16 0.42424 0.41257 0.43239 0.42088 0.44681 0.43559 0.94740
LOF 13 0.39394 0.38165 0.33474 0.32126 0.42795 0.41635 0.86270
LOF 16 0.39394 0.38165 0.33748 0.32405 0.43519 0.42373 0.85685
LOF 27 0.38384 0.37135 0.35775 0.34473 0.39153 0.37920 0.91831
LOF 100 0.32323 0.30951 0.34520 0.33193 0.41975 0.40799 0.95828
SimplifiedLOF 21 0.38384 0.37135 0.34908 0.33588 0.43556 0.42411 0.85429
SimplifiedLOF 34 0.39394 0.38165 0.37983 0.36726 0.41525 0.40340 0.89223
SimplifiedLOF 46 0.41414 0.40226 0.36901 0.35622 0.41414 0.40226 0.91981
SimplifiedLOF 100 0.35354 0.34043 0.36895 0.35615 0.42294 0.41124 0.96018
LoOP 22 0.39394 0.38165 0.31446 0.30056 0.39394 0.38165 0.85801
LoOP 37 0.36364 0.35073 0.35970 0.34672 0.41420 0.40232 0.88886
LoOP 40 0.36364 0.35073 0.36254 0.34962 0.40816 0.39616 0.89332
LoOP 100 0.34343 0.33012 0.35514 0.34206 0.38132 0.36878 0.95156
LDOF 71 0.39394 0.38165 0.40681 0.39478 0.41434 0.40247 0.94866
LDOF 75 0.37374 0.36104 0.40676 0.39473 0.43515 0.42369 0.95080
LDOF 82 0.36364 0.35073 0.41488 0.40301 0.42735 0.41574 0.95291
LDOF 100 0.36364 0.35073 0.41280 0.40090 0.42857 0.41699 0.95915
ODIN 44 0.42424 0.41257 0.28593 0.27145 0.42424 0.41257 0.87216
ODIN 47 0.41181 0.39989 0.29268 0.27834 0.42553 0.41388 0.87473
ODIN 99 0.38384 0.37135 0.35622 0.34317 0.38384 0.37135 0.92997
FastABOD 10 0.41414 0.40226 0.35180 0.33866 0.44693 0.43571 0.84896
FastABOD 33 0.42424 0.41257 0.35655 0.34351 0.43158 0.42005 0.85137
FastABOD 35 0.42424 0.41257 0.35719 0.34416 0.43386 0.42238 0.85146
FastABOD 62 0.42424 0.41257 0.36140 0.34845 0.44211 0.43079 0.85020
KDEOS 35 0.06061 0.04156 0.05321 0.03401 0.11610 0.09818 0.78009
KDEOS 56 0.05051 0.03125 0.06659 0.04767 0.15789 0.14082 0.81332
KDEOS 69 0.04040 0.02095 0.07303 0.05423 0.14205 0.12465 0.81889
KDEOS 100 0.05051 0.03125 0.06900 0.05012 0.13176 0.11415 0.83632
LDF 20 0.41414 0.40226 0.35653 0.34349 0.41414 0.40226 0.91298
LDF 55 0.30303 0.28890 0.37310 0.36039 0.42105 0.40931 0.95643
LDF 64 0.31313 0.29921 0.36112 0.34817 0.41499 0.40312 0.95729
LDF 80 0.35354 0.34043 0.36304 0.35013 0.42236 0.41065 0.95649
INFLO 14 0.39394 0.38165 0.30540 0.29132 0.40000 0.38784 0.80587
INFLO 22 0.36364 0.35073 0.32841 0.31479 0.41975 0.40799 0.78611
INFLO 99 0.33333 0.31982 0.34438 0.33109 0.41638 0.40455 0.90711
INFLO 100 0.33333 0.31982 0.34463 0.33134 0.41924 0.40747 0.90245
COF 29 0.42424 0.41257 0.38119 0.36864 0.45652 0.44550 0.83100
COF 32 0.44444 0.43318 0.38465 0.37217 0.45128 0.44016 0.84447
COF 33 0.43434 0.42288 0.38783 0.37542 0.44571 0.43448 0.84946
COF 100 0.38384 0.37135 0.36650 0.35365 0.40642 0.39438 0.93653

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.3 MB) Download raw algorithm evaluation table (62.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.45000 0.43881 0.37226 0.35948 0.47059 0.45981 0.87989
KNN 14 0.41000 0.39799 0.40423 0.39210 0.44037 0.42898 0.93855
KNN 42 0.35000 0.33677 0.38800 0.37554 0.38710 0.37462 0.94686
KNNW 6 0.44000 0.42860 0.36349 0.35053 0.45503 0.44393 0.87905
KNNW 26 0.41000 0.39799 0.38936 0.37693 0.45581 0.44474 0.93765
KNNW 74 0.37000 0.35718 0.39890 0.38667 0.39326 0.38091 0.94756
LOF 19 0.37000 0.35718 0.31346 0.29948 0.40964 0.39762 0.84979
LOF 27 0.35000 0.33677 0.33284 0.31926 0.39669 0.38441 0.82790
LOF 100 0.32000 0.30616 0.31340 0.29943 0.33858 0.32512 0.94979
SimplifiedLOF 17 0.40000 0.38779 0.30382 0.28965 0.41048 0.39848 0.86772
SimplifiedLOF 20 0.39000 0.37758 0.33143 0.31783 0.42857 0.41694 0.86665
SimplifiedLOF 32 0.38000 0.36738 0.35456 0.34142 0.40336 0.39122 0.84122
SimplifiedLOF 100 0.33000 0.31636 0.33123 0.31762 0.35498 0.34185 0.92348
LoOP 34 0.37000 0.35718 0.27130 0.25647 0.40506 0.39295 0.83315
LoOP 36 0.38000 0.36738 0.28224 0.26763 0.40329 0.39115 0.83470
LoOP 56 0.34000 0.32657 0.29745 0.28315 0.37600 0.36330 0.83833
LoOP 100 0.33000 0.31636 0.28528 0.27073 0.35577 0.34266 0.90567
LDOF 37 0.36000 0.34697 0.29676 0.28244 0.44000 0.42860 0.90840
LDOF 60 0.40000 0.38779 0.31188 0.29788 0.41921 0.40739 0.91102
LDOF 100 0.33000 0.31636 0.31892 0.30506 0.39841 0.38616 0.93354
ODIN 65 0.32692 0.31322 0.22801 0.21230 0.36681 0.35392 0.81845
ODIN 80 0.35000 0.33677 0.23245 0.21682 0.35000 0.33677 0.84901
ODIN 100 0.31600 0.30208 0.24123 0.22579 0.34510 0.33177 0.88165
FastABOD 3 0.34000 0.32657 0.23354 0.21794 0.34667 0.33337 0.85429
FastABOD 15 0.40000 0.38779 0.34709 0.33380 0.40212 0.38995 0.84871
FastABOD 73 0.39000 0.37758 0.35625 0.34314 0.41212 0.40016 0.84604
FastABOD 81 0.39000 0.37758 0.35699 0.34390 0.41212 0.40016 0.84583
KDEOS 37 0.07000 0.05107 0.04835 0.02898 0.10483 0.08661 0.74795
KDEOS 63 0.05000 0.03066 0.06278 0.04371 0.16450 0.14750 0.77561
KDEOS 76 0.06000 0.04087 0.06446 0.04542 0.15798 0.14084 0.77443
KDEOS 98 0.04000 0.02046 0.06122 0.04212 0.14099 0.12351 0.78247
LDF 5 0.37000 0.35718 0.24567 0.23032 0.37615 0.36345 0.78683
LDF 100 0.35000 0.33677 0.39468 0.38236 0.46640 0.45554 0.96046
INFLO 22 0.39000 0.37758 0.28406 0.26949 0.39614 0.38384 0.79697
INFLO 25 0.36000 0.34697 0.29511 0.28077 0.39669 0.38441 0.80601
INFLO 28 0.37000 0.35718 0.30472 0.29057 0.40171 0.38953 0.80016
INFLO 35 0.34000 0.32657 0.30942 0.29537 0.39200 0.37962 0.77579
COF 27 0.45000 0.43881 0.32029 0.30645 0.45455 0.44344 0.78206
COF 28 0.43000 0.41840 0.31979 0.30594 0.45933 0.44833 0.77013
COF 29 0.43000 0.41840 0.32378 0.31002 0.45320 0.44207 0.76164
COF 93 0.36000 0.34697 0.30765 0.29356 0.39506 0.38275 0.84333

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.2 MB) Download raw algorithm evaluation table (66.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.20202 0.18584 0.15393 0.13678 0.20408 0.18794 0.65707
KNN 4 0.19192 0.17554 0.15390 0.13675 0.22500 0.20929 0.65687
KNNW 1 0.19192 0.17554 0.15393 0.13678 0.21519 0.19928 0.62104
KNNW 4 0.20202 0.18584 0.15322 0.13605 0.21053 0.19452 0.64670
KNNW 5 0.20202 0.18584 0.15263 0.13545 0.21739 0.20152 0.64980
KNNW 26 0.14141 0.12401 0.13519 0.11766 0.17476 0.15803 0.65441
LOF 40 0.44444 0.43318 0.41345 0.40156 0.45055 0.43941 0.93407
LOF 41 0.44444 0.43318 0.41441 0.40254 0.44944 0.43828 0.93452
LOF 42 0.43434 0.42288 0.41376 0.40187 0.45198 0.44087 0.93449
LOF 63 0.38384 0.37135 0.37762 0.36500 0.41071 0.39877 0.93676
SimplifiedLOF 42 0.45455 0.44349 0.42095 0.40921 0.46078 0.44985 0.92018
SimplifiedLOF 44 0.45455 0.44349 0.41863 0.40685 0.46995 0.45920 0.92360
SimplifiedLOF 49 0.45455 0.44349 0.42229 0.41057 0.46739 0.45659 0.93258
SimplifiedLOF 68 0.42424 0.41257 0.39610 0.38386 0.43781 0.42641 0.93965
LoOP 59 0.46465 0.45379 0.41415 0.40228 0.46701 0.45620 0.93216
LoOP 67 0.45455 0.44349 0.41409 0.40221 0.48128 0.47077 0.93627
LoOP 69 0.45455 0.44349 0.41511 0.40325 0.47619 0.46557 0.93684
LoOP 83 0.44444 0.43318 0.41007 0.39811 0.44944 0.43828 0.93983
LDOF 72 0.41414 0.40226 0.41385 0.40197 0.43636 0.42494 0.93867
LDOF 79 0.40404 0.39196 0.41769 0.40589 0.44068 0.42934 0.94344
LDOF 81 0.39394 0.38165 0.41537 0.40351 0.44571 0.43448 0.94463
LDOF 99 0.41414 0.40226 0.41121 0.39928 0.43850 0.42712 0.94814
ODIN 98 0.34343 0.33012 0.33836 0.32494 0.36242 0.34949 0.92519
ODIN 99 0.34596 0.33270 0.33915 0.32575 0.36486 0.35199 0.92498
ODIN 100 0.35101 0.33785 0.33971 0.32632 0.36486 0.35199 0.92489
FastABOD 3 0.16162 0.14462 0.13177 0.11417 0.18421 0.16767 0.53336
FastABOD 4 0.16162 0.14462 0.13005 0.11242 0.18571 0.16921 0.53337
FastABOD 6 0.16162 0.14462 0.12893 0.11127 0.19118 0.17478 0.53222
KDEOS 46 0.06061 0.04156 0.04471 0.02534 0.09472 0.07636 0.73865
KDEOS 100 0.06061 0.04156 0.06682 0.04790 0.11917 0.10131 0.80136
LDF 19 0.41414 0.40226 0.41112 0.39918 0.44068 0.42934 0.91022
LDF 20 0.41414 0.40226 0.41702 0.40520 0.43011 0.41855 0.91347
LDF 21 0.42424 0.41257 0.41020 0.39824 0.43299 0.42149 0.91707
LDF 33 0.39394 0.38165 0.40633 0.39429 0.42169 0.40996 0.93461
INFLO 39 0.43434 0.42288 0.38407 0.37159 0.43925 0.42788 0.86649
INFLO 42 0.43434 0.42288 0.38653 0.37409 0.44554 0.43430 0.87438
INFLO 47 0.43434 0.42288 0.38411 0.37162 0.45652 0.44550 0.88221
INFLO 67 0.40404 0.39196 0.37692 0.36428 0.42795 0.41635 0.92510
COF 35 0.44444 0.43318 0.39314 0.38084 0.45833 0.44735 0.88078
COF 43 0.43434 0.42288 0.40427 0.39219 0.46626 0.45544 0.90109
COF 51 0.41414 0.40226 0.40723 0.39521 0.43850 0.42712 0.91286

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (63.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.14000 0.12250 0.07567 0.05685 0.14286 0.12541 0.65260
KNN 5 0.10000 0.08168 0.06963 0.05069 0.11354 0.09549 0.65432
KNNW 3 0.12000 0.10209 0.07434 0.05550 0.13201 0.11435 0.63729
KNNW 4 0.13000 0.11229 0.07425 0.05540 0.13265 0.11500 0.64313
KNNW 10 0.10000 0.08168 0.06935 0.05041 0.11722 0.09925 0.65029
LOF 35 0.46000 0.44901 0.41150 0.39952 0.47368 0.46297 0.89306
LOF 37 0.46000 0.44901 0.42001 0.40820 0.50000 0.48982 0.89667
LOF 40 0.46000 0.44901 0.41796 0.40611 0.51397 0.50407 0.89570
LOF 98 0.41000 0.39799 0.33240 0.31881 0.45022 0.43903 0.92581
SimplifiedLOF 37 0.46000 0.44901 0.42785 0.41620 0.48402 0.47352 0.90322
SimplifiedLOF 38 0.47000 0.45921 0.42406 0.41234 0.48182 0.47127 0.90324
SimplifiedLOF 45 0.47000 0.45921 0.41548 0.40358 0.48677 0.47633 0.90851
SimplifiedLOF 100 0.40000 0.38779 0.34447 0.33113 0.46324 0.45231 0.92469
LoOP 58 0.47000 0.45921 0.39746 0.38519 0.49180 0.48146 0.91369
LoOP 67 0.47000 0.45921 0.40583 0.39374 0.48913 0.47873 0.91817
LoOP 100 0.44000 0.42860 0.39483 0.38252 0.46341 0.45249 0.93071
LDOF 38 0.44000 0.42860 0.34690 0.33361 0.44670 0.43544 0.91697
LDOF 55 0.40000 0.38779 0.35762 0.34454 0.46154 0.45058 0.92302
LDOF 84 0.42000 0.40819 0.37749 0.36482 0.44915 0.43794 0.94093
LDOF 100 0.44000 0.42860 0.37708 0.36440 0.44086 0.42948 0.94715
ODIN 61 0.39000 0.37758 0.31460 0.30065 0.40426 0.39213 0.88245
ODIN 74 0.37727 0.36460 0.33878 0.32532 0.43373 0.42221 0.90041
ODIN 100 0.39000 0.37758 0.37200 0.35922 0.43137 0.41980 0.92255
FastABOD 3 0.09000 0.07148 0.04150 0.02199 0.10738 0.08921 0.52472
FastABOD 5 0.08000 0.06127 0.05252 0.03324 0.11429 0.09626 0.52711
KDEOS 11 0.09000 0.07148 0.05154 0.03223 0.09184 0.07335 0.66082
KDEOS 99 0.04000 0.02046 0.05550 0.03628 0.11016 0.09205 0.77131
KDEOS 100 0.04000 0.02046 0.05485 0.03561 0.11152 0.09344 0.77260
LDF 11 0.43000 0.41840 0.39126 0.37887 0.44560 0.43431 0.91838
LDF 14 0.44000 0.42860 0.42242 0.41067 0.48750 0.47707 0.91036
LDF 17 0.45000 0.43881 0.43646 0.42499 0.47826 0.46764 0.89888
LDF 25 0.46000 0.44901 0.42078 0.40899 0.46875 0.45794 0.88233
INFLO 38 0.47000 0.45921 0.39753 0.38527 0.47619 0.46553 0.83627
INFLO 45 0.46000 0.44901 0.39814 0.38589 0.48936 0.47897 0.83150
INFLO 73 0.43000 0.41840 0.37839 0.36574 0.46512 0.45423 0.88851
COF 17 0.32000 0.30616 0.31474 0.30079 0.35165 0.33845 0.89442
COF 37 0.49000 0.47962 0.41219 0.40023 0.50000 0.48982 0.85694
COF 45 0.46000 0.44901 0.43510 0.42360 0.50867 0.49867 0.87352
COF 91 0.46000 0.44901 0.38677 0.37429 0.52814 0.51853 0.84819

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO