Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#09)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.2 MB) Download raw algorithm evaluation table (66.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.46465 0.45379 0.42193 0.41021 0.50867 0.49871 0.91403
KNN 10 0.44444 0.43318 0.42833 0.41674 0.47059 0.45985 0.91638
KNN 16 0.45455 0.44349 0.43309 0.42160 0.46073 0.44980 0.91452
KNNW 5 0.46465 0.45379 0.40553 0.39348 0.51685 0.50706 0.90629
KNNW 24 0.44444 0.43318 0.42878 0.41720 0.47312 0.46244 0.91589
KNNW 25 0.44444 0.43318 0.42880 0.41722 0.47059 0.45985 0.91574
LOF 21 0.45455 0.44349 0.37147 0.35873 0.47321 0.46253 0.90140
LOF 23 0.43434 0.42288 0.38198 0.36945 0.47577 0.46514 0.90984
LOF 27 0.44444 0.43318 0.38920 0.37681 0.46018 0.44923 0.91523
LOF 96 0.38384 0.37135 0.35611 0.34305 0.44813 0.43694 0.94809
SimplifiedLOF 29 0.47475 0.46410 0.39733 0.38511 0.48454 0.47409 0.88978
SimplifiedLOF 47 0.43434 0.42288 0.40063 0.38848 0.47059 0.45985 0.92276
SimplifiedLOF 100 0.43434 0.42288 0.38991 0.37754 0.45098 0.43985 0.95035
LoOP 40 0.44444 0.43318 0.34946 0.33627 0.45714 0.44614 0.90600
LoOP 55 0.42424 0.41257 0.35320 0.34008 0.45902 0.44805 0.92655
LoOP 77 0.43434 0.42288 0.36514 0.35227 0.44565 0.43441 0.93907
LoOP 99 0.42424 0.41257 0.35735 0.34432 0.43925 0.42788 0.94701
LDOF 84 0.43434 0.42288 0.40348 0.39139 0.45977 0.44882 0.95605
LDOF 95 0.45455 0.44349 0.41316 0.40126 0.45455 0.44349 0.95817
LDOF 99 0.44444 0.43318 0.41413 0.40225 0.44693 0.43571 0.95869
ODIN 89 0.41751 0.40570 0.34004 0.32666 0.44131 0.42999 0.91698
ODIN 97 0.42873 0.41715 0.34450 0.33121 0.43158 0.42005 0.92154
ODIN 100 0.42172 0.40999 0.34977 0.33659 0.42922 0.41765 0.92343
FastABOD 3 0.37374 0.36104 0.32100 0.30724 0.39216 0.37983 0.82653
FastABOD 11 0.45455 0.44349 0.36612 0.35327 0.45556 0.44452 0.81908
FastABOD 19 0.44444 0.43318 0.36683 0.35399 0.46591 0.45508 0.81840
FastABOD 53 0.45455 0.44349 0.36998 0.35721 0.45918 0.44822 0.81825
KDEOS 69 0.02020 0.00034 0.05982 0.04076 0.15493 0.13780 0.80940
KDEOS 96 0.04040 0.02095 0.06083 0.04179 0.12780 0.11011 0.82238
KDEOS 97 0.05051 0.03125 0.06079 0.04175 0.12632 0.10860 0.82235
KDEOS 100 0.05051 0.03125 0.06074 0.04170 0.12944 0.11179 0.82291
LDF 18 0.46465 0.45379 0.38483 0.37236 0.47423 0.46357 0.90652
LDF 22 0.46465 0.45379 0.39717 0.38495 0.46875 0.45798 0.91964
LDF 64 0.36364 0.35073 0.36387 0.35097 0.44280 0.43151 0.94391
INFLO 22 0.43434 0.42288 0.33699 0.32354 0.44693 0.43571 0.80005
INFLO 29 0.42424 0.41257 0.35084 0.33768 0.46667 0.45585 0.80150
INFLO 92 0.38384 0.37135 0.35246 0.33933 0.42202 0.41030 0.90506
INFLO 99 0.38384 0.37135 0.35835 0.34534 0.43165 0.42013 0.90103
COF 76 0.48485 0.47440 0.42073 0.40898 0.49038 0.48005 0.91063
COF 98 0.46465 0.45379 0.43180 0.42028 0.49761 0.48742 0.91642
COF 99 0.46465 0.45379 0.43204 0.42052 0.49102 0.48070 0.91699
COF 100 0.47475 0.46410 0.43181 0.42029 0.48619 0.47577 0.91741

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.3 MB) Download raw algorithm evaluation table (62.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.41000 0.39799 0.33192 0.31832 0.43114 0.41956 0.85242
KNN 7 0.40000 0.38779 0.35877 0.34572 0.42324 0.41150 0.91984
KNN 38 0.35000 0.33677 0.32469 0.31094 0.36364 0.35068 0.93298
KNNW 4 0.41000 0.39799 0.34054 0.32712 0.43529 0.42380 0.87847
KNNW 6 0.41000 0.39799 0.35126 0.33805 0.43931 0.42789 0.89578
KNNW 18 0.40000 0.38779 0.35564 0.34252 0.40310 0.39095 0.92554
KNNW 63 0.36000 0.34697 0.34358 0.33022 0.37288 0.36012 0.93411
LOF 15 0.41000 0.39799 0.32020 0.30637 0.41000 0.39799 0.81171
LOF 22 0.39000 0.37758 0.31435 0.30039 0.41053 0.39853 0.81917
LOF 100 0.31000 0.29596 0.27714 0.26242 0.33645 0.32294 0.95286
SimplifiedLOF 22 0.42000 0.40819 0.33463 0.32108 0.42857 0.41694 0.82692
SimplifiedLOF 25 0.43000 0.41840 0.33081 0.31719 0.43000 0.41840 0.83906
SimplifiedLOF 28 0.42000 0.40819 0.33227 0.31868 0.43386 0.42234 0.83340
SimplifiedLOF 100 0.37000 0.35718 0.28979 0.27533 0.37186 0.35907 0.93607
LoOP 27 0.40000 0.38779 0.31164 0.29763 0.40394 0.39181 0.82297
LoOP 30 0.40000 0.38779 0.32339 0.30962 0.40625 0.39416 0.82258
LoOP 35 0.40000 0.38779 0.31770 0.30381 0.42553 0.41384 0.81706
LoOP 100 0.35000 0.33677 0.28037 0.26572 0.38278 0.37021 0.92259
LDOF 39 0.41000 0.39799 0.36377 0.35082 0.42328 0.41154 0.91346
LDOF 40 0.40000 0.38779 0.36504 0.35212 0.42781 0.41616 0.91454
LDOF 49 0.39000 0.37758 0.37048 0.35766 0.41573 0.40384 0.90977
LDOF 100 0.38000 0.36738 0.35879 0.34574 0.40376 0.39162 0.94096
ODIN 50 0.37000 0.35718 0.23824 0.22273 0.37186 0.35907 0.81698
ODIN 89 0.36200 0.34901 0.29982 0.28557 0.39080 0.37840 0.88502
ODIN 97 0.36200 0.34901 0.30443 0.29027 0.37931 0.36668 0.89367
ODIN 100 0.36667 0.35378 0.30360 0.28943 0.37500 0.36228 0.89737
FastABOD 6 0.44000 0.42860 0.33188 0.31828 0.44670 0.43544 0.87169
FastABOD 16 0.44000 0.42860 0.33622 0.32271 0.46316 0.45223 0.87602
FastABOD 29 0.44000 0.42860 0.34363 0.33027 0.46073 0.44976 0.87796
FastABOD 62 0.44000 0.42860 0.34483 0.33150 0.45989 0.44890 0.87683
KDEOS 66 0.06000 0.04087 0.06120 0.04209 0.15228 0.13503 0.75926
KDEOS 68 0.06000 0.04087 0.06205 0.04295 0.15065 0.13336 0.76036
KDEOS 80 0.08000 0.06127 0.05953 0.04039 0.14706 0.12970 0.76812
KDEOS 100 0.04000 0.02046 0.05786 0.03868 0.12438 0.10656 0.78460
LDF 7 0.42000 0.40819 0.30774 0.29365 0.44444 0.43314 0.86351
LDF 13 0.40000 0.38779 0.33267 0.31908 0.40758 0.39552 0.82275
LDF 80 0.28000 0.26535 0.29562 0.28128 0.39871 0.38648 0.95584
INFLO 15 0.40000 0.38779 0.29988 0.28563 0.41451 0.40259 0.75213
INFLO 22 0.38000 0.36738 0.30485 0.29070 0.42529 0.41359 0.75963
INFLO 24 0.38000 0.36738 0.30193 0.28772 0.42775 0.41610 0.76368
INFLO 100 0.34000 0.32657 0.26145 0.24642 0.35714 0.34406 0.83223
COF 20 0.42000 0.40819 0.33069 0.31707 0.44318 0.43185 0.78544
COF 27 0.42000 0.40819 0.34766 0.33438 0.45304 0.44191 0.75310
COF 28 0.42000 0.40819 0.34547 0.33214 0.46328 0.45235 0.74352
COF 98 0.36000 0.34697 0.28212 0.26750 0.37500 0.36228 0.88902

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.2 MB) Download raw algorithm evaluation table (66.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.20202 0.18584 0.13462 0.11708 0.22619 0.21050 0.63259
KNN 2 0.21212 0.19615 0.14379 0.12643 0.22222 0.20645 0.64673
KNN 3 0.21212 0.19615 0.14474 0.12740 0.22111 0.20531 0.64794
KNN 15 0.17172 0.15492 0.13158 0.11398 0.17172 0.15492 0.65928
KNNW 1 0.21212 0.19615 0.13467 0.11713 0.24000 0.22459 0.62540
KNNW 6 0.21212 0.19615 0.14126 0.12385 0.22105 0.20526 0.64741
KNNW 30 0.16162 0.14462 0.13150 0.11389 0.17467 0.15794 0.65621
LOF 39 0.44444 0.43318 0.39538 0.38312 0.44898 0.43781 0.94141
LOF 49 0.43434 0.42288 0.39767 0.38546 0.46739 0.45659 0.94484
LOF 50 0.43434 0.42288 0.39878 0.38659 0.46486 0.45402 0.94508
SimplifiedLOF 44 0.47475 0.46410 0.40120 0.38906 0.47761 0.46702 0.92882
SimplifiedLOF 49 0.46465 0.45379 0.40924 0.39727 0.49462 0.48438 0.93811
SimplifiedLOF 53 0.46465 0.45379 0.41264 0.40073 0.48649 0.47608 0.94280
SimplifiedLOF 68 0.42424 0.41257 0.40090 0.38875 0.44776 0.43656 0.94737
LoOP 62 0.46465 0.45379 0.39901 0.38682 0.46486 0.45402 0.94078
LoOP 71 0.45455 0.44349 0.40420 0.39212 0.48913 0.47877 0.94348
LoOP 77 0.46465 0.45379 0.41573 0.40389 0.48387 0.47341 0.94430
LoOP 80 0.46465 0.45379 0.41313 0.40123 0.47668 0.46607 0.94472
LDOF 70 0.45455 0.44349 0.40636 0.39433 0.45918 0.44822 0.93591
LDOF 86 0.45455 0.44349 0.42206 0.41034 0.48315 0.47267 0.94676
LDOF 97 0.43434 0.42288 0.42544 0.41380 0.46512 0.45427 0.94970
LDOF 99 0.42424 0.41257 0.42565 0.41401 0.45882 0.44785 0.94970
ODIN 95 0.31313 0.29921 0.30079 0.28662 0.34545 0.33218 0.92198
ODIN 96 0.31313 0.29921 0.30111 0.28694 0.34862 0.33542 0.92165
ODIN 100 0.33517 0.32169 0.30795 0.29392 0.34742 0.33419 0.92137
FastABOD 3 0.17172 0.15492 0.11154 0.09353 0.18085 0.16424 0.54259
FastABOD 4 0.17172 0.15492 0.11211 0.09411 0.18605 0.16954 0.54139
FastABOD 11 0.16162 0.14462 0.11500 0.09706 0.18045 0.16384 0.53907
KDEOS 38 0.08081 0.06217 0.04242 0.02301 0.08734 0.06883 0.71516
KDEOS 100 0.06061 0.04156 0.05578 0.03663 0.12343 0.10565 0.80481
LDF 32 0.42424 0.41257 0.40784 0.39583 0.44681 0.43559 0.93954
LDF 33 0.42424 0.41257 0.40447 0.39240 0.45455 0.44349 0.94007
INFLO 48 0.46465 0.45379 0.37667 0.36403 0.46809 0.45730 0.89078
INFLO 50 0.46465 0.45379 0.37878 0.36618 0.47917 0.46861 0.89847
INFLO 64 0.41414 0.40226 0.38321 0.37070 0.44444 0.43318 0.92849
INFLO 68 0.41414 0.40226 0.38258 0.37006 0.44860 0.43742 0.92855
COF 40 0.42424 0.41257 0.39633 0.38409 0.48810 0.47772 0.90224
COF 54 0.44444 0.43318 0.41208 0.40016 0.46667 0.45585 0.92506
COF 56 0.45455 0.44349 0.40991 0.39794 0.46486 0.45402 0.92642

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (61.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.17000 0.15311 0.12631 0.10853 0.18537 0.16878 0.58511
KNN 4 0.17000 0.15311 0.12834 0.11060 0.20000 0.18372 0.59042
KNN 10 0.14000 0.12250 0.11750 0.09954 0.15517 0.13798 0.59490
KNNW 1 0.15000 0.13270 0.11786 0.09990 0.17937 0.16267 0.54559
KNNW 4 0.16000 0.14290 0.12366 0.10583 0.17021 0.15332 0.57473
KNNW 5 0.16000 0.14290 0.12721 0.10944 0.17391 0.15710 0.57894
KNNW 17 0.15000 0.13270 0.12005 0.10214 0.15517 0.13798 0.58999
LOF 28 0.46000 0.44901 0.37937 0.36674 0.46231 0.45137 0.91011
LOF 37 0.46000 0.44901 0.39493 0.38261 0.46995 0.45916 0.91918
LOF 41 0.46000 0.44901 0.39917 0.38694 0.46316 0.45223 0.91389
LOF 47 0.46000 0.44901 0.39257 0.38021 0.47179 0.46104 0.91802
SimplifiedLOF 48 0.50000 0.48982 0.40695 0.39488 0.50000 0.48982 0.92082
SimplifiedLOF 53 0.47000 0.45921 0.40930 0.39728 0.48168 0.47113 0.92163
SimplifiedLOF 56 0.46000 0.44901 0.40713 0.39506 0.46766 0.45683 0.92253
LoOP 68 0.49000 0.47962 0.40874 0.39670 0.49495 0.48467 0.91530
LoOP 71 0.48000 0.46942 0.41015 0.39814 0.48515 0.47467 0.91595
LoOP 95 0.42000 0.40819 0.40145 0.38927 0.44828 0.43705 0.91806
LDOF 66 0.47000 0.45921 0.40941 0.39739 0.48168 0.47113 0.92357
LDOF 75 0.44000 0.42860 0.41756 0.40570 0.49123 0.48087 0.92855
LDOF 87 0.46000 0.44901 0.42202 0.41026 0.47727 0.46663 0.93319
LDOF 100 0.45000 0.43881 0.41370 0.40177 0.46377 0.45285 0.93792
ODIN 93 0.38438 0.37184 0.32886 0.31520 0.41860 0.40677 0.90834
ODIN 100 0.37545 0.36274 0.33305 0.31948 0.40491 0.39280 0.91322
FastABOD 3 0.12000 0.10209 0.06830 0.04934 0.15493 0.13773 0.47095
FastABOD 4 0.12000 0.10209 0.08517 0.06655 0.15873 0.14161 0.47526
FastABOD 62 0.12000 0.10209 0.10688 0.08870 0.15714 0.13999 0.45850
KDEOS 88 0.07000 0.05107 0.04754 0.02816 0.09926 0.08093 0.76287
KDEOS 100 0.07000 0.05107 0.05325 0.03398 0.11793 0.09998 0.77743
LDF 16 0.46000 0.44901 0.39337 0.38102 0.46000 0.44901 0.91059
LDF 26 0.43000 0.41840 0.40239 0.39022 0.47778 0.46715 0.91678
LDF 27 0.44000 0.42860 0.41265 0.40070 0.46561 0.45473 0.91930
LDF 30 0.42000 0.40819 0.40125 0.38907 0.46429 0.45338 0.92130
INFLO 46 0.47000 0.45921 0.37436 0.36162 0.47573 0.46506 0.86706
INFLO 47 0.47000 0.45921 0.37446 0.36173 0.47959 0.46900 0.86713
INFLO 56 0.45000 0.43881 0.38343 0.37089 0.46602 0.45515 0.89423
INFLO 66 0.41000 0.39799 0.37176 0.35897 0.44643 0.43516 0.89525
COF 38 0.42000 0.40819 0.37971 0.36708 0.42640 0.41472 0.88672
COF 42 0.41000 0.39799 0.38886 0.37642 0.41667 0.40479 0.89994
COF 47 0.42000 0.40819 0.38022 0.36761 0.43299 0.42145 0.88601
COF 52 0.40000 0.38779 0.38035 0.36774 0.41341 0.40147 0.91028

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO