Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#07)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.2 MB) Download raw algorithm evaluation table (66.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.43434 0.42288 0.37530 0.36263 0.47399 0.46332 0.90793
KNN 4 0.45455 0.44349 0.39558 0.38332 0.46667 0.45585 0.93403
KNN 18 0.40404 0.39196 0.37343 0.36073 0.42391 0.41223 0.94081
KNNW 2 0.43434 0.42288 0.37538 0.36271 0.48235 0.47186 0.88180
KNNW 7 0.43434 0.42288 0.38810 0.37569 0.46591 0.45508 0.92794
KNNW 33 0.41414 0.40226 0.37743 0.36480 0.42781 0.41621 0.94094
LOF 73 0.42424 0.41257 0.36172 0.34878 0.42640 0.41477 0.95643
LOF 80 0.42424 0.41257 0.36478 0.35191 0.43750 0.42610 0.95745
LOF 99 0.40404 0.39196 0.37805 0.36544 0.42581 0.41417 0.96018
LOF 100 0.39394 0.38165 0.37934 0.36676 0.43011 0.41855 0.96018
SimplifiedLOF 66 0.39394 0.38165 0.35567 0.34260 0.43094 0.41940 0.94743
SimplifiedLOF 96 0.41414 0.40226 0.38292 0.37040 0.42105 0.40931 0.95869
SimplifiedLOF 100 0.40404 0.39196 0.38673 0.37430 0.42149 0.40976 0.95953
LoOP 65 0.40404 0.39196 0.34477 0.33148 0.41176 0.39984 0.93405
LoOP 83 0.39394 0.38165 0.35267 0.33955 0.42775 0.41614 0.94560
LoOP 100 0.39394 0.38165 0.35909 0.34609 0.42236 0.41065 0.95181
LDOF 30 0.40404 0.39196 0.29802 0.28379 0.40838 0.39638 0.89419
LDOF 80 0.37374 0.36104 0.37164 0.35890 0.42857 0.41699 0.95552
LDOF 100 0.38384 0.37135 0.38968 0.37731 0.41522 0.40337 0.96118
ODIN 62 0.40067 0.38852 0.26012 0.24512 0.40206 0.38994 0.88406
ODIN 100 0.36364 0.35073 0.30203 0.28788 0.38278 0.37026 0.92429
FastABOD 4 0.40404 0.39196 0.34277 0.32944 0.40758 0.39557 0.85462
FastABOD 5 0.40404 0.39196 0.35046 0.33729 0.41791 0.40611 0.85480
FastABOD 98 0.40404 0.39196 0.36142 0.34847 0.42718 0.41557 0.85044
FastABOD 100 0.40404 0.39196 0.36220 0.34927 0.42512 0.41347 0.85043
KDEOS 47 0.06061 0.04156 0.05144 0.03221 0.11340 0.09543 0.76620
KDEOS 51 0.05051 0.03125 0.05381 0.03463 0.11545 0.09752 0.77564
KDEOS 65 0.02020 0.00034 0.05254 0.03333 0.12356 0.10579 0.79248
KDEOS 99 0.03030 0.01064 0.05131 0.03208 0.10870 0.09062 0.80853
LDF 14 0.41414 0.40226 0.31543 0.30155 0.41624 0.40441 0.85309
LDF 88 0.39394 0.38165 0.39977 0.38760 0.46209 0.45119 0.96044
LDF 89 0.39394 0.38165 0.39807 0.38586 0.45878 0.44781 0.96044
LDF 99 0.38384 0.37135 0.41021 0.39826 0.45038 0.43924 0.96006
INFLO 73 0.39394 0.38165 0.35019 0.33702 0.43114 0.41960 0.89120
INFLO 74 0.40404 0.39196 0.35166 0.33852 0.43114 0.41960 0.89702
INFLO 91 0.39394 0.38165 0.36762 0.35480 0.42500 0.41334 0.93199
INFLO 100 0.38384 0.37135 0.37437 0.36168 0.41718 0.40536 0.92732
COF 76 0.44444 0.43318 0.40839 0.39640 0.46561 0.45477 0.93086
COF 85 0.44444 0.43318 0.41321 0.40132 0.47337 0.46270 0.93676
COF 90 0.43434 0.42288 0.41652 0.40469 0.46243 0.45153 0.94000
COF 99 0.42424 0.41257 0.41131 0.39937 0.45198 0.44087 0.94175

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.4 MB) Download raw algorithm evaluation table (62.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.44000 0.42860 0.31999 0.30615 0.44565 0.43437 0.84992
KNN 56 0.34000 0.32657 0.34810 0.33483 0.39437 0.38204 0.92638
KNN 61 0.34000 0.32657 0.34982 0.33659 0.40989 0.39788 0.92612
KNNW 9 0.42000 0.40819 0.30795 0.29387 0.42640 0.41472 0.85608
KNNW 17 0.42000 0.40819 0.32108 0.30727 0.42786 0.41622 0.88834
KNNW 81 0.38000 0.36738 0.34548 0.33215 0.39024 0.37783 0.92663
KNNW 95 0.38000 0.36738 0.34714 0.33385 0.38647 0.37399 0.92639
LOF 23 0.40000 0.38779 0.32210 0.30830 0.44068 0.42929 0.80005
LOF 25 0.42000 0.40819 0.31995 0.30611 0.42529 0.41359 0.79972
LOF 29 0.41000 0.39799 0.32407 0.31031 0.43011 0.41851 0.80385
LOF 100 0.31000 0.29596 0.27723 0.26252 0.33853 0.32507 0.95336
SimplifiedLOF 27 0.45000 0.43881 0.33833 0.32486 0.46009 0.44910 0.83603
SimplifiedLOF 31 0.44000 0.42860 0.33946 0.32602 0.47826 0.46764 0.82650
SimplifiedLOF 41 0.43000 0.41840 0.34632 0.33302 0.44335 0.43202 0.80107
SimplifiedLOF 100 0.36000 0.34697 0.28982 0.27537 0.38356 0.37101 0.93268
LoOP 50 0.35000 0.33677 0.29570 0.28137 0.41935 0.40754 0.81772
LoOP 56 0.36000 0.34697 0.30346 0.28928 0.40889 0.39686 0.83643
LoOP 69 0.40000 0.38779 0.29543 0.28109 0.40404 0.39191 0.87455
LoOP 100 0.34000 0.32657 0.28843 0.27395 0.37719 0.36452 0.92209
LDOF 57 0.45000 0.43881 0.33066 0.31703 0.45545 0.44436 0.90067
LDOF 61 0.44000 0.42860 0.33073 0.31711 0.46561 0.45473 0.90265
LDOF 99 0.38000 0.36738 0.34718 0.33389 0.42066 0.40887 0.93364
LDOF 100 0.38000 0.36738 0.34706 0.33377 0.42063 0.40884 0.93401
ODIN 78 0.38200 0.36942 0.25923 0.24415 0.41379 0.40186 0.87194
ODIN 94 0.39222 0.37985 0.27607 0.26133 0.41341 0.40147 0.89564
ODIN 98 0.39750 0.38524 0.27219 0.25737 0.40758 0.39552 0.89931
ODIN 100 0.39750 0.38524 0.27207 0.25725 0.41346 0.40152 0.90011
FastABOD 3 0.34000 0.32657 0.23991 0.22444 0.36464 0.35171 0.80482
FastABOD 53 0.39000 0.37758 0.26555 0.25060 0.39000 0.37758 0.78883
FastABOD 82 0.39000 0.37758 0.26833 0.25343 0.39796 0.38571 0.78655
FastABOD 100 0.39000 0.37758 0.26995 0.25509 0.39583 0.38354 0.78571
KDEOS 5 0.07000 0.05107 0.03660 0.01699 0.07650 0.05771 0.60387
KDEOS 100 0.03000 0.01026 0.05441 0.03516 0.13689 0.11933 0.76032
LDF 11 0.38000 0.36738 0.35888 0.34583 0.41558 0.40369 0.80398
LDF 23 0.43000 0.41840 0.32942 0.31577 0.43094 0.41936 0.78556
LDF 24 0.41000 0.39799 0.32517 0.31144 0.43333 0.42180 0.78759
LDF 82 0.30000 0.28575 0.30948 0.29542 0.39883 0.38659 0.95291
INFLO 36 0.35000 0.33677 0.28746 0.27295 0.40625 0.39416 0.76798
INFLO 47 0.38000 0.36738 0.29107 0.27664 0.39252 0.38016 0.77442
INFLO 50 0.39000 0.37758 0.28357 0.26899 0.39810 0.38585 0.77330
INFLO 100 0.33000 0.31636 0.26640 0.25147 0.37131 0.35851 0.84097
COF 16 0.33000 0.31636 0.26097 0.24593 0.34211 0.32871 0.81836
COF 39 0.44000 0.42860 0.34183 0.32843 0.47561 0.46494 0.74252
COF 43 0.42000 0.40819 0.35712 0.34403 0.46857 0.45775 0.74740

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.1 MB) Download raw algorithm evaluation table (64.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.18182 0.16523 0.13318 0.11561 0.18994 0.17352 0.61971
KNN 27 0.12121 0.10340 0.11154 0.09353 0.16901 0.15217 0.62759
KNNW 1 0.17172 0.15492 0.12741 0.10972 0.19178 0.17539 0.60766
KNNW 4 0.17172 0.15492 0.13248 0.11489 0.18321 0.16665 0.61762
KNNW 54 0.13131 0.11370 0.11086 0.09283 0.16901 0.15217 0.62671
LOF 36 0.40404 0.39196 0.41412 0.40224 0.46914 0.45837 0.91706
LOF 56 0.43434 0.42288 0.41116 0.39922 0.43878 0.42740 0.93844
LOF 65 0.41414 0.40226 0.39761 0.38540 0.43011 0.41855 0.94009
SimplifiedLOF 42 0.43434 0.42288 0.42519 0.41354 0.45783 0.44684 0.89153
SimplifiedLOF 45 0.43434 0.42288 0.42159 0.40987 0.46154 0.45062 0.89893
SimplifiedLOF 67 0.44444 0.43318 0.40876 0.39677 0.44670 0.43548 0.93701
SimplifiedLOF 100 0.37374 0.36104 0.37328 0.36057 0.44444 0.43318 0.93976
LoOP 67 0.43434 0.42288 0.42803 0.41644 0.47399 0.46332 0.92690
LoOP 78 0.44444 0.43318 0.43347 0.42198 0.46784 0.45705 0.93222
LoOP 79 0.44444 0.43318 0.43369 0.42220 0.46784 0.45705 0.93271
LoOP 100 0.43434 0.42288 0.42578 0.41414 0.45000 0.43885 0.93905
LDOF 37 0.42424 0.41257 0.38522 0.37276 0.42424 0.41257 0.86516
LDOF 85 0.41414 0.40226 0.43401 0.42254 0.45283 0.44174 0.93600
LDOF 90 0.41414 0.40226 0.43355 0.42207 0.45455 0.44349 0.93977
LDOF 100 0.41414 0.40226 0.43109 0.41956 0.44828 0.43709 0.94575
ODIN 98 0.38740 0.37498 0.35142 0.33827 0.40252 0.39040 0.92472
ODIN 99 0.39538 0.38312 0.35061 0.33744 0.40252 0.39040 0.92522
ODIN 100 0.39538 0.38312 0.35228 0.33915 0.40252 0.39040 0.92544
FastABOD 3 0.14141 0.12401 0.11080 0.09278 0.17219 0.15540 0.50488
FastABOD 7 0.14141 0.12401 0.11350 0.09553 0.17500 0.15827 0.50522
FastABOD 13 0.14141 0.12401 0.11190 0.09390 0.17518 0.15846 0.50450
KDEOS 6 0.07071 0.05187 0.03690 0.01737 0.08108 0.06245 0.63906
KDEOS 100 0.06061 0.04156 0.05763 0.03853 0.12996 0.11232 0.79528
LDF 12 0.42424 0.41257 0.37109 0.35834 0.43386 0.42238 0.86261
LDF 24 0.41414 0.40226 0.40894 0.39696 0.46541 0.45457 0.89388
LDF 56 0.41414 0.40226 0.38303 0.37052 0.43182 0.42030 0.92944
INFLO 42 0.41414 0.40226 0.40144 0.38930 0.45238 0.44128 0.83702
INFLO 54 0.43434 0.42288 0.39321 0.38090 0.46739 0.45659 0.86132
INFLO 66 0.44444 0.43318 0.39714 0.38492 0.45128 0.44016 0.91051
INFLO 67 0.44444 0.43318 0.39917 0.38699 0.44670 0.43548 0.91075
COF 42 0.42424 0.41257 0.40997 0.39801 0.46591 0.45508 0.86722
COF 44 0.41414 0.40226 0.40729 0.39528 0.47205 0.46135 0.87151
COF 62 0.41414 0.40226 0.39298 0.38068 0.45122 0.44009 0.89533
COF 67 0.43434 0.42288 0.39586 0.38361 0.44444 0.43318 0.89325

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (61.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.13000 0.11229 0.08633 0.06773 0.14815 0.13081 0.57348
KNN 4 0.13000 0.11229 0.10084 0.08253 0.13636 0.11879 0.61034
KNN 6 0.12000 0.10209 0.10247 0.08420 0.15176 0.13450 0.60880
KNN 9 0.13000 0.11229 0.09883 0.08049 0.15534 0.13815 0.60910
KNNW 1 0.14000 0.12250 0.08989 0.07137 0.17323 0.15640 0.56941
KNNW 10 0.13000 0.11229 0.10101 0.08271 0.14155 0.12408 0.60439
KNNW 15 0.12000 0.10209 0.09761 0.07925 0.14646 0.12909 0.60519
LOF 35 0.46000 0.44901 0.40201 0.38984 0.48387 0.47337 0.92335
LOF 36 0.47000 0.45921 0.40620 0.39411 0.47761 0.46698 0.92093
LOF 41 0.48000 0.46942 0.39697 0.38470 0.48756 0.47713 0.91930
LOF 43 0.47000 0.45921 0.39643 0.38415 0.50267 0.49255 0.91769
SimplifiedLOF 37 0.45000 0.43881 0.41107 0.39908 0.46927 0.45847 0.91687
SimplifiedLOF 43 0.50000 0.48982 0.40469 0.39257 0.50256 0.49244 0.91774
SimplifiedLOF 55 0.47000 0.45921 0.39885 0.38661 0.47525 0.46457 0.92720
LoOP 54 0.45000 0.43881 0.39220 0.37983 0.47442 0.46372 0.91830
LoOP 59 0.46000 0.44901 0.38899 0.37656 0.47312 0.46239 0.92120
LoOP 69 0.48000 0.46942 0.39079 0.37839 0.48677 0.47633 0.91977
LoOP 73 0.48000 0.46942 0.38900 0.37657 0.49231 0.48197 0.91965
LDOF 55 0.46000 0.44901 0.37585 0.36314 0.48341 0.47290 0.91523
LDOF 57 0.47000 0.45921 0.37839 0.36573 0.47805 0.46742 0.91802
LDOF 89 0.43000 0.41840 0.40755 0.39549 0.45887 0.44786 0.93776
LDOF 100 0.42000 0.40819 0.39534 0.38303 0.45333 0.44221 0.94112
ODIN 87 0.40000 0.38779 0.32902 0.31536 0.41935 0.40754 0.90674
ODIN 92 0.39000 0.37758 0.33323 0.31966 0.42857 0.41694 0.90863
ODIN 100 0.39000 0.37758 0.33569 0.32216 0.42391 0.41219 0.91143
FastABOD 3 0.11000 0.09188 0.05512 0.03589 0.14966 0.13235 0.49431
FastABOD 6 0.10000 0.08168 0.07513 0.05631 0.15038 0.13308 0.48380
FastABOD 12 0.10000 0.08168 0.07563 0.05682 0.13986 0.12235 0.48169
KDEOS 9 0.08000 0.06127 0.03283 0.01314 0.08696 0.06837 0.61780
KDEOS 100 0.08000 0.06127 0.06467 0.04563 0.11807 0.10012 0.80139
LDF 21 0.46000 0.44901 0.40691 0.39484 0.48128 0.47073 0.91977
LDF 23 0.46000 0.44901 0.41462 0.40270 0.47568 0.46500 0.92149
LDF 28 0.44000 0.42860 0.41128 0.39929 0.46512 0.45423 0.92318
INFLO 38 0.45000 0.43881 0.37540 0.36269 0.46226 0.45132 0.87226
INFLO 50 0.47000 0.45921 0.37429 0.36156 0.47059 0.45981 0.88455
INFLO 51 0.46000 0.44901 0.37279 0.36003 0.47291 0.46218 0.87927
INFLO 64 0.43000 0.41840 0.34949 0.33624 0.44565 0.43437 0.88631
COF 37 0.49000 0.47962 0.38341 0.37086 0.49261 0.48228 0.87038
COF 47 0.48000 0.46942 0.39183 0.37945 0.48000 0.46942 0.88544
COF 52 0.41000 0.39799 0.38578 0.37328 0.44172 0.43035 0.89273

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO