Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#10)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.2 MB) Download raw algorithm evaluation table (67.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 8 0.45455 0.44349 0.40339 0.39129 0.47059 0.45985 0.94662
KNN 9 0.46465 0.45379 0.40096 0.38882 0.46939 0.45863 0.94738
KNN 14 0.44444 0.43318 0.42554 0.41390 0.46154 0.45062 0.94584
KNNW 12 0.44444 0.43318 0.39176 0.37943 0.47059 0.45985 0.94450
KNNW 23 0.45455 0.44349 0.41179 0.39987 0.45685 0.44584 0.94629
KNNW 40 0.43434 0.42288 0.42183 0.41011 0.46575 0.45492 0.94595
LOF 20 0.40404 0.39196 0.29618 0.28191 0.41206 0.40014 0.88991
LOF 94 0.38384 0.37135 0.36220 0.34926 0.45714 0.44614 0.96030
LOF 97 0.37374 0.36104 0.36456 0.35168 0.44755 0.43635 0.96060
LOF 100 0.37374 0.36104 0.36851 0.35571 0.45487 0.44382 0.96057
SimplifiedLOF 34 0.43434 0.42288 0.34797 0.33475 0.44086 0.42952 0.88646
SimplifiedLOF 35 0.43434 0.42288 0.34953 0.33635 0.44103 0.42969 0.88975
SimplifiedLOF 100 0.39394 0.38165 0.38097 0.36842 0.41199 0.40006 0.95768
LoOP 54 0.41414 0.40226 0.30792 0.29389 0.41414 0.40226 0.91355
LoOP 62 0.40404 0.39196 0.31894 0.30513 0.42000 0.40824 0.92110
LoOP 100 0.37374 0.36104 0.34063 0.32726 0.39819 0.38599 0.94967
LDOF 65 0.41414 0.40226 0.35354 0.34043 0.43243 0.42093 0.94177
LDOF 70 0.41414 0.40226 0.35669 0.34365 0.44531 0.43407 0.94597
LDOF 99 0.39394 0.38165 0.38863 0.37624 0.42697 0.41535 0.95755
LDOF 100 0.40404 0.39196 0.38796 0.37555 0.43220 0.42069 0.95796
ODIN 78 0.28716 0.27270 0.23602 0.22053 0.38596 0.37352 0.91174
ODIN 100 0.35354 0.34043 0.26203 0.24707 0.37209 0.35936 0.92875
FastABOD 4 0.36364 0.35073 0.29217 0.27782 0.36735 0.35452 0.84232
FastABOD 13 0.33333 0.31982 0.30348 0.28936 0.38095 0.36840 0.84761
FastABOD 15 0.33333 0.31982 0.30307 0.28894 0.38788 0.37547 0.84718
KDEOS 3 0.05051 0.03125 0.02808 0.00838 0.06863 0.04974 0.58601
KDEOS 64 0.01010 -0.00997 0.05297 0.03377 0.12010 0.10227 0.79705
KDEOS 85 0.00000 -0.02027 0.05184 0.03262 0.12779 0.11010 0.80433
KDEOS 100 0.00000 -0.02027 0.05281 0.03360 0.11473 0.09678 0.81069
LDF 79 0.38384 0.37135 0.38345 0.37095 0.47210 0.46140 0.96069
LDF 82 0.39394 0.38165 0.38898 0.37659 0.46610 0.45528 0.96097
LDF 96 0.42424 0.41257 0.39772 0.38551 0.46364 0.45276 0.96041
LDF 99 0.41414 0.40226 0.39894 0.38676 0.45946 0.44850 0.96035
INFLO 34 0.38384 0.37135 0.29705 0.28280 0.39640 0.38416 0.79357
INFLO 40 0.38384 0.37135 0.30520 0.29111 0.40952 0.39755 0.81584
INFLO 100 0.35354 0.34043 0.35061 0.33744 0.39482 0.38255 0.93395
COF 28 0.43434 0.42288 0.30657 0.29251 0.43750 0.42610 0.79545
COF 31 0.42424 0.41257 0.32329 0.30957 0.45614 0.44511 0.80541
COF 86 0.40404 0.39196 0.35412 0.34103 0.41808 0.40628 0.92878
COF 100 0.39394 0.38165 0.34788 0.33465 0.42857 0.41699 0.93457

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.2 MB) Download raw algorithm evaluation table (62.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 9 0.41000 0.39799 0.35650 0.34341 0.41206 0.40009 0.83520
KNN 40 0.31000 0.29596 0.36485 0.35192 0.38961 0.37719 0.93215
KNN 78 0.31000 0.29596 0.37747 0.36480 0.43446 0.42295 0.92872
KNN 82 0.30000 0.28575 0.37592 0.36322 0.44106 0.42969 0.92836
KNNW 6 0.40000 0.38779 0.32194 0.30813 0.41758 0.40573 0.82237
KNNW 74 0.33000 0.31636 0.37122 0.35843 0.37037 0.35755 0.93312
KNNW 99 0.33000 0.31636 0.37515 0.36244 0.40248 0.39031 0.93241
LOF 14 0.44000 0.42860 0.36781 0.35494 0.46083 0.44986 0.84890
LOF 16 0.44000 0.42860 0.38248 0.36991 0.48077 0.47020 0.84563
LOF 100 0.29000 0.27555 0.32427 0.31052 0.33195 0.31835 0.94428
SimplifiedLOF 17 0.45000 0.43881 0.38029 0.36768 0.45771 0.44667 0.85717
SimplifiedLOF 22 0.43000 0.41840 0.38819 0.37574 0.45714 0.44609 0.83762
SimplifiedLOF 25 0.43000 0.41840 0.38116 0.36857 0.47534 0.46466 0.82519
SimplifiedLOF 100 0.33000 0.31636 0.32540 0.31167 0.36123 0.34823 0.91947
LoOP 31 0.41000 0.39799 0.31096 0.29693 0.41206 0.40009 0.81303
LoOP 53 0.34000 0.32657 0.33132 0.31771 0.36036 0.34734 0.79656
LoOP 100 0.32000 0.30616 0.30634 0.29222 0.35945 0.34641 0.90339
LDOF 26 0.41000 0.39799 0.32773 0.31404 0.42529 0.41359 0.84029
LDOF 27 0.40000 0.38779 0.32667 0.31296 0.43192 0.42036 0.83919
LDOF 94 0.36000 0.34697 0.37127 0.35847 0.38095 0.36835 0.93144
LDOF 100 0.36000 0.34697 0.36892 0.35608 0.38938 0.37695 0.93508
ODIN 36 0.33000 0.31636 0.23513 0.21956 0.36279 0.34982 0.78528
ODIN 41 0.35000 0.33677 0.23460 0.21903 0.35000 0.33677 0.79022
ODIN 84 0.31750 0.30361 0.26865 0.25376 0.34395 0.33060 0.86851
ODIN 100 0.31200 0.29800 0.26104 0.24600 0.33065 0.31702 0.88850
FastABOD 4 0.41000 0.39799 0.26479 0.24982 0.41315 0.40120 0.79051
FastABOD 5 0.42000 0.40819 0.30234 0.28814 0.43299 0.42145 0.78617
FastABOD 6 0.41000 0.39799 0.32210 0.30830 0.42105 0.40927 0.78624
KDEOS 15 0.11000 0.09188 0.05022 0.03089 0.11518 0.09717 0.71515
KDEOS 91 0.09000 0.07148 0.08032 0.06160 0.18132 0.16466 0.75722
KDEOS 94 0.09000 0.07148 0.08142 0.06273 0.16842 0.15149 0.75994
KDEOS 100 0.10000 0.08168 0.07428 0.05544 0.15775 0.14060 0.76239
LDF 12 0.46000 0.44901 0.39244 0.38008 0.46231 0.45137 0.84215
LDF 15 0.45000 0.43881 0.39616 0.38387 0.45455 0.44344 0.84105
LDF 93 0.33000 0.31636 0.39104 0.37865 0.47945 0.46886 0.95372
INFLO 16 0.43000 0.41840 0.34109 0.32767 0.43216 0.42060 0.78829
COF 13 0.29000 0.27555 0.24170 0.22627 0.33198 0.31839 0.84468
COF 24 0.46000 0.44901 0.34535 0.33203 0.47059 0.45981 0.77908
COF 32 0.44000 0.42860 0.36027 0.34725 0.45902 0.44801 0.76885
COF 34 0.45000 0.43881 0.35872 0.34567 0.49180 0.48146 0.75892

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.2 MB) Download raw algorithm evaluation table (67.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.12121 0.10340 0.08454 0.06598 0.13505 0.11751 0.63194
KNN 3 0.13131 0.11370 0.08425 0.06569 0.14218 0.12479 0.63581
KNN 9 0.10101 0.08278 0.08205 0.06344 0.15054 0.13332 0.64733
KNNW 1 0.13131 0.11370 0.08662 0.06810 0.13986 0.12242 0.57270
KNNW 15 0.10101 0.08278 0.07978 0.06112 0.12836 0.11069 0.64331
LOF 16 0.47475 0.46410 0.42263 0.41092 0.47716 0.46656 0.92315
LOF 20 0.46465 0.45379 0.43118 0.41965 0.49711 0.48691 0.92220
LOF 57 0.39394 0.38165 0.35229 0.33916 0.41584 0.40400 0.94644
SimplifiedLOF 22 0.46465 0.45379 0.41232 0.40041 0.48087 0.47035 0.92255
SimplifiedLOF 29 0.45455 0.44349 0.40866 0.39667 0.48837 0.47800 0.93305
SimplifiedLOF 66 0.40404 0.39196 0.35849 0.34548 0.42105 0.40931 0.95342
LoOP 38 0.45455 0.44349 0.38576 0.37331 0.49123 0.48091 0.92586
LoOP 50 0.47475 0.46410 0.39373 0.38144 0.47959 0.46904 0.93964
LoOP 52 0.47475 0.46410 0.39427 0.38199 0.48454 0.47409 0.94202
LoOP 79 0.45455 0.44349 0.38590 0.37345 0.46667 0.45585 0.95193
LDOF 56 0.47475 0.46410 0.39011 0.37774 0.47959 0.46904 0.93562
LDOF 77 0.45455 0.44349 0.39588 0.38363 0.48276 0.47227 0.95252
LDOF 83 0.45455 0.44349 0.39725 0.38503 0.46995 0.45920 0.95484
LDOF 99 0.42424 0.41257 0.38423 0.37175 0.44340 0.43211 0.95733
ODIN 87 0.42496 0.41331 0.38889 0.37650 0.44578 0.43455 0.93206
ODIN 92 0.42057 0.40882 0.38660 0.37417 0.44970 0.43855 0.93466
ODIN 100 0.42424 0.41257 0.39415 0.38187 0.46784 0.45705 0.93444
FastABOD 3 0.10101 0.08278 0.06613 0.04719 0.11050 0.09246 0.49583
FastABOD 6 0.10101 0.08278 0.06855 0.04966 0.12270 0.10491 0.49306
FastABOD 7 0.10101 0.08278 0.06865 0.04977 0.11696 0.09906 0.49348
KDEOS 72 0.10101 0.08278 0.07337 0.05459 0.11844 0.10057 0.82229
KDEOS 78 0.07071 0.05187 0.07681 0.05809 0.12245 0.10466 0.82830
KDEOS 98 0.10101 0.08278 0.07182 0.05300 0.13859 0.12113 0.83753
LDF 12 0.49495 0.48471 0.44809 0.43690 0.51042 0.50049 0.91936
LDF 14 0.46465 0.45379 0.45122 0.44009 0.51220 0.50231 0.92702
LDF 15 0.47475 0.46410 0.45253 0.44143 0.51111 0.50120 0.92459
LDF 29 0.42424 0.41257 0.39350 0.38121 0.44199 0.43068 0.93927
INFLO 26 0.41414 0.40226 0.37948 0.36689 0.46429 0.45342 0.86270
INFLO 37 0.45455 0.44349 0.37504 0.36237 0.46707 0.45626 0.88659
INFLO 47 0.45455 0.44349 0.36079 0.34783 0.47872 0.46815 0.87233
INFLO 67 0.40404 0.39196 0.34347 0.33016 0.42391 0.41223 0.91656
COF 28 0.42424 0.41257 0.38154 0.36900 0.44444 0.43318 0.90464
COF 36 0.45455 0.44349 0.37926 0.36668 0.45714 0.44614 0.90355
COF 60 0.42424 0.41257 0.35487 0.34179 0.43299 0.42149 0.92835

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (59.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.14000 0.12250 0.12010 0.10219 0.18792 0.17139 0.57741
KNN 2 0.15000 0.13270 0.12319 0.10534 0.17886 0.16215 0.58107
KNN 3 0.16000 0.14290 0.12275 0.10490 0.18462 0.16802 0.58975
KNN 8 0.16000 0.14290 0.11577 0.09777 0.16901 0.15210 0.59816
KNNW 1 0.16000 0.14290 0.12217 0.10430 0.19310 0.17668 0.56870
KNNW 5 0.16000 0.14290 0.12218 0.10432 0.18182 0.16516 0.58396
KNNW 33 0.14000 0.12250 0.10928 0.09115 0.16783 0.15089 0.59441
LOF 33 0.44000 0.42860 0.41239 0.40043 0.45503 0.44393 0.91961
LOF 37 0.44000 0.42860 0.40806 0.39602 0.45596 0.44489 0.92255
LOF 39 0.46000 0.44901 0.40696 0.39488 0.46231 0.45137 0.92057
LOF 41 0.46000 0.44901 0.40643 0.39435 0.47514 0.46445 0.91987
SimplifiedLOF 34 0.45000 0.43881 0.42271 0.41096 0.47120 0.46044 0.91828
SimplifiedLOF 39 0.48000 0.46942 0.41491 0.40300 0.48515 0.47467 0.91963
SimplifiedLOF 46 0.47000 0.45921 0.41228 0.40032 0.49462 0.48434 0.92674
SimplifiedLOF 59 0.45000 0.43881 0.41471 0.40279 0.45771 0.44667 0.93502
LoOP 56 0.48000 0.46942 0.40796 0.39591 0.48619 0.47573 0.92109
LoOP 61 0.48000 0.46942 0.41014 0.39813 0.49735 0.48712 0.92357
LoOP 74 0.47000 0.45921 0.41017 0.39817 0.48352 0.47300 0.92883
LoOP 76 0.46000 0.44901 0.41174 0.39977 0.48087 0.47031 0.92828
LDOF 63 0.45000 0.43881 0.41080 0.39881 0.48913 0.47873 0.93323
LDOF 68 0.47000 0.45921 0.41066 0.39866 0.47059 0.45981 0.93650
LDOF 78 0.45000 0.43881 0.42172 0.40995 0.47826 0.46764 0.94439
LDOF 100 0.46000 0.44901 0.40451 0.39239 0.46000 0.44901 0.95315
ODIN 83 0.35067 0.33745 0.33634 0.32283 0.40764 0.39559 0.90616
ODIN 99 0.37769 0.36503 0.34481 0.33147 0.40000 0.38779 0.91216
ODIN 100 0.37769 0.36503 0.34452 0.33118 0.40000 0.38779 0.91303
FastABOD 3 0.14000 0.12250 0.07291 0.05404 0.16883 0.15191 0.48388
FastABOD 5 0.14000 0.12250 0.09567 0.07726 0.17931 0.16261 0.48447
FastABOD 40 0.13000 0.11229 0.10400 0.08576 0.17391 0.15710 0.47093
KDEOS 96 0.09000 0.07148 0.05459 0.03535 0.11175 0.09367 0.78337
KDEOS 100 0.08000 0.06127 0.05784 0.03867 0.11585 0.09786 0.78832
LDF 16 0.46000 0.44901 0.42952 0.41791 0.48087 0.47031 0.91012
LDF 17 0.45000 0.43881 0.43098 0.41940 0.48352 0.47300 0.91434
LDF 18 0.45000 0.43881 0.43139 0.41982 0.48045 0.46987 0.91158
LDF 26 0.40000 0.38779 0.41637 0.40449 0.42857 0.41694 0.91858
INFLO 41 0.48000 0.46942 0.38255 0.36999 0.48276 0.47223 0.84307
INFLO 47 0.47000 0.45921 0.38419 0.37165 0.49198 0.48164 0.85145
INFLO 66 0.44000 0.42860 0.38898 0.37654 0.46561 0.45473 0.89365
COF 41 0.43000 0.41840 0.39301 0.38066 0.47399 0.46328 0.87850
COF 44 0.43000 0.41840 0.39574 0.38344 0.46512 0.45423 0.88943
COF 45 0.43000 0.41840 0.39704 0.38476 0.46626 0.45539 0.88733
COF 87 0.46000 0.44901 0.35708 0.34400 0.46000 0.44901 0.82781

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO