Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#08)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.3 MB) Download raw algorithm evaluation table (67.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.48485 0.47440 0.44871 0.43754 0.50000 0.48986 0.95229
KNN 7 0.45455 0.44349 0.43120 0.41967 0.47826 0.46768 0.95302
KNNW 6 0.46465 0.45379 0.42116 0.40943 0.49724 0.48704 0.94729
KNNW 12 0.45455 0.44349 0.43022 0.41867 0.48087 0.47035 0.95291
KNNW 21 0.45455 0.44349 0.43737 0.42597 0.47568 0.46505 0.95221
LOF 41 0.42424 0.41257 0.37367 0.36097 0.42424 0.41257 0.95664
LOF 99 0.41414 0.40226 0.39020 0.37783 0.44944 0.43828 0.96819
LOF 100 0.41414 0.40226 0.39262 0.38030 0.44776 0.43656 0.96824
SimplifiedLOF 98 0.44444 0.43318 0.40695 0.39493 0.45223 0.44112 0.96929
SimplifiedLOF 100 0.44444 0.43318 0.41101 0.39907 0.45367 0.44260 0.96959
LoOP 42 0.44444 0.43318 0.35822 0.34520 0.44898 0.43781 0.92776
LoOP 93 0.43434 0.42288 0.39219 0.37987 0.45652 0.44550 0.96458
LoOP 94 0.43434 0.42288 0.39257 0.38025 0.45503 0.44398 0.96449
LoOP 100 0.43434 0.42288 0.39189 0.37956 0.45055 0.43941 0.96543
LDOF 61 0.45455 0.44349 0.39522 0.38296 0.47847 0.46790 0.95856
LDOF 62 0.47475 0.46410 0.39694 0.38471 0.47475 0.46410 0.95909
LDOF 93 0.46465 0.45379 0.42635 0.41472 0.47154 0.46083 0.96951
LDOF 100 0.44444 0.43318 0.42536 0.41371 0.47660 0.46598 0.97023
ODIN 93 0.40404 0.39196 0.33227 0.31873 0.42857 0.41699 0.94126
ODIN 99 0.41975 0.40799 0.33821 0.32479 0.42697 0.41535 0.94517
ODIN 100 0.42424 0.41257 0.33667 0.32322 0.42636 0.41473 0.94581
FastABOD 7 0.40404 0.39196 0.33957 0.32618 0.43396 0.42249 0.88316
FastABOD 67 0.42424 0.41257 0.35127 0.33811 0.45198 0.44087 0.88243
FastABOD 93 0.44444 0.43318 0.35553 0.34247 0.44571 0.43448 0.88248
FastABOD 100 0.44444 0.43318 0.35685 0.34381 0.44776 0.43656 0.88247
KDEOS 26 0.05051 0.03125 0.04011 0.02065 0.08794 0.06945 0.73901
KDEOS 99 0.02020 0.00034 0.05311 0.03391 0.11496 0.09702 0.82361
KDEOS 100 0.02020 0.00034 0.05320 0.03400 0.11560 0.09767 0.82355
LDF 24 0.42424 0.41257 0.37097 0.35821 0.42640 0.41477 0.95111
LDF 61 0.37374 0.36104 0.39891 0.38672 0.43825 0.42686 0.96492
LDF 94 0.40404 0.39196 0.42493 0.41327 0.46575 0.45492 0.96199
LDF 100 0.40404 0.39196 0.43266 0.42116 0.45794 0.44695 0.96169
INFLO 61 0.44444 0.43318 0.37539 0.36273 0.45361 0.44253 0.92616
INFLO 67 0.44444 0.43318 0.37089 0.35813 0.45503 0.44398 0.92079
INFLO 89 0.43434 0.42288 0.39696 0.38474 0.44330 0.43201 0.94918
INFLO 100 0.43434 0.42288 0.40971 0.39775 0.44571 0.43448 0.94510
COF 72 0.42424 0.41257 0.39206 0.37973 0.48000 0.46946 0.94921
COF 93 0.42424 0.41257 0.40757 0.39556 0.44324 0.43196 0.95309
COF 96 0.43434 0.42288 0.41824 0.40644 0.45000 0.43885 0.95253
COF 97 0.45455 0.44349 0.41201 0.40009 0.45685 0.44584 0.95210

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.4 MB) Download raw algorithm evaluation table (62.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.41000 0.39799 0.33530 0.32177 0.42623 0.41455 0.85595
KNN 9 0.40000 0.38779 0.33865 0.32519 0.41121 0.39923 0.89842
KNN 19 0.38000 0.36738 0.33847 0.32501 0.40394 0.39181 0.92488
KNN 84 0.33000 0.31636 0.33158 0.31798 0.43750 0.42605 0.91155
KNNW 20 0.40000 0.38779 0.33905 0.32559 0.42157 0.40980 0.90854
KNNW 24 0.42000 0.40819 0.33620 0.32269 0.42574 0.41405 0.91508
KNNW 52 0.37000 0.35718 0.33596 0.32245 0.40196 0.38979 0.92383
LOF 15 0.39000 0.37758 0.29997 0.28572 0.41111 0.39912 0.84863
LOF 100 0.31000 0.29596 0.28508 0.27053 0.33871 0.32525 0.95427
SimplifiedLOF 20 0.40000 0.38779 0.30905 0.29498 0.40000 0.38779 0.85875
SimplifiedLOF 24 0.40000 0.38779 0.30651 0.29240 0.41237 0.40041 0.86020
SimplifiedLOF 28 0.39000 0.37758 0.31375 0.29978 0.39791 0.38565 0.85377
SimplifiedLOF 100 0.33000 0.31636 0.30434 0.29018 0.35754 0.34447 0.94681
LoOP 27 0.38000 0.36738 0.26050 0.24545 0.38384 0.37130 0.85124
LoOP 30 0.35000 0.33677 0.26421 0.24924 0.38857 0.37613 0.85215
LoOP 99 0.34000 0.32657 0.28554 0.27100 0.35897 0.34593 0.93944
LoOP 100 0.33000 0.31636 0.28531 0.27077 0.35802 0.34496 0.93989
LDOF 67 0.40000 0.38779 0.31710 0.30320 0.40201 0.38984 0.92507
LDOF 86 0.38000 0.36738 0.33102 0.31741 0.41702 0.40516 0.94457
LDOF 99 0.36000 0.34697 0.33443 0.32089 0.41600 0.40411 0.95060
LDOF 100 0.36000 0.34697 0.33422 0.32067 0.40800 0.39595 0.95077
ODIN 57 0.34722 0.33394 0.21441 0.19842 0.36279 0.34982 0.86345
ODIN 85 0.35000 0.33677 0.23922 0.22374 0.35354 0.34038 0.90894
ODIN 99 0.32333 0.30956 0.24507 0.22970 0.34921 0.33596 0.92050
ODIN 100 0.32250 0.30871 0.24397 0.22858 0.34921 0.33596 0.92067
FastABOD 4 0.31000 0.29596 0.18068 0.16401 0.32692 0.31322 0.78930
FastABOD 41 0.35000 0.33677 0.25733 0.24222 0.36842 0.35557 0.77609
FastABOD 59 0.36000 0.34697 0.25903 0.24395 0.36842 0.35557 0.77495
FastABOD 100 0.35000 0.33677 0.26370 0.24872 0.36471 0.35178 0.77390
KDEOS 3 0.07000 0.05107 0.02586 0.00603 0.07035 0.05143 0.53028
KDEOS 84 0.02000 0.00005 0.05351 0.03424 0.13008 0.11237 0.77857
KDEOS 100 0.04000 0.02046 0.05500 0.03577 0.11728 0.09932 0.79170
LDF 11 0.39000 0.37758 0.30863 0.29455 0.43590 0.42442 0.85530
LDF 13 0.39000 0.37758 0.34107 0.32766 0.41509 0.40319 0.85916
LDF 73 0.29000 0.27555 0.30415 0.28998 0.42202 0.41025 0.95249
LDF 97 0.27000 0.25514 0.32552 0.31179 0.44361 0.43228 0.94863
INFLO 14 0.37000 0.35718 0.25966 0.24459 0.37363 0.36088 0.81548
INFLO 15 0.37000 0.35718 0.26408 0.24910 0.39773 0.38547 0.80213
INFLO 47 0.32000 0.30616 0.28329 0.26870 0.36290 0.34994 0.81862
INFLO 100 0.33000 0.31636 0.26994 0.25508 0.34074 0.32732 0.85193
COF 20 0.43000 0.41840 0.28613 0.27160 0.44828 0.43705 0.78048
COF 22 0.44000 0.42860 0.30588 0.29175 0.44444 0.43314 0.77531
COF 30 0.42000 0.40819 0.33240 0.31881 0.42857 0.41694 0.77657
COF 94 0.34000 0.32657 0.27976 0.26510 0.36585 0.35295 0.90333

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.1 MB) Download raw algorithm evaluation table (65.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.15152 0.13431 0.09698 0.07867 0.15247 0.13528 0.64169
KNN 4 0.13131 0.11370 0.09079 0.07235 0.14365 0.12628 0.65523
KNNW 1 0.14141 0.12401 0.10011 0.08187 0.16522 0.14829 0.62292
KNNW 8 0.13131 0.11370 0.09326 0.07487 0.14607 0.12875 0.65154
LOF 43 0.46465 0.45379 0.43526 0.42381 0.46927 0.45851 0.93570
LOF 46 0.45455 0.44349 0.43803 0.42664 0.47872 0.46815 0.93743
LOF 50 0.45455 0.44349 0.43735 0.42594 0.48235 0.47186 0.94064
LOF 59 0.45455 0.44349 0.43256 0.42105 0.46067 0.44974 0.94253
SimplifiedLOF 39 0.49495 0.48471 0.43051 0.41896 0.49495 0.48471 0.91646
SimplifiedLOF 45 0.47475 0.46410 0.44169 0.43037 0.48421 0.47375 0.92867
SimplifiedLOF 69 0.44444 0.43318 0.42321 0.41152 0.45405 0.44299 0.95091
LoOP 57 0.47475 0.46410 0.40622 0.39418 0.48649 0.47608 0.92877
LoOP 64 0.47475 0.46410 0.41133 0.39939 0.50000 0.48986 0.93542
LoOP 88 0.45455 0.44349 0.43206 0.42055 0.47368 0.46301 0.94123
LoOP 100 0.45455 0.44349 0.41950 0.40773 0.46632 0.45550 0.94227
LDOF 88 0.47475 0.46410 0.40624 0.39420 0.47475 0.46410 0.94672
LDOF 91 0.46465 0.45379 0.40870 0.39672 0.46766 0.45687 0.94885
LDOF 93 0.46465 0.45379 0.40585 0.39381 0.47668 0.46607 0.94952
LDOF 100 0.46465 0.45379 0.40401 0.39193 0.46701 0.45620 0.95170
ODIN 83 0.34654 0.33329 0.34327 0.32995 0.41060 0.39865 0.91206
ODIN 100 0.37734 0.36472 0.35615 0.34309 0.39726 0.38504 0.92389
FastABOD 5 0.10101 0.08278 0.07262 0.05382 0.13534 0.11781 0.51546
FastABOD 6 0.11111 0.09309 0.07235 0.05354 0.13793 0.12045 0.51467
FastABOD 100 0.10101 0.08278 0.07534 0.05659 0.14493 0.12759 0.51221
KDEOS 89 0.09091 0.07248 0.05614 0.03700 0.11000 0.09196 0.79554
KDEOS 100 0.08081 0.06217 0.05987 0.04081 0.10581 0.08768 0.80376
LDF 43 0.44444 0.43318 0.44030 0.42896 0.44444 0.43318 0.93682
LDF 45 0.43434 0.42288 0.44010 0.42875 0.45581 0.44478 0.93741
LDF 52 0.41414 0.40226 0.42974 0.41818 0.46388 0.45301 0.93512
LDF 67 0.45455 0.44349 0.38469 0.37222 0.45887 0.44790 0.93203
INFLO 56 0.46465 0.45379 0.40601 0.39396 0.48936 0.47901 0.87487
INFLO 58 0.47475 0.46410 0.40820 0.39620 0.48454 0.47409 0.89477
INFLO 66 0.45455 0.44349 0.41353 0.40164 0.47872 0.46815 0.91771
INFLO 100 0.44444 0.43318 0.38652 0.37408 0.46640 0.45558 0.93076
COF 53 0.44444 0.43318 0.44345 0.43216 0.47742 0.46682 0.94293
COF 62 0.42424 0.41257 0.44288 0.43158 0.47561 0.46498 0.94493
COF 86 0.47475 0.46410 0.41767 0.40587 0.48515 0.47471 0.93398
COF 91 0.45455 0.44349 0.41149 0.39956 0.50000 0.48986 0.93013

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (60.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.16000 0.14290 0.09585 0.07745 0.17476 0.15796 0.59563
KNNW 4 0.14000 0.12250 0.09136 0.07287 0.14070 0.12321 0.57994
KNNW 8 0.11000 0.09188 0.09271 0.07424 0.12945 0.11173 0.58819
KNNW 9 0.11000 0.09188 0.09218 0.07370 0.13201 0.11435 0.58838
LOF 35 0.45000 0.43881 0.37393 0.36119 0.45000 0.43881 0.89077
LOF 36 0.44000 0.42860 0.37702 0.36434 0.44776 0.43652 0.89269
LOF 95 0.34000 0.32657 0.30441 0.29025 0.41606 0.40417 0.90177
SimplifiedLOF 36 0.44000 0.42860 0.38469 0.37216 0.45977 0.44877 0.88939
SimplifiedLOF 42 0.47000 0.45921 0.38122 0.36862 0.47120 0.46044 0.89290
SimplifiedLOF 49 0.47000 0.45921 0.38312 0.37057 0.47716 0.46652 0.90053
SimplifiedLOF 59 0.42000 0.40819 0.37270 0.35993 0.44444 0.43314 0.91335
LoOP 52 0.45000 0.43881 0.36765 0.35478 0.45614 0.44507 0.89684
LoOP 59 0.44000 0.42860 0.37092 0.35811 0.46739 0.45655 0.90787
LoOP 84 0.44000 0.42860 0.38003 0.36741 0.44444 0.43314 0.91115
LoOP 96 0.42000 0.40819 0.36897 0.35612 0.42478 0.41307 0.91258
LDOF 58 0.43000 0.41840 0.36330 0.35034 0.44944 0.43823 0.91508
LDOF 76 0.43000 0.41840 0.38065 0.36805 0.45745 0.44640 0.93052
LDOF 100 0.42000 0.40819 0.36644 0.35354 0.42982 0.41822 0.93504
ODIN 99 0.36600 0.35310 0.32607 0.31235 0.39490 0.38259 0.90297
ODIN 100 0.36750 0.35463 0.32708 0.31339 0.39490 0.38259 0.90363
FastABOD 3 0.08000 0.06127 0.04181 0.02231 0.11111 0.09302 0.46501
FastABOD 10 0.08000 0.06127 0.07649 0.05769 0.11966 0.10174 0.46149
FastABOD 12 0.07000 0.05107 0.07696 0.05817 0.11966 0.10174 0.46163
KDEOS 98 0.06000 0.04087 0.05161 0.03230 0.10750 0.08933 0.76179
KDEOS 99 0.07000 0.05107 0.05214 0.03285 0.10588 0.08768 0.76279
KDEOS 100 0.06000 0.04087 0.05222 0.03293 0.10588 0.08768 0.76378
LDF 14 0.45000 0.43881 0.35359 0.34043 0.45226 0.44111 0.88813
LDF 21 0.45000 0.43881 0.38299 0.37043 0.47059 0.45981 0.86915
LDF 28 0.41000 0.39799 0.36862 0.35577 0.43386 0.42234 0.88979
INFLO 41 0.42000 0.40819 0.36014 0.34712 0.44444 0.43314 0.83609
INFLO 48 0.46000 0.44901 0.35711 0.34403 0.46465 0.45375 0.84869
INFLO 49 0.46000 0.44901 0.35890 0.34585 0.47179 0.46104 0.84931
INFLO 66 0.42000 0.40819 0.35596 0.34285 0.44068 0.42929 0.88684
COF 41 0.43000 0.41840 0.38285 0.37029 0.47126 0.46050 0.86242
COF 50 0.44000 0.42860 0.38553 0.37302 0.44000 0.42860 0.86917
COF 54 0.42000 0.40819 0.38674 0.37426 0.43023 0.41864 0.87359
COF 62 0.41000 0.39799 0.36511 0.35219 0.41905 0.40722 0.87805

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO