Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#06)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (71.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 93 0.42597 0.28246 0.44453 0.30567 0.43237 0.29047 0.68291
KNN 97 0.42233 0.27791 0.44788 0.30985 0.43690 0.29613 0.68531
KNN 100 0.42233 0.27791 0.44892 0.31114 0.43522 0.29403 0.68657
KNNW 84 0.41262 0.26578 0.39588 0.24486 0.41422 0.26777 0.63943
KNNW 87 0.41262 0.26578 0.39736 0.24670 0.41565 0.26956 0.64109
KNNW 100 0.41262 0.26578 0.40331 0.25414 0.41472 0.26840 0.64747
LOF 100 0.34951 0.18689 0.31302 0.14128 0.41297 0.26621 0.67036
SimplifiedLOF 18 0.32039 0.15049 0.28975 0.11219 0.34572 0.18215 0.58076
SimplifiedLOF 21 0.32039 0.15049 0.29161 0.11451 0.35165 0.18956 0.58421
SimplifiedLOF 100 0.31068 0.13835 0.28224 0.10280 0.36269 0.20336 0.61006
LoOP 16 0.32282 0.15352 0.27502 0.09377 0.33686 0.17108 0.56900
LoOP 27 0.30097 0.12621 0.27993 0.09991 0.33677 0.17096 0.57572
LoOP 100 0.30097 0.12621 0.27484 0.09355 0.35596 0.19495 0.60406
LDOF 20 0.31311 0.14138 0.27982 0.09978 0.33374 0.16717 0.56368
LDOF 22 0.30340 0.12925 0.28407 0.10509 0.34372 0.17965 0.56707
LDOF 99 0.28883 0.11104 0.26154 0.07693 0.34790 0.18487 0.57148
LDOF 100 0.29126 0.11408 0.26184 0.07730 0.34483 0.18103 0.57198
ODIN 86 0.30372 0.12965 0.27663 0.09579 0.37975 0.22468 0.62316
ODIN 100 0.29029 0.11286 0.28369 0.10462 0.38994 0.23743 0.63707
FastABOD 65 0.29126 0.11408 0.26339 0.07923 0.34392 0.17990 0.57116
FastABOD 100 0.28883 0.11104 0.26702 0.08378 0.34594 0.18242 0.57655
KDEOS 15 0.23301 0.04126 0.23123 0.03903 0.33552 0.16940 0.53830
KDEOS 26 0.23058 0.03823 0.22464 0.03080 0.34760 0.18450 0.54971
KDEOS 34 0.23544 0.04430 0.22492 0.03114 0.34087 0.17609 0.55113
KDEOS 96 0.25000 0.06250 0.22749 0.03436 0.34607 0.18258 0.54728
LDF 97 0.34709 0.18386 0.42198 0.27747 0.42363 0.27954 0.69925
LDF 99 0.35194 0.18993 0.42740 0.28425 0.42220 0.27775 0.70143
LDF 100 0.34709 0.18386 0.42836 0.28545 0.42140 0.27676 0.70217
INFLO 95 0.32282 0.15352 0.28137 0.10171 0.40595 0.25744 0.63060
INFLO 99 0.31553 0.14442 0.28302 0.10378 0.40710 0.25888 0.63262
INFLO 100 0.31796 0.14745 0.28318 0.10397 0.40604 0.25754 0.63113
COF 19 0.29126 0.11408 0.28077 0.10096 0.34690 0.18363 0.57403
COF 22 0.29854 0.12318 0.28401 0.10501 0.34590 0.18238 0.57892
COF 43 0.31553 0.14442 0.28709 0.10887 0.33333 0.16667 0.55427
COF 54 0.29369 0.11711 0.29903 0.12378 0.33347 0.16684 0.54163

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 58 0.41646 0.27085 0.41403 0.26780 0.41697 0.27148 0.64879
KNN 98 0.40920 0.26177 0.44599 0.30773 0.42356 0.27971 0.67909
KNN 100 0.41646 0.27085 0.44699 0.30899 0.42200 0.27776 0.68018
KNNW 90 0.40678 0.25874 0.39839 0.24827 0.41313 0.26668 0.62948
KNNW 98 0.40678 0.25874 0.40181 0.25253 0.41541 0.26952 0.63404
KNNW 100 0.40678 0.25874 0.40257 0.25349 0.41541 0.26952 0.63502
LOF 99 0.35835 0.19823 0.31256 0.14101 0.40660 0.25852 0.66162
LOF 100 0.36077 0.20126 0.31328 0.14191 0.40520 0.25677 0.66260
SimplifiedLOF 18 0.33172 0.16495 0.29194 0.11524 0.33843 0.17333 0.57877
SimplifiedLOF 24 0.30751 0.13470 0.29502 0.11910 0.34140 0.17705 0.58185
SimplifiedLOF 99 0.30993 0.13772 0.28364 0.10487 0.36491 0.20643 0.60373
SimplifiedLOF 100 0.30993 0.13772 0.28410 0.10545 0.36379 0.20503 0.60441
LoOP 15 0.31235 0.14075 0.27437 0.09329 0.33293 0.16647 0.56252
LoOP 24 0.30751 0.13470 0.28384 0.10512 0.33392 0.16770 0.57378
LoOP 94 0.30751 0.13470 0.27460 0.09357 0.35809 0.19790 0.59517
LoOP 100 0.30993 0.13772 0.27676 0.09628 0.35726 0.19687 0.59975
LDOF 23 0.30993 0.13772 0.27821 0.09809 0.33425 0.16811 0.56224
LDOF 24 0.30508 0.13167 0.28115 0.10177 0.33553 0.16971 0.56380
LDOF 100 0.30266 0.12865 0.26346 0.07966 0.34208 0.17790 0.56743
ODIN 93 0.30292 0.12896 0.28192 0.10272 0.38095 0.22647 0.62805
ODIN 99 0.29725 0.12188 0.28582 0.10760 0.38606 0.23286 0.63373
ODIN 100 0.29855 0.12350 0.28643 0.10836 0.38509 0.23165 0.63493
FastABOD 98 0.28814 0.11049 0.26911 0.08672 0.34278 0.17877 0.56840
FastABOD 99 0.29056 0.11352 0.26920 0.08683 0.34278 0.17877 0.56856
FastABOD 100 0.29056 0.11352 0.26930 0.08695 0.34278 0.17877 0.56873
KDEOS 16 0.24213 0.05301 0.23226 0.04067 0.34286 0.17887 0.54065
KDEOS 17 0.23729 0.04696 0.23084 0.03890 0.34611 0.18293 0.54512
KDEOS 25 0.22276 0.02880 0.22857 0.03607 0.33984 0.17509 0.55043
KDEOS 90 0.24939 0.06208 0.22317 0.02931 0.34238 0.17828 0.53851
LDF 59 0.34140 0.17705 0.31072 0.13872 0.36743 0.20958 0.62418
LDF 100 0.33656 0.17100 0.42247 0.27835 0.41585 0.27008 0.69837
INFLO 96 0.33172 0.16495 0.28184 0.10262 0.39264 0.24107 0.62165
INFLO 100 0.32688 0.15890 0.28774 0.10999 0.40657 0.25848 0.63410
COF 19 0.29782 0.12259 0.28766 0.10990 0.34661 0.18356 0.55951
COF 24 0.30024 0.12562 0.29556 0.11977 0.33974 0.17497 0.57415
COF 41 0.31719 0.14680 0.30439 0.13080 0.34010 0.17543 0.55434
COF 46 0.29056 0.11352 0.30777 0.13502 0.33800 0.17280 0.54551

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 87 0.40777 0.25971 0.36057 0.20072 0.41761 0.27201 0.68776
KNN 99 0.40777 0.25971 0.36187 0.20234 0.41880 0.27350 0.68962
KNN 100 0.40291 0.25364 0.36246 0.20308 0.41880 0.27350 0.69001
KNNW 96 0.39563 0.24454 0.34544 0.18180 0.41766 0.27207 0.67726
KNNW 97 0.39563 0.24454 0.34566 0.18208 0.41801 0.27251 0.67744
KNNW 100 0.39563 0.24454 0.34616 0.18270 0.41786 0.27233 0.67794
LOF 87 0.43932 0.29915 0.34874 0.18593 0.46318 0.32897 0.70035
LOF 99 0.44417 0.30522 0.35572 0.19465 0.46073 0.32592 0.70787
LOF 100 0.44417 0.30522 0.35615 0.19519 0.46311 0.32889 0.70834
SimplifiedLOF 98 0.38350 0.22937 0.32411 0.15514 0.43936 0.29920 0.66995
SimplifiedLOF 100 0.38592 0.23240 0.32468 0.15585 0.43750 0.29688 0.67022
LoOP 90 0.38835 0.23544 0.31457 0.14321 0.42801 0.28501 0.65208
LoOP 91 0.38592 0.23240 0.31559 0.14449 0.43083 0.28854 0.65345
LoOP 99 0.38592 0.23240 0.31873 0.14842 0.42956 0.28695 0.65472
LoOP 100 0.38592 0.23240 0.31894 0.14868 0.42915 0.28644 0.65455
LDOF 99 0.36650 0.20813 0.31084 0.13855 0.41851 0.27314 0.64844
LDOF 100 0.36408 0.20510 0.31109 0.13887 0.41870 0.27337 0.64913
ODIN 98 0.40049 0.25061 0.34364 0.17955 0.43566 0.29458 0.67869
ODIN 100 0.39979 0.24974 0.34562 0.18203 0.43744 0.29680 0.67974
FastABOD 41 0.28398 0.10498 0.26954 0.08692 0.35553 0.19441 0.58107
FastABOD 94 0.28641 0.10801 0.27390 0.09238 0.35418 0.19272 0.58528
FastABOD 99 0.28398 0.10498 0.27421 0.09276 0.35410 0.19262 0.58564
FastABOD 100 0.28398 0.10498 0.27420 0.09275 0.35391 0.19239 0.58566
KDEOS 72 0.25728 0.07160 0.25234 0.06543 0.36865 0.21081 0.59881
KDEOS 99 0.25243 0.06553 0.25764 0.07204 0.37899 0.22374 0.61202
KDEOS 100 0.25728 0.07160 0.25759 0.07198 0.38140 0.22674 0.61217
LDF 97 0.42718 0.28398 0.36966 0.21208 0.43732 0.29665 0.71614
LDF 100 0.42476 0.28095 0.37124 0.21405 0.43836 0.29795 0.71790
INFLO 71 0.39320 0.24150 0.31560 0.14450 0.42844 0.28555 0.66035
INFLO 99 0.38592 0.23240 0.32677 0.15846 0.44970 0.31213 0.67612
COF 74 0.32524 0.15655 0.29251 0.11564 0.36581 0.20726 0.58293
COF 85 0.34223 0.17779 0.29740 0.12175 0.35628 0.19536 0.59269
COF 87 0.34466 0.18083 0.29635 0.12043 0.35067 0.18834 0.59154
COF 89 0.33495 0.16869 0.29482 0.11853 0.35294 0.19118 0.59297

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.9 MB) Download raw algorithm evaluation table (75.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 97 0.40194 0.25269 0.35982 0.20006 0.41113 0.26419 0.68045
KNN 100 0.40194 0.25269 0.36036 0.20074 0.40914 0.26169 0.68079
KNNW 88 0.38499 0.23151 0.34179 0.17753 0.40335 0.25445 0.66426
KNNW 97 0.38499 0.23151 0.34369 0.17991 0.40622 0.25805 0.66641
KNNW 100 0.38499 0.23151 0.34434 0.18072 0.40614 0.25795 0.66703
LOF 98 0.43584 0.29505 0.35092 0.18894 0.45578 0.31997 0.69423
LOF 99 0.43826 0.29808 0.35169 0.18991 0.45455 0.31843 0.69482
LOF 100 0.43826 0.29808 0.35206 0.19036 0.45389 0.31760 0.69514
SimplifiedLOF 98 0.39225 0.24059 0.32321 0.15432 0.43043 0.28830 0.65984
SimplifiedLOF 99 0.39467 0.24362 0.32378 0.15503 0.43015 0.28795 0.65994
SimplifiedLOF 100 0.39467 0.24362 0.32391 0.15519 0.42872 0.28615 0.66031
LoOP 94 0.38257 0.22849 0.31679 0.14630 0.42073 0.27618 0.65036
LoOP 95 0.38499 0.23151 0.31726 0.14688 0.42010 0.27539 0.64988
LoOP 98 0.38499 0.23151 0.31854 0.14848 0.42116 0.27671 0.65033
LoOP 99 0.38499 0.23151 0.31937 0.14952 0.42065 0.27608 0.65017
LDOF 96 0.35593 0.19521 0.30837 0.13577 0.41780 0.27252 0.63977
LDOF 100 0.36320 0.20428 0.30983 0.13760 0.41296 0.26647 0.64169
ODIN 100 0.40231 0.25316 0.34360 0.17979 0.42626 0.28309 0.66926
FastABOD 50 0.29298 0.11654 0.27252 0.09098 0.35288 0.19139 0.57746
FastABOD 100 0.28571 0.10747 0.27508 0.09418 0.35539 0.19453 0.58097
KDEOS 83 0.25424 0.06813 0.24963 0.06238 0.36416 0.20549 0.59855
KDEOS 99 0.24939 0.06208 0.25131 0.06448 0.37485 0.21885 0.60423
KDEOS 100 0.25424 0.06813 0.25193 0.06525 0.37566 0.21986 0.60421
LDF 96 0.41889 0.27387 0.36433 0.20570 0.42562 0.28229 0.70088
LDF 99 0.41646 0.27085 0.36656 0.20848 0.42658 0.28349 0.70296
LDF 100 0.41404 0.26782 0.36676 0.20873 0.42612 0.28292 0.70345
INFLO 82 0.38983 0.23756 0.31346 0.14213 0.42115 0.27670 0.64998
INFLO 94 0.38257 0.22849 0.31976 0.15001 0.42899 0.28650 0.66220
INFLO 100 0.38257 0.22849 0.32051 0.15094 0.42907 0.28660 0.66109
COF 73 0.29298 0.11654 0.28610 0.10795 0.35419 0.19302 0.57956
COF 85 0.30751 0.13470 0.29166 0.11490 0.34726 0.18438 0.58448
COF 96 0.31477 0.14377 0.28692 0.10898 0.34715 0.18423 0.58951
COF 98 0.32203 0.15285 0.28941 0.11208 0.34652 0.18345 0.58915

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO