Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#07)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (72.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 85 0.42476 0.28095 0.42898 0.28622 0.43649 0.29561 0.67539
KNN 95 0.43204 0.29005 0.43453 0.29316 0.43519 0.29398 0.67988
KNN 100 0.43204 0.29005 0.43733 0.29667 0.43458 0.29323 0.68296
KNNW 74 0.41019 0.26274 0.38845 0.23556 0.42530 0.28163 0.62835
KNNW 98 0.41505 0.26881 0.39836 0.24795 0.42207 0.27759 0.64138
KNNW 100 0.41505 0.26881 0.39894 0.24867 0.42164 0.27705 0.64226
LOF 99 0.34951 0.18689 0.31234 0.14042 0.40552 0.25690 0.66492
LOF 100 0.35194 0.18993 0.31293 0.14117 0.40434 0.25543 0.66563
SimplifiedLOF 17 0.33981 0.17476 0.30501 0.13126 0.37374 0.21717 0.60636
SimplifiedLOF 22 0.35922 0.19903 0.30514 0.13142 0.36853 0.21066 0.60705
SimplifiedLOF 33 0.33252 0.16566 0.30791 0.13489 0.36032 0.20040 0.60570
SimplifiedLOF 100 0.31553 0.14442 0.29342 0.11678 0.36687 0.20859 0.61661
LoOP 19 0.33495 0.16869 0.28854 0.11067 0.34959 0.18699 0.59039
LoOP 33 0.32039 0.15049 0.29160 0.11450 0.34545 0.18182 0.59188
LoOP 81 0.31553 0.14442 0.28426 0.10532 0.36608 0.20760 0.60518
LoOP 100 0.31553 0.14442 0.28597 0.10746 0.36500 0.20625 0.61170
LDOF 22 0.31311 0.14138 0.28365 0.10457 0.34534 0.18167 0.57597
LDOF 81 0.32767 0.15959 0.28237 0.10296 0.36601 0.20752 0.59399
LDOF 82 0.32524 0.15655 0.28283 0.10353 0.36629 0.20786 0.59418
LDOF 92 0.31311 0.14138 0.28253 0.10317 0.36724 0.20905 0.59233
ODIN 87 0.31299 0.14124 0.28650 0.10813 0.37681 0.22101 0.62493
ODIN 94 0.30939 0.13674 0.28976 0.11219 0.38457 0.23071 0.63158
ODIN 100 0.30635 0.13293 0.29416 0.11770 0.38271 0.22839 0.63704
FastABOD 42 0.27427 0.09284 0.27016 0.08770 0.34719 0.18399 0.57865
FastABOD 82 0.29612 0.12015 0.27751 0.09689 0.34557 0.18196 0.58638
FastABOD 99 0.29612 0.12015 0.27964 0.09954 0.34620 0.18275 0.58871
FastABOD 100 0.29612 0.12015 0.27959 0.09949 0.34600 0.18250 0.58877
KDEOS 16 0.24515 0.05643 0.24089 0.05111 0.35009 0.18762 0.56340
KDEOS 17 0.25000 0.06250 0.23880 0.04851 0.35000 0.18750 0.56614
KDEOS 29 0.22330 0.02913 0.22833 0.03542 0.35550 0.19437 0.56068
KDEOS 99 0.25485 0.06857 0.23816 0.04771 0.34492 0.18115 0.56351
LDF 6 0.35437 0.19296 0.32726 0.15908 0.36866 0.21083 0.61028
LDF 100 0.34951 0.18689 0.38975 0.23719 0.41321 0.26651 0.69525
INFLO 98 0.33738 0.17172 0.29338 0.11673 0.41803 0.27254 0.64460
INFLO 100 0.33495 0.16869 0.29447 0.11809 0.41803 0.27254 0.64622
COF 15 0.32767 0.15959 0.30564 0.13205 0.36181 0.20226 0.59534
COF 41 0.33738 0.17172 0.30495 0.13119 0.34023 0.17528 0.56836
COF 56 0.30825 0.13532 0.31584 0.14480 0.33333 0.16667 0.55312

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 94 0.41525 0.26933 0.44658 0.30847 0.42544 0.28206 0.67803
KNN 100 0.41162 0.26479 0.44844 0.31080 0.42492 0.28141 0.68033
KNNW 85 0.40436 0.25572 0.39828 0.24813 0.41758 0.27224 0.63137
KNNW 97 0.40920 0.26177 0.40438 0.25574 0.41655 0.27095 0.63780
KNNW 100 0.40678 0.25874 0.40541 0.25703 0.41598 0.27024 0.63925
LOF 100 0.36804 0.21033 0.31634 0.14574 0.40825 0.26058 0.66670
SimplifiedLOF 18 0.33898 0.17403 0.29881 0.12383 0.35818 0.19802 0.59495
SimplifiedLOF 22 0.33656 0.17100 0.30463 0.13111 0.36364 0.20483 0.59706
SimplifiedLOF 100 0.32688 0.15890 0.28960 0.11232 0.37262 0.21606 0.61340
LoOP 15 0.32930 0.16193 0.28094 0.10150 0.34282 0.17882 0.58049
LoOP 24 0.31719 0.14680 0.29211 0.11546 0.35430 0.19316 0.58915
LoOP 100 0.32446 0.15588 0.28079 0.10132 0.36490 0.20641 0.60754
LDOF 23 0.29298 0.11654 0.28655 0.10851 0.34811 0.18544 0.57704
LDOF 84 0.30751 0.13470 0.26934 0.08701 0.35756 0.19724 0.57874
LDOF 99 0.31719 0.14680 0.27147 0.08967 0.35584 0.19509 0.57957
ODIN 93 0.31477 0.14377 0.28716 0.10928 0.38163 0.22732 0.63378
ODIN 100 0.29831 0.12320 0.29070 0.11369 0.38723 0.23432 0.63999
FastABOD 73 0.29782 0.12259 0.27764 0.09738 0.35128 0.18939 0.58875
FastABOD 87 0.29540 0.11957 0.27875 0.09876 0.35307 0.19163 0.59117
FastABOD 100 0.29540 0.11957 0.28014 0.10050 0.35161 0.18981 0.59304
KDEOS 14 0.23245 0.04090 0.23315 0.04178 0.33947 0.17464 0.54117
KDEOS 16 0.24697 0.05906 0.23302 0.04163 0.34683 0.18384 0.54963
KDEOS 24 0.23002 0.03788 0.22932 0.03699 0.35448 0.19339 0.55671
LDF 40 0.33898 0.17403 0.27254 0.09101 0.36228 0.20313 0.59347
LDF 100 0.33172 0.16495 0.41657 0.27097 0.41347 0.26711 0.69541
INFLO 96 0.33898 0.17403 0.29005 0.11289 0.40163 0.25231 0.63623
INFLO 100 0.33656 0.17100 0.29350 0.11719 0.40980 0.26252 0.64074
COF 15 0.32203 0.15285 0.30131 0.12696 0.34476 0.18124 0.58449
COF 23 0.30751 0.13470 0.30418 0.13054 0.34829 0.18565 0.59238
COF 30 0.28571 0.10747 0.30973 0.13747 0.35214 0.19047 0.59060
COF 32 0.29298 0.11654 0.31669 0.14617 0.34911 0.18668 0.59228

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 64 0.40534 0.25667 0.36023 0.20029 0.42222 0.27778 0.67638
KNN 90 0.41505 0.26881 0.36623 0.20778 0.42082 0.27603 0.68379
KNN 96 0.41262 0.26578 0.36727 0.20908 0.42174 0.27717 0.68464
KNN 100 0.41019 0.26274 0.36707 0.20884 0.42094 0.27617 0.68545
KNNW 96 0.40534 0.25667 0.35087 0.18859 0.41549 0.26937 0.67040
KNNW 100 0.40534 0.25667 0.35144 0.18930 0.41452 0.26815 0.67119
LOF 76 0.42233 0.27791 0.34074 0.17593 0.45984 0.32480 0.68596
LOF 93 0.44417 0.30522 0.35191 0.18988 0.45943 0.32429 0.69869
LOF 100 0.44417 0.30522 0.35486 0.19358 0.45829 0.32287 0.70187
SimplifiedLOF 99 0.39078 0.23847 0.32541 0.15676 0.43226 0.29033 0.66385
SimplifiedLOF 100 0.39563 0.24454 0.32598 0.15747 0.43193 0.28992 0.66414
LoOP 99 0.38835 0.23544 0.32038 0.15048 0.42613 0.28266 0.65034
LoOP 100 0.39320 0.24150 0.32101 0.15126 0.42467 0.28084 0.65048
LDOF 83 0.37136 0.21420 0.30232 0.12790 0.40219 0.25274 0.62787
LDOF 98 0.37136 0.21420 0.31241 0.14052 0.42261 0.27827 0.64110
LDOF 100 0.36650 0.20813 0.31380 0.14225 0.41873 0.27342 0.64227
ODIN 100 0.39777 0.24722 0.34097 0.17621 0.42248 0.27810 0.66848
FastABOD 43 0.29854 0.12318 0.27586 0.09483 0.35907 0.19884 0.58649
FastABOD 86 0.30583 0.13228 0.27989 0.09986 0.35736 0.19669 0.59019
FastABOD 100 0.30340 0.12925 0.28077 0.10097 0.35757 0.19696 0.59130
KDEOS 73 0.27184 0.08981 0.25432 0.06790 0.36618 0.20772 0.60271
KDEOS 94 0.26699 0.08374 0.25548 0.06935 0.37731 0.22164 0.60879
KDEOS 99 0.26699 0.08374 0.25691 0.07114 0.37517 0.21896 0.60913
KDEOS 100 0.26942 0.08677 0.25657 0.07072 0.37633 0.22041 0.60927
LDF 57 0.41505 0.26881 0.34705 0.18381 0.43743 0.29679 0.67687
LDF 65 0.42961 0.28701 0.35353 0.19191 0.42961 0.28701 0.68424
LDF 100 0.42961 0.28701 0.37396 0.21745 0.43649 0.29561 0.70880
INFLO 95 0.39320 0.24150 0.32307 0.15384 0.43440 0.29299 0.66630
INFLO 98 0.39320 0.24150 0.32447 0.15559 0.44113 0.30142 0.67098
INFLO 99 0.39320 0.24150 0.32594 0.15743 0.44065 0.30082 0.67294
COF 16 0.30583 0.13228 0.27238 0.09047 0.35365 0.19206 0.59654
COF 76 0.31068 0.13835 0.29486 0.11857 0.36484 0.20605 0.58650
COF 78 0.31311 0.14138 0.29355 0.11694 0.36116 0.20145 0.58695

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.9 MB) Download raw algorithm evaluation table (75.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 92 0.40436 0.25572 0.36060 0.20104 0.41603 0.27031 0.67809
KNN 98 0.40436 0.25572 0.36174 0.20246 0.41631 0.27065 0.67988
KNN 100 0.40436 0.25572 0.36209 0.20291 0.41612 0.27042 0.68030
KNNW 93 0.39709 0.24664 0.34632 0.18320 0.41065 0.26358 0.66575
KNNW 100 0.39709 0.24664 0.34735 0.18448 0.41096 0.26397 0.66705
LOF 84 0.43584 0.29505 0.34404 0.18035 0.45808 0.32284 0.68483
LOF 100 0.44794 0.31018 0.35264 0.19110 0.45618 0.32047 0.69540
SimplifiedLOF 96 0.39709 0.24664 0.32565 0.15737 0.42961 0.28728 0.66434
SimplifiedLOF 98 0.39467 0.24362 0.32620 0.15805 0.43203 0.29030 0.66419
SimplifiedLOF 99 0.39467 0.24362 0.32663 0.15859 0.43161 0.28977 0.66425
LoOP 97 0.39467 0.24362 0.32078 0.15129 0.42254 0.27843 0.65119
LoOP 98 0.39225 0.24059 0.32127 0.15190 0.42369 0.27988 0.65155
LoOP 99 0.39467 0.24362 0.32177 0.15252 0.42510 0.28164 0.65128
LoOP 100 0.39467 0.24362 0.32158 0.15229 0.42626 0.28309 0.65096
LDOF 91 0.37046 0.21336 0.30999 0.13781 0.41839 0.27326 0.64118
LDOF 99 0.36562 0.20731 0.31492 0.14396 0.42192 0.27766 0.64482
LDOF 100 0.36562 0.20731 0.31494 0.14398 0.41968 0.27487 0.64507
ODIN 99 0.39989 0.25013 0.34146 0.17712 0.42742 0.28453 0.66826
ODIN 100 0.40496 0.25647 0.34172 0.17745 0.42742 0.28453 0.66939
FastABOD 51 0.30024 0.12562 0.28231 0.10321 0.36441 0.20580 0.59769
FastABOD 98 0.29782 0.12259 0.28550 0.10719 0.36869 0.21115 0.60172
FastABOD 100 0.29782 0.12259 0.28569 0.10743 0.36764 0.20983 0.60194
KDEOS 99 0.26877 0.08629 0.25568 0.06993 0.37722 0.22181 0.61011
KDEOS 100 0.26150 0.07721 0.25681 0.07135 0.37849 0.22339 0.61022
LDF 14 0.35593 0.19521 0.31033 0.13822 0.42784 0.28506 0.65668
LDF 58 0.41646 0.27085 0.34129 0.17691 0.42093 0.27642 0.66605
LDF 100 0.41404 0.26782 0.36627 0.20812 0.42007 0.27535 0.70065
INFLO 98 0.39952 0.24967 0.32776 0.16000 0.44296 0.30396 0.67636
INFLO 100 0.39225 0.24059 0.32908 0.16165 0.44493 0.30642 0.68031
COF 76 0.34140 0.17705 0.30441 0.13082 0.36229 0.20315 0.60158
COF 77 0.33656 0.17100 0.30624 0.13312 0.36313 0.20421 0.60446
COF 86 0.33172 0.16495 0.30262 0.12859 0.37121 0.21430 0.60612
COF 87 0.33414 0.16798 0.29950 0.12470 0.37216 0.21549 0.60365

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO