Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#03)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (71.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 56 0.42476 0.28095 0.41384 0.26730 0.42595 0.28244 0.64976
KNN 95 0.41990 0.27488 0.44611 0.30764 0.43458 0.29323 0.67950
KNN 100 0.42233 0.27791 0.44884 0.31105 0.43285 0.29107 0.68282
KNNW 67 0.41990 0.27488 0.38931 0.23664 0.42394 0.27993 0.61975
KNNW 87 0.41748 0.27184 0.40005 0.25007 0.42607 0.28258 0.63231
KNNW 100 0.41748 0.27184 0.40566 0.25707 0.42424 0.28030 0.63910
LOF 99 0.34223 0.17779 0.30650 0.13313 0.40144 0.25180 0.66313
LOF 100 0.34223 0.17779 0.30755 0.13444 0.40170 0.25213 0.66431
SimplifiedLOF 19 0.33981 0.17476 0.29106 0.11383 0.35307 0.19134 0.59214
SimplifiedLOF 25 0.33010 0.16262 0.29547 0.11934 0.35413 0.19266 0.59450
SimplifiedLOF 100 0.30583 0.13228 0.27885 0.09856 0.36364 0.20455 0.60616
LoOP 15 0.32524 0.15655 0.27493 0.09366 0.33833 0.17292 0.57422
LoOP 28 0.31796 0.14745 0.28204 0.10255 0.35261 0.19076 0.58580
LoOP 100 0.30825 0.13532 0.27263 0.09079 0.35783 0.19728 0.60120
LDOF 21 0.31068 0.13835 0.27474 0.09342 0.34015 0.17518 0.56636
LDOF 24 0.30583 0.13228 0.27830 0.09787 0.34263 0.17829 0.56942
LDOF 25 0.30825 0.13532 0.27744 0.09680 0.34950 0.18688 0.57090
LDOF 32 0.29369 0.11711 0.27190 0.08988 0.34350 0.17938 0.57435
ODIN 99 0.29832 0.12290 0.28235 0.10294 0.38646 0.23307 0.63292
ODIN 100 0.30127 0.12659 0.28309 0.10386 0.38601 0.23251 0.63396
FastABOD 81 0.29126 0.11408 0.27010 0.08762 0.34532 0.18165 0.57618
FastABOD 86 0.29126 0.11408 0.27119 0.08898 0.34594 0.18242 0.57709
FastABOD 100 0.28883 0.11104 0.27266 0.09082 0.34505 0.18131 0.57900
KDEOS 16 0.23786 0.04733 0.24002 0.05002 0.33929 0.17411 0.55019
KDEOS 17 0.24757 0.05947 0.23553 0.04441 0.33912 0.17390 0.54929
KDEOS 23 0.22087 0.02609 0.23351 0.04189 0.34475 0.18094 0.55757
KDEOS 32 0.22573 0.03216 0.22699 0.03374 0.34800 0.18500 0.55507
LDF 8 0.34466 0.18083 0.32177 0.15221 0.34856 0.18569 0.59831
LDF 100 0.34466 0.18083 0.41893 0.27367 0.42355 0.27943 0.69953
INFLO 78 0.32767 0.15959 0.26808 0.08510 0.38223 0.22779 0.60820
INFLO 98 0.32282 0.15352 0.27799 0.09749 0.40164 0.25206 0.62634
INFLO 100 0.32039 0.15049 0.27836 0.09795 0.39836 0.24795 0.62433
COF 22 0.29126 0.11408 0.28705 0.10881 0.34437 0.18046 0.58603
COF 29 0.28398 0.10498 0.29527 0.11908 0.34893 0.18616 0.57720
COF 36 0.30097 0.12621 0.30564 0.13205 0.33824 0.17280 0.57542
COF 48 0.31311 0.14138 0.30070 0.12587 0.33806 0.17258 0.55347

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 68 0.40920 0.26177 0.40133 0.25193 0.41114 0.26419 0.65330
KNN 98 0.40920 0.26177 0.41969 0.27488 0.41896 0.27397 0.67165
KNN 100 0.40920 0.26177 0.42097 0.27648 0.41743 0.27205 0.67269
KNNW 91 0.39467 0.24362 0.37934 0.22446 0.40890 0.26139 0.62613
KNNW 94 0.39709 0.24664 0.38037 0.22574 0.40833 0.26068 0.62768
KNNW 100 0.39709 0.24664 0.38283 0.22882 0.40789 0.26014 0.63070
LOF 99 0.34625 0.18310 0.30207 0.12790 0.39559 0.24476 0.66047
LOF 100 0.34625 0.18310 0.30294 0.12899 0.39738 0.24699 0.66160
SimplifiedLOF 19 0.33656 0.17100 0.29625 0.12063 0.34612 0.18295 0.58903
SimplifiedLOF 22 0.34140 0.17705 0.29454 0.11849 0.34924 0.18685 0.58933
SimplifiedLOF 89 0.30508 0.13167 0.27885 0.09889 0.36723 0.20933 0.60580
SimplifiedLOF 100 0.30993 0.13772 0.28173 0.10248 0.36549 0.20715 0.61185
LoOP 12 0.32203 0.15285 0.27963 0.09987 0.33293 0.16647 0.57135
LoOP 17 0.31719 0.14680 0.28346 0.10465 0.34089 0.17641 0.57691
LoOP 100 0.31235 0.14075 0.27545 0.09464 0.36602 0.20781 0.60734
LDOF 20 0.31235 0.14075 0.27426 0.09316 0.33641 0.17081 0.55760
LDOF 24 0.29782 0.12259 0.27686 0.09640 0.33606 0.17038 0.56311
LDOF 84 0.30266 0.12865 0.26726 0.08441 0.35568 0.19489 0.58178
LDOF 86 0.30751 0.13470 0.26757 0.08479 0.35215 0.19048 0.58253
ODIN 86 0.30470 0.13119 0.27831 0.09821 0.37493 0.21894 0.62231
ODIN 100 0.29621 0.12058 0.28411 0.10546 0.38407 0.23037 0.63397
FastABOD 75 0.29540 0.11957 0.27617 0.09555 0.34913 0.18670 0.58621
FastABOD 82 0.29782 0.12259 0.27730 0.09696 0.34772 0.18494 0.58734
FastABOD 100 0.29540 0.11957 0.27914 0.09926 0.34799 0.18528 0.58968
KDEOS 11 0.25182 0.06511 0.23633 0.04576 0.33564 0.16985 0.54009
KDEOS 16 0.25182 0.06511 0.24097 0.05156 0.34402 0.18032 0.55650
KDEOS 21 0.25182 0.06511 0.23655 0.04603 0.34149 0.17716 0.56120
KDEOS 26 0.23245 0.04090 0.23442 0.04337 0.34811 0.18543 0.56018
LDF 95 0.33414 0.16798 0.36662 0.20856 0.41160 0.26476 0.68250
LDF 99 0.33172 0.16495 0.37692 0.22143 0.41379 0.26751 0.68861
LDF 100 0.33414 0.16798 0.38056 0.22598 0.41325 0.26682 0.68975
INFLO 85 0.32688 0.15890 0.27510 0.09420 0.40027 0.25061 0.62600
INFLO 100 0.32688 0.15890 0.28101 0.10159 0.40574 0.25744 0.63074
COF 14 0.30508 0.13167 0.28948 0.11217 0.34082 0.17632 0.58228
COF 21 0.27603 0.09536 0.28485 0.10638 0.34806 0.18537 0.57526
COF 39 0.31477 0.14377 0.29744 0.12212 0.33876 0.17375 0.56538
COF 54 0.30024 0.12562 0.29927 0.12441 0.33403 0.16784 0.55115

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 85 0.41019 0.26274 0.36298 0.20372 0.41818 0.27273 0.68249
KNN 96 0.41019 0.26274 0.36459 0.20574 0.41845 0.27307 0.68490
KNN 99 0.40777 0.25971 0.36488 0.20610 0.41530 0.26913 0.68517
KNN 100 0.40534 0.25667 0.36488 0.20609 0.41749 0.27186 0.68548
KNNW 86 0.39806 0.24757 0.34628 0.18284 0.40660 0.25825 0.66811
KNNW 96 0.39806 0.24757 0.34856 0.18570 0.40951 0.26188 0.67044
KNNW 100 0.39806 0.24757 0.34936 0.18670 0.40930 0.26163 0.67138
LOF 95 0.43932 0.29915 0.34998 0.18748 0.45283 0.31604 0.69742
LOF 97 0.43932 0.29915 0.35080 0.18851 0.45367 0.31708 0.69833
LOF 100 0.43932 0.29915 0.35209 0.19011 0.45130 0.31413 0.69996
SimplifiedLOF 99 0.39078 0.23847 0.32422 0.15527 0.43061 0.28827 0.66398
SimplifiedLOF 100 0.39320 0.24150 0.32471 0.15589 0.42944 0.28679 0.66434
LoOP 92 0.39078 0.23847 0.31677 0.14596 0.41543 0.26929 0.65052
LoOP 100 0.38835 0.23544 0.32014 0.15017 0.41901 0.27376 0.64999
LDOF 83 0.36408 0.20510 0.30176 0.12720 0.39937 0.24922 0.63071
LDOF 100 0.35922 0.19903 0.31330 0.14162 0.41472 0.26840 0.64439
ODIN 100 0.40240 0.25300 0.34089 0.17612 0.42549 0.28186 0.67495
FastABOD 15 0.25728 0.07160 0.27100 0.08875 0.35770 0.19713 0.57711
FastABOD 92 0.28641 0.10801 0.27479 0.09349 0.35509 0.19386 0.58215
FastABOD 100 0.28641 0.10801 0.27566 0.09457 0.35532 0.19415 0.58284
KDEOS 96 0.25243 0.06553 0.25453 0.06816 0.38070 0.22588 0.60692
KDEOS 99 0.25485 0.06857 0.25535 0.06919 0.37975 0.22468 0.60803
KDEOS 100 0.25971 0.07464 0.25515 0.06894 0.38053 0.22566 0.60824
LDF 98 0.42476 0.28095 0.36782 0.20978 0.42527 0.28159 0.70784
LDF 100 0.41990 0.27488 0.36831 0.21038 0.42548 0.28185 0.70917
INFLO 95 0.38350 0.22937 0.32202 0.15252 0.43653 0.29566 0.66398
INFLO 99 0.37864 0.22330 0.32540 0.15675 0.44527 0.30658 0.67112
INFLO 100 0.37864 0.22330 0.32617 0.15771 0.44527 0.30658 0.67338
COF 76 0.30825 0.13532 0.29547 0.11934 0.36755 0.20944 0.59988
COF 83 0.31311 0.14138 0.30071 0.12589 0.36196 0.20245 0.60614
COF 90 0.32282 0.15352 0.29595 0.11994 0.35692 0.19615 0.60646
COF 99 0.32039 0.15049 0.29576 0.11971 0.35900 0.19875 0.60887

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.9 MB) Download raw algorithm evaluation table (75.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 92 0.39709 0.24664 0.35590 0.19517 0.40741 0.25953 0.67446
KNN 95 0.40194 0.25269 0.35632 0.19569 0.40700 0.25902 0.67537
KNN 100 0.40194 0.25269 0.35701 0.19656 0.40693 0.25893 0.67619
KNNW 89 0.38741 0.23454 0.34084 0.17635 0.40207 0.25286 0.66084
KNNW 95 0.39225 0.24059 0.34180 0.17755 0.40104 0.25157 0.66207
KNNW 100 0.39225 0.24059 0.34258 0.17852 0.40000 0.25027 0.66296
LOF 88 0.42857 0.28597 0.33829 0.17316 0.44318 0.30423 0.68285
LOF 96 0.42615 0.28295 0.34333 0.17946 0.44728 0.30936 0.68913
LOF 100 0.42857 0.28597 0.34518 0.18177 0.44562 0.30727 0.69130
SimplifiedLOF 99 0.38983 0.23756 0.31932 0.14946 0.41896 0.27396 0.65667
SimplifiedLOF 100 0.39225 0.24059 0.31974 0.14999 0.41968 0.27486 0.65661
LoOP 91 0.38257 0.22849 0.31115 0.13925 0.41317 0.26673 0.64286
LoOP 98 0.38015 0.22546 0.31404 0.14286 0.41096 0.26397 0.64375
LoOP 100 0.38257 0.22849 0.31488 0.14391 0.41398 0.26774 0.64290
LDOF 94 0.36562 0.20731 0.30529 0.13193 0.40812 0.26042 0.63602
LDOF 99 0.36562 0.20731 0.30765 0.13488 0.41118 0.26424 0.63799
LDOF 100 0.36562 0.20731 0.30821 0.13557 0.41020 0.26302 0.63794
ODIN 100 0.39168 0.23988 0.33544 0.16961 0.41973 0.27492 0.66487
FastABOD 94 0.29782 0.12259 0.29304 0.11662 0.37129 0.21440 0.60859
FastABOD 99 0.29782 0.12259 0.29330 0.11694 0.37094 0.21396 0.60885
FastABOD 100 0.29782 0.12259 0.29326 0.11690 0.37116 0.21423 0.60889
KDEOS 88 0.27603 0.09536 0.25726 0.07192 0.37328 0.21688 0.60630
KDEOS 97 0.26877 0.08629 0.25799 0.07282 0.37509 0.21914 0.60888
KDEOS 100 0.26877 0.08629 0.25942 0.07462 0.37444 0.21833 0.60936
LDF 92 0.40678 0.25874 0.35685 0.19635 0.40876 0.26122 0.69528
LDF 100 0.40436 0.25572 0.36200 0.20279 0.41522 0.26930 0.70137
INFLO 95 0.38741 0.23454 0.31773 0.14747 0.43125 0.28933 0.66042
INFLO 98 0.38499 0.23151 0.31929 0.14942 0.43485 0.29382 0.66399
INFLO 99 0.38499 0.23151 0.31996 0.15026 0.43699 0.29649 0.66353
INFLO 100 0.38257 0.22849 0.32040 0.15081 0.43504 0.29406 0.66344
COF 82 0.34625 0.18310 0.30328 0.12941 0.35932 0.19944 0.59558
COF 89 0.35109 0.18916 0.29826 0.12314 0.36364 0.20483 0.59550
COF 91 0.34140 0.17705 0.29805 0.12288 0.36566 0.20737 0.59677
COF 100 0.34867 0.18613 0.30032 0.12572 0.36417 0.20550 0.59960

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO