Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#02)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 89 0.43204 0.29005 0.42727 0.28409 0.43256 0.29070 0.67640
KNN 97 0.42718 0.28398 0.43140 0.28925 0.43506 0.29383 0.67951
KNN 100 0.42718 0.28398 0.43314 0.29142 0.43431 0.29289 0.68141
KNNW 86 0.40534 0.25667 0.38417 0.23021 0.42308 0.27885 0.63170
KNNW 99 0.41019 0.26274 0.38960 0.23700 0.42207 0.27759 0.63853
KNNW 100 0.41019 0.26274 0.39013 0.23766 0.42207 0.27759 0.63902
LOF 99 0.33981 0.17476 0.30058 0.12572 0.40563 0.25704 0.66316
LOF 100 0.33981 0.17476 0.30251 0.12814 0.40728 0.25911 0.66398
SimplifiedLOF 15 0.32282 0.15352 0.28416 0.10520 0.33964 0.17455 0.57135
SimplifiedLOF 18 0.32039 0.15049 0.28570 0.10713 0.33937 0.17421 0.57741
SimplifiedLOF 98 0.30340 0.12925 0.27863 0.09829 0.36856 0.21070 0.60871
SimplifiedLOF 100 0.30097 0.12621 0.27942 0.09928 0.36846 0.21057 0.60974
LoOP 16 0.32282 0.15352 0.27393 0.09242 0.33333 0.16667 0.56320
LoOP 92 0.31068 0.13835 0.27017 0.08771 0.35751 0.19689 0.59924
LoOP 100 0.30825 0.13532 0.27236 0.09045 0.35719 0.19649 0.60405
LDOF 18 0.28641 0.10801 0.27304 0.09130 0.33387 0.16734 0.55762
LDOF 81 0.30583 0.13228 0.26534 0.08168 0.35394 0.19242 0.57980
LDOF 85 0.29612 0.12015 0.26525 0.08157 0.36215 0.20268 0.57991
LDOF 100 0.30340 0.12925 0.26760 0.08450 0.35868 0.19835 0.58094
ODIN 82 0.30751 0.13438 0.27274 0.09092 0.37464 0.21830 0.61651
ODIN 96 0.29885 0.12356 0.27948 0.09935 0.38737 0.23421 0.62867
ODIN 100 0.29722 0.12152 0.28178 0.10223 0.38554 0.23193 0.63214
FastABOD 86 0.27913 0.09891 0.26632 0.08290 0.35224 0.19029 0.58421
FastABOD 89 0.27427 0.09284 0.26660 0.08325 0.35320 0.19150 0.58468
FastABOD 100 0.27913 0.09891 0.26841 0.08552 0.35204 0.19005 0.58639
KDEOS 12 0.22573 0.03216 0.23366 0.04207 0.33580 0.16975 0.53251
KDEOS 19 0.25000 0.06250 0.22910 0.03637 0.33719 0.17149 0.54974
KDEOS 58 0.20874 0.01092 0.21738 0.02173 0.34547 0.18184 0.54107
KDEOS 100 0.24029 0.05036 0.23004 0.03755 0.34480 0.18100 0.55173
LDF 100 0.35437 0.19296 0.38986 0.23732 0.41970 0.27463 0.69687
INFLO 94 0.32039 0.15049 0.27656 0.09570 0.40648 0.25810 0.62736
INFLO 98 0.31553 0.14442 0.28048 0.10059 0.41701 0.27126 0.63477
INFLO 100 0.32039 0.15049 0.28073 0.10091 0.41417 0.26771 0.63292
COF 20 0.26699 0.08374 0.27592 0.09490 0.35375 0.19219 0.56687
COF 34 0.27184 0.08981 0.28731 0.10914 0.34314 0.17892 0.57676
COF 38 0.30583 0.13228 0.29191 0.11488 0.34334 0.17917 0.57246
COF 82 0.28398 0.10498 0.30138 0.12673 0.33333 0.16667 0.54294

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 99 0.42131 0.27690 0.42850 0.28588 0.43516 0.29421 0.67963
KNN 100 0.42615 0.28295 0.42915 0.28669 0.43421 0.29302 0.68017
KNNW 80 0.40920 0.26177 0.37908 0.22413 0.41545 0.26957 0.62319
KNNW 94 0.40920 0.26177 0.38524 0.23182 0.42149 0.27712 0.63166
KNNW 100 0.40920 0.26177 0.38756 0.23473 0.42033 0.27567 0.63482
LOF 97 0.34140 0.17705 0.30359 0.12981 0.39641 0.24579 0.65579
LOF 100 0.34140 0.17705 0.30613 0.13298 0.39824 0.24807 0.65899
SimplifiedLOF 32 0.30266 0.12865 0.28935 0.11200 0.35120 0.18930 0.58870
SimplifiedLOF 97 0.32688 0.15890 0.28472 0.10622 0.36243 0.20332 0.60412
SimplifiedLOF 100 0.31961 0.14982 0.28525 0.10689 0.36302 0.20407 0.60530
LoOP 57 0.32203 0.15285 0.27470 0.09370 0.34270 0.17868 0.57967
LoOP 82 0.31477 0.14377 0.27556 0.09478 0.35595 0.19522 0.59156
LoOP 100 0.31961 0.14982 0.27867 0.09867 0.35474 0.19371 0.59989
LDOF 96 0.30266 0.12865 0.27165 0.08989 0.35968 0.19989 0.57835
LDOF 97 0.30508 0.13167 0.27234 0.09075 0.35798 0.19776 0.57861
LDOF 100 0.30266 0.12865 0.27241 0.09084 0.35502 0.19407 0.57771
ODIN 88 0.30730 0.13444 0.27962 0.09985 0.37173 0.21495 0.61769
ODIN 96 0.30024 0.12562 0.28446 0.10590 0.37757 0.22225 0.62419
ODIN 100 0.29891 0.12396 0.28638 0.10830 0.37576 0.21998 0.62762
FastABOD 68 0.27845 0.09839 0.26067 0.07617 0.34361 0.17981 0.57042
FastABOD 88 0.28329 0.10444 0.26375 0.08003 0.34327 0.17938 0.57426
FastABOD 100 0.28087 0.10142 0.26459 0.08107 0.34254 0.17847 0.57586
KDEOS 28 0.21308 0.01670 0.22132 0.02700 0.34513 0.18171 0.54602
KDEOS 80 0.24213 0.05301 0.22727 0.03444 0.33991 0.17518 0.54679
KDEOS 98 0.23729 0.04696 0.23250 0.04098 0.34152 0.17720 0.55124
KDEOS 99 0.24213 0.05301 0.23185 0.04016 0.34152 0.17720 0.55156
LDF 97 0.33656 0.17100 0.36932 0.21193 0.41584 0.27007 0.68948
LDF 100 0.33656 0.17100 0.37503 0.21907 0.41696 0.27147 0.69278
INFLO 98 0.33414 0.16798 0.28117 0.10178 0.39394 0.24270 0.61991
INFLO 99 0.32930 0.16193 0.28311 0.10422 0.39889 0.24889 0.62327
COF 16 0.30508 0.13167 0.27805 0.09788 0.34378 0.18002 0.56542
COF 22 0.29298 0.11654 0.27351 0.09222 0.34786 0.18513 0.56671
COF 24 0.30266 0.12865 0.27518 0.09430 0.34436 0.18075 0.56827
COF 55 0.28571 0.10747 0.30345 0.12963 0.33347 0.16714 0.53723

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (74.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 81 0.39320 0.24150 0.35656 0.19570 0.41768 0.27210 0.68165
KNN 93 0.40777 0.25971 0.35805 0.19756 0.41640 0.27050 0.68393
KNN 100 0.40049 0.25061 0.35964 0.19954 0.41621 0.27026 0.68586
KNNW 70 0.37621 0.22027 0.33499 0.16874 0.41039 0.26299 0.66389
KNNW 99 0.39320 0.24150 0.34256 0.17820 0.40816 0.26020 0.67139
KNNW 100 0.39320 0.24150 0.34276 0.17844 0.40816 0.26020 0.67157
LOF 96 0.44175 0.30218 0.34694 0.18367 0.45867 0.32334 0.69793
LOF 100 0.44175 0.30218 0.34847 0.18559 0.46328 0.32910 0.70012
SimplifiedLOF 99 0.39078 0.23847 0.31752 0.14689 0.43260 0.29074 0.66308
LoOP 96 0.37379 0.21723 0.30946 0.13683 0.42594 0.28243 0.64865
LoOP 97 0.38107 0.22633 0.31001 0.13751 0.42442 0.28052 0.64946
LoOP 99 0.38107 0.22633 0.31079 0.13849 0.42308 0.27885 0.64953
LoOP 100 0.37621 0.22027 0.31091 0.13863 0.42186 0.27733 0.64891
LDOF 87 0.35922 0.19903 0.29471 0.11839 0.40195 0.25244 0.63312
LDOF 98 0.35194 0.18993 0.30181 0.12726 0.40978 0.26223 0.64148
LDOF 100 0.35680 0.19600 0.30215 0.12769 0.40886 0.26107 0.64191
ODIN 100 0.39229 0.24037 0.33396 0.16745 0.42985 0.28731 0.67279
FastABOD 62 0.29126 0.11408 0.27249 0.09061 0.35673 0.19591 0.58752
FastABOD 100 0.28883 0.11104 0.27406 0.09258 0.35917 0.19897 0.59016
KDEOS 86 0.25485 0.06857 0.24854 0.06068 0.37191 0.21489 0.60059
KDEOS 100 0.23786 0.04733 0.25059 0.06324 0.37831 0.22289 0.60441
LDF 54 0.39806 0.24757 0.33190 0.16487 0.42411 0.28014 0.66720
LDF 92 0.41748 0.27184 0.35679 0.19599 0.41955 0.27443 0.70084
LDF 100 0.41262 0.26578 0.36211 0.20264 0.42334 0.27918 0.70762
INFLO 90 0.38835 0.23544 0.31715 0.14644 0.44248 0.30310 0.66734
INFLO 98 0.38592 0.23240 0.32200 0.15251 0.44937 0.31171 0.67676
COF 81 0.33981 0.17476 0.30298 0.12872 0.36381 0.20476 0.60461
COF 88 0.35194 0.18993 0.30285 0.12856 0.35714 0.19643 0.60857
COF 96 0.32767 0.15959 0.29675 0.12094 0.37066 0.21333 0.60808
COF 100 0.33010 0.16262 0.29815 0.12269 0.36598 0.20748 0.61162

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (75.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 92 0.40436 0.25572 0.35423 0.19308 0.41901 0.27402 0.67816
KNN 93 0.40678 0.25874 0.35461 0.19355 0.41855 0.27346 0.67833
KNN 99 0.40436 0.25572 0.35536 0.19450 0.41722 0.27179 0.67963
KNN 100 0.40436 0.25572 0.35546 0.19462 0.41676 0.27121 0.67954
KNNW 99 0.39225 0.24059 0.33814 0.17298 0.40646 0.25835 0.66317
KNNW 100 0.39225 0.24059 0.33842 0.17333 0.40782 0.26004 0.66344
LOF 95 0.43584 0.29505 0.34193 0.17771 0.45073 0.31367 0.68518
LOF 100 0.43341 0.29202 0.34453 0.18096 0.45279 0.31623 0.68877
SimplifiedLOF 100 0.39225 0.24059 0.31742 0.14708 0.42593 0.28267 0.65479
LoOP 96 0.38499 0.23151 0.31411 0.14295 0.41433 0.26817 0.64302
LoOP 99 0.39225 0.24059 0.31535 0.14450 0.41582 0.27004 0.64272
LoOP 100 0.39225 0.24059 0.31560 0.14482 0.41443 0.26831 0.64223
LDOF 89 0.36804 0.21033 0.30581 0.13258 0.40858 0.26100 0.63326
LDOF 100 0.36562 0.20731 0.31058 0.13853 0.41419 0.26800 0.63969
ODIN 100 0.39758 0.24725 0.34327 0.17939 0.42915 0.28670 0.66578
FastABOD 29 0.29056 0.11352 0.27092 0.08898 0.35449 0.19340 0.58222
FastABOD 99 0.28814 0.11049 0.27541 0.09459 0.35733 0.19695 0.58662
FastABOD 100 0.28814 0.11049 0.27560 0.09482 0.35686 0.19637 0.58667
KDEOS 70 0.25666 0.07116 0.25157 0.06480 0.36305 0.20410 0.59083
KDEOS 100 0.24939 0.06208 0.25430 0.06822 0.37533 0.21945 0.60299
LDF 91 0.40436 0.25572 0.34747 0.18464 0.41983 0.27505 0.68737
LDF 92 0.40920 0.26177 0.34847 0.18588 0.41915 0.27420 0.68870
LDF 100 0.40920 0.26177 0.35407 0.19288 0.41915 0.27420 0.69546
INFLO 96 0.39225 0.24059 0.31918 0.14929 0.43102 0.28903 0.66274
INFLO 97 0.38983 0.23756 0.31923 0.14935 0.43199 0.29024 0.66115
INFLO 100 0.39225 0.24059 0.31996 0.15026 0.43006 0.28783 0.66299
COF 48 0.28814 0.11049 0.28118 0.10180 0.34473 0.18121 0.57233
COF 84 0.30993 0.13772 0.28016 0.10053 0.34872 0.18619 0.57540
COF 91 0.32446 0.15588 0.27532 0.09447 0.34638 0.18327 0.57675
COF 95 0.31235 0.14075 0.27332 0.09198 0.34704 0.18410 0.57750

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO