Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (20% of outliers version#02)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 305 objects, 61 outliers (20.00%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (52.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 17 0.59016 0.48770 0.69748 0.62185 0.61261 0.51577 0.85347
KNN 23 0.59016 0.48770 0.69697 0.62121 0.62857 0.53571 0.85071
KNN 84 0.60656 0.50820 0.70208 0.62759 0.61682 0.52103 0.85078
KNN 90 0.60656 0.50820 0.70313 0.62891 0.61871 0.52338 0.85219
KNNW 5 0.59016 0.48770 0.67082 0.58853 0.60606 0.50758 0.84285
KNNW 81 0.59016 0.48770 0.69662 0.62078 0.61947 0.52434 0.84957
KNNW 95 0.59016 0.48770 0.69789 0.62236 0.61682 0.52103 0.85031
KNNW 96 0.59016 0.48770 0.69818 0.62272 0.61682 0.52103 0.85024
LOF 31 0.59016 0.48770 0.66072 0.57590 0.63380 0.54225 0.84520
LOF 39 0.60656 0.50820 0.66897 0.58621 0.62937 0.53671 0.84359
LOF 60 0.59016 0.48770 0.68220 0.60275 0.63014 0.53767 0.84802
LOF 83 0.59016 0.48770 0.68796 0.60995 0.61745 0.52181 0.84802
SimplifiedLOF 44 0.60656 0.50820 0.65896 0.57371 0.61429 0.51786 0.84258
SimplifiedLOF 86 0.57377 0.46721 0.68165 0.60206 0.64789 0.55986 0.84910
SimplifiedLOF 93 0.59016 0.48770 0.68655 0.60819 0.64336 0.55420 0.85044
SimplifiedLOF 94 0.59016 0.48770 0.68578 0.60723 0.64336 0.55420 0.85051
LoOP 57 0.60656 0.50820 0.66455 0.58069 0.62857 0.53571 0.84420
LoOP 86 0.57377 0.46721 0.67703 0.59629 0.64789 0.55986 0.84796
LoOP 92 0.57377 0.46721 0.68359 0.60449 0.64789 0.55986 0.84970
LoOP 93 0.57377 0.46721 0.68370 0.60463 0.64789 0.55986 0.84970
LDOF 87 0.59016 0.48770 0.66533 0.58166 0.64286 0.55357 0.84561
LDOF 90 0.59016 0.48770 0.66790 0.58487 0.64286 0.55357 0.84682
LDOF 98 0.59016 0.48770 0.66831 0.58539 0.62937 0.53671 0.84473
LDOF 99 0.60656 0.50820 0.66651 0.58314 0.63448 0.54310 0.84446
ODIN 96 0.57923 0.47404 0.57905 0.47382 0.62500 0.53125 0.83442
ODIN 99 0.59016 0.48770 0.59433 0.49292 0.61972 0.52465 0.83472
ODIN 100 0.59016 0.48770 0.59312 0.49140 0.61972 0.52465 0.83492
FastABOD 40 0.59016 0.48770 0.63112 0.53890 0.61429 0.51786 0.83633
FastABOD 62 0.57377 0.46721 0.64265 0.55331 0.62319 0.52899 0.83822
FastABOD 67 0.55738 0.44672 0.64758 0.55947 0.61871 0.52338 0.83875
FastABOD 99 0.57377 0.46721 0.65442 0.56802 0.61111 0.51389 0.83875
KDEOS 98 0.34426 0.18033 0.34902 0.18628 0.52514 0.40642 0.75410
KDEOS 99 0.36066 0.20082 0.34898 0.18622 0.53191 0.41489 0.75410
LDF 58 0.54098 0.42623 0.66233 0.57792 0.58824 0.48529 0.84440
LDF 61 0.55738 0.44672 0.66195 0.57743 0.61765 0.52206 0.83983
LDF 62 0.57377 0.46721 0.66001 0.57502 0.60294 0.50368 0.83002
LDF 68 0.57377 0.46721 0.67890 0.59863 0.59829 0.49786 0.83069
INFLO 69 0.60656 0.50820 0.68044 0.60055 0.65248 0.56560 0.84903
INFLO 92 0.60656 0.50820 0.68524 0.60655 0.64336 0.55420 0.84897
INFLO 93 0.62295 0.52869 0.68460 0.60575 0.64336 0.55420 0.84829
INFLO 100 0.62295 0.52869 0.68369 0.60461 0.64336 0.55420 0.85239
COF 6 0.52459 0.40574 0.61071 0.51339 0.56410 0.45513 0.82800
COF 35 0.55738 0.44672 0.61192 0.51490 0.57746 0.47183 0.80576
COF 40 0.55738 0.44672 0.63143 0.53929 0.58667 0.48333 0.80892

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 305 objects, 61 outliers (20.00%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (52.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.62295 0.52869 0.66110 0.57638 0.64567 0.55709 0.85357
KNN 2 0.63934 0.54918 0.67239 0.59049 0.63934 0.54918 0.85266
KNN 3 0.63934 0.54918 0.67510 0.59388 0.65600 0.57000 0.85273
KNNW 3 0.63934 0.54918 0.66216 0.57770 0.64000 0.55000 0.85683
KNNW 7 0.63934 0.54918 0.67225 0.59031 0.65152 0.56439 0.85548
KNNW 10 0.63934 0.54918 0.67476 0.59345 0.65152 0.56439 0.85508
LOF 15 0.59016 0.48770 0.64708 0.55885 0.64706 0.55882 0.86200
LOF 20 0.62295 0.52869 0.64844 0.56055 0.64662 0.55827 0.86032
LOF 44 0.60656 0.50820 0.64828 0.56035 0.66667 0.58333 0.84205
LOF 79 0.62295 0.52869 0.65601 0.57002 0.64407 0.55508 0.83936
SimplifiedLOF 18 0.60656 0.50820 0.64844 0.56055 0.65693 0.57117 0.86576
SimplifiedLOF 43 0.62295 0.52869 0.65015 0.56269 0.66667 0.58333 0.85938
SimplifiedLOF 49 0.60656 0.50820 0.65522 0.56902 0.67647 0.59559 0.85770
SimplifiedLOF 64 0.62295 0.52869 0.66114 0.57642 0.66667 0.58333 0.85669
LoOP 23 0.60656 0.50820 0.65238 0.56548 0.65714 0.57143 0.86314
LoOP 32 0.60656 0.50820 0.64379 0.55474 0.67164 0.58955 0.85783
LoOP 52 0.62295 0.52869 0.64652 0.55815 0.66667 0.58333 0.85528
LoOP 88 0.62295 0.52869 0.65993 0.57491 0.66667 0.58333 0.85363
LDOF 24 0.57377 0.46721 0.64809 0.56012 0.66667 0.58333 0.86838
LDOF 43 0.60656 0.50820 0.64744 0.55930 0.68657 0.60821 0.86180
LDOF 63 0.62295 0.52869 0.65944 0.57430 0.68657 0.60821 0.85898
LDOF 73 0.63934 0.54918 0.65356 0.56695 0.67647 0.59559 0.85696
ODIN 38 0.57377 0.46721 0.58020 0.47525 0.62162 0.52703 0.84749
ODIN 48 0.61639 0.52049 0.59794 0.49743 0.63514 0.54392 0.84285
ODIN 88 0.57923 0.47404 0.61362 0.51702 0.64748 0.55935 0.83543
ODIN 93 0.58470 0.48087 0.61970 0.52462 0.64286 0.55357 0.83590
FastABOD 6 0.60656 0.50820 0.63163 0.53954 0.65672 0.57090 0.86623
FastABOD 10 0.62295 0.52869 0.62177 0.52721 0.63492 0.54365 0.85891
FastABOD 59 0.62295 0.52869 0.63813 0.54767 0.64234 0.55292 0.84802
KDEOS 11 0.39344 0.24180 0.46198 0.32747 0.46789 0.33486 0.74321
KDEOS 49 0.47541 0.34426 0.38914 0.23642 0.54023 0.42529 0.78709
KDEOS 83 0.40984 0.26230 0.40639 0.25799 0.60494 0.50617 0.79851
KDEOS 89 0.47541 0.34426 0.41728 0.27160 0.59494 0.49367 0.80355
LDF 74 0.47541 0.34426 0.49627 0.37033 0.49624 0.37030 0.75423
LDF 100 0.42623 0.28279 0.53300 0.41625 0.54545 0.43182 0.75927
INFLO 22 0.59016 0.48770 0.63840 0.54800 0.67606 0.59507 0.85736
INFLO 44 0.63934 0.54918 0.64965 0.56207 0.64336 0.55420 0.86516
INFLO 51 0.62295 0.52869 0.65309 0.56637 0.64122 0.55153 0.86764
INFLO 84 0.62295 0.52869 0.66481 0.58101 0.63309 0.54137 0.86301
COF 10 0.52459 0.40574 0.61382 0.51728 0.55944 0.44930 0.83183
COF 50 0.59016 0.48770 0.59101 0.48876 0.59016 0.48770 0.78581
COF 77 0.59016 0.48770 0.60151 0.50188 0.60504 0.50630 0.79871
COF 89 0.55738 0.44672 0.61914 0.52392 0.57627 0.47034 0.79448

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO