Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (10% of outliers version#04)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (45.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.48148 0.42410 0.53905 0.48804 0.56410 0.51587 0.77353
KNN 48 0.48148 0.42410 0.56233 0.51390 0.58537 0.53948 0.78339
KNN 88 0.48148 0.42410 0.56512 0.51700 0.57143 0.52400 0.78825
KNN 96 0.48148 0.42410 0.56600 0.51797 0.57143 0.52400 0.78734
KNNW 1 0.44444 0.38297 0.54158 0.49085 0.57895 0.53236 0.76154
KNNW 9 0.48148 0.42410 0.54358 0.49307 0.56410 0.51587 0.77717
KNNW 58 0.48148 0.42410 0.55973 0.51101 0.56410 0.51587 0.78203
KNNW 94 0.48148 0.42410 0.55871 0.50988 0.57143 0.52400 0.78370
LOF 2 0.44444 0.38297 0.45858 0.39867 0.47059 0.41201 0.78522
LOF 3 0.48148 0.42410 0.47693 0.41905 0.48148 0.42410 0.76776
LOF 93 0.48148 0.42410 0.55840 0.50954 0.57143 0.52400 0.77975
LOF 96 0.48148 0.42410 0.56014 0.51147 0.57143 0.52400 0.78112
SimplifiedLOF 5 0.48148 0.42410 0.51479 0.46110 0.54545 0.49516 0.74909
SimplifiedLOF 42 0.44444 0.38297 0.54624 0.49602 0.56410 0.51587 0.78005
SimplifiedLOF 76 0.44444 0.38297 0.55353 0.50413 0.56410 0.51587 0.78522
SimplifiedLOF 88 0.48148 0.42410 0.54939 0.49952 0.56410 0.51587 0.78613
LoOP 4 0.48148 0.42410 0.48893 0.43238 0.52174 0.46882 0.75607
LoOP 43 0.44444 0.38297 0.54118 0.49041 0.56410 0.51587 0.77687
LoOP 79 0.44444 0.38297 0.55251 0.50299 0.56410 0.51587 0.78355
LoOP 86 0.44444 0.38297 0.54867 0.49873 0.56410 0.51587 0.78666
LDOF 5 0.48148 0.42410 0.43474 0.37219 0.50980 0.45556 0.74499
LDOF 93 0.44444 0.38297 0.55046 0.50072 0.57143 0.52400 0.78522
LDOF 100 0.44444 0.38297 0.55205 0.50249 0.56410 0.51587 0.78810
ODIN 74 0.44444 0.38297 0.35836 0.28735 0.45902 0.39915 0.77694
ODIN 81 0.44444 0.38297 0.41539 0.35070 0.48000 0.42246 0.78218
ODIN 82 0.44444 0.38297 0.41632 0.35173 0.48000 0.42246 0.78294
ODIN 100 0.44444 0.38297 0.44207 0.38033 0.45283 0.39228 0.78241
FastABOD 5 0.48148 0.42410 0.49839 0.44289 0.50000 0.44467 0.78673
FastABOD 23 0.40741 0.34183 0.54420 0.49376 0.57895 0.53236 0.78324
FastABOD 99 0.44444 0.38297 0.55099 0.50131 0.57895 0.53236 0.78218
KDEOS 11 0.25926 0.17729 0.16918 0.07724 0.28866 0.20995 0.67547
KDEOS 98 0.22222 0.13616 0.19930 0.11070 0.33333 0.25956 0.71478
KDEOS 100 0.18519 0.09502 0.20683 0.11906 0.33071 0.25665 0.72116
LDF 79 0.55556 0.50638 0.53246 0.48072 0.57778 0.53106 0.76609
LDF 89 0.55556 0.50638 0.59468 0.54983 0.65116 0.61256 0.78658
LDF 91 0.55556 0.50638 0.60033 0.55610 0.65116 0.61256 0.77945
LDF 100 0.51852 0.46524 0.57746 0.53070 0.57143 0.52400 0.78992
INFLO 5 0.48148 0.42410 0.50434 0.44949 0.52174 0.46882 0.74514
INFLO 93 0.48148 0.42410 0.55663 0.50756 0.60000 0.55574 0.80540
INFLO 99 0.48148 0.42410 0.55919 0.51041 0.60000 0.55574 0.80844
INFLO 100 0.48148 0.42410 0.56267 0.51427 0.60000 0.55574 0.80806
COF 2 0.48148 0.42410 0.47932 0.42171 0.51163 0.45759 0.74438
COF 4 0.48148 0.42410 0.53233 0.48059 0.55319 0.50375 0.77034
COF 10 0.44444 0.38297 0.50027 0.44498 0.50000 0.44467 0.78355

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (45.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.40741 0.34183 0.47607 0.41809 0.51282 0.45891 0.78597
KNN 3 0.44444 0.38297 0.50610 0.45145 0.51163 0.45759 0.80814
KNN 87 0.48148 0.42410 0.49207 0.43586 0.51282 0.45891 0.80373
KNNW 3 0.44444 0.38297 0.48140 0.42401 0.48780 0.43113 0.78476
KNNW 5 0.44444 0.38297 0.49397 0.43797 0.51282 0.45891 0.79417
KNNW 12 0.40741 0.34183 0.49957 0.44420 0.51282 0.45891 0.79630
KNNW 100 0.44444 0.38297 0.49145 0.43517 0.51282 0.45891 0.80267
LOF 6 0.44444 0.38297 0.49935 0.44395 0.48780 0.43113 0.81967
LOF 7 0.40741 0.34183 0.50324 0.44827 0.50000 0.44467 0.82240
LOF 9 0.40741 0.34183 0.50471 0.44991 0.50000 0.44467 0.81709
LOF 22 0.40741 0.34183 0.49939 0.44400 0.55000 0.50020 0.80146
SimplifiedLOF 9 0.44444 0.38297 0.48870 0.43213 0.47619 0.41823 0.80738
SimplifiedLOF 21 0.40741 0.34183 0.50729 0.45277 0.52381 0.47112 0.81740
SimplifiedLOF 22 0.40741 0.34183 0.51189 0.45788 0.53659 0.48531 0.81618
SimplifiedLOF 29 0.40741 0.34183 0.48837 0.43175 0.55000 0.50020 0.80313
LoOP 20 0.40741 0.34183 0.49681 0.44113 0.51282 0.45891 0.81322
LoOP 21 0.40741 0.34183 0.49749 0.44189 0.51282 0.45891 0.81132
LoOP 30 0.40741 0.34183 0.47802 0.42026 0.55000 0.50020 0.79607
LoOP 58 0.44444 0.38297 0.48880 0.43223 0.55000 0.50020 0.80024
LDOF 21 0.44444 0.38297 0.49552 0.43970 0.51282 0.45891 0.82620
LDOF 22 0.44444 0.38297 0.49737 0.44175 0.51282 0.45891 0.82832
LDOF 37 0.40741 0.34183 0.48550 0.42857 0.55000 0.50020 0.81345
ODIN 32 0.43210 0.36926 0.37303 0.30365 0.44828 0.38722 0.81110
ODIN 38 0.40741 0.34183 0.36593 0.29577 0.42105 0.35699 0.81299
ODIN 77 0.40741 0.34183 0.42573 0.36218 0.48889 0.43233 0.79819
ODIN 84 0.40741 0.34183 0.42815 0.36487 0.47826 0.42053 0.79592
FastABOD 4 0.51852 0.46524 0.48285 0.42562 0.51852 0.46524 0.74924
FastABOD 13 0.40741 0.34183 0.48516 0.42819 0.52381 0.47112 0.78446
FastABOD 71 0.44444 0.38297 0.48861 0.43202 0.50000 0.44467 0.79189
FastABOD 100 0.44444 0.38297 0.47961 0.42203 0.50000 0.44467 0.79645
KDEOS 23 0.25926 0.17729 0.36111 0.29041 0.36559 0.29539 0.77277
KDEOS 33 0.33333 0.25956 0.29044 0.21192 0.36893 0.29910 0.76897
KDEOS 47 0.29630 0.21843 0.24710 0.16379 0.38636 0.31846 0.78719
KDEOS 72 0.25926 0.17729 0.22885 0.14352 0.40816 0.34267 0.77717
LDF 14 0.33333 0.25956 0.42523 0.36163 0.41026 0.34500 0.75319
LDF 15 0.33333 0.25956 0.43111 0.36816 0.40000 0.33361 0.73679
LDF 37 0.37037 0.30070 0.26821 0.18724 0.38462 0.31652 0.69642
LDF 94 0.29630 0.21843 0.36730 0.29728 0.42105 0.35699 0.71084
INFLO 46 0.44444 0.38297 0.48534 0.42839 0.55000 0.50020 0.82028
INFLO 83 0.48148 0.42410 0.50170 0.44656 0.55000 0.50020 0.83182
COF 14 0.44444 0.38297 0.49521 0.43935 0.52381 0.47112 0.77034
COF 17 0.44444 0.38297 0.48534 0.42839 0.50000 0.44467 0.77171
COF 42 0.48148 0.42410 0.44032 0.37839 0.48148 0.42410 0.70370

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO