Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (10% of outliers version#01)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (48.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.44444 0.38297 0.45383 0.39340 0.46429 0.40501 0.83895
KNN 9 0.40741 0.34183 0.48556 0.42864 0.45000 0.38914 0.84123
KNN 19 0.40741 0.34183 0.47991 0.42236 0.46667 0.40765 0.83880
KNN 46 0.40741 0.34183 0.48699 0.43022 0.46512 0.40593 0.83698
KNNW 3 0.44444 0.38297 0.44752 0.38638 0.44828 0.38722 0.83834
KNNW 6 0.44444 0.38297 0.45591 0.39570 0.47273 0.41438 0.83743
KNNW 73 0.40741 0.34183 0.48832 0.43170 0.46512 0.40593 0.83880
KNNW 89 0.40741 0.34183 0.48652 0.42970 0.46512 0.40593 0.83940
LOF 11 0.44444 0.38297 0.42530 0.36171 0.44444 0.38297 0.81330
LOF 38 0.37037 0.30070 0.45142 0.39072 0.45977 0.39999 0.83667
LOF 44 0.37037 0.30070 0.45612 0.39594 0.45455 0.39419 0.83986
LOF 97 0.40741 0.34183 0.48104 0.42361 0.44444 0.38297 0.83394
SimplifiedLOF 26 0.37037 0.30070 0.45331 0.39282 0.45902 0.39915 0.82256
SimplifiedLOF 65 0.37037 0.30070 0.46460 0.40536 0.44681 0.38559 0.84563
SimplifiedLOF 90 0.44444 0.38297 0.47608 0.41811 0.45070 0.38992 0.84259
SimplifiedLOF 99 0.44444 0.38297 0.48043 0.42294 0.45283 0.39228 0.84259
LoOP 26 0.37037 0.30070 0.45075 0.38997 0.46667 0.40765 0.82225
LoOP 71 0.37037 0.30070 0.45962 0.39983 0.45161 0.39093 0.84593
LoOP 97 0.44444 0.38297 0.48137 0.42398 0.44444 0.38297 0.84244
LoOP 99 0.44444 0.38297 0.48247 0.42520 0.44444 0.38297 0.84290
LDOF 42 0.44444 0.38297 0.45177 0.39111 0.45283 0.39228 0.83424
LDOF 43 0.44444 0.38297 0.44987 0.38899 0.46154 0.40195 0.83637
LDOF 67 0.44444 0.38297 0.46956 0.41086 0.45283 0.39228 0.84821
LDOF 81 0.40741 0.34183 0.47332 0.41504 0.44156 0.37976 0.84517
ODIN 99 0.40741 0.34183 0.41405 0.34921 0.43902 0.37695 0.83675
ODIN 100 0.40741 0.34183 0.41426 0.34945 0.45000 0.38914 0.83698
FastABOD 6 0.40741 0.34183 0.42724 0.36386 0.43011 0.36705 0.82817
FastABOD 56 0.29630 0.21843 0.44563 0.38429 0.44776 0.38665 0.83667
FastABOD 59 0.29630 0.21843 0.44610 0.38481 0.44118 0.37934 0.83789
FastABOD 100 0.37037 0.30070 0.45261 0.39203 0.42105 0.35699 0.83349
KDEOS 12 0.29630 0.21843 0.21673 0.13006 0.30189 0.22464 0.72693
KDEOS 15 0.22222 0.13616 0.24200 0.15812 0.31579 0.24008 0.73103
KDEOS 23 0.18519 0.09502 0.22056 0.13431 0.42667 0.36322 0.76017
KDEOS 100 0.25926 0.17729 0.22862 0.14326 0.33333 0.25956 0.77080
LDF 48 0.44444 0.38297 0.35313 0.28155 0.44444 0.38297 0.83804
LDF 82 0.48148 0.42410 0.49667 0.44097 0.48276 0.42552 0.80829
LDF 83 0.44444 0.38297 0.48135 0.42395 0.51064 0.45649 0.81223
LDF 91 0.48148 0.42410 0.49697 0.44130 0.49057 0.43419 0.82316
INFLO 8 0.44444 0.38297 0.42786 0.36455 0.46154 0.40195 0.81330
INFLO 70 0.40741 0.34183 0.48230 0.42501 0.47619 0.41823 0.85307
INFLO 94 0.44444 0.38297 0.49392 0.43792 0.47059 0.41201 0.85519
INFLO 97 0.44444 0.38297 0.49378 0.43776 0.47059 0.41201 0.85610
COF 3 0.40741 0.34183 0.45986 0.40009 0.43137 0.36845 0.80768
COF 4 0.40741 0.34183 0.47826 0.42053 0.43333 0.37063 0.83743

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (46.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.44444 0.38297 0.46825 0.40941 0.52381 0.47112 0.81535
KNN 7 0.40741 0.34183 0.45565 0.39541 0.46809 0.40923 0.82605
KNNW 1 0.44444 0.38297 0.48119 0.42378 0.52381 0.47112 0.80776
KNNW 2 0.44444 0.38297 0.48621 0.42935 0.52381 0.47112 0.81436
KNNW 9 0.44444 0.38297 0.46041 0.40070 0.50000 0.44467 0.81952
LOF 2 0.40741 0.34183 0.47990 0.42235 0.46512 0.40593 0.77383
LOF 4 0.48148 0.42410 0.47386 0.41564 0.48148 0.42410 0.78628
LOF 11 0.40741 0.34183 0.46330 0.40391 0.48780 0.43113 0.82878
LOF 49 0.40741 0.34183 0.45476 0.39443 0.51282 0.45891 0.79781
SimplifiedLOF 4 0.44444 0.38297 0.45369 0.39324 0.47059 0.41201 0.79068
SimplifiedLOF 7 0.44444 0.38297 0.48897 0.43242 0.50000 0.44467 0.80525
SimplifiedLOF 18 0.40741 0.34183 0.46563 0.40649 0.52381 0.47112 0.81694
SimplifiedLOF 23 0.40741 0.34183 0.46796 0.40909 0.52381 0.47112 0.81831
LoOP 4 0.44444 0.38297 0.44833 0.38728 0.47059 0.41201 0.78886
LoOP 7 0.40741 0.34183 0.48465 0.42763 0.47619 0.41823 0.80601
LoOP 19 0.40741 0.34183 0.46530 0.40613 0.52381 0.47112 0.81846
LoOP 22 0.40741 0.34183 0.47125 0.41275 0.52381 0.47112 0.81998
LDOF 7 0.44444 0.38297 0.42818 0.36490 0.44444 0.38297 0.81815
LDOF 23 0.40741 0.34183 0.47828 0.42055 0.50000 0.44467 0.82180
LDOF 39 0.40741 0.34183 0.45240 0.39181 0.51282 0.45891 0.81208
ODIN 17 0.44444 0.38297 0.35287 0.28126 0.45455 0.39419 0.84062
ODIN 25 0.48560 0.42868 0.39830 0.33172 0.49057 0.43419 0.81573
ODIN 35 0.48148 0.42410 0.43632 0.37395 0.56522 0.51711 0.81535
FastABOD 4 0.44444 0.38297 0.44235 0.38065 0.45902 0.39915 0.80222
FastABOD 16 0.40741 0.34183 0.47365 0.41541 0.47826 0.42053 0.81982
KDEOS 10 0.29630 0.21843 0.29326 0.21506 0.34848 0.27639 0.76427
KDEOS 11 0.29630 0.21843 0.31740 0.24187 0.38095 0.31245 0.76154
KDEOS 51 0.18519 0.09502 0.21863 0.13216 0.41975 0.35555 0.77641
KDEOS 94 0.25926 0.17729 0.24561 0.16214 0.38095 0.31245 0.78689
LDF 14 0.37037 0.30070 0.35554 0.28423 0.37037 0.30070 0.65149
LDF 17 0.29630 0.21843 0.35698 0.28582 0.35821 0.28719 0.66864
LDF 41 0.33333 0.25956 0.30692 0.23022 0.35897 0.28804 0.75471
LDF 73 0.25926 0.17729 0.31954 0.24425 0.37363 0.30431 0.72268
INFLO 7 0.40741 0.34183 0.47496 0.41686 0.48780 0.43113 0.81557
INFLO 8 0.44444 0.38297 0.45431 0.39393 0.46154 0.40195 0.80677
INFLO 30 0.40741 0.34183 0.46785 0.40896 0.53659 0.48531 0.81102
INFLO 56 0.40741 0.34183 0.46761 0.40870 0.51282 0.45891 0.83030
COF 1 0.33333 0.25956 0.42989 0.36681 0.43243 0.36963 0.80791
COF 5 0.44444 0.38297 0.48552 0.42859 0.53333 0.48169 0.75015
COF 6 0.44444 0.38297 0.47957 0.42198 0.53659 0.48531 0.75213
COF 9 0.48148 0.42410 0.45766 0.39765 0.50000 0.44467 0.76746

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO