Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (5% of outliers version#03)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (40.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.33333 0.30055 0.43898 0.41138 0.50000 0.47541 0.78005
KNN 2 0.33333 0.30055 0.44154 0.41408 0.50000 0.47541 0.76742
KNN 31 0.41667 0.38798 0.43568 0.40793 0.50000 0.47541 0.75478
KNNW 1 0.33333 0.30055 0.41665 0.38796 0.50000 0.47541 0.77425
KNNW 2 0.33333 0.30055 0.42348 0.39512 0.50000 0.47541 0.77903
KNNW 7 0.33333 0.30055 0.43722 0.40954 0.50000 0.47541 0.77049
LOF 2 0.33333 0.30055 0.38031 0.34983 0.47059 0.44455 0.76469
LOF 3 0.33333 0.30055 0.42445 0.39615 0.50000 0.47541 0.77015
LOF 99 0.33333 0.30055 0.42880 0.40070 0.50000 0.47541 0.75205
SimplifiedLOF 2 0.33333 0.30055 0.32126 0.28788 0.38095 0.35051 0.78347
SimplifiedLOF 3 0.33333 0.30055 0.42158 0.39314 0.50000 0.47541 0.76776
SimplifiedLOF 6 0.33333 0.30055 0.43603 0.40829 0.50000 0.47541 0.75820
LoOP 4 0.33333 0.30055 0.40346 0.37413 0.44444 0.41712 0.77527
LoOP 5 0.33333 0.30055 0.42441 0.39610 0.50000 0.47541 0.76469
LoOP 6 0.41667 0.38798 0.43141 0.40344 0.50000 0.47541 0.75478
LDOF 6 0.33333 0.30055 0.37574 0.34503 0.40000 0.37049 0.74214
LDOF 11 0.33333 0.30055 0.40289 0.37352 0.50000 0.47541 0.73087
LDOF 86 0.33333 0.30055 0.41282 0.38395 0.50000 0.47541 0.75410
LDOF 100 0.33333 0.30055 0.41562 0.38688 0.50000 0.47541 0.75273
ODIN 90 0.33333 0.30055 0.23985 0.20247 0.38095 0.35051 0.73839
ODIN 93 0.33333 0.30055 0.24226 0.20500 0.38095 0.35051 0.74283
ODIN 94 0.35417 0.32240 0.24449 0.20734 0.38095 0.35051 0.74266
FastABOD 3 0.33333 0.30055 0.20650 0.16747 0.42105 0.39258 0.62807
FastABOD 5 0.33333 0.30055 0.38905 0.35900 0.50000 0.47541 0.70014
FastABOD 15 0.33333 0.30055 0.38371 0.35340 0.47059 0.44455 0.75342
FastABOD 100 0.33333 0.30055 0.40865 0.37957 0.50000 0.47541 0.75102
KDEOS 11 0.16667 0.12568 0.11792 0.07454 0.22222 0.18397 0.72643
KDEOS 17 0.16667 0.12568 0.19329 0.15362 0.22222 0.18397 0.76093
KDEOS 19 0.16667 0.12568 0.19992 0.16057 0.21429 0.17564 0.74966
KDEOS 26 0.08333 0.03825 0.11279 0.06915 0.25000 0.21311 0.72199
LDF 44 0.41667 0.38798 0.33394 0.30119 0.47619 0.45043 0.81660
LDF 95 0.50000 0.47541 0.45358 0.42670 0.50000 0.47541 0.74317
LDF 99 0.50000 0.47541 0.47192 0.44595 0.57143 0.55035 0.75171
LDF 100 0.50000 0.47541 0.47321 0.44731 0.57143 0.55035 0.75205
INFLO 2 0.33333 0.30055 0.32802 0.29497 0.40000 0.37049 0.74317
INFLO 3 0.33333 0.30055 0.41107 0.38210 0.50000 0.47541 0.72268
INFLO 7 0.33333 0.30055 0.41632 0.38762 0.50000 0.47541 0.75581
INFLO 95 0.33333 0.30055 0.42590 0.39767 0.50000 0.47541 0.75171
COF 3 0.33333 0.30055 0.43434 0.40652 0.50000 0.47541 0.81523
COF 4 0.33333 0.30055 0.44832 0.42118 0.50000 0.47541 0.83163
COF 37 0.41667 0.38798 0.40224 0.37284 0.41667 0.38798 0.75137

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (40.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.41667 0.38798 0.38498 0.35473 0.47619 0.45043 0.77647
KNN 4 0.41667 0.38798 0.38636 0.35619 0.50000 0.47541 0.77135
KNN 69 0.41667 0.38798 0.38316 0.35282 0.52632 0.50302 0.75717
KNNW 2 0.33333 0.30055 0.41839 0.38978 0.47059 0.44455 0.78005
KNNW 3 0.41667 0.38798 0.42351 0.39516 0.47059 0.44455 0.77152
KNNW 54 0.41667 0.38798 0.37123 0.34031 0.50000 0.47541 0.74624
LOF 3 0.33333 0.30055 0.42227 0.39385 0.50000 0.47541 0.80089
LOF 4 0.33333 0.30055 0.42330 0.39493 0.50000 0.47541 0.81728
LOF 5 0.33333 0.30055 0.42776 0.39961 0.50000 0.47541 0.81694
LOF 95 0.41667 0.38798 0.35588 0.32420 0.44444 0.41712 0.75410
SimplifiedLOF 3 0.33333 0.30055 0.40767 0.37854 0.50000 0.47541 0.75922
SimplifiedLOF 5 0.33333 0.30055 0.42681 0.39863 0.50000 0.47541 0.79406
SimplifiedLOF 6 0.33333 0.30055 0.40701 0.37785 0.47059 0.44455 0.80430
SimplifiedLOF 9 0.41667 0.38798 0.39945 0.36991 0.47059 0.44455 0.78484
LoOP 4 0.33333 0.30055 0.37714 0.34650 0.47059 0.44455 0.78449
LoOP 5 0.33333 0.30055 0.40636 0.37716 0.47059 0.44455 0.78962
LoOP 6 0.33333 0.30055 0.38499 0.35475 0.47059 0.44455 0.80260
LoOP 9 0.41667 0.38798 0.36829 0.33722 0.47059 0.44455 0.77971
LDOF 9 0.41667 0.38798 0.34586 0.31369 0.44444 0.41712 0.74488
LDOF 13 0.33333 0.30055 0.35403 0.32226 0.47059 0.44455 0.75990
LDOF 35 0.33333 0.30055 0.33553 0.30285 0.44444 0.41712 0.78176
LDOF 55 0.41667 0.38798 0.35938 0.32788 0.44444 0.41712 0.76298
ODIN 9 0.10526 0.06126 0.10829 0.06443 0.22581 0.18773 0.76793
ODIN 28 0.41667 0.38798 0.25084 0.21400 0.43478 0.40699 0.74249
ODIN 32 0.41667 0.38798 0.29597 0.26135 0.50000 0.47541 0.74915
ODIN 90 0.41667 0.38798 0.34367 0.31139 0.47059 0.44455 0.75188
FastABOD 4 0.33333 0.30055 0.34617 0.31401 0.47059 0.44455 0.77220
FastABOD 13 0.41667 0.38798 0.34002 0.30756 0.45455 0.42772 0.74180
FastABOD 24 0.41667 0.38798 0.35249 0.32064 0.50000 0.47541 0.75239
FastABOD 80 0.41667 0.38798 0.37936 0.34884 0.47619 0.45043 0.76161
KDEOS 11 0.25000 0.21311 0.22885 0.19092 0.30000 0.26557 0.75102
KDEOS 12 0.16667 0.12568 0.24282 0.20558 0.36364 0.33234 0.76776
KDEOS 14 0.25000 0.21311 0.29151 0.25667 0.36364 0.33234 0.78074
KDEOS 18 0.16667 0.12568 0.15400 0.11240 0.30769 0.27364 0.78723
LDF 2 0.16667 0.12568 0.14495 0.10290 0.25000 0.21311 0.78091
LDF 10 0.33333 0.30055 0.42447 0.39616 0.50000 0.47541 0.73668
LDF 42 0.41667 0.38798 0.20527 0.16618 0.41667 0.38798 0.72643
INFLO 4 0.33333 0.30055 0.34646 0.31432 0.47059 0.44455 0.78928
INFLO 9 0.41667 0.38798 0.35789 0.32632 0.47059 0.44455 0.75273
INFLO 52 0.33333 0.30055 0.35087 0.31894 0.44444 0.41712 0.79269
INFLO 96 0.41667 0.38798 0.35912 0.32760 0.44444 0.41712 0.79184
COF 3 0.33333 0.30055 0.41253 0.38363 0.50000 0.47541 0.78928
COF 6 0.41667 0.38798 0.46250 0.43607 0.50000 0.47541 0.82889
COF 9 0.41667 0.38798 0.43312 0.40524 0.50000 0.47541 0.84221

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO