Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (2% of outliers version#01)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (33.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.25000 0.23770 0.27756 0.26572 0.40000 0.39016 0.72387
KNN 95 0.25000 0.23770 0.29908 0.28759 0.40000 0.39016 0.79303
KNNW 1 0.25000 0.23770 0.27799 0.26615 0.40000 0.39016 0.71875
KNNW 100 0.25000 0.23770 0.28730 0.27562 0.40000 0.39016 0.76742
LOF 1 0.25000 0.23770 0.28861 0.27695 0.40000 0.39016 0.70287
LOF 99 0.25000 0.23770 0.28563 0.27392 0.40000 0.39016 0.76742
SimplifiedLOF 1 0.25000 0.23770 0.29034 0.27871 0.40000 0.39016 0.70031
SimplifiedLOF 97 0.25000 0.23770 0.28264 0.27088 0.40000 0.39016 0.75000
LoOP 1 0.25000 0.23770 0.29034 0.27871 0.40000 0.39016 0.70031
LoOP 97 0.25000 0.23770 0.28253 0.27077 0.40000 0.39016 0.75000
LDOF 2 0.25000 0.23770 0.26418 0.25211 0.40000 0.39016 0.51537
LDOF 37 0.25000 0.23770 0.28437 0.27264 0.40000 0.39016 0.75512
ODIN 71 0.25000 0.23770 0.09326 0.07839 0.25000 0.23770 0.76281
ODIN 80 0.25000 0.23770 0.11242 0.09787 0.28571 0.27400 0.75666
ODIN 87 0.25000 0.23770 0.11407 0.09954 0.28571 0.27400 0.76537
ODIN 88 0.25000 0.23770 0.11397 0.09944 0.28571 0.27400 0.76639
FastABOD 4 0.25000 0.23770 0.26989 0.25792 0.40000 0.39016 0.63012
FastABOD 15 0.25000 0.23770 0.29435 0.28279 0.40000 0.39016 0.77152
KDEOS 2 0.00000 -0.01639 0.01582 -0.00032 0.03226 0.01639 0.40779
KDEOS 10 0.00000 -0.01639 0.03770 0.02192 0.11111 0.09654 0.65164
KDEOS 97 0.00000 -0.01639 0.03042 0.01453 0.06897 0.05370 0.66803
LDF 9 0.00000 -0.01639 0.07676 0.06163 0.19048 0.17721 0.85656
LDF 33 0.25000 0.23770 0.08696 0.07200 0.25000 0.23770 0.72746
LDF 59 0.25000 0.23770 0.28789 0.27622 0.40000 0.39016 0.76332
LDF 76 0.25000 0.23770 0.34556 0.33483 0.40000 0.39016 0.70697
INFLO 1 0.25000 0.23770 0.28204 0.27027 0.40000 0.39016 0.70492
INFLO 97 0.25000 0.23770 0.28936 0.27771 0.40000 0.39016 0.79508
COF 1 0.25000 0.23770 0.29034 0.27871 0.40000 0.39016 0.70031
COF 3 0.25000 0.23770 0.28852 0.27685 0.40000 0.39016 0.76639

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (35.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.25000 0.23770 0.15814 0.14434 0.33333 0.32240 0.76537
KNN 80 0.25000 0.23770 0.17734 0.16386 0.33333 0.32240 0.83504
KNN 83 0.25000 0.23770 0.17891 0.16545 0.33333 0.32240 0.83402
KNNW 1 0.25000 0.23770 0.28483 0.27311 0.40000 0.39016 0.74283
KNNW 81 0.25000 0.23770 0.17030 0.15670 0.33333 0.32240 0.80840
LOF 1 0.25000 0.23770 0.27584 0.26397 0.40000 0.39016 0.58402
LOF 5 0.25000 0.23770 0.28144 0.26966 0.40000 0.39016 0.75512
LOF 99 0.25000 0.23770 0.17489 0.16137 0.33333 0.32240 0.82787
SimplifiedLOF 1 0.25000 0.23770 0.27913 0.26732 0.40000 0.39016 0.72643
SimplifiedLOF 5 0.25000 0.23770 0.27937 0.26756 0.40000 0.39016 0.67725
SimplifiedLOF 99 0.25000 0.23770 0.16842 0.15478 0.33333 0.32240 0.78996
LoOP 1 0.25000 0.23770 0.27913 0.26732 0.40000 0.39016 0.72643
LoOP 8 0.25000 0.23770 0.28123 0.26944 0.40000 0.39016 0.68033
LoOP 99 0.25000 0.23770 0.16864 0.15501 0.33333 0.32240 0.79303
LDOF 4 0.25000 0.23770 0.16023 0.14647 0.33333 0.32240 0.69980
LDOF 99 0.25000 0.23770 0.17029 0.15669 0.33333 0.32240 0.78381
ODIN 24 0.25000 0.23770 0.09406 0.07920 0.25000 0.23770 0.73668
ODIN 46 0.25000 0.23770 0.11428 0.09976 0.28571 0.27400 0.76281
ODIN 99 0.25000 0.23770 0.12624 0.11192 0.28571 0.27400 0.81404
FastABOD 3 0.25000 0.23770 0.27459 0.26270 0.40000 0.39016 0.69980
FastABOD 9 0.25000 0.23770 0.32492 0.31386 0.40000 0.39016 0.79201
FastABOD 14 0.25000 0.23770 0.34028 0.32946 0.40000 0.39016 0.77459
KDEOS 4 0.25000 0.23770 0.07700 0.06187 0.25000 0.23770 0.52971
KDEOS 12 0.25000 0.23770 0.28912 0.27746 0.40000 0.39016 0.69365
KDEOS 100 0.00000 -0.01639 0.03605 0.02025 0.07500 0.05984 0.70594
LDF 10 0.25000 0.23770 0.13642 0.12226 0.28571 0.27400 0.87193
LDF 12 0.25000 0.23770 0.30310 0.29167 0.40000 0.39016 0.83709
LDF 15 0.25000 0.23770 0.36963 0.35929 0.40000 0.39016 0.90369
INFLO 1 0.25000 0.23770 0.15599 0.14215 0.33333 0.32240 0.77971
INFLO 2 0.25000 0.23770 0.27658 0.26472 0.40000 0.39016 0.65984
INFLO 4 0.25000 0.23770 0.28049 0.26869 0.40000 0.39016 0.66086
INFLO 97 0.25000 0.23770 0.17618 0.16268 0.33333 0.32240 0.81250
COF 1 0.25000 0.23770 0.27913 0.26732 0.40000 0.39016 0.72643
COF 78 0.25000 0.23770 0.31086 0.29956 0.40000 0.39016 0.83914
COF 84 0.25000 0.23770 0.30455 0.29314 0.40000 0.39016 0.84324

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO