Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (2% of outliers version#05)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (30.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.50000 0.49180 0.52541 0.51763 0.66667 0.66120 0.85553
KNN 3 0.50000 0.49180 0.52906 0.52134 0.66667 0.66120 0.86680
KNNW 1 0.50000 0.49180 0.52885 0.52112 0.66667 0.66120 0.87090
LOF 1 0.50000 0.49180 0.54405 0.53657 0.66667 0.66120 0.86270
LOF 4 0.50000 0.49180 0.53766 0.53008 0.66667 0.66120 0.88422
SimplifiedLOF 1 0.50000 0.49180 0.34979 0.33913 0.57143 0.56440 0.94365
SimplifiedLOF 3 0.50000 0.49180 0.55329 0.54597 0.66667 0.66120 0.87807
SimplifiedLOF 4 0.50000 0.49180 0.55329 0.54597 0.66667 0.66120 0.90061
LoOP 1 0.50000 0.49180 0.34979 0.33913 0.57143 0.56440 0.94365
LoOP 3 0.50000 0.49180 0.55893 0.55170 0.66667 0.66120 0.87705
LDOF 2 0.25000 0.23770 0.29226 0.28066 0.44444 0.43534 0.97439
LDOF 10 0.50000 0.49180 0.47909 0.47055 0.57143 0.56440 0.87602
LDOF 11 0.50000 0.49180 0.55205 0.54471 0.66667 0.66120 0.86066
ODIN 49 0.15385 0.13997 0.10401 0.08933 0.23529 0.22276 0.86680
ODIN 80 0.50000 0.49180 0.26864 0.25665 0.50000 0.49180 0.81148
ODIN 100 0.50000 0.49180 0.35222 0.34160 0.57143 0.56440 0.80225
FastABOD 6 0.50000 0.49180 0.40224 0.39244 0.50000 0.49180 0.87398
FastABOD 9 0.50000 0.49180 0.53040 0.52270 0.66667 0.66120 0.88934
KDEOS 8 0.25000 0.23770 0.10502 0.09035 0.25000 0.23770 0.80328
KDEOS 15 0.00000 -0.01639 0.09864 0.08386 0.20000 0.18689 0.87910
KDEOS 24 0.25000 0.23770 0.28405 0.27231 0.40000 0.39016 0.80430
KDEOS 26 0.25000 0.23770 0.28477 0.27305 0.40000 0.39016 0.80738
LDF 70 0.25000 0.23770 0.39233 0.38237 0.40000 0.39016 0.86783
LDF 74 0.50000 0.49180 0.42642 0.41701 0.50000 0.49180 0.84734
LDF 91 0.50000 0.49180 0.53139 0.52370 0.57143 0.56440 0.86373
LDF 99 0.50000 0.49180 0.52407 0.51627 0.66667 0.66120 0.82275
INFLO 1 0.50000 0.49180 0.53408 0.52645 0.66667 0.66120 0.83811
INFLO 2 0.50000 0.49180 0.57503 0.56806 0.66667 0.66120 0.84631
INFLO 4 0.50000 0.49180 0.56124 0.55404 0.66667 0.66120 0.90061
COF 1 0.50000 0.49180 0.34979 0.33913 0.57143 0.56440 0.94365
COF 3 0.50000 0.49180 0.52376 0.51595 0.66667 0.66120 0.81660

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (27.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.50000 0.49180 0.53306 0.52541 0.66667 0.66120 0.89857
KNN 7 0.50000 0.49180 0.53556 0.52795 0.66667 0.66120 0.90676
KNNW 1 0.50000 0.49180 0.53442 0.52678 0.66667 0.66120 0.89242
KNNW 11 0.50000 0.49180 0.53362 0.52597 0.66667 0.66120 0.90061
LOF 1 0.50000 0.49180 0.54391 0.53643 0.66667 0.66120 0.86066
LOF 4 0.50000 0.49180 0.57599 0.56904 0.66667 0.66120 0.86475
LOF 21 0.50000 0.49180 0.54925 0.54187 0.66667 0.66120 0.93033
SimplifiedLOF 1 0.50000 0.49180 0.54161 0.53410 0.66667 0.66120 0.84836
SimplifiedLOF 11 0.50000 0.49180 0.59722 0.59062 0.66667 0.66120 0.92418
SimplifiedLOF 24 0.50000 0.49180 0.55897 0.55174 0.66667 0.66120 0.94160
LoOP 1 0.50000 0.49180 0.54161 0.53410 0.66667 0.66120 0.84836
LoOP 9 0.50000 0.49180 0.59762 0.59102 0.66667 0.66120 0.92623
LoOP 21 0.50000 0.49180 0.56207 0.55490 0.66667 0.66120 0.93852
LDOF 2 0.50000 0.49180 0.46279 0.45398 0.50000 0.49180 0.94160
LDOF 3 0.25000 0.23770 0.46275 0.45394 0.46154 0.45271 0.96004
LDOF 4 0.50000 0.49180 0.60890 0.60249 0.66667 0.66120 0.93135
LDOF 5 0.50000 0.49180 0.64628 0.64048 0.66667 0.66120 0.95287
ODIN 14 0.21429 0.20141 0.18703 0.17370 0.33333 0.32240 0.95389
ODIN 46 0.50000 0.49180 0.31131 0.30002 0.50000 0.49180 0.94570
FastABOD 3 0.50000 0.49180 0.52816 0.52042 0.66667 0.66120 0.87807
FastABOD 11 0.50000 0.49180 0.57989 0.57300 0.66667 0.66120 0.95389
KDEOS 4 0.25000 0.23770 0.17374 0.16020 0.33333 0.32240 0.86578
KDEOS 7 0.00000 -0.01639 0.18813 0.17483 0.30769 0.29634 0.95184
KDEOS 13 0.25000 0.23770 0.40346 0.39368 0.44444 0.43534 0.87705
LDF 27 0.50000 0.49180 0.42732 0.41793 0.57143 0.56440 0.67111
LDF 94 0.50000 0.49180 0.39663 0.38674 0.50000 0.49180 0.84016
LDF 96 0.50000 0.49180 0.52038 0.51252 0.66667 0.66120 0.83197
INFLO 1 0.50000 0.49180 0.56177 0.55458 0.66667 0.66120 0.86783
INFLO 5 0.50000 0.49180 0.59409 0.58743 0.66667 0.66120 0.90266
INFLO 21 0.50000 0.49180 0.56439 0.55725 0.66667 0.66120 0.94365
COF 1 0.50000 0.49180 0.54161 0.53410 0.66667 0.66120 0.84836
COF 9 0.50000 0.49180 0.52764 0.51989 0.66667 0.66120 0.87807

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO