Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (5% of outliers version#07)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (39.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.50000 0.47541 0.55297 0.53099 0.63158 0.61346 0.84785
KNN 6 0.50000 0.47541 0.56721 0.54592 0.63158 0.61346 0.85417
KNN 16 0.50000 0.47541 0.56182 0.54027 0.60000 0.58033 0.85861
KNNW 1 0.50000 0.47541 0.58207 0.56152 0.66667 0.65027 0.86236
LOF 10 0.58333 0.56284 0.57401 0.55306 0.63636 0.61848 0.80430
LOF 11 0.58333 0.56284 0.57625 0.55541 0.63636 0.61848 0.80943
LOF 87 0.50000 0.47541 0.55461 0.53271 0.60000 0.58033 0.85143
SimplifiedLOF 10 0.50000 0.47541 0.58921 0.56901 0.66667 0.65027 0.81045
SimplifiedLOF 11 0.58333 0.56284 0.58287 0.56236 0.63158 0.61346 0.81113
SimplifiedLOF 97 0.50000 0.47541 0.57488 0.55398 0.63158 0.61346 0.84870
LoOP 10 0.58333 0.56284 0.59233 0.57228 0.66667 0.65027 0.80567
LoOP 96 0.50000 0.47541 0.57418 0.55324 0.63158 0.61346 0.84836
LDOF 10 0.58333 0.56284 0.57223 0.55119 0.63158 0.61346 0.81421
LDOF 20 0.50000 0.47541 0.57924 0.55855 0.66667 0.65027 0.80089
LDOF 21 0.50000 0.47541 0.57948 0.55879 0.66667 0.65027 0.80191
LDOF 100 0.50000 0.47541 0.57159 0.55052 0.63158 0.61346 0.84392
ODIN 72 0.49242 0.46746 0.40966 0.38062 0.52632 0.50302 0.83419
ODIN 79 0.46667 0.44044 0.45843 0.43180 0.55556 0.53370 0.83794
ODIN 84 0.46667 0.44044 0.48599 0.46071 0.50000 0.47541 0.84153
ODIN 85 0.46667 0.44044 0.48558 0.46028 0.50000 0.47541 0.84187
FastABOD 9 0.58333 0.56284 0.52345 0.50002 0.58333 0.56284 0.83982
FastABOD 20 0.50000 0.47541 0.54596 0.52363 0.63158 0.61346 0.82548
FastABOD 26 0.50000 0.47541 0.56181 0.54026 0.63158 0.61346 0.83333
FastABOD 84 0.50000 0.47541 0.54213 0.51961 0.58824 0.56798 0.84597
KDEOS 11 0.33333 0.30055 0.19224 0.15252 0.37037 0.33940 0.76127
KDEOS 12 0.25000 0.21311 0.18283 0.14264 0.38462 0.35435 0.75205
KDEOS 13 0.33333 0.30055 0.19561 0.15605 0.33333 0.30055 0.76264
LDF 37 0.25000 0.21311 0.28066 0.24528 0.48276 0.45732 0.89686
LDF 76 0.50000 0.47541 0.46385 0.43748 0.50000 0.47541 0.88046
LDF 96 0.41667 0.38798 0.49814 0.47346 0.52632 0.50302 0.82548
LDF 99 0.50000 0.47541 0.53380 0.51088 0.52174 0.49822 0.85178
INFLO 9 0.58333 0.56284 0.58425 0.56380 0.63636 0.61848 0.79816
INFLO 10 0.58333 0.56284 0.60089 0.58127 0.66667 0.65027 0.80123
INFLO 11 0.58333 0.56284 0.60335 0.58384 0.66667 0.65027 0.79986
INFLO 100 0.50000 0.47541 0.56052 0.53891 0.63158 0.61346 0.86066
COF 3 0.41667 0.38798 0.50904 0.48489 0.58824 0.56798 0.82719
COF 4 0.50000 0.47541 0.51801 0.49431 0.58824 0.56798 0.81592
COF 36 0.41667 0.38798 0.46358 0.43720 0.52632 0.50302 0.86134

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (39.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.58333 0.56284 0.55832 0.53659 0.60870 0.58945 0.83641
KNN 4 0.58333 0.56284 0.55465 0.53274 0.66667 0.65027 0.85075
KNN 15 0.58333 0.56284 0.55254 0.53053 0.66667 0.65027 0.87193
KNNW 1 0.58333 0.56284 0.57515 0.55426 0.61538 0.59647 0.84853
KNNW 10 0.58333 0.56284 0.56556 0.54420 0.66667 0.65027 0.85348
KNNW 33 0.58333 0.56284 0.55375 0.53181 0.66667 0.65027 0.86646
LOF 6 0.50000 0.47541 0.57264 0.55162 0.57143 0.55035 0.83675
LOF 7 0.58333 0.56284 0.53283 0.50985 0.58333 0.56284 0.83948
LOF 12 0.58333 0.56284 0.54217 0.51965 0.66667 0.65027 0.85280
LOF 21 0.58333 0.56284 0.53209 0.50908 0.66667 0.65027 0.85997
SimplifiedLOF 7 0.50000 0.47541 0.56795 0.54670 0.57143 0.55035 0.84460
SimplifiedLOF 8 0.58333 0.56284 0.53270 0.50972 0.60870 0.58945 0.83846
SimplifiedLOF 10 0.58333 0.56284 0.54057 0.51798 0.66667 0.65027 0.83743
SimplifiedLOF 89 0.58333 0.56284 0.54190 0.51937 0.66667 0.65027 0.86066
LoOP 8 0.58333 0.56284 0.51915 0.49550 0.60000 0.58033 0.82872
LoOP 12 0.58333 0.56284 0.53322 0.51026 0.66667 0.65027 0.81677
LoOP 53 0.58333 0.56284 0.54420 0.52178 0.66667 0.65027 0.85143
LoOP 89 0.58333 0.56284 0.54232 0.51982 0.66667 0.65027 0.86202
LDOF 10 0.58333 0.56284 0.49842 0.47375 0.58333 0.56284 0.84699
LDOF 15 0.58333 0.56284 0.52050 0.49692 0.66667 0.65027 0.82992
LDOF 88 0.58333 0.56284 0.55050 0.52839 0.66667 0.65027 0.85383
LDOF 89 0.58333 0.56284 0.55210 0.53007 0.66667 0.65027 0.85314
ODIN 68 0.58333 0.56284 0.47216 0.44620 0.60000 0.58033 0.85212
ODIN 88 0.58333 0.56284 0.47272 0.44679 0.63636 0.61848 0.85383
ODIN 94 0.58333 0.56284 0.50720 0.48297 0.63636 0.61848 0.86202
ODIN 99 0.58333 0.56284 0.49515 0.47032 0.63636 0.61848 0.86424
FastABOD 3 0.50000 0.47541 0.47802 0.45235 0.55556 0.53370 0.78108
FastABOD 4 0.50000 0.47541 0.49671 0.47196 0.60000 0.58033 0.78893
FastABOD 60 0.50000 0.47541 0.52228 0.49878 0.60000 0.58033 0.85690
KDEOS 11 0.25000 0.21311 0.20951 0.17063 0.27273 0.23696 0.75410
KDEOS 98 0.16667 0.12568 0.18362 0.14347 0.35000 0.31803 0.82684
LDF 70 0.25000 0.21311 0.37851 0.34794 0.40000 0.37049 0.83094
LDF 72 0.33333 0.30055 0.42273 0.39434 0.50000 0.47541 0.80840
LDF 91 0.41667 0.38798 0.44232 0.41489 0.50000 0.47541 0.77459
LDF 97 0.50000 0.47541 0.41522 0.38646 0.50000 0.47541 0.77937
INFLO 9 0.58333 0.56284 0.51078 0.48672 0.60000 0.58033 0.81421
INFLO 23 0.58333 0.56284 0.52410 0.50069 0.66667 0.65027 0.84665
INFLO 68 0.58333 0.56284 0.54228 0.51977 0.66667 0.65027 0.85827
INFLO 89 0.58333 0.56284 0.54222 0.51970 0.66667 0.65027 0.87432
COF 2 0.50000 0.47541 0.50461 0.48024 0.58824 0.56798 0.75137
COF 5 0.50000 0.47541 0.53915 0.51649 0.58824 0.56798 0.77664
COF 51 0.41667 0.38798 0.46500 0.43868 0.47619 0.45043 0.86510

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO