Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (2% of outliers version#04)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (28.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.50000 0.49180 0.53114 0.52345 0.66667 0.66120 0.82787
KNN 19 0.50000 0.49180 0.53986 0.53231 0.66667 0.66120 0.84221
KNN 33 0.50000 0.49180 0.54440 0.53693 0.66667 0.66120 0.83811
KNNW 1 0.50000 0.49180 0.53275 0.52509 0.66667 0.66120 0.84426
KNNW 40 0.50000 0.49180 0.53955 0.53200 0.66667 0.66120 0.83607
LOF 1 0.50000 0.49180 0.53139 0.52371 0.66667 0.66120 0.89344
LOF 100 0.50000 0.49180 0.53810 0.53053 0.66667 0.66120 0.79713
SimplifiedLOF 1 0.50000 0.49180 0.44850 0.43946 0.57143 0.56440 0.89344
SimplifiedLOF 2 0.50000 0.49180 0.52592 0.51814 0.66667 0.66120 0.84221
SimplifiedLOF 11 0.50000 0.49180 0.55475 0.54745 0.66667 0.66120 0.86066
LoOP 1 0.50000 0.49180 0.44850 0.43946 0.57143 0.56440 0.89344
LoOP 2 0.50000 0.49180 0.52480 0.51701 0.66667 0.66120 0.83504
LoOP 12 0.50000 0.49180 0.55787 0.55063 0.66667 0.66120 0.86168
LDOF 3 0.50000 0.49180 0.46129 0.45246 0.57143 0.56440 0.78791
LDOF 4 0.50000 0.49180 0.54502 0.53756 0.66667 0.66120 0.85041
LDOF 12 0.50000 0.49180 0.56292 0.55575 0.66667 0.66120 0.88320
LDOF 18 0.50000 0.49180 0.56603 0.55891 0.66667 0.66120 0.87090
ODIN 28 0.10526 0.09060 0.08917 0.07424 0.20000 0.18689 0.83402
ODIN 99 0.50000 0.49180 0.28453 0.27280 0.50000 0.49180 0.79867
ODIN 100 0.50000 0.49180 0.28693 0.27524 0.50000 0.49180 0.80020
FastABOD 3 0.50000 0.49180 0.45721 0.44831 0.57143 0.56440 0.82480
FastABOD 4 0.50000 0.49180 0.53820 0.53062 0.66667 0.66120 0.80020
FastABOD 8 0.50000 0.49180 0.56138 0.55419 0.66667 0.66120 0.86168
FastABOD 12 0.50000 0.49180 0.58983 0.58310 0.66667 0.66120 0.84016
KDEOS 3 0.25000 0.23770 0.07791 0.06280 0.25000 0.23770 0.56250
KDEOS 12 0.00000 -0.01639 0.14788 0.13391 0.33333 0.32240 0.84836
KDEOS 13 0.00000 -0.01639 0.10856 0.09395 0.23077 0.21816 0.85348
LDF 43 0.25000 0.23770 0.32362 0.31254 0.40000 0.39016 0.88012
LDF 76 0.50000 0.49180 0.40004 0.39020 0.50000 0.49180 0.80123
LDF 99 0.50000 0.49180 0.59896 0.59238 0.66667 0.66120 0.80225
INFLO 1 0.50000 0.49180 0.52183 0.51399 0.66667 0.66120 0.84324
INFLO 5 0.50000 0.49180 0.54116 0.53363 0.66667 0.66120 0.86373
INFLO 12 0.50000 0.49180 0.55781 0.55056 0.66667 0.66120 0.86066
COF 1 0.50000 0.49180 0.44850 0.43946 0.57143 0.56440 0.89344
COF 2 0.50000 0.49180 0.53036 0.52266 0.66667 0.66120 0.86270

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (29.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.50000 0.49180 0.31607 0.30485 0.57143 0.56440 0.78279
KNN 15 0.50000 0.49180 0.28833 0.27667 0.50000 0.49180 0.85553
KNNW 1 0.50000 0.49180 0.39109 0.38111 0.50000 0.49180 0.74334
KNNW 2 0.50000 0.49180 0.43707 0.42784 0.57143 0.56440 0.76537
KNNW 58 0.50000 0.49180 0.28900 0.27735 0.50000 0.49180 0.84734
LOF 1 0.50000 0.49180 0.43054 0.42120 0.57143 0.56440 0.68545
LOF 2 0.50000 0.49180 0.52413 0.51633 0.66667 0.66120 0.85758
LOF 5 0.50000 0.49180 0.52777 0.52003 0.66667 0.66120 0.79303
SimplifiedLOF 2 0.50000 0.49180 0.52309 0.51528 0.66667 0.66120 0.85246
SimplifiedLOF 7 0.50000 0.49180 0.53586 0.52825 0.66667 0.66120 0.87500
LoOP 2 0.50000 0.49180 0.52393 0.51613 0.66667 0.66120 0.85758
LoOP 7 0.50000 0.49180 0.54135 0.53383 0.66667 0.66120 0.88115
LDOF 3 0.50000 0.49180 0.34110 0.33030 0.57143 0.56440 0.92418
LDOF 5 0.50000 0.49180 0.55329 0.54597 0.66667 0.66120 0.90061
LDOF 7 0.50000 0.49180 0.56667 0.55956 0.66667 0.66120 0.93033
LDOF 10 0.50000 0.49180 0.49397 0.48567 0.57143 0.56440 0.94057
ODIN 10 0.09524 0.08041 0.07864 0.06354 0.16000 0.14623 0.86783
ODIN 24 0.50000 0.49180 0.29155 0.27994 0.50000 0.49180 0.84939
ODIN 46 0.50000 0.49180 0.35963 0.34913 0.57143 0.56440 0.83811
ODIN 98 0.50000 0.49180 0.36597 0.35557 0.57143 0.56440 0.84375
FastABOD 3 0.50000 0.49180 0.52190 0.51407 0.66667 0.66120 0.84426
FastABOD 4 0.50000 0.49180 0.53588 0.52827 0.66667 0.66120 0.80738
FastABOD 10 0.50000 0.49180 0.46163 0.45280 0.57143 0.56440 0.84836
KDEOS 9 0.25000 0.23770 0.19268 0.17944 0.31579 0.30457 0.87193
KDEOS 10 0.25000 0.23770 0.38417 0.37408 0.40000 0.39016 0.89344
KDEOS 11 0.25000 0.23770 0.36464 0.35423 0.40000 0.39016 0.89549
LDF 87 0.00000 -0.01639 0.11294 0.09840 0.30769 0.29634 0.79918
LDF 97 0.50000 0.49180 0.28633 0.27463 0.50000 0.49180 0.77766
INFLO 1 0.50000 0.49180 0.43422 0.42495 0.57143 0.56440 0.80225
INFLO 2 0.50000 0.49180 0.52528 0.51749 0.66667 0.66120 0.86475
INFLO 4 0.50000 0.49180 0.53450 0.52687 0.66667 0.66120 0.87602
INFLO 5 0.50000 0.49180 0.45637 0.44746 0.57143 0.56440 0.90574
COF 2 0.50000 0.49180 0.51792 0.51002 0.66667 0.66120 0.80635
COF 4 0.50000 0.49180 0.52487 0.51708 0.66667 0.66120 0.80943
COF 95 0.25000 0.23770 0.18364 0.17025 0.30769 0.29634 0.83607

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO