Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (5% of outliers version#01)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (40.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.25000 0.21311 0.36465 0.33340 0.40000 0.37049 0.80225
KNN 15 0.25000 0.21311 0.37661 0.34595 0.40000 0.37049 0.81831
KNN 46 0.33333 0.30055 0.38442 0.35415 0.40000 0.37049 0.81557
KNNW 1 0.25000 0.21311 0.37124 0.34031 0.40000 0.37049 0.81472
KNNW 46 0.25000 0.21311 0.37590 0.34520 0.40000 0.37049 0.81557
KNNW 50 0.25000 0.21311 0.37536 0.34464 0.40000 0.37049 0.81592
LOF 1 0.25000 0.21311 0.20268 0.16346 0.25000 0.21311 0.74898
LOF 12 0.25000 0.21311 0.33106 0.29816 0.40000 0.37049 0.77459
LOF 69 0.25000 0.21311 0.37145 0.34054 0.40000 0.37049 0.81045
LOF 100 0.25000 0.21311 0.37252 0.34166 0.40000 0.37049 0.80430
SimplifiedLOF 4 0.25000 0.21311 0.27893 0.24347 0.35294 0.32112 0.76366
SimplifiedLOF 10 0.25000 0.21311 0.32977 0.29681 0.40000 0.37049 0.76878
SimplifiedLOF 69 0.25000 0.21311 0.36469 0.33344 0.40000 0.37049 0.81694
SimplifiedLOF 100 0.25000 0.21311 0.36844 0.33738 0.40000 0.37049 0.80977
LoOP 3 0.25000 0.21311 0.22688 0.18885 0.27273 0.23696 0.74898
LoOP 10 0.25000 0.21311 0.32991 0.29695 0.40000 0.37049 0.77049
LoOP 69 0.25000 0.21311 0.36323 0.33192 0.40000 0.37049 0.81421
LoOP 100 0.25000 0.21311 0.36596 0.33478 0.40000 0.37049 0.79986
LDOF 6 0.25000 0.21311 0.20680 0.16779 0.26087 0.22452 0.75273
LDOF 21 0.25000 0.21311 0.32298 0.28968 0.40000 0.37049 0.73770
LDOF 92 0.25000 0.21311 0.35600 0.32433 0.40000 0.37049 0.80738
LDOF 95 0.25000 0.21311 0.35579 0.32411 0.40000 0.37049 0.80874
ODIN 38 0.25000 0.21311 0.15809 0.11668 0.28571 0.25059 0.78535
ODIN 95 0.25000 0.21311 0.36127 0.32986 0.40000 0.37049 0.80499
ODIN 97 0.25000 0.21311 0.36260 0.33126 0.40000 0.37049 0.80721
ODIN 99 0.25000 0.21311 0.36405 0.33277 0.40000 0.37049 0.80635
FastABOD 4 0.25000 0.21311 0.22154 0.18325 0.28571 0.25059 0.71721
FastABOD 18 0.25000 0.21311 0.34358 0.31129 0.40000 0.37049 0.77903
FastABOD 76 0.25000 0.21311 0.36859 0.33754 0.40000 0.37049 0.80089
FastABOD 77 0.25000 0.21311 0.36909 0.33806 0.40000 0.37049 0.80089
KDEOS 10 0.16667 0.12568 0.13514 0.09261 0.20833 0.16940 0.77835
KDEOS 12 0.00000 -0.04918 0.11440 0.07084 0.25974 0.22333 0.80123
KDEOS 13 0.08333 0.03825 0.12023 0.07696 0.27273 0.23696 0.78723
LDF 46 0.41667 0.38798 0.33351 0.30074 0.45455 0.42772 0.88183
LDF 99 0.33333 0.30055 0.34244 0.31010 0.47059 0.44455 0.75717
INFLO 3 0.25000 0.21311 0.24249 0.20523 0.30000 0.26557 0.73156
INFLO 14 0.25000 0.21311 0.33284 0.30003 0.40000 0.37049 0.77493
INFLO 100 0.25000 0.21311 0.36805 0.33697 0.40000 0.37049 0.82480
COF 31 0.41667 0.38798 0.33084 0.29793 0.41667 0.38798 0.80669
COF 50 0.33333 0.30055 0.44117 0.41369 0.44444 0.41712 0.81796
COF 54 0.33333 0.30055 0.45662 0.42990 0.50000 0.47541 0.80669
COF 55 0.33333 0.30055 0.46158 0.43510 0.50000 0.47541 0.80908

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (40.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.25000 0.21311 0.25578 0.21918 0.37500 0.34426 0.77083
KNN 3 0.25000 0.21311 0.23848 0.20103 0.35294 0.32112 0.78893
KNNW 1 0.25000 0.21311 0.29176 0.25693 0.35294 0.32112 0.77561
KNNW 2 0.25000 0.21311 0.25909 0.22265 0.37500 0.34426 0.77117
KNNW 8 0.25000 0.21311 0.24032 0.20296 0.35294 0.32112 0.77801
LOF 2 0.25000 0.21311 0.29287 0.25810 0.31579 0.28214 0.75546
LOF 5 0.25000 0.21311 0.29924 0.26478 0.35294 0.32112 0.76844
LOF 9 0.25000 0.21311 0.23553 0.19794 0.35294 0.32112 0.79064
SimplifiedLOF 2 0.25000 0.21311 0.27365 0.23793 0.28571 0.25059 0.75171
SimplifiedLOF 6 0.25000 0.21311 0.29941 0.26496 0.35294 0.32112 0.77937
SimplifiedLOF 15 0.25000 0.21311 0.22315 0.18495 0.35294 0.32112 0.79816
LoOP 2 0.25000 0.21311 0.26961 0.23369 0.28571 0.25059 0.75444
LoOP 6 0.25000 0.21311 0.30057 0.26617 0.35294 0.32112 0.78381
LoOP 15 0.25000 0.21311 0.22390 0.18573 0.35294 0.32112 0.79918
LDOF 2 0.25000 0.21311 0.15724 0.11579 0.25000 0.21311 0.75034
LDOF 3 0.25000 0.21311 0.22241 0.18417 0.28571 0.25059 0.83470
LDOF 5 0.25000 0.21311 0.32625 0.29311 0.33333 0.30055 0.79235
LDOF 12 0.25000 0.21311 0.20379 0.16463 0.35294 0.32112 0.77220
ODIN 10 0.20000 0.16066 0.14405 0.10195 0.25000 0.21311 0.79406
ODIN 23 0.31667 0.28306 0.19911 0.15972 0.32000 0.28656 0.77664
ODIN 36 0.25000 0.21311 0.23517 0.19755 0.35294 0.32112 0.75700
ODIN 62 0.25000 0.21311 0.25369 0.21699 0.35294 0.32112 0.77527
FastABOD 3 0.25000 0.21311 0.29331 0.25855 0.37500 0.34426 0.80089
FastABOD 4 0.25000 0.21311 0.35142 0.31952 0.37500 0.34426 0.79508
KDEOS 6 0.00000 -0.04918 0.11171 0.06803 0.23729 0.19978 0.79747
KDEOS 10 0.08333 0.03825 0.12683 0.08388 0.20000 0.16066 0.74556
KDEOS 11 0.16667 0.12568 0.12458 0.08152 0.23256 0.19482 0.76673
KDEOS 100 0.08333 0.03825 0.12279 0.07965 0.30000 0.26557 0.75922
LDF 2 0.00000 -0.04918 0.08677 0.04186 0.18182 0.14158 0.72370
LDF 13 0.16667 0.12568 0.22078 0.18246 0.28571 0.25059 0.58982
LDF 44 0.25000 0.21311 0.15140 0.10966 0.26087 0.22452 0.70253
LDF 100 0.25000 0.21311 0.16403 0.12292 0.33333 0.30055 0.65574
INFLO 2 0.25000 0.21311 0.28325 0.24800 0.31579 0.28214 0.75410
INFLO 4 0.25000 0.21311 0.29822 0.26371 0.30000 0.26557 0.81557
INFLO 5 0.25000 0.21311 0.28179 0.24647 0.35294 0.32112 0.81079
COF 5 0.25000 0.21311 0.33241 0.29957 0.40000 0.37049 0.75102
COF 9 0.33333 0.30055 0.32916 0.29616 0.34783 0.31575 0.79918
COF 11 0.25000 0.21311 0.32398 0.29073 0.33333 0.30055 0.81660

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO