Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (2% of outliers version#05)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (30.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 14 0.33333 0.32000 0.24359 0.22846 0.37500 0.36250 0.95111
KNN 53 0.33333 0.32000 0.44286 0.43171 0.60000 0.59200 0.98222
KNN 57 0.33333 0.32000 0.41111 0.39933 0.66667 0.66000 0.98222
KNNW 1 0.00000 -0.02000 0.02659 0.00712 0.06522 0.04652 0.44000
KNNW 72 0.00000 -0.02000 0.25694 0.24208 0.44444 0.43333 0.95556
KNNW 98 0.00000 -0.02000 0.26587 0.25119 0.44444 0.43333 0.96000
LOF 56 0.33333 0.32000 0.23660 0.22133 0.33333 0.32000 0.94667
LOF 76 0.33333 0.32000 0.34444 0.33133 0.50000 0.49000 0.97333
SimplifiedLOF 1 0.00000 -0.02000 0.01815 -0.00149 0.03896 0.01974 0.35778
SimplifiedLOF 94 0.00000 -0.02000 0.15132 0.13435 0.25000 0.23500 0.91111
SimplifiedLOF 99 0.00000 -0.02000 0.15275 0.13580 0.25000 0.23500 0.91333
SimplifiedLOF 100 0.00000 -0.02000 0.15499 0.13809 0.25000 0.23500 0.91333
LoOP 1 0.00000 -0.02000 0.01797 -0.00167 0.03846 0.01923 0.35111
LoOP 93 0.00000 -0.02000 0.16768 0.15103 0.26667 0.25200 0.92667
LoOP 95 0.00000 -0.02000 0.16727 0.15062 0.28571 0.27143 0.92222
LDOF 2 0.00000 -0.02000 0.02128 0.00171 0.05128 0.03231 0.31778
LDOF 95 0.00000 -0.02000 0.11089 0.09311 0.22222 0.20667 0.88444
LDOF 100 0.00000 -0.02000 0.11909 0.10147 0.20000 0.18400 0.89111
ODIN 87 0.33333 0.32000 0.28889 0.27467 0.44444 0.43333 0.96000
ODIN 93 0.33333 0.32000 0.32778 0.31433 0.50000 0.49000 0.97000
FastABOD 3 0.00000 -0.02000 0.02769 0.00824 0.06977 0.05116 0.54889
FastABOD 91 0.00000 -0.02000 0.06922 0.05061 0.16216 0.14541 0.82667
KDEOS 2 0.00000 -0.02000 0.03480 0.01549 0.10526 0.08737 0.44889
KDEOS 98 0.00000 -0.02000 0.03519 0.01589 0.09091 0.07273 0.64889
LDF 31 0.66667 0.66000 0.53175 0.52238 0.66667 0.66000 0.98667
LDF 47 0.33333 0.32000 0.70000 0.69400 0.75000 0.74500 0.99111
LDF 66 0.66667 0.66000 0.75556 0.75067 0.75000 0.74500 0.99333
INFLO 1 0.00000 -0.02000 0.04119 0.02201 0.10000 0.08200 0.62333
INFLO 81 0.00000 -0.02000 0.21329 0.19756 0.40000 0.38800 0.94889
INFLO 94 0.00000 -0.02000 0.21667 0.20100 0.36364 0.35091 0.94222
COF 48 0.66667 0.66000 0.51389 0.50417 0.66667 0.66000 0.98444
COF 63 0.66667 0.66000 0.63889 0.63167 0.85714 0.85429 0.99333
COF 84 0.66667 0.66000 0.74359 0.73846 0.80000 0.79600 0.97778

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (37.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02000 0.05126 0.03229 0.14286 0.12571 0.68667
KNN 2 0.00000 -0.02000 0.05141 0.03244 0.11429 0.09657 0.74444
KNNW 1 0.00000 -0.02000 0.06893 0.05031 0.16000 0.14320 0.76444
LOF 1 0.33333 0.32000 0.20553 0.18964 0.40000 0.38800 0.77222
SimplifiedLOF 1 0.00000 -0.02000 0.07045 0.05186 0.16667 0.15000 0.76889
SimplifiedLOF 2 0.33333 0.32000 0.19205 0.17589 0.40000 0.38800 0.71333
LoOP 1 0.00000 -0.02000 0.07508 0.05659 0.18182 0.16545 0.77111
LoOP 2 0.33333 0.32000 0.19101 0.17483 0.40000 0.38800 0.70222
LoOP 3 0.33333 0.32000 0.19116 0.17498 0.40000 0.38800 0.68889
LDOF 2 0.00000 -0.02000 0.08988 0.07168 0.22222 0.20667 0.75778
LDOF 3 0.33333 0.32000 0.19372 0.17760 0.40000 0.38800 0.73778
ODIN 2 0.05263 0.03368 0.04879 0.02977 0.09091 0.07273 0.80444
ODIN 3 0.10000 0.08200 0.06315 0.04441 0.15385 0.13692 0.76222
ODIN 7 0.00000 -0.02000 0.08381 0.06549 0.20000 0.18400 0.73778
FastABOD 3 0.33333 0.32000 0.28333 0.26900 0.40000 0.38800 0.92444
KDEOS 3 0.00000 -0.02000 0.10648 0.08861 0.26667 0.25200 0.80667
KDEOS 16 0.33333 0.32000 0.14032 0.12313 0.33333 0.32000 0.68667
KDEOS 20 0.33333 0.32000 0.19965 0.18364 0.40000 0.38800 0.70444
LDF 1 0.00000 -0.02000 0.09453 0.07642 0.22222 0.20667 0.77667
INFLO 2 0.33333 0.32000 0.20751 0.19166 0.40000 0.38800 0.82000
COF 2 0.33333 0.32000 0.13449 0.11718 0.33333 0.32000 0.67778
COF 7 0.00000 -0.02000 0.07764 0.05919 0.19048 0.17429 0.77333

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO