Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (2% of outliers version#09)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (19.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.66667 0.66000 0.69697 0.69091 0.80000 0.79600 0.93333
KNN 3 0.66667 0.66000 0.75556 0.75067 0.75000 0.74500 0.99333
KNNW 1 0.66667 0.66000 0.71429 0.70857 0.80000 0.79600 0.96000
KNNW 7 0.66667 0.66000 0.75758 0.75273 0.80000 0.79600 0.98222
KNNW 32 0.66667 0.66000 0.69841 0.69238 0.66667 0.66000 0.98889
LOF 26 0.66667 0.66000 0.62698 0.61952 0.66667 0.66000 0.97333
LOF 30 0.66667 0.66000 0.69841 0.69238 0.66667 0.66000 0.98889
SimplifiedLOF 37 0.33333 0.32000 0.21296 0.19722 0.33333 0.32000 0.90667
SimplifiedLOF 91 0.33333 0.32000 0.42063 0.40905 0.60000 0.59200 0.98222
SimplifiedLOF 92 0.33333 0.32000 0.47619 0.46571 0.60000 0.59200 0.98444
LoOP 38 0.33333 0.32000 0.22365 0.20812 0.33333 0.32000 0.92889
LoOP 92 0.33333 0.32000 0.47619 0.46571 0.60000 0.59200 0.98444
LDOF 44 0.33333 0.32000 0.20370 0.18778 0.33333 0.32000 0.92000
LDOF 97 0.33333 0.32000 0.37778 0.36533 0.57143 0.56286 0.97556
LDOF 100 0.33333 0.32000 0.38889 0.37667 0.57143 0.56286 0.97778
ODIN 50 0.66667 0.66000 0.48889 0.47867 0.66667 0.66000 0.98111
ODIN 52 0.66667 0.66000 0.65556 0.64867 0.66667 0.66000 0.98222
ODIN 80 0.66667 0.66000 0.53175 0.52238 0.66667 0.66000 0.98778
FastABOD 12 0.66667 0.66000 0.47980 0.46939 0.66667 0.66000 0.97778
FastABOD 29 0.66667 0.66000 0.68056 0.67417 0.66667 0.66000 0.98667
FastABOD 36 0.66667 0.66000 0.55556 0.54667 0.66667 0.66000 0.98889
KDEOS 4 0.33333 0.32000 0.35232 0.33937 0.50000 0.49000 0.62222
KDEOS 100 0.00000 -0.02000 0.10020 0.08220 0.20000 0.18400 0.84444
LDF 19 0.66667 0.66000 0.86667 0.86400 0.80000 0.79600 0.99556
INFLO 24 0.33333 0.32000 0.21429 0.19857 0.33333 0.32000 0.93333
INFLO 94 0.00000 -0.02000 0.38333 0.37100 0.66667 0.66000 0.98000
INFLO 95 0.33333 0.32000 0.41111 0.39933 0.66667 0.66000 0.98222
INFLO 99 0.33333 0.32000 0.42063 0.40905 0.60000 0.59200 0.98222
COF 18 0.66667 0.66000 0.58056 0.57217 0.66667 0.66000 0.91556
COF 77 0.66667 0.66000 0.91667 0.91500 0.85714 0.85429 0.99778

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (17.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.33333 0.32000 0.27778 0.26333 0.40000 0.38800 0.96222
KNNW 1 0.00000 -0.02000 0.18516 0.16886 0.40000 0.38800 0.89333
KNNW 5 0.00000 -0.02000 0.23413 0.21881 0.40000 0.38800 0.95778
KNNW 8 0.33333 0.32000 0.24864 0.23361 0.37500 0.36250 0.95333
LOF 1 0.33333 0.32000 0.13497 0.11767 0.33333 0.32000 0.68444
LOF 8 0.33333 0.32000 0.27489 0.26039 0.40000 0.38800 0.93778
LOF 38 0.33333 0.32000 0.24359 0.22846 0.37500 0.36250 0.95111
SimplifiedLOF 16 0.33333 0.32000 0.22435 0.20884 0.33333 0.32000 0.94000
SimplifiedLOF 46 0.33333 0.32000 0.23810 0.22286 0.35294 0.34000 0.94889
LoOP 17 0.33333 0.32000 0.22222 0.20667 0.33333 0.32000 0.94000
LoOP 36 0.33333 0.32000 0.26905 0.25443 0.40000 0.38800 0.92444
LoOP 37 0.33333 0.32000 0.27333 0.25880 0.40000 0.38800 0.93111
LoOP 80 0.33333 0.32000 0.23838 0.22315 0.33333 0.32000 0.94889
LDOF 17 0.33333 0.32000 0.20819 0.19235 0.33333 0.32000 0.93111
LDOF 36 0.33333 0.32000 0.25595 0.24107 0.40000 0.38800 0.92444
LDOF 46 0.33333 0.32000 0.26340 0.24867 0.40000 0.38800 0.93111
LDOF 91 0.33333 0.32000 0.23422 0.21890 0.33333 0.32000 0.94667
ODIN 26 0.00000 -0.02000 0.16389 0.14717 0.36364 0.35091 0.89889
ODIN 75 0.33333 0.32000 0.21429 0.19857 0.33333 0.32000 0.94000
ODIN 80 0.33333 0.32000 0.23054 0.21515 0.33333 0.32000 0.94556
ODIN 81 0.33333 0.32000 0.23054 0.21515 0.33333 0.32000 0.94778
FastABOD 3 0.00000 -0.02000 0.11554 0.09785 0.24000 0.22480 0.90222
FastABOD 9 0.00000 -0.02000 0.24206 0.22690 0.50000 0.49000 0.96000
FastABOD 24 0.00000 -0.02000 0.26190 0.24714 0.50000 0.49000 0.96444
KDEOS 6 0.33333 0.32000 0.18566 0.16937 0.40000 0.38800 0.59778
KDEOS 7 0.33333 0.32000 0.19008 0.17388 0.40000 0.38800 0.66889
KDEOS 88 0.00000 -0.02000 0.14949 0.13248 0.28571 0.27143 0.90889
LDF 1 0.33333 0.32000 0.19642 0.18035 0.40000 0.38800 0.73111
LDF 8 0.33333 0.32000 0.27778 0.26333 0.40000 0.38800 0.94222
LDF 44 0.33333 0.32000 0.26587 0.25119 0.36364 0.35091 0.95778
INFLO 8 0.33333 0.32000 0.22619 0.21071 0.40000 0.38800 0.88222
INFLO 36 0.33333 0.32000 0.28977 0.27557 0.40000 0.38800 0.94889
INFLO 58 0.33333 0.32000 0.24315 0.22801 0.35294 0.34000 0.95111
COF 2 0.33333 0.32000 0.13864 0.12141 0.33333 0.32000 0.68222
COF 30 0.33333 0.32000 0.36905 0.35643 0.57143 0.56286 0.93778
COF 61 0.33333 0.32000 0.40476 0.39286 0.57143 0.56286 0.96889

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO