Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (2% of outliers version#03)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (28.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.33333 0.32000 0.36386 0.35114 0.50000 0.49000 0.76889
KNN 16 0.33333 0.32000 0.20217 0.18621 0.40000 0.38800 0.80222
KNN 23 0.33333 0.32000 0.36604 0.35336 0.50000 0.49000 0.78667
KNNW 1 0.33333 0.32000 0.35217 0.33922 0.50000 0.49000 0.62000
KNNW 28 0.33333 0.32000 0.36362 0.35089 0.50000 0.49000 0.76889
LOF 27 0.33333 0.32000 0.19581 0.17973 0.40000 0.38800 0.75556
LOF 63 0.33333 0.32000 0.36223 0.34948 0.50000 0.49000 0.75333
LOF 67 0.33333 0.32000 0.36408 0.35137 0.50000 0.49000 0.76444
SimplifiedLOF 63 0.33333 0.32000 0.13575 0.11847 0.33333 0.32000 0.70889
SimplifiedLOF 66 0.33333 0.32000 0.19234 0.17619 0.40000 0.38800 0.72222
SimplifiedLOF 83 0.33333 0.32000 0.19483 0.17872 0.40000 0.38800 0.74667
LoOP 56 0.33333 0.32000 0.13636 0.11908 0.33333 0.32000 0.71556
LoOP 63 0.33333 0.32000 0.19171 0.17554 0.40000 0.38800 0.71556
LoOP 83 0.33333 0.32000 0.19476 0.17865 0.40000 0.38800 0.74667
LDOF 83 0.33333 0.32000 0.13756 0.12031 0.33333 0.32000 0.72889
LDOF 85 0.33333 0.32000 0.13804 0.12080 0.33333 0.32000 0.73333
LDOF 86 0.33333 0.32000 0.19270 0.17656 0.40000 0.38800 0.72667
ODIN 30 0.33333 0.32000 0.13894 0.12171 0.33333 0.32000 0.74667
ODIN 34 0.33333 0.32000 0.36111 0.34833 0.50000 0.49000 0.75111
ODIN 100 0.33333 0.32000 0.19710 0.18104 0.40000 0.38800 0.76556
FastABOD 4 0.00000 -0.02000 0.08939 0.07118 0.20000 0.18400 0.82222
FastABOD 5 0.33333 0.32000 0.19668 0.18062 0.40000 0.38800 0.76444
FastABOD 6 0.33333 0.32000 0.36540 0.35271 0.50000 0.49000 0.78222
FastABOD 75 0.33333 0.32000 0.37240 0.35985 0.50000 0.49000 0.82000
KDEOS 2 0.33333 0.32000 0.37363 0.36110 0.50000 0.49000 0.71889
LDF 17 0.00000 -0.02000 0.06870 0.05007 0.13793 0.12069 0.82000
LDF 19 0.33333 0.32000 0.36018 0.34738 0.50000 0.49000 0.73556
LDF 63 0.33333 0.32000 0.38291 0.37057 0.50000 0.49000 0.81556
INFLO 25 0.33333 0.32000 0.13915 0.12193 0.33333 0.32000 0.74444
INFLO 80 0.33333 0.32000 0.35662 0.34375 0.50000 0.49000 0.58667
INFLO 98 0.33333 0.32000 0.37071 0.35812 0.50000 0.49000 0.81333
INFLO 99 0.33333 0.32000 0.20490 0.18900 0.40000 0.38800 0.81556
COF 23 0.33333 0.32000 0.14532 0.12823 0.33333 0.32000 0.78889
COF 25 0.33333 0.32000 0.36701 0.35435 0.50000 0.49000 0.78889
COF 77 0.33333 0.32000 0.37619 0.36371 0.50000 0.49000 0.84000

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (28.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02000 0.10900 0.09118 0.25000 0.23500 0.75000
KNN 4 0.00000 -0.02000 0.08161 0.06324 0.19048 0.17429 0.75333
KNN 10 0.00000 -0.02000 0.11240 0.09465 0.25000 0.23500 0.70667
KNNW 1 0.00000 -0.02000 0.11982 0.10222 0.25000 0.23500 0.80000
KNNW 3 0.00000 -0.02000 0.12132 0.10374 0.26667 0.25200 0.75556
LOF 1 0.00000 -0.02000 0.03849 0.01926 0.09677 0.07871 0.68000
LOF 6 0.00000 -0.02000 0.10037 0.08237 0.25000 0.23500 0.78000
SimplifiedLOF 1 0.00000 -0.02000 0.04368 0.02455 0.10909 0.09127 0.72000
SimplifiedLOF 5 0.00000 -0.02000 0.09307 0.07494 0.20000 0.18400 0.80222
SimplifiedLOF 6 0.00000 -0.02000 0.11095 0.09317 0.25000 0.23500 0.78444
SimplifiedLOF 7 0.00000 -0.02000 0.09722 0.07917 0.26667 0.25200 0.76667
LoOP 1 0.00000 -0.02000 0.04368 0.02455 0.10909 0.09127 0.72000
LoOP 5 0.00000 -0.02000 0.09041 0.07222 0.20000 0.18400 0.80444
LoOP 6 0.00000 -0.02000 0.09323 0.07509 0.20000 0.18400 0.77778
LoOP 7 0.00000 -0.02000 0.09321 0.07508 0.25000 0.23500 0.76889
LDOF 2 0.00000 -0.02000 0.02608 0.00660 0.05769 0.03885 0.52000
LDOF 5 0.00000 -0.02000 0.06679 0.04813 0.15000 0.13300 0.82222
LDOF 20 0.00000 -0.02000 0.07780 0.05936 0.20000 0.18400 0.62444
ODIN 1 0.03846 0.01923 0.03441 0.01510 0.07273 0.05418 0.71000
ODIN 5 0.00000 -0.02000 0.05330 0.03437 0.10811 0.09027 0.77778
ODIN 16 0.00000 -0.02000 0.08552 0.06723 0.20000 0.18400 0.63556
FastABOD 6 0.00000 -0.02000 0.19048 0.17429 0.40000 0.38800 0.80222
FastABOD 7 0.00000 -0.02000 0.20594 0.19006 0.44444 0.43333 0.79778
FastABOD 13 0.33333 0.32000 0.18192 0.16556 0.33333 0.32000 0.76444
KDEOS 2 0.00000 -0.02000 0.03882 0.01960 0.09091 0.07273 0.71889
KDEOS 9 0.00000 -0.02000 0.06194 0.04317 0.13333 0.11600 0.80222
KDEOS 60 0.00000 -0.02000 0.06781 0.04916 0.19048 0.17429 0.68000
KDEOS 63 0.00000 -0.02000 0.06959 0.05098 0.17391 0.15739 0.68000
LDF 3 0.33333 0.32000 0.14140 0.12423 0.33333 0.32000 0.66444
LDF 7 0.00000 -0.02000 0.09910 0.08108 0.22222 0.20667 0.79111
INFLO 1 0.00000 -0.02000 0.03885 0.01963 0.09524 0.07714 0.66889
INFLO 7 0.00000 -0.02000 0.08577 0.06749 0.25000 0.23500 0.62667
INFLO 11 0.00000 -0.02000 0.08319 0.06485 0.19048 0.17429 0.77778
COF 1 0.00000 -0.02000 0.04269 0.02355 0.10714 0.08929 0.71333
COF 7 0.00000 -0.02000 0.14279 0.12564 0.33333 0.32000 0.81111
COF 9 0.00000 -0.02000 0.15045 0.13346 0.33333 0.32000 0.75333
COF 15 0.00000 -0.02000 0.13893 0.12171 0.33333 0.32000 0.84889

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO