Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (2% of outliers version#06)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (26.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.33333 0.32000 0.24864 0.23361 0.37500 0.36250 0.95333
KNN 3 0.33333 0.32000 0.28472 0.27042 0.44444 0.43333 0.95778
KNN 4 0.33333 0.32000 0.36869 0.35606 0.44444 0.43333 0.97111
KNNW 1 0.00000 -0.02000 0.11545 0.09776 0.25000 0.23500 0.82222
KNNW 72 0.00000 -0.02000 0.24444 0.22933 0.44444 0.43333 0.95556
LOF 1 0.00000 -0.02000 0.04369 0.02456 0.10526 0.08737 0.58000
LOF 31 0.00000 -0.02000 0.24028 0.22508 0.44444 0.43333 0.95333
LOF 34 0.00000 -0.02000 0.26190 0.24714 0.40000 0.38800 0.96222
SimplifiedLOF 1 0.33333 0.32000 0.34658 0.33351 0.50000 0.49000 0.51111
SimplifiedLOF 99 0.00000 -0.02000 0.21746 0.20181 0.40000 0.38800 0.94667
LoOP 1 0.33333 0.32000 0.34641 0.33333 0.50000 0.49000 0.50444
LoOP 98 0.00000 -0.02000 0.23660 0.22133 0.44444 0.43333 0.95111
LDOF 2 0.33333 0.32000 0.35362 0.34069 0.50000 0.49000 0.63556
LDOF 98 0.00000 -0.02000 0.17778 0.16133 0.30769 0.29385 0.93778
ODIN 2 0.03846 0.01923 0.03288 0.01354 0.06897 0.05034 0.69444
ODIN 71 0.00000 -0.02000 0.28333 0.26900 0.46154 0.45077 0.96222
ODIN 74 0.00000 -0.02000 0.23889 0.22367 0.46154 0.45077 0.96444
ODIN 75 0.00000 -0.02000 0.27778 0.26333 0.50000 0.49000 0.96444
FastABOD 9 0.33333 0.32000 0.21212 0.19636 0.33333 0.32000 0.93111
FastABOD 56 0.33333 0.32000 0.36111 0.34833 0.50000 0.49000 0.97111
KDEOS 3 0.33333 0.32000 0.35437 0.34145 0.50000 0.49000 0.58222
KDEOS 100 0.00000 -0.02000 0.11059 0.09280 0.20690 0.19103 0.89333
LDF 19 0.33333 0.32000 0.45741 0.44656 0.50000 0.49000 0.94667
LDF 67 0.00000 -0.02000 0.25000 0.23500 0.46154 0.45077 0.96222
INFLO 1 0.00000 -0.02000 0.03050 0.01111 0.09524 0.07714 0.40444
INFLO 46 0.00000 -0.02000 0.14217 0.12501 0.30769 0.29385 0.91333
INFLO 100 0.00000 -0.02000 0.16111 0.14433 0.26667 0.25200 0.92889
COF 1 0.33333 0.32000 0.34667 0.33360 0.50000 0.49000 0.51111
COF 54 0.00000 -0.02000 0.26190 0.24714 0.50000 0.49000 0.96444

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (28.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02000 0.09954 0.08153 0.25000 0.23500 0.81444
KNN 2 0.00000 -0.02000 0.09975 0.08174 0.26667 0.25200 0.80333
KNN 45 0.00000 -0.02000 0.09592 0.07784 0.20000 0.18400 0.87778
KNNW 1 0.00000 -0.02000 0.14576 0.12867 0.33333 0.32000 0.84222
KNNW 88 0.00000 -0.02000 0.08890 0.07068 0.19048 0.17429 0.86667
LOF 1 0.33333 0.32000 0.17657 0.16010 0.33333 0.32000 0.75667
LOF 71 0.00000 -0.02000 0.08433 0.06601 0.19355 0.17742 0.86222
SimplifiedLOF 1 0.00000 -0.02000 0.07575 0.05726 0.16667 0.15000 0.78222
SimplifiedLOF 4 0.00000 -0.02000 0.08823 0.06999 0.21053 0.19474 0.72444
SimplifiedLOF 100 0.00000 -0.02000 0.07416 0.05565 0.16000 0.14320 0.83778
LoOP 1 0.00000 -0.02000 0.07575 0.05726 0.16667 0.15000 0.78222
LoOP 4 0.00000 -0.02000 0.08130 0.06293 0.18182 0.16545 0.70889
LoOP 16 0.00000 -0.02000 0.07585 0.05736 0.20000 0.18400 0.74444
LoOP 79 0.00000 -0.02000 0.07386 0.05533 0.16667 0.15000 0.82667
LDOF 2 0.00000 -0.02000 0.04420 0.02508 0.10256 0.08462 0.69778
LDOF 38 0.00000 -0.02000 0.07997 0.06157 0.21053 0.19474 0.77333
LDOF 88 0.00000 -0.02000 0.06973 0.05112 0.15385 0.13692 0.82222
ODIN 4 0.20000 0.18400 0.10359 0.08566 0.25000 0.23500 0.70000
ODIN 8 0.00000 -0.02000 0.15938 0.14257 0.33333 0.32000 0.79000
ODIN 99 0.00000 -0.02000 0.08488 0.06658 0.20000 0.18400 0.86444
FastABOD 3 0.00000 -0.02000 0.11389 0.09617 0.22222 0.20667 0.80667
FastABOD 4 0.00000 -0.02000 0.17593 0.15944 0.33333 0.32000 0.86444
FastABOD 5 0.00000 -0.02000 0.16818 0.15155 0.36364 0.35091 0.86222
FastABOD 72 0.00000 -0.02000 0.12870 0.11128 0.30769 0.29385 0.88222
KDEOS 2 0.00000 -0.02000 0.02144 0.00187 0.04211 0.02295 0.46333
KDEOS 40 0.00000 -0.02000 0.15105 0.13407 0.33333 0.32000 0.76667
KDEOS 56 0.00000 -0.02000 0.12235 0.10479 0.26667 0.25200 0.77556
LDF 1 0.66667 0.66000 0.39931 0.38729 0.66667 0.66000 0.79000
LDF 69 0.00000 -0.02000 0.09962 0.08161 0.24000 0.22480 0.88444
INFLO 1 0.00000 -0.02000 0.03128 0.01190 0.06557 0.04689 0.60000
INFLO 4 0.00000 -0.02000 0.07671 0.05824 0.21053 0.19474 0.77111
INFLO 80 0.00000 -0.02000 0.07451 0.05600 0.16667 0.15000 0.84222
COF 2 0.33333 0.32000 0.15395 0.13703 0.33333 0.32000 0.67778
COF 4 0.33333 0.32000 0.28798 0.27374 0.57143 0.56286 0.78000
COF 98 0.00000 -0.02000 0.14679 0.12973 0.30769 0.29385 0.91556

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO