Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (2% of outliers version#01)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (32.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02000 0.03705 0.01779 0.08451 0.06620 0.66889
KNN 54 0.00000 -0.02000 0.09206 0.07390 0.22222 0.20667 0.85556
KNN 62 0.00000 -0.02000 0.10159 0.08362 0.22222 0.20667 0.86444
KNNW 1 0.00000 -0.02000 0.03884 0.01961 0.10000 0.08200 0.63778
KNNW 72 0.00000 -0.02000 0.07270 0.05415 0.14286 0.12571 0.83111
LOF 1 0.00000 -0.02000 0.02349 0.00396 0.06742 0.04876 0.44889
LOF 75 0.00000 -0.02000 0.09281 0.07467 0.22222 0.20667 0.85333
LOF 77 0.00000 -0.02000 0.09500 0.07690 0.23529 0.22000 0.85111
SimplifiedLOF 1 0.00000 -0.02000 0.01825 -0.00138 0.03947 0.02026 0.35556
SimplifiedLOF 92 0.00000 -0.02000 0.05540 0.03651 0.12121 0.10364 0.78000
SimplifiedLOF 95 0.00000 -0.02000 0.05530 0.03641 0.12500 0.10750 0.77778
SimplifiedLOF 100 0.00000 -0.02000 0.05581 0.03693 0.12500 0.10750 0.78000
LoOP 1 0.00000 -0.02000 0.01790 -0.00174 0.03846 0.01923 0.34667
LoOP 90 0.00000 -0.02000 0.05886 0.04004 0.13333 0.11600 0.79111
LoOP 98 0.00000 -0.02000 0.06052 0.04173 0.12903 0.11161 0.79778
LoOP 99 0.00000 -0.02000 0.06088 0.04210 0.12903 0.11161 0.79778
LDOF 2 0.00000 -0.02000 0.02452 0.00501 0.05556 0.03667 0.49111
LDOF 66 0.00000 -0.02000 0.04500 0.02590 0.11538 0.09769 0.72889
LDOF 98 0.00000 -0.02000 0.05085 0.03186 0.11429 0.09657 0.75778
ODIN 1 0.01852 -0.00111 0.02016 0.00056 0.04110 0.02192 0.38333
ODIN 81 0.00000 -0.02000 0.08179 0.06342 0.20000 0.18400 0.84000
ODIN 99 0.00000 -0.02000 0.09477 0.07667 0.20000 0.18400 0.85444
FastABOD 3 0.00000 -0.02000 0.04594 0.02686 0.10909 0.09127 0.73333
FastABOD 98 0.00000 -0.02000 0.14803 0.13099 0.28571 0.27143 0.89111
KDEOS 2 0.00000 -0.02000 0.01606 -0.00362 0.04082 0.02163 0.32111
KDEOS 99 0.00000 -0.02000 0.03948 0.02027 0.10345 0.08552 0.68444
LDF 67 0.33333 0.32000 0.17063 0.15405 0.33333 0.32000 0.88000
LDF 73 0.33333 0.32000 0.17323 0.15669 0.33333 0.32000 0.88222
LDF 76 0.00000 -0.02000 0.16164 0.14488 0.28571 0.27143 0.89333
INFLO 1 0.00000 -0.02000 0.01729 -0.00236 0.04444 0.02533 0.26889
INFLO 100 0.00000 -0.02000 0.07096 0.05238 0.15385 0.13692 0.83333
COF 45 0.33333 0.32000 0.17009 0.15349 0.33333 0.32000 0.87556
COF 69 0.33333 0.32000 0.38437 0.37205 0.50000 0.49000 0.86222
COF 79 0.33333 0.32000 0.39262 0.38047 0.50000 0.49000 0.88667
COF 85 0.33333 0.32000 0.23401 0.21869 0.40000 0.38800 0.89556

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (30.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02000 0.06966 0.05105 0.15385 0.13692 0.78889
KNN 7 0.00000 -0.02000 0.06713 0.04847 0.13333 0.11600 0.79556
KNNW 1 0.00000 -0.02000 0.07710 0.05864 0.14286 0.12571 0.83333
LOF 2 0.33333 0.32000 0.13842 0.12118 0.33333 0.32000 0.73556
LOF 14 0.00000 -0.02000 0.06210 0.04335 0.11765 0.10000 0.79556
SimplifiedLOF 1 0.00000 -0.02000 0.01856 -0.00107 0.04255 0.02340 0.32889
SimplifiedLOF 2 0.00000 -0.02000 0.05613 0.03726 0.16667 0.15000 0.60444
SimplifiedLOF 27 0.00000 -0.02000 0.05754 0.03869 0.11765 0.10000 0.76444
SimplifiedLOF 32 0.00000 -0.02000 0.05517 0.03627 0.10000 0.08200 0.76667
LoOP 1 0.00000 -0.02000 0.02065 0.00106 0.04255 0.02340 0.41556
LoOP 2 0.00000 -0.02000 0.04516 0.02607 0.12500 0.10750 0.60667
LoOP 96 0.00000 -0.02000 0.05407 0.03515 0.11765 0.10000 0.77333
LoOP 100 0.00000 -0.02000 0.05442 0.03551 0.12121 0.10364 0.77333
LDOF 2 0.00000 -0.02000 0.02595 0.00647 0.07143 0.05286 0.51111
LDOF 42 0.00000 -0.02000 0.06236 0.04361 0.15000 0.13300 0.80667
LDOF 43 0.00000 -0.02000 0.06210 0.04335 0.15385 0.13692 0.80667
ODIN 1 0.01961 0.00000 0.01933 -0.00029 0.04082 0.02163 0.48333
ODIN 90 0.00000 -0.02000 0.06341 0.04467 0.12245 0.10490 0.77556
ODIN 95 0.00000 -0.02000 0.05713 0.03827 0.13636 0.11909 0.79444
ODIN 100 0.00000 -0.02000 0.05763 0.03878 0.14634 0.12927 0.79444
FastABOD 3 0.00000 -0.02000 0.05808 0.03924 0.12121 0.10364 0.73111
FastABOD 10 0.00000 -0.02000 0.06377 0.04505 0.16667 0.15000 0.71778
FastABOD 26 0.00000 -0.02000 0.06473 0.04602 0.16667 0.15000 0.72667
FastABOD 83 0.00000 -0.02000 0.06231 0.04355 0.15385 0.13692 0.73778
KDEOS 2 0.00000 -0.02000 0.02144 0.00187 0.04211 0.02295 0.46111
KDEOS 64 0.00000 -0.02000 0.05635 0.03747 0.14286 0.12571 0.70667
KDEOS 97 0.00000 -0.02000 0.05158 0.03261 0.10256 0.08462 0.75333
LDF 2 0.33333 0.32000 0.26429 0.24957 0.40000 0.38800 0.93333
INFLO 1 0.00000 -0.02000 0.02439 0.00487 0.05000 0.03100 0.44222
INFLO 75 0.00000 -0.02000 0.06530 0.04660 0.14286 0.12571 0.81778
INFLO 80 0.00000 -0.02000 0.06604 0.04737 0.13333 0.11600 0.81778
INFLO 95 0.00000 -0.02000 0.06123 0.04245 0.15000 0.13300 0.80444
COF 1 0.00000 -0.02000 0.01853 -0.00109 0.04167 0.02250 0.33222
COF 8 0.00000 -0.02000 0.17041 0.15382 0.30769 0.29385 0.87333
COF 99 0.00000 -0.02000 0.10765 0.08981 0.20000 0.18400 0.87556

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO