Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (2% of outliers version#10)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (35.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02000 0.02130 0.00172 0.04839 0.02935 0.39889
KNN 12 0.00000 -0.02000 0.03708 0.01782 0.08824 0.07000 0.66222
KNN 15 0.00000 -0.02000 0.03841 0.01918 0.08333 0.06500 0.66444
KNNW 1 0.00000 -0.02000 0.02294 0.00339 0.04800 0.02896 0.43222
KNNW 16 0.00000 -0.02000 0.03101 0.01163 0.07500 0.05650 0.59556
KNNW 20 0.00000 -0.02000 0.03218 0.01282 0.07500 0.05650 0.60889
KNNW 22 0.00000 -0.02000 0.03235 0.01299 0.07500 0.05650 0.60889
LOF 1 0.00000 -0.02000 0.02030 0.00070 0.04348 0.02435 0.33556
LOF 15 0.00000 -0.02000 0.03715 0.01789 0.07407 0.05556 0.64222
LOF 16 0.00000 -0.02000 0.03650 0.01723 0.07317 0.05463 0.64889
LOF 17 0.00000 -0.02000 0.03529 0.01600 0.07895 0.06053 0.64889
SimplifiedLOF 1 0.00000 -0.02000 0.05734 0.03849 0.18182 0.16545 0.58222
LoOP 1 0.00000 -0.02000 0.05721 0.03836 0.18182 0.16545 0.57889
LoOP 90 0.00000 -0.02000 0.02965 0.01025 0.07500 0.05650 0.58222
LDOF 2 0.00000 -0.02000 0.08194 0.06358 0.22222 0.20667 0.71556
ODIN 1 0.01852 -0.00111 0.01874 -0.00089 0.03922 0.02000 0.44000
ODIN 38 0.00000 -0.02000 0.03018 0.01078 0.07317 0.05463 0.59667
ODIN 42 0.00000 -0.02000 0.03123 0.01185 0.07143 0.05286 0.54778
ODIN 79 0.00000 -0.02000 0.03038 0.01099 0.06977 0.05116 0.59889
FastABOD 3 0.00000 -0.02000 0.02517 0.00567 0.05405 0.03514 0.46000
FastABOD 97 0.00000 -0.02000 0.03744 0.01819 0.08333 0.06500 0.62667
FastABOD 99 0.00000 -0.02000 0.03758 0.01833 0.08333 0.06500 0.62889
KDEOS 2 0.00000 -0.02000 0.02436 0.00485 0.05660 0.03774 0.54667
KDEOS 3 0.00000 -0.02000 0.05650 0.03763 0.14815 0.13111 0.70222
KDEOS 7 0.00000 -0.02000 0.06792 0.04927 0.22222 0.20667 0.40667
KDEOS 10 0.00000 -0.02000 0.07003 0.05143 0.22222 0.20667 0.49333
LDF 1 0.00000 -0.02000 0.02301 0.00347 0.05714 0.03829 0.36222
LDF 9 0.00000 -0.02000 0.04471 0.02560 0.13333 0.11600 0.58667
LDF 17 0.00000 -0.02000 0.05988 0.04107 0.13333 0.11600 0.78000
INFLO 1 0.00000 -0.02000 0.02966 0.01025 0.06593 0.04725 0.57778
INFLO 90 0.00000 -0.02000 0.04154 0.02237 0.11538 0.09769 0.69556
COF 1 0.00000 -0.02000 0.05734 0.03849 0.18182 0.16545 0.58222
COF 62 0.00000 -0.02000 0.04641 0.02734 0.10000 0.08200 0.70000

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 153 objects, 3 outliers (1.96%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (37.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02000 0.01782 -0.00183 0.03947 0.02026 0.25222
KNN 7 0.00000 -0.02000 0.02412 0.00461 0.05660 0.03774 0.47556
KNN 27 0.00000 -0.02000 0.02239 0.00284 0.06316 0.04442 0.43000
KNNW 1 0.00000 -0.02000 0.02098 0.00140 0.04651 0.02744 0.34556
KNNW 25 0.00000 -0.02000 0.02211 0.00255 0.05556 0.03667 0.43333
KNNW 58 0.00000 -0.02000 0.02192 0.00236 0.06061 0.04182 0.41778
LOF 1 0.00000 -0.02000 0.02735 0.00789 0.06250 0.04375 0.53222
LOF 10 0.00000 -0.02000 0.02823 0.00880 0.06742 0.04876 0.56222
LOF 14 0.00000 -0.02000 0.02737 0.00792 0.07059 0.05200 0.54444
SimplifiedLOF 1 0.00000 -0.02000 0.03480 0.01550 0.08000 0.06160 0.51444
LoOP 1 0.00000 -0.02000 0.03476 0.01545 0.08000 0.06160 0.59556
LDOF 2 0.00000 -0.02000 0.06909 0.05047 0.15385 0.13692 0.71111
ODIN 2 0.04762 0.02857 0.02957 0.01016 0.08333 0.06500 0.38222
ODIN 9 0.00000 -0.02000 0.04666 0.02759 0.14286 0.12571 0.56778
ODIN 13 0.00000 -0.02000 0.03447 0.01516 0.08333 0.06500 0.59667
FastABOD 3 0.00000 -0.02000 0.02443 0.00492 0.05714 0.03829 0.41556
KDEOS 3 0.00000 -0.02000 0.10377 0.08585 0.25000 0.23500 0.62667
KDEOS 5 0.33333 0.32000 0.12614 0.10866 0.33333 0.32000 0.51778
KDEOS 6 0.33333 0.32000 0.34773 0.33468 0.50000 0.49000 0.49778
LDF 1 0.00000 -0.02000 0.03114 0.01177 0.06897 0.05034 0.55444
LDF 2 0.00000 -0.02000 0.08305 0.06471 0.25000 0.23500 0.53333
LDF 3 0.00000 -0.02000 0.04959 0.03059 0.12500 0.10750 0.60444
INFLO 1 0.00000 -0.02000 0.02016 0.00057 0.04598 0.02690 0.36556
INFLO 4 0.00000 -0.02000 0.04103 0.02185 0.10000 0.08200 0.70222
INFLO 6 0.00000 -0.02000 0.03629 0.01701 0.10811 0.09027 0.52667
COF 1 0.00000 -0.02000 0.03398 0.01466 0.07692 0.05846 0.51556
COF 11 0.00000 -0.02000 0.04108 0.02190 0.12500 0.10750 0.50667
COF 12 0.00000 -0.02000 0.03562 0.01634 0.09524 0.07714 0.54222

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO