Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (10% of outliers version#09)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (44.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 22 0.56250 0.51583 0.42044 0.35862 0.56250 0.51583 0.89458
KNN 24 0.50000 0.44667 0.42368 0.36221 0.55000 0.50200 0.89625
KNN 30 0.43750 0.37750 0.43552 0.37531 0.57143 0.52571 0.88667
KNN 31 0.43750 0.37750 0.46834 0.41163 0.57143 0.52571 0.88542
KNNW 32 0.37500 0.30833 0.41803 0.35595 0.55000 0.50200 0.88333
KNNW 48 0.43750 0.37750 0.40457 0.34106 0.53333 0.48356 0.88083
KNNW 83 0.43750 0.37750 0.44859 0.38977 0.55000 0.50200 0.88792
KNNW 87 0.43750 0.37750 0.44868 0.38987 0.53659 0.48715 0.88792
LOF 56 0.43750 0.37750 0.37964 0.31346 0.50000 0.44667 0.88333
LOF 87 0.37500 0.30833 0.45568 0.39762 0.60000 0.55733 0.90083
LOF 91 0.37500 0.30833 0.46887 0.41221 0.60000 0.55733 0.90125
SimplifiedLOF 91 0.31250 0.23917 0.30906 0.23536 0.46154 0.40410 0.85167
SimplifiedLOF 94 0.37500 0.30833 0.31730 0.24448 0.44828 0.38943 0.85458
SimplifiedLOF 100 0.37500 0.30833 0.33785 0.26722 0.44898 0.39020 0.86208
LoOP 91 0.37500 0.30833 0.32013 0.24761 0.45614 0.39813 0.85333
LoOP 96 0.37500 0.30833 0.32881 0.25722 0.46429 0.40714 0.85708
LoOP 100 0.31250 0.23917 0.36482 0.29706 0.45614 0.39813 0.85958
LDOF 42 0.31250 0.23917 0.19575 0.10996 0.31250 0.23917 0.73708
LDOF 95 0.18750 0.10083 0.30261 0.22822 0.43636 0.37624 0.82917
LDOF 100 0.25000 0.17000 0.28662 0.21052 0.44444 0.38519 0.83875
ODIN 69 0.37500 0.30833 0.35879 0.29040 0.55814 0.51101 0.88083
ODIN 80 0.50000 0.44667 0.36840 0.30103 0.50000 0.44667 0.87833
ODIN 93 0.37500 0.30833 0.41436 0.35189 0.54054 0.49153 0.89229
ODIN 96 0.39583 0.33139 0.41802 0.35594 0.53333 0.48356 0.89104
FastABOD 87 0.56250 0.51583 0.50948 0.45716 0.58824 0.54431 0.89458
FastABOD 92 0.62500 0.58500 0.50539 0.45263 0.62500 0.58500 0.89250
KDEOS 5 0.25000 0.17000 0.12444 0.03105 0.25000 0.17000 0.51042
KDEOS 96 0.18750 0.10083 0.19581 0.11003 0.33333 0.26222 0.75542
KDEOS 100 0.18750 0.10083 0.20205 0.11693 0.33333 0.26222 0.76375
LDF 21 0.56250 0.51583 0.60659 0.56463 0.61111 0.56963 0.91625
LDF 22 0.56250 0.51583 0.60119 0.55865 0.57895 0.53404 0.91958
LDF 24 0.56250 0.51583 0.58519 0.54094 0.62857 0.58895 0.91917
INFLO 65 0.37500 0.30833 0.27496 0.19762 0.40000 0.33600 0.83458
INFLO 76 0.31250 0.23917 0.31026 0.23669 0.46154 0.40410 0.86208
INFLO 97 0.25000 0.17000 0.36244 0.29443 0.49057 0.43623 0.83500
INFLO 99 0.25000 0.17000 0.36934 0.30207 0.48387 0.42882 0.83750
COF 57 0.56250 0.51583 0.51501 0.46328 0.58065 0.53591 0.89833
COF 87 0.37500 0.30833 0.55116 0.50328 0.52632 0.47579 0.90875
COF 88 0.37500 0.30833 0.55476 0.50727 0.51064 0.45844 0.90708

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (42.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.37500 0.30833 0.28168 0.20506 0.50000 0.44667 0.75458
KNN 4 0.25000 0.17000 0.27726 0.20017 0.52632 0.47579 0.75104
KNN 7 0.37500 0.30833 0.29108 0.21547 0.48780 0.43317 0.76208
KNNW 3 0.43750 0.37750 0.27856 0.20161 0.43750 0.37750 0.75042
KNNW 5 0.31250 0.23917 0.27389 0.19644 0.44444 0.38519 0.75833
KNNW 12 0.31250 0.23917 0.28692 0.21086 0.52632 0.47579 0.75833
KNNW 19 0.31250 0.23917 0.28739 0.21138 0.52632 0.47579 0.75667
LOF 28 0.37500 0.30833 0.24357 0.16289 0.40000 0.33600 0.71792
LOF 64 0.25000 0.17000 0.24577 0.16532 0.43478 0.37449 0.74583
LOF 68 0.25000 0.17000 0.24330 0.16259 0.45833 0.40056 0.74792
LOF 74 0.25000 0.17000 0.23916 0.15800 0.42308 0.36154 0.75000
SimplifiedLOF 25 0.25000 0.17000 0.24245 0.16164 0.40000 0.33600 0.72583
SimplifiedLOF 43 0.31250 0.23917 0.23850 0.15728 0.40909 0.34606 0.72125
SimplifiedLOF 60 0.31250 0.23917 0.24425 0.16363 0.41667 0.35444 0.72083
SimplifiedLOF 97 0.25000 0.17000 0.24074 0.15976 0.43478 0.37449 0.72333
LoOP 11 0.31250 0.23917 0.18890 0.10239 0.32787 0.25617 0.65812
LoOP 44 0.25000 0.17000 0.25328 0.17363 0.41860 0.35659 0.73958
LoOP 62 0.31250 0.23917 0.25357 0.17395 0.41176 0.34902 0.73625
LoOP 95 0.25000 0.17000 0.23121 0.14920 0.42553 0.36426 0.72625
LDOF 7 0.31250 0.23917 0.18802 0.10141 0.31250 0.23917 0.67667
LDOF 47 0.25000 0.17000 0.23485 0.15324 0.40000 0.33600 0.71292
LDOF 50 0.18750 0.10083 0.22850 0.14621 0.41667 0.35444 0.70542
LDOF 98 0.25000 0.17000 0.22831 0.14600 0.41667 0.35444 0.71417
ODIN 3 0.34250 0.27237 0.23224 0.15034 0.34483 0.27494 0.69667
ODIN 11 0.33333 0.26222 0.23412 0.15243 0.40000 0.33600 0.70896
ODIN 90 0.25000 0.17000 0.23048 0.14839 0.39130 0.32638 0.73625
ODIN 91 0.21875 0.13542 0.22708 0.14463 0.40816 0.34503 0.72896
FastABOD 10 0.43750 0.37750 0.27708 0.19997 0.43750 0.37750 0.73167
FastABOD 21 0.37500 0.30833 0.28571 0.20952 0.46154 0.40410 0.74583
FastABOD 98 0.37500 0.30833 0.28827 0.21236 0.43243 0.37189 0.75458
KDEOS 16 0.25000 0.17000 0.28044 0.20369 0.32000 0.24747 0.61708
KDEOS 39 0.31250 0.23917 0.21299 0.12904 0.31250 0.23917 0.67875
KDEOS 83 0.12500 0.03167 0.18684 0.10010 0.38596 0.32047 0.69500
KDEOS 98 0.12500 0.03167 0.18892 0.10240 0.35294 0.28392 0.69792
LDF 10 0.31250 0.23917 0.27111 0.19336 0.47368 0.41754 0.77292
LDF 13 0.43750 0.37750 0.28259 0.20607 0.46154 0.40410 0.76625
LDF 26 0.43750 0.37750 0.28443 0.20810 0.45714 0.39924 0.75792
LDF 41 0.31250 0.23917 0.27278 0.19521 0.48889 0.43437 0.75750
INFLO 7 0.31250 0.23917 0.22845 0.14615 0.37838 0.31207 0.66208
INFLO 43 0.25000 0.17000 0.24090 0.15993 0.41667 0.35444 0.71521
INFLO 45 0.25000 0.17000 0.24171 0.16082 0.41667 0.35444 0.75146
INFLO 98 0.25000 0.17000 0.23157 0.14960 0.39286 0.32810 0.78625
COF 94 0.56250 0.51583 0.41076 0.34790 0.58065 0.53591 0.79250
COF 95 0.56250 0.51583 0.37816 0.31184 0.56250 0.51583 0.79458
COF 98 0.56250 0.51583 0.42139 0.35967 0.60000 0.55733 0.78958
COF 99 0.56250 0.51583 0.40821 0.34509 0.62069 0.58023 0.78687

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO