Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (10% of outliers version#04)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (45.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 33 0.50000 0.44667 0.44191 0.38238 0.50000 0.44667 0.86042
KNN 55 0.50000 0.44667 0.45114 0.39260 0.51613 0.46452 0.87917
KNN 71 0.50000 0.44667 0.45455 0.39637 0.54545 0.49697 0.87542
KNN 75 0.50000 0.44667 0.46616 0.40922 0.54545 0.49697 0.87417
KNNW 89 0.43750 0.37750 0.41203 0.34931 0.45714 0.39924 0.86125
KNNW 90 0.43750 0.37750 0.41228 0.34959 0.45714 0.39924 0.86167
KNNW 92 0.43750 0.37750 0.41437 0.35191 0.45714 0.39924 0.86167
KNNW 95 0.43750 0.37750 0.41115 0.34834 0.47059 0.41412 0.86083
LOF 74 0.50000 0.44667 0.42014 0.35829 0.51613 0.46452 0.87458
LOF 86 0.43750 0.37750 0.44866 0.38985 0.51852 0.46716 0.87708
LOF 100 0.50000 0.44667 0.45870 0.40096 0.51852 0.46716 0.88333
SimplifiedLOF 86 0.31250 0.23917 0.26579 0.18747 0.35714 0.28857 0.79708
SimplifiedLOF 100 0.31250 0.23917 0.30747 0.23360 0.38462 0.31897 0.82000
LoOP 81 0.31250 0.23917 0.26783 0.18973 0.35088 0.28164 0.79542
LoOP 99 0.31250 0.23917 0.31695 0.24409 0.37931 0.31310 0.82250
LoOP 100 0.31250 0.23917 0.31864 0.24596 0.37931 0.31310 0.82458
LDOF 92 0.31250 0.23917 0.23497 0.15337 0.32258 0.25032 0.77833
LDOF 97 0.31250 0.23917 0.27171 0.19403 0.37037 0.30321 0.79458
LDOF 98 0.31250 0.23917 0.27602 0.19879 0.37037 0.30321 0.79625
LDOF 100 0.31250 0.23917 0.27143 0.19372 0.37037 0.30321 0.79833
ODIN 92 0.37500 0.30833 0.39178 0.32691 0.48649 0.43171 0.86208
ODIN 94 0.43750 0.37750 0.40875 0.34568 0.47059 0.41412 0.86208
ODIN 97 0.45833 0.40056 0.40387 0.34028 0.47059 0.41412 0.86229
ODIN 98 0.45833 0.40056 0.40097 0.33708 0.47059 0.41412 0.86333
FastABOD 87 0.56250 0.51583 0.52158 0.47055 0.56250 0.51583 0.90208
FastABOD 92 0.56250 0.51583 0.52996 0.47982 0.60000 0.55733 0.90333
FastABOD 94 0.56250 0.51583 0.53070 0.48064 0.60000 0.55733 0.90333
KDEOS 7 0.31250 0.23917 0.20762 0.12310 0.34483 0.27494 0.58833
KDEOS 100 0.06250 -0.03750 0.16020 0.07062 0.31579 0.24281 0.71500
LDF 26 0.50000 0.44667 0.43075 0.37003 0.50000 0.44667 0.86167
LDF 67 0.50000 0.44667 0.58473 0.54043 0.59459 0.55135 0.92042
LDF 69 0.50000 0.44667 0.60736 0.56548 0.58537 0.54114 0.91833
INFLO 84 0.43750 0.37750 0.37081 0.30370 0.43750 0.37750 0.85708
INFLO 98 0.37500 0.30833 0.40151 0.33767 0.50000 0.44667 0.88333
INFLO 100 0.37500 0.30833 0.40122 0.33735 0.52459 0.47388 0.88833
COF 54 0.50000 0.44667 0.47430 0.41822 0.50000 0.44667 0.88750
COF 67 0.50000 0.44667 0.54369 0.49502 0.56522 0.51884 0.92125
COF 92 0.43750 0.37750 0.62048 0.58000 0.58333 0.53889 0.90875

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (45.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.12500 0.03167 0.16161 0.07218 0.33333 0.26222 0.67792
KNN 4 0.12500 0.03167 0.16151 0.07207 0.34783 0.27826 0.66500
KNN 10 0.12500 0.03167 0.16453 0.07541 0.32432 0.25225 0.67104
KNNW 1 0.18750 0.10083 0.19284 0.10674 0.30769 0.23385 0.70854
KNNW 8 0.12500 0.03167 0.16134 0.07188 0.34286 0.27276 0.67625
LOF 2 0.25000 0.17000 0.14799 0.05711 0.29412 0.21882 0.50938
LOF 3 0.25000 0.17000 0.16882 0.08016 0.30303 0.22869 0.56458
LOF 10 0.12500 0.03167 0.16344 0.07421 0.29730 0.22234 0.68750
LOF 17 0.12500 0.03167 0.16162 0.07219 0.32877 0.25717 0.67667
SimplifiedLOF 3 0.25000 0.17000 0.18295 0.09580 0.30303 0.22869 0.61292
SimplifiedLOF 5 0.18750 0.10083 0.16732 0.07851 0.30769 0.23385 0.62750
SimplifiedLOF 29 0.06250 -0.03750 0.15063 0.06003 0.25714 0.17790 0.64958
LoOP 3 0.25000 0.17000 0.17575 0.08782 0.30000 0.22533 0.60708
LoOP 4 0.12500 0.03167 0.18321 0.09608 0.27907 0.20217 0.62875
LoOP 5 0.18750 0.10083 0.15918 0.06949 0.32000 0.24747 0.62000
LoOP 38 0.06250 -0.03750 0.14199 0.05047 0.25974 0.18078 0.64125
LDOF 3 0.18750 0.10083 0.18106 0.09371 0.29787 0.22298 0.66958
LDOF 4 0.25000 0.17000 0.19137 0.10512 0.27273 0.19515 0.62167
LDOF 5 0.18750 0.10083 0.18782 0.10119 0.28571 0.20952 0.68625
ODIN 5 0.23864 0.15742 0.15632 0.06633 0.24242 0.16162 0.58313
ODIN 9 0.21429 0.13048 0.14953 0.05881 0.27273 0.19515 0.59979
ODIN 12 0.18750 0.10083 0.15677 0.06683 0.27027 0.19243 0.65750
ODIN 33 0.06250 -0.03750 0.14690 0.05591 0.26087 0.18203 0.65875
FastABOD 3 0.18750 0.10083 0.18208 0.09483 0.35000 0.28067 0.67500
FastABOD 4 0.25000 0.17000 0.19398 0.10801 0.30000 0.22533 0.70542
KDEOS 4 0.18750 0.10083 0.21012 0.12587 0.26923 0.19128 0.60958
KDEOS 15 0.31250 0.23917 0.16551 0.07650 0.34286 0.27276 0.61708
KDEOS 22 0.31250 0.23917 0.19256 0.10644 0.34483 0.27494 0.61250
KDEOS 90 0.12500 0.03167 0.15213 0.06169 0.30769 0.23385 0.65917
LDF 1 0.31250 0.23917 0.21596 0.13232 0.31250 0.23917 0.66083
LDF 7 0.12500 0.03167 0.16435 0.07521 0.32877 0.25717 0.66375
LDF 12 0.12500 0.03167 0.16816 0.07943 0.29787 0.22298 0.68083
INFLO 2 0.25000 0.17000 0.15689 0.06695 0.29630 0.22123 0.57729
INFLO 3 0.25000 0.17000 0.18408 0.09705 0.35556 0.28681 0.63458
INFLO 4 0.18750 0.10083 0.19037 0.10401 0.32432 0.25225 0.63042
INFLO 47 0.12500 0.03167 0.14930 0.05855 0.32609 0.25420 0.69563
COF 4 0.25000 0.17000 0.19188 0.10568 0.29630 0.22123 0.61000
COF 78 0.12500 0.03167 0.17448 0.08642 0.32877 0.25717 0.71208
COF 84 0.12500 0.03167 0.16479 0.07570 0.35135 0.28216 0.70000

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO