Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (10% of outliers version#03)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (45.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 44 0.50000 0.44667 0.43686 0.37679 0.53333 0.48356 0.85833
KNN 57 0.50000 0.44667 0.47883 0.42324 0.55172 0.50391 0.85667
KNN 64 0.56250 0.51583 0.46409 0.40693 0.60000 0.55733 0.85000
KNNW 78 0.43750 0.37750 0.38537 0.31981 0.45714 0.39924 0.84125
KNNW 100 0.43750 0.37750 0.40498 0.34151 0.48276 0.42759 0.84500
LOF 85 0.50000 0.44667 0.46363 0.40642 0.57143 0.52571 0.86542
LOF 88 0.56250 0.51583 0.46561 0.40861 0.57143 0.52571 0.86583
LOF 98 0.56250 0.51583 0.47054 0.41406 0.57143 0.52571 0.86708
SimplifiedLOF 91 0.18750 0.10083 0.30721 0.23331 0.38095 0.31492 0.78167
SimplifiedLOF 95 0.31250 0.23917 0.29080 0.21516 0.38095 0.31492 0.79292
SimplifiedLOF 97 0.31250 0.23917 0.29206 0.21654 0.40000 0.33600 0.79458
SimplifiedLOF 99 0.31250 0.23917 0.29507 0.21988 0.40000 0.33600 0.79833
LoOP 90 0.25000 0.17000 0.32012 0.24760 0.39024 0.32520 0.79229
LoOP 98 0.37500 0.30833 0.30729 0.23340 0.40000 0.33600 0.80292
LoOP 100 0.37500 0.30833 0.31202 0.23863 0.42424 0.36283 0.80542
LDOF 50 0.18750 0.10083 0.16710 0.07826 0.28037 0.20361 0.65708
LDOF 96 0.18750 0.10083 0.28108 0.20440 0.33333 0.26222 0.75042
LDOF 99 0.18750 0.10083 0.26060 0.18173 0.35556 0.28681 0.76042
LDOF 100 0.18750 0.10083 0.26137 0.18258 0.35556 0.28681 0.76250
ODIN 90 0.43750 0.37750 0.41495 0.35254 0.47368 0.41754 0.82979
ODIN 92 0.43750 0.37750 0.42107 0.35931 0.48649 0.43171 0.82896
ODIN 97 0.46875 0.41208 0.41420 0.35171 0.48485 0.42990 0.82937
ODIN 100 0.45833 0.40056 0.42407 0.36264 0.48485 0.42990 0.82917
FastABOD 56 0.18750 0.10083 0.32240 0.25013 0.45902 0.40131 0.83417
FastABOD 78 0.25000 0.17000 0.33670 0.26595 0.45902 0.40131 0.84458
FastABOD 79 0.31250 0.23917 0.33615 0.26534 0.45902 0.40131 0.84417
KDEOS 3 0.18750 0.10083 0.12893 0.03602 0.21429 0.13048 0.46875
KDEOS 38 0.18750 0.10083 0.16568 0.07669 0.21769 0.13424 0.50333
KDEOS 100 0.12500 0.03167 0.15258 0.06219 0.31250 0.23917 0.66958
LDF 30 0.56250 0.51583 0.43612 0.37597 0.56250 0.51583 0.88250
LDF 59 0.50000 0.44667 0.57152 0.52582 0.61538 0.57436 0.90125
LDF 62 0.50000 0.44667 0.58221 0.53765 0.61538 0.57436 0.90417
LDF 63 0.50000 0.44667 0.57366 0.52819 0.59259 0.54914 0.90500
INFLO 79 0.43750 0.37750 0.33439 0.26339 0.43750 0.37750 0.79563
INFLO 87 0.43750 0.37750 0.39997 0.33596 0.50000 0.44667 0.84333
INFLO 94 0.43750 0.37750 0.40973 0.34676 0.50000 0.44667 0.85021
INFLO 99 0.43750 0.37750 0.42054 0.35873 0.48276 0.42759 0.81896
COF 45 0.43750 0.37750 0.32024 0.24774 0.43750 0.37750 0.84250
COF 65 0.31250 0.23917 0.41099 0.34816 0.54167 0.49278 0.88167
COF 66 0.37500 0.30833 0.42487 0.36352 0.53061 0.48054 0.88333
COF 86 0.43750 0.37750 0.52366 0.47285 0.46667 0.40978 0.87625

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (44.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.25000 0.17000 0.22874 0.14648 0.38889 0.32370 0.79625
KNN 9 0.31250 0.23917 0.22668 0.14419 0.40000 0.33600 0.78417
KNNW 4 0.12500 0.03167 0.20847 0.12404 0.40816 0.34503 0.77583
KNNW 9 0.18750 0.10083 0.21377 0.12990 0.37736 0.31094 0.78167
KNNW 10 0.25000 0.17000 0.21541 0.13172 0.38710 0.32172 0.78125
KNNW 18 0.25000 0.17000 0.22279 0.13989 0.40000 0.33600 0.78042
LOF 14 0.00000 -0.10667 0.18315 0.09602 0.38356 0.31781 0.75542
LOF 16 0.06250 -0.03750 0.18912 0.10263 0.37838 0.31207 0.76750
LOF 18 0.06250 -0.03750 0.19043 0.10407 0.36923 0.30195 0.76667
LOF 28 0.25000 0.17000 0.17965 0.09215 0.30556 0.23148 0.73292
SimplifiedLOF 20 0.00000 -0.10667 0.18139 0.09408 0.39394 0.32929 0.74333
SimplifiedLOF 24 0.06250 -0.03750 0.19610 0.11036 0.37288 0.30599 0.77000
SimplifiedLOF 78 0.18750 0.10083 0.17357 0.08542 0.33333 0.26222 0.71083
LoOP 20 0.06250 -0.03750 0.18284 0.09568 0.40000 0.33600 0.74625
LoOP 23 0.06250 -0.03750 0.19198 0.10579 0.36620 0.29859 0.76208
LoOP 24 0.06250 -0.03750 0.19238 0.10624 0.34615 0.27641 0.76042
LoOP 58 0.18750 0.10083 0.16775 0.07898 0.32258 0.25032 0.71437
LDOF 28 0.00000 -0.10667 0.16815 0.07941 0.35714 0.28857 0.72792
LDOF 31 0.00000 -0.10667 0.17618 0.08831 0.34615 0.27641 0.74417
LDOF 90 0.12500 0.03167 0.15537 0.06528 0.31818 0.24545 0.68167
ODIN 9 0.25000 0.17000 0.16943 0.08084 0.27778 0.20074 0.66979
ODIN 28 0.06250 -0.03750 0.16854 0.07985 0.30303 0.22869 0.71062
ODIN 29 0.07292 -0.02597 0.17143 0.08305 0.31250 0.23917 0.70833
FastABOD 12 0.12500 0.03167 0.22175 0.13873 0.42623 0.36503 0.78417
FastABOD 13 0.18750 0.10083 0.22231 0.13935 0.41379 0.35126 0.78250
FastABOD 20 0.18750 0.10083 0.23041 0.14832 0.41935 0.35742 0.79125
KDEOS 3 0.31250 0.23917 0.36013 0.29188 0.41667 0.35444 0.68083
KDEOS 80 0.00000 -0.10667 0.17019 0.08167 0.34146 0.27122 0.73042
LDF 11 0.31250 0.23917 0.22515 0.14250 0.37931 0.31310 0.77458
LDF 13 0.25000 0.17000 0.23131 0.14932 0.40741 0.34420 0.79125
INFLO 17 0.12500 0.03167 0.19233 0.10618 0.38889 0.32370 0.75208
INFLO 31 0.06250 -0.03750 0.19288 0.10679 0.35556 0.28681 0.78000
INFLO 63 0.18750 0.10083 0.17155 0.08318 0.34091 0.27061 0.71375
COF 14 0.12500 0.03167 0.21720 0.13370 0.43636 0.37624 0.76292
COF 51 0.18750 0.10083 0.22579 0.14321 0.39216 0.32732 0.79583
COF 71 0.31250 0.23917 0.25052 0.17057 0.40000 0.33600 0.77917
COF 90 0.31250 0.23917 0.25666 0.17737 0.40816 0.34503 0.77125

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO