Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (10% of outliers version#07)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (42.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.37500 0.30833 0.41324 0.35065 0.38462 0.31897 0.75813
KNN 13 0.31250 0.23917 0.35990 0.29162 0.41935 0.35742 0.84417
KNN 56 0.37500 0.30833 0.40998 0.34705 0.48000 0.42453 0.82708
KNN 76 0.43750 0.37750 0.39710 0.33279 0.48000 0.42453 0.83292
KNNW 5 0.31250 0.23917 0.40218 0.33841 0.37037 0.30321 0.77500
KNNW 51 0.37500 0.30833 0.36688 0.29934 0.38710 0.32172 0.82292
KNNW 82 0.37500 0.30833 0.37941 0.31322 0.41667 0.35444 0.82583
KNNW 85 0.37500 0.30833 0.37557 0.30896 0.42857 0.36762 0.82583
LOF 24 0.43750 0.37750 0.38428 0.31861 0.46667 0.40978 0.81750
LOF 26 0.43750 0.37750 0.41757 0.35545 0.45161 0.39312 0.81667
LOF 82 0.37500 0.30833 0.38718 0.32181 0.46154 0.40410 0.82875
SimplifiedLOF 34 0.37500 0.30833 0.32129 0.24890 0.40000 0.33600 0.73833
SimplifiedLOF 36 0.37500 0.30833 0.31398 0.24081 0.41176 0.34902 0.74167
SimplifiedLOF 78 0.25000 0.17000 0.36356 0.29567 0.38095 0.31492 0.79417
SimplifiedLOF 100 0.37500 0.30833 0.34836 0.27885 0.37838 0.31207 0.81417
LoOP 34 0.43750 0.37750 0.32486 0.25285 0.45161 0.39312 0.75042
LoOP 76 0.25000 0.17000 0.36726 0.29977 0.38095 0.31492 0.79833
LoOP 100 0.37500 0.30833 0.35242 0.28334 0.37500 0.30833 0.81708
LDOF 49 0.31250 0.23917 0.28193 0.20534 0.32258 0.25032 0.74250
LDOF 96 0.25000 0.17000 0.36153 0.29343 0.37209 0.30512 0.79625
LDOF 100 0.31250 0.23917 0.34852 0.27902 0.41026 0.34735 0.80333
ODIN 14 0.37500 0.30833 0.22905 0.14682 0.37500 0.30833 0.72542
ODIN 60 0.32812 0.25646 0.38673 0.32132 0.41026 0.34735 0.81104
ODIN 92 0.37500 0.30833 0.34745 0.27784 0.44444 0.38519 0.81521
ODIN 99 0.37500 0.30833 0.34749 0.27789 0.40000 0.33600 0.81833
FastABOD 5 0.37500 0.30833 0.42054 0.35873 0.40000 0.33600 0.81625
FastABOD 8 0.37500 0.30833 0.43904 0.37921 0.43478 0.37449 0.80958
FastABOD 82 0.31250 0.23917 0.37038 0.30323 0.38961 0.32450 0.82875
KDEOS 52 0.31250 0.23917 0.19185 0.10565 0.32258 0.25032 0.70167
KDEOS 92 0.31250 0.23917 0.21612 0.13251 0.33333 0.26222 0.74750
KDEOS 97 0.31250 0.23917 0.23381 0.15209 0.33333 0.26222 0.75958
KDEOS 99 0.31250 0.23917 0.23467 0.15304 0.33333 0.26222 0.75833
LDF 19 0.43750 0.37750 0.52267 0.47175 0.51852 0.46716 0.84375
LDF 20 0.50000 0.44667 0.50190 0.44877 0.51852 0.46716 0.82917
INFLO 24 0.43750 0.37750 0.35039 0.28110 0.44444 0.38519 0.78000
INFLO 84 0.37500 0.30833 0.33626 0.26546 0.42105 0.35930 0.84875
INFLO 97 0.37500 0.30833 0.39604 0.33161 0.45455 0.39636 0.81792
INFLO 100 0.37500 0.30833 0.41268 0.35003 0.45455 0.39636 0.82208
COF 13 0.31250 0.23917 0.25371 0.17411 0.32653 0.25469 0.64500
COF 41 0.31250 0.23917 0.41650 0.35426 0.42623 0.36503 0.83458
COF 66 0.25000 0.17000 0.38246 0.31659 0.41935 0.35742 0.84417
COF 74 0.18750 0.10083 0.38451 0.31885 0.47273 0.41648 0.84417

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (43.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.25000 0.17000 0.24146 0.16055 0.43243 0.37189 0.75187
KNN 8 0.25000 0.17000 0.24188 0.16101 0.42105 0.35930 0.74958
KNN 10 0.31250 0.23917 0.23635 0.15489 0.40909 0.34606 0.73646
KNNW 12 0.18750 0.10083 0.22539 0.14276 0.40000 0.33600 0.73875
KNNW 14 0.25000 0.17000 0.22997 0.14783 0.40000 0.33600 0.73917
KNNW 27 0.31250 0.23917 0.22181 0.13880 0.38889 0.32370 0.72042
LOF 12 0.18750 0.10083 0.21072 0.12653 0.36364 0.29576 0.75125
LOF 17 0.18750 0.10083 0.21878 0.13545 0.36000 0.29173 0.74917
LOF 23 0.31250 0.23917 0.21429 0.13048 0.35897 0.29060 0.72542
LOF 27 0.25000 0.17000 0.20942 0.12509 0.37838 0.31207 0.71250
SimplifiedLOF 8 0.25000 0.17000 0.14521 0.05403 0.29412 0.21882 0.57333
SimplifiedLOF 19 0.18750 0.10083 0.22406 0.14129 0.42553 0.36426 0.73375
SimplifiedLOF 21 0.18750 0.10083 0.22207 0.13909 0.43478 0.37449 0.73042
LoOP 8 0.25000 0.17000 0.14632 0.05526 0.30303 0.22869 0.57792
LoOP 19 0.18750 0.10083 0.23219 0.15029 0.44444 0.38519 0.73833
LoOP 21 0.18750 0.10083 0.22840 0.14610 0.45455 0.39636 0.73771
LoOP 69 0.18750 0.10083 0.21026 0.12602 0.38298 0.31716 0.74333
LDOF 12 0.25000 0.17000 0.14605 0.05496 0.25000 0.17000 0.60500
LDOF 28 0.12500 0.03167 0.21177 0.12769 0.44898 0.39020 0.71875
LDOF 29 0.12500 0.03167 0.21779 0.13436 0.43137 0.37072 0.72708
LDOF 66 0.12500 0.03167 0.20266 0.11762 0.38298 0.31716 0.73208
ODIN 11 0.31250 0.23917 0.20475 0.11992 0.32258 0.25032 0.71354
ODIN 16 0.22500 0.14233 0.21876 0.13543 0.33333 0.26222 0.73396
ODIN 18 0.23438 0.15271 0.20982 0.12553 0.35556 0.28681 0.72896
ODIN 21 0.17188 0.08354 0.19885 0.11339 0.34146 0.27122 0.73479
FastABOD 4 0.18750 0.10083 0.16392 0.07474 0.29787 0.22298 0.68125
FastABOD 6 0.18750 0.10083 0.17517 0.08718 0.33333 0.26222 0.67792
FastABOD 18 0.18750 0.10083 0.18868 0.10213 0.29787 0.22298 0.68833
FastABOD 87 0.18750 0.10083 0.18383 0.09677 0.29508 0.21989 0.69375
KDEOS 22 0.25000 0.17000 0.26672 0.18850 0.36364 0.29576 0.62542
KDEOS 27 0.25000 0.17000 0.30931 0.23564 0.33333 0.26222 0.66042
KDEOS 36 0.31250 0.23917 0.26817 0.19011 0.31250 0.23917 0.67583
KDEOS 83 0.18750 0.10083 0.24562 0.16516 0.34286 0.27276 0.72958
LDF 8 0.18750 0.10083 0.22372 0.14091 0.41860 0.35659 0.74625
LDF 10 0.25000 0.17000 0.24419 0.16357 0.40000 0.33600 0.77417
LDF 13 0.31250 0.23917 0.23457 0.15292 0.37931 0.31310 0.74333
INFLO 6 0.25000 0.17000 0.16000 0.07040 0.28125 0.20458 0.61750
INFLO 17 0.18750 0.10083 0.23040 0.14831 0.41860 0.35659 0.72417
INFLO 29 0.25000 0.17000 0.22867 0.14640 0.37736 0.31094 0.78333
COF 14 0.12500 0.03167 0.18091 0.09354 0.32099 0.24856 0.71896
COF 59 0.18750 0.10083 0.17811 0.09045 0.35484 0.28602 0.69125
COF 77 0.31250 0.23917 0.22128 0.13822 0.31579 0.24281 0.71208
COF 93 0.25000 0.17000 0.24813 0.16793 0.34043 0.27007 0.67708

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO