Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (5% of outliers version#08)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (38.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.57143 0.55143 0.47755 0.45317 0.61538 0.59744 0.87619
KNN 3 0.57143 0.55143 0.51758 0.49507 0.66667 0.65111 0.87048
KNN 4 0.57143 0.55143 0.56617 0.54592 0.66667 0.65111 0.91000
KNN 79 0.42857 0.40190 0.46819 0.44337 0.54545 0.52424 0.93048
KNNW 3 0.57143 0.55143 0.40607 0.37835 0.57143 0.55143 0.86095
KNNW 9 0.57143 0.55143 0.48592 0.46193 0.61538 0.59744 0.88571
KNNW 35 0.42857 0.40190 0.48449 0.46044 0.55556 0.53481 0.91429
LOF 29 0.42857 0.40190 0.28622 0.25291 0.42857 0.40190 0.88381
LOF 70 0.42857 0.40190 0.47228 0.44766 0.53333 0.51156 0.90952
LOF 94 0.42857 0.40190 0.45194 0.42636 0.55556 0.53481 0.89429
SimplifiedLOF 74 0.42857 0.40190 0.28260 0.24912 0.47059 0.44588 0.88571
SimplifiedLOF 98 0.42857 0.40190 0.36238 0.33262 0.45455 0.42909 0.91048
LoOP 74 0.42857 0.40190 0.29855 0.26581 0.53333 0.51156 0.89048
LoOP 97 0.42857 0.40190 0.37035 0.34097 0.47059 0.44588 0.91429
LDOF 80 0.28571 0.25238 0.21885 0.18239 0.40000 0.37200 0.86571
LDOF 83 0.28571 0.25238 0.22621 0.19010 0.42105 0.39404 0.86571
LDOF 96 0.28571 0.25238 0.26425 0.22991 0.40000 0.37200 0.88952
LDOF 98 0.28571 0.25238 0.28502 0.25166 0.38462 0.35590 0.88857
ODIN 51 0.42857 0.40190 0.28703 0.25375 0.42857 0.40190 0.89810
ODIN 74 0.42857 0.40190 0.46206 0.43696 0.50000 0.47667 0.90619
ODIN 90 0.42857 0.40190 0.36165 0.33186 0.46154 0.43641 0.90667
FastABOD 56 0.57143 0.55143 0.51147 0.48867 0.57143 0.55143 0.91619
FastABOD 91 0.57143 0.55143 0.57287 0.55294 0.66667 0.65111 0.92095
FastABOD 97 0.57143 0.55143 0.57438 0.55451 0.66667 0.65111 0.92286
KDEOS 3 0.14286 0.10286 0.09233 0.04998 0.20000 0.16267 0.52667
KDEOS 85 0.14286 0.10286 0.11790 0.07673 0.25000 0.21500 0.78476
KDEOS 98 0.14286 0.10286 0.13601 0.09569 0.22222 0.18593 0.79238
KDEOS 100 0.14286 0.10286 0.12595 0.08516 0.22857 0.19257 0.79429
LDF 27 0.57143 0.55143 0.47345 0.44888 0.57143 0.55143 0.90381
LDF 31 0.42857 0.40190 0.46610 0.44118 0.50000 0.47667 0.91238
LDF 53 0.57143 0.55143 0.49704 0.47357 0.61538 0.59744 0.90571
LDF 76 0.57143 0.55143 0.56729 0.54709 0.61538 0.59744 0.88952
INFLO 62 0.42857 0.40190 0.28123 0.24768 0.42857 0.40190 0.91048
INFLO 84 0.28571 0.25238 0.42361 0.39671 0.50000 0.47667 0.93143
INFLO 94 0.28571 0.25238 0.43953 0.41338 0.52174 0.49942 0.93048
INFLO 95 0.28571 0.25238 0.43999 0.41386 0.52174 0.49942 0.93143
COF 48 0.71429 0.70095 0.64190 0.62518 0.71429 0.70095 0.92000
COF 55 0.71429 0.70095 0.68260 0.66778 0.71429 0.70095 0.90667
COF 72 0.57143 0.55143 0.66602 0.65043 0.72727 0.71455 0.92762
COF 80 0.57143 0.55143 0.66092 0.64510 0.66667 0.65111 0.93810

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (38.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.14286 0.10286 0.14624 0.10640 0.31579 0.28386 0.80762
KNN 9 0.14286 0.10286 0.18857 0.15070 0.34783 0.31739 0.83714
KNNW 3 0.00000 -0.04667 0.17297 0.13438 0.34783 0.31739 0.84762
KNNW 4 0.00000 -0.04667 0.16818 0.12936 0.36364 0.33394 0.83810
KNNW 5 0.14286 0.10286 0.17314 0.13455 0.34783 0.31739 0.83905
KNNW 7 0.14286 0.10286 0.18734 0.14942 0.33333 0.30222 0.83333
LOF 1 0.14286 0.10286 0.10134 0.05940 0.18182 0.14364 0.72667
LOF 17 0.14286 0.10286 0.17490 0.13640 0.26415 0.22981 0.84571
LOF 23 0.14286 0.10286 0.17109 0.13241 0.27273 0.23879 0.82286
SimplifiedLOF 6 0.14286 0.10286 0.13589 0.09556 0.25532 0.22057 0.82000
SimplifiedLOF 18 0.14286 0.10286 0.18151 0.14331 0.33333 0.30222 0.82667
SimplifiedLOF 19 0.14286 0.10286 0.18255 0.14441 0.32258 0.29097 0.83238
LoOP 6 0.14286 0.10286 0.13119 0.09064 0.27273 0.23879 0.80571
LoOP 15 0.14286 0.10286 0.18060 0.14237 0.32258 0.29097 0.82667
LoOP 18 0.14286 0.10286 0.19326 0.15562 0.35714 0.32714 0.82476
LDOF 3 0.14286 0.10286 0.11179 0.07034 0.20000 0.16267 0.68381
LDOF 21 0.14286 0.10286 0.16039 0.12120 0.31579 0.28386 0.78286
LDOF 33 0.14286 0.10286 0.15802 0.11873 0.24490 0.20966 0.79238
LDOF 41 0.14286 0.10286 0.17714 0.13874 0.24000 0.20453 0.77524
ODIN 8 0.14286 0.10286 0.15123 0.11162 0.29268 0.25967 0.79905
ODIN 12 0.00000 -0.04667 0.18586 0.14786 0.31579 0.28386 0.81524
ODIN 15 0.03571 -0.00929 0.13846 0.09826 0.28571 0.25238 0.82048
FastABOD 3 0.28571 0.25238 0.22931 0.19334 0.34783 0.31739 0.86571
FastABOD 12 0.14286 0.10286 0.19093 0.15317 0.40000 0.37200 0.82857
KDEOS 3 0.14286 0.10286 0.17319 0.13460 0.28571 0.25238 0.64667
KDEOS 50 0.00000 -0.04667 0.13059 0.09002 0.33333 0.30222 0.79619
KDEOS 54 0.00000 -0.04667 0.14565 0.10578 0.32258 0.29097 0.81429
LDF 4 0.42857 0.40190 0.24471 0.20947 0.42857 0.40190 0.79524
LDF 7 0.14286 0.10286 0.20792 0.17096 0.33333 0.30222 0.83048
INFLO 4 0.00000 -0.04667 0.12676 0.08601 0.31818 0.28636 0.82952
INFLO 6 0.14286 0.10286 0.14116 0.10108 0.25000 0.21500 0.81429
INFLO 11 0.14286 0.10286 0.17407 0.13553 0.27273 0.23879 0.84286
INFLO 42 0.14286 0.10286 0.18160 0.14341 0.27273 0.23879 0.79619
COF 21 0.14286 0.10286 0.18338 0.14527 0.34483 0.31425 0.87286
COF 49 0.28571 0.25238 0.22378 0.18755 0.32258 0.29097 0.82762
COF 78 0.28571 0.25238 0.24587 0.21067 0.33333 0.30222 0.86000
COF 95 0.14286 0.10286 0.19294 0.15527 0.40000 0.37200 0.81143

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO