Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (5% of outliers version#06)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (39.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 6 0.42857 0.40190 0.31065 0.27848 0.42857 0.40190 0.86571
KNN 7 0.28571 0.25238 0.26407 0.22973 0.36364 0.33394 0.88714
KNN 20 0.42857 0.40190 0.24775 0.21265 0.46154 0.43641 0.86857
KNNW 17 0.28571 0.25238 0.26464 0.23032 0.36364 0.33394 0.87619
KNNW 23 0.28571 0.25238 0.28887 0.25569 0.40000 0.37200 0.87429
KNNW 57 0.42857 0.40190 0.27571 0.24191 0.42857 0.40190 0.87429
LOF 44 0.28571 0.25238 0.27570 0.24190 0.37500 0.34583 0.87143
LOF 60 0.42857 0.40190 0.27211 0.23815 0.42857 0.40190 0.87714
LOF 61 0.42857 0.40190 0.27256 0.23861 0.42857 0.40190 0.87810
SimplifiedLOF 93 0.28571 0.25238 0.22460 0.18841 0.40000 0.37200 0.86476
SimplifiedLOF 96 0.42857 0.40190 0.23616 0.20051 0.42857 0.40190 0.86190
SimplifiedLOF 98 0.42857 0.40190 0.23742 0.20184 0.42857 0.40190 0.86381
LoOP 94 0.28571 0.25238 0.22997 0.19403 0.40000 0.37200 0.86286
LoOP 95 0.42857 0.40190 0.23626 0.20062 0.42857 0.40190 0.86190
LoOP 98 0.42857 0.40190 0.23698 0.20138 0.42857 0.40190 0.86286
LDOF 51 0.14286 0.10286 0.13564 0.09530 0.24242 0.20707 0.81524
LDOF 94 0.14286 0.10286 0.20196 0.16472 0.33333 0.30222 0.85333
LDOF 100 0.14286 0.10286 0.21048 0.17364 0.37500 0.34583 0.85238
ODIN 74 0.42857 0.40190 0.22164 0.18532 0.42857 0.40190 0.85762
ODIN 88 0.42857 0.40190 0.23261 0.19680 0.42857 0.40190 0.87048
ODIN 94 0.42857 0.40190 0.26463 0.23032 0.46154 0.43641 0.87000
FastABOD 7 0.28571 0.25238 0.21000 0.17313 0.28571 0.25238 0.84762
FastABOD 10 0.28571 0.25238 0.30729 0.27496 0.33333 0.30222 0.86286
FastABOD 15 0.28571 0.25238 0.28416 0.25075 0.40000 0.37200 0.87810
FastABOD 21 0.28571 0.25238 0.29826 0.26551 0.40000 0.37200 0.89238
KDEOS 7 0.28571 0.25238 0.22609 0.18997 0.30769 0.27538 0.54857
KDEOS 100 0.14286 0.10286 0.12187 0.08089 0.22222 0.18593 0.79238
LDF 18 0.28571 0.25238 0.36611 0.33653 0.37037 0.34099 0.90667
LDF 19 0.42857 0.40190 0.47924 0.45493 0.53333 0.51156 0.90667
INFLO 32 0.14286 0.10286 0.16105 0.12190 0.33333 0.30222 0.82762
INFLO 66 0.28571 0.25238 0.18024 0.14198 0.28571 0.25238 0.84857
INFLO 92 0.28571 0.25238 0.21971 0.18329 0.33333 0.30222 0.87048
INFLO 99 0.28571 0.25238 0.21006 0.17320 0.33333 0.30222 0.87333
COF 32 0.42857 0.40190 0.31829 0.28648 0.46154 0.43641 0.90571
COF 33 0.42857 0.40190 0.35516 0.32507 0.46154 0.43641 0.92000
COF 92 0.42857 0.40190 0.53765 0.51607 0.60000 0.58133 0.89143
COF 93 0.42857 0.40190 0.54467 0.52342 0.60000 0.58133 0.89143

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (39.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.04667 0.10868 0.06708 0.20000 0.16267 0.77000
KNN 8 0.00000 -0.04667 0.15409 0.11462 0.34783 0.31739 0.83476
KNNW 1 0.14286 0.10286 0.10376 0.06193 0.18182 0.14364 0.73524
KNNW 8 0.00000 -0.04667 0.13417 0.09376 0.28571 0.25238 0.81143
KNNW 13 0.00000 -0.04667 0.13776 0.09753 0.28571 0.25238 0.81905
KNNW 18 0.00000 -0.04667 0.13788 0.09764 0.28571 0.25238 0.81429
LOF 1 0.00000 -0.04667 0.05048 0.00617 0.10169 0.05977 0.48333
LOF 14 0.00000 -0.04667 0.12060 0.07956 0.29630 0.26346 0.77333
LOF 20 0.00000 -0.04667 0.12725 0.08653 0.29630 0.26346 0.79619
LOF 22 0.00000 -0.04667 0.13236 0.09187 0.27586 0.24207 0.79524
SimplifiedLOF 1 0.00000 -0.04667 0.05445 0.01033 0.12821 0.08752 0.53810
SimplifiedLOF 22 0.00000 -0.04667 0.12310 0.08218 0.29630 0.26346 0.78190
SimplifiedLOF 26 0.00000 -0.04667 0.12853 0.08786 0.32000 0.28827 0.78095
LoOP 1 0.00000 -0.04667 0.05517 0.01108 0.12821 0.08752 0.57429
LoOP 22 0.00000 -0.04667 0.12095 0.07993 0.28571 0.25238 0.77429
LoOP 23 0.00000 -0.04667 0.12304 0.08212 0.32000 0.28827 0.75905
LoOP 27 0.00000 -0.04667 0.12718 0.08645 0.32000 0.28827 0.77429
LDOF 16 0.14286 0.10286 0.09112 0.04870 0.16949 0.13073 0.66857
LDOF 31 0.00000 -0.04667 0.11223 0.07080 0.26316 0.22877 0.74762
LDOF 45 0.00000 -0.04667 0.10788 0.06625 0.25641 0.22171 0.77143
LDOF 80 0.00000 -0.04667 0.11098 0.06949 0.27778 0.24407 0.76286
ODIN 2 0.09524 0.05302 0.07586 0.03273 0.14286 0.10286 0.68619
ODIN 11 0.00000 -0.04667 0.13880 0.09861 0.27273 0.23879 0.77333
ODIN 13 0.00000 -0.04667 0.10601 0.06429 0.20000 0.16267 0.78905
FastABOD 3 0.00000 -0.04667 0.12452 0.08366 0.24490 0.20966 0.79238
FastABOD 100 0.00000 -0.04667 0.12545 0.08464 0.23529 0.19961 0.79048
KDEOS 3 0.14286 0.10286 0.12979 0.08918 0.23077 0.19487 0.75619
KDEOS 4 0.14286 0.10286 0.25031 0.21533 0.30000 0.26733 0.65143
KDEOS 91 0.00000 -0.04667 0.10960 0.06805 0.21739 0.18087 0.77429
LDF 5 0.14286 0.10286 0.16611 0.12719 0.27586 0.24207 0.81714
LDF 9 0.00000 -0.04667 0.17527 0.13678 0.33333 0.30222 0.85810
INFLO 1 0.00000 -0.04667 0.06410 0.02042 0.15094 0.11132 0.60286
INFLO 25 0.00000 -0.04667 0.11227 0.07084 0.28571 0.25238 0.70952
INFLO 31 0.00000 -0.04667 0.11961 0.07852 0.25806 0.22344 0.79905
COF 72 0.14286 0.10286 0.16396 0.12495 0.33333 0.30222 0.82286
COF 87 0.00000 -0.04667 0.14968 0.10999 0.27586 0.24207 0.83048
COF 96 0.28571 0.25238 0.14897 0.10926 0.28571 0.25238 0.80286
COF 97 0.28571 0.25238 0.16453 0.12554 0.30769 0.27538 0.81524

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO