Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (5% of outliers version#09)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (42.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 30 0.42857 0.40190 0.26440 0.23007 0.44444 0.41852 0.88000
KNN 54 0.42857 0.40190 0.28587 0.25255 0.42857 0.40190 0.88857
KNN 68 0.28571 0.25238 0.26869 0.23457 0.47619 0.45175 0.89048
KNN 69 0.28571 0.25238 0.27259 0.23865 0.47619 0.45175 0.89143
KNNW 54 0.14286 0.10286 0.17567 0.13720 0.34783 0.31739 0.84857
KNNW 63 0.28571 0.25238 0.18649 0.14853 0.33333 0.30222 0.85143
KNNW 79 0.28571 0.25238 0.20594 0.16888 0.34783 0.31739 0.86000
KNNW 99 0.28571 0.25238 0.20780 0.17083 0.34783 0.31739 0.85905
LOF 56 0.28571 0.25238 0.19466 0.15708 0.34483 0.31425 0.85810
LOF 80 0.28571 0.25238 0.27207 0.23810 0.44444 0.41852 0.89810
LOF 100 0.28571 0.25238 0.27962 0.24601 0.44444 0.41852 0.90000
SimplifiedLOF 98 0.28571 0.25238 0.15914 0.11990 0.28571 0.25238 0.81619
SimplifiedLOF 99 0.28571 0.25238 0.16222 0.12312 0.28571 0.25238 0.81905
LoOP 85 0.28571 0.25238 0.15384 0.11435 0.28571 0.25238 0.81048
LoOP 96 0.28571 0.25238 0.16938 0.13062 0.30769 0.27538 0.82095
LoOP 99 0.28571 0.25238 0.18355 0.14545 0.30769 0.27538 0.82762
LDOF 2 0.14286 0.10286 0.07423 0.03102 0.16667 0.12778 0.54190
LDOF 98 0.14286 0.10286 0.13465 0.09426 0.26667 0.23244 0.78952
LDOF 100 0.14286 0.10286 0.13605 0.09573 0.26667 0.23244 0.79238
ODIN 71 0.28571 0.25238 0.21107 0.17425 0.34783 0.31739 0.85333
ODIN 91 0.28571 0.25238 0.24541 0.21019 0.42105 0.39404 0.88143
ODIN 93 0.28571 0.25238 0.27049 0.23644 0.40000 0.37200 0.88381
ODIN 100 0.28571 0.25238 0.26403 0.22968 0.41667 0.38944 0.88524
FastABOD 55 0.14286 0.10286 0.12961 0.08899 0.27907 0.24543 0.82190
FastABOD 61 0.14286 0.10286 0.13214 0.09164 0.30000 0.26733 0.82476
FastABOD 98 0.14286 0.10286 0.14224 0.10221 0.30000 0.26733 0.83619
KDEOS 2 0.00000 -0.04667 0.03623 -0.00875 0.08805 0.04549 0.32190
KDEOS 3 0.00000 -0.04667 0.06504 0.02141 0.20000 0.16267 0.44190
KDEOS 100 0.00000 -0.04667 0.06451 0.02085 0.17949 0.14120 0.63333
LDF 47 0.57143 0.55143 0.37087 0.34151 0.57143 0.55143 0.91429
LDF 53 0.57143 0.55143 0.40629 0.37859 0.61538 0.59744 0.92190
LDF 54 0.57143 0.55143 0.40866 0.38106 0.61538 0.59744 0.92381
LDF 74 0.42857 0.40190 0.36722 0.33769 0.52174 0.49942 0.92571
INFLO 65 0.28571 0.25238 0.15451 0.11505 0.28571 0.25238 0.81048
INFLO 87 0.28571 0.25238 0.20008 0.16275 0.33333 0.30222 0.85238
INFLO 100 0.28571 0.25238 0.25106 0.21611 0.33333 0.30222 0.87524
COF 56 0.42857 0.40190 0.27332 0.23941 0.46154 0.43641 0.86381
COF 88 0.28571 0.25238 0.33550 0.30449 0.55556 0.53481 0.89048
COF 100 0.42857 0.40190 0.37281 0.34355 0.62500 0.60750 0.86952

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (39.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.04667 0.09169 0.04930 0.20000 0.16267 0.67810
KNN 9 0.00000 -0.04667 0.15691 0.11757 0.37037 0.34099 0.78619
KNN 18 0.00000 -0.04667 0.15003 0.11036 0.32258 0.29097 0.79238
KNNW 1 0.00000 -0.04667 0.10855 0.06695 0.25000 0.21500 0.72238
KNNW 9 0.00000 -0.04667 0.13916 0.09899 0.34783 0.31739 0.76190
KNNW 14 0.00000 -0.04667 0.14440 0.10447 0.34783 0.31739 0.77238
KNNW 18 0.00000 -0.04667 0.14443 0.10450 0.34783 0.31739 0.77238
LOF 1 0.00000 -0.04667 0.04634 0.00184 0.11236 0.07094 0.49000
LOF 11 0.00000 -0.04667 0.11828 0.07713 0.29412 0.26118 0.76762
LOF 25 0.00000 -0.04667 0.13248 0.09199 0.31579 0.28386 0.74000
SimplifiedLOF 1 0.00000 -0.04667 0.04851 0.00411 0.12698 0.08624 0.49524
SimplifiedLOF 19 0.00000 -0.04667 0.11987 0.07880 0.29412 0.26118 0.77048
SimplifiedLOF 20 0.00000 -0.04667 0.12234 0.08138 0.28571 0.25238 0.78095
SimplifiedLOF 43 0.00000 -0.04667 0.12511 0.08429 0.28571 0.25238 0.76286
LoOP 1 0.00000 -0.04667 0.05015 0.00583 0.12698 0.08624 0.53238
LoOP 19 0.00000 -0.04667 0.12353 0.08262 0.30303 0.27051 0.78190
LoOP 20 0.00000 -0.04667 0.12564 0.08484 0.30303 0.27051 0.79429
LDOF 2 0.00000 -0.04667 0.09293 0.05060 0.23529 0.19961 0.64952
LDOF 44 0.00000 -0.04667 0.10929 0.06772 0.23077 0.19487 0.75333
LDOF 69 0.00000 -0.04667 0.12042 0.07937 0.25000 0.21500 0.73905
LDOF 80 0.00000 -0.04667 0.11640 0.07516 0.27778 0.24407 0.73429
ODIN 2 0.10526 0.06351 0.06922 0.02578 0.15385 0.11436 0.61429
ODIN 12 0.00000 -0.04667 0.14369 0.10373 0.27027 0.23622 0.78286
ODIN 51 0.00000 -0.04667 0.12666 0.08591 0.31579 0.28386 0.71429
FastABOD 3 0.00000 -0.04667 0.08555 0.04288 0.21277 0.17603 0.65333
FastABOD 67 0.00000 -0.04667 0.12808 0.08739 0.32258 0.29097 0.73714
FastABOD 100 0.00000 -0.04667 0.13447 0.09408 0.32258 0.29097 0.74190
KDEOS 14 0.14286 0.10286 0.07224 0.02895 0.15385 0.11436 0.57238
KDEOS 64 0.00000 -0.04667 0.11499 0.07369 0.26667 0.23244 0.74000
KDEOS 67 0.00000 -0.04667 0.11075 0.06925 0.28571 0.25238 0.73810
KDEOS 80 0.00000 -0.04667 0.11416 0.07282 0.28571 0.25238 0.74952
LDF 5 0.14286 0.10286 0.12066 0.07962 0.25000 0.21500 0.69905
LDF 13 0.00000 -0.04667 0.16358 0.12455 0.38095 0.35206 0.75905
LDF 15 0.00000 -0.04667 0.15956 0.12034 0.36364 0.33394 0.77333
INFLO 1 0.00000 -0.04667 0.06274 0.01900 0.17391 0.13536 0.49905
INFLO 18 0.00000 -0.04667 0.12947 0.08885 0.24000 0.20453 0.81333
INFLO 19 0.00000 -0.04667 0.13044 0.08986 0.25806 0.22344 0.81238
INFLO 26 0.00000 -0.04667 0.11923 0.07813 0.27273 0.23879 0.71810
COF 55 0.14286 0.10286 0.24030 0.20485 0.42105 0.39404 0.81143
COF 56 0.14286 0.10286 0.20441 0.16729 0.42105 0.39404 0.81524
COF 96 0.28571 0.25238 0.23979 0.20431 0.50000 0.47667 0.76381
COF 98 0.35714 0.32714 0.22935 0.19339 0.44444 0.41852 0.76143

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO