Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (10% of outliers version#08)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (44.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.37500 0.30833 0.44295 0.38353 0.42857 0.36762 0.83854
KNN 18 0.43750 0.37750 0.34229 0.27214 0.43750 0.37750 0.86604
KNN 35 0.43750 0.37750 0.39812 0.33391 0.48276 0.42759 0.85708
KNN 55 0.37500 0.30833 0.41597 0.35368 0.46154 0.40410 0.87250
KNNW 6 0.37500 0.30833 0.40256 0.33883 0.42857 0.36762 0.82250
KNNW 48 0.43750 0.37750 0.39067 0.32567 0.47059 0.41412 0.85792
KNNW 74 0.43750 0.37750 0.38750 0.32217 0.47059 0.41412 0.85958
KNNW 79 0.43750 0.37750 0.38218 0.31628 0.48485 0.42990 0.85875
LOF 77 0.37500 0.30833 0.39993 0.33592 0.44444 0.38519 0.87292
LOF 81 0.43750 0.37750 0.39699 0.33267 0.43750 0.37750 0.86958
LOF 85 0.43750 0.37750 0.40247 0.33874 0.46667 0.40978 0.86792
LOF 100 0.43750 0.37750 0.40647 0.34316 0.46154 0.40410 0.86958
SimplifiedLOF 98 0.31250 0.23917 0.29258 0.21712 0.40000 0.33600 0.82042
SimplifiedLOF 99 0.31250 0.23917 0.29740 0.22245 0.41176 0.34902 0.82292
LoOP 96 0.31250 0.23917 0.29489 0.21968 0.40909 0.34606 0.82167
LoOP 97 0.37500 0.30833 0.29754 0.22261 0.42424 0.36283 0.82083
LDOF 81 0.25000 0.17000 0.20641 0.12176 0.35294 0.28392 0.76000
LDOF 99 0.18750 0.10083 0.26558 0.18724 0.42857 0.36762 0.79917
LDOF 100 0.18750 0.10083 0.26701 0.18882 0.43902 0.37919 0.79875
ODIN 81 0.37500 0.30833 0.33797 0.26735 0.41509 0.35270 0.84125
ODIN 84 0.34375 0.27375 0.30538 0.23129 0.44898 0.39020 0.84083
ODIN 91 0.37500 0.30833 0.35999 0.29172 0.43137 0.37072 0.85083
ODIN 97 0.37500 0.30833 0.32557 0.25363 0.42308 0.36154 0.85396
FastABOD 22 0.43750 0.37750 0.43071 0.36998 0.43750 0.37750 0.86375
FastABOD 89 0.43750 0.37750 0.45629 0.39829 0.48889 0.43437 0.88583
FastABOD 95 0.43750 0.37750 0.45894 0.40123 0.48889 0.43437 0.88625
FastABOD 100 0.43750 0.37750 0.45881 0.40109 0.48889 0.43437 0.88667
KDEOS 5 0.25000 0.17000 0.17443 0.08637 0.25000 0.17000 0.50833
KDEOS 91 0.18750 0.10083 0.16754 0.07875 0.32184 0.24950 0.72542
KDEOS 100 0.18750 0.10083 0.17723 0.08947 0.31250 0.23917 0.73875
LDF 65 0.50000 0.44667 0.54003 0.49097 0.55000 0.50200 0.91000
LDF 67 0.50000 0.44667 0.54814 0.49994 0.58537 0.54114 0.91208
LDF 68 0.50000 0.44667 0.55247 0.50473 0.60000 0.55733 0.91125
INFLO 76 0.37500 0.30833 0.28909 0.21326 0.42105 0.35930 0.83792
INFLO 90 0.43750 0.37750 0.36476 0.29700 0.45161 0.39312 0.82042
INFLO 100 0.43750 0.37750 0.35601 0.28732 0.48485 0.42990 0.78417
COF 60 0.50000 0.44667 0.54437 0.49577 0.54545 0.49697 0.90417
COF 63 0.43750 0.37750 0.55712 0.50988 0.50909 0.45673 0.90667

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (44.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.12500 0.03167 0.15339 0.06309 0.27273 0.19515 0.65687
KNN 4 0.12500 0.03167 0.15439 0.06419 0.27907 0.20217 0.66792
KNN 6 0.12500 0.03167 0.15453 0.06434 0.27586 0.19862 0.66187
KNN 17 0.12500 0.03167 0.13922 0.04740 0.29213 0.21663 0.63604
KNNW 1 0.18750 0.10083 0.18073 0.09334 0.30769 0.23385 0.70417
LOF 1 0.12500 0.03167 0.15242 0.06202 0.23529 0.15373 0.57208
LOF 6 0.12500 0.03167 0.14394 0.05263 0.26829 0.19024 0.66625
LOF 16 0.12500 0.03167 0.13467 0.04237 0.28125 0.20458 0.59500
SimplifiedLOF 1 0.12500 0.03167 0.13742 0.04541 0.24390 0.16325 0.55208
SimplifiedLOF 5 0.06250 -0.03750 0.14441 0.05315 0.22642 0.14390 0.57292
SimplifiedLOF 62 0.12500 0.03167 0.12940 0.03653 0.27586 0.19862 0.61083
SimplifiedLOF 72 0.12500 0.03167 0.13173 0.03911 0.26263 0.18397 0.61667
LoOP 1 0.12500 0.03167 0.12829 0.03531 0.24390 0.16325 0.56500
LoOP 5 0.06250 -0.03750 0.14962 0.05892 0.25532 0.17589 0.57625
LoOP 12 0.06250 -0.03750 0.13384 0.04144 0.27273 0.19515 0.59521
LoOP 94 0.06250 -0.03750 0.12868 0.03574 0.26087 0.18203 0.62042
LDOF 4 0.12500 0.03167 0.12525 0.03194 0.20833 0.12389 0.56167
LDOF 71 0.06250 -0.03750 0.12533 0.03203 0.27273 0.19515 0.60708
LDOF 80 0.06250 -0.03750 0.12638 0.03319 0.28261 0.20609 0.60417
LDOF 96 0.06250 -0.03750 0.12864 0.03569 0.26263 0.18397 0.60500
ODIN 8 0.21429 0.13048 0.13535 0.04312 0.23529 0.15373 0.56208
ODIN 12 0.14062 0.04896 0.13443 0.04210 0.27848 0.20152 0.59562
ODIN 28 0.12500 0.03167 0.15045 0.05983 0.23684 0.15544 0.59646
ODIN 35 0.08333 -0.01444 0.13437 0.04203 0.23762 0.15630 0.60583
FastABOD 3 0.12500 0.03167 0.18186 0.09460 0.28571 0.20952 0.68833
FastABOD 4 0.12500 0.03167 0.16467 0.07556 0.29508 0.21989 0.69333
FastABOD 12 0.06250 -0.03750 0.15836 0.06859 0.31034 0.23678 0.68000
KDEOS 6 0.25000 0.17000 0.23529 0.15372 0.35294 0.28392 0.60708
KDEOS 95 0.06250 -0.03750 0.13436 0.04202 0.26087 0.18203 0.62417
LDF 6 0.25000 0.17000 0.17207 0.08376 0.28571 0.20952 0.64208
LDF 9 0.18750 0.10083 0.15786 0.06803 0.29851 0.22368 0.62000
LDF 12 0.12500 0.03167 0.15702 0.06711 0.28571 0.20952 0.65208
INFLO 1 0.06250 -0.03750 0.17460 0.08656 0.25532 0.17589 0.57604
INFLO 2 0.18750 0.10083 0.15767 0.06782 0.24000 0.15893 0.54563
INFLO 35 0.12500 0.03167 0.14881 0.05802 0.34146 0.27122 0.65375
INFLO 100 0.06250 -0.03750 0.14453 0.05327 0.32877 0.25717 0.66271
COF 11 0.31250 0.23917 0.22283 0.13993 0.35714 0.28857 0.72875

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO