Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (10% of outliers version#02)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (45.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 51 0.31250 0.23917 0.38419 0.31850 0.53659 0.48715 0.86458
KNN 65 0.43750 0.37750 0.39384 0.32918 0.50980 0.45752 0.87250
KNN 70 0.43750 0.37750 0.42920 0.36832 0.50000 0.44667 0.87750
KNN 71 0.43750 0.37750 0.40245 0.33871 0.50000 0.44667 0.87792
KNNW 29 0.37500 0.30833 0.31890 0.24625 0.37838 0.31207 0.81708
KNNW 97 0.37500 0.30833 0.36199 0.29393 0.46809 0.41135 0.85708
KNNW 98 0.37500 0.30833 0.36399 0.29615 0.46809 0.41135 0.85750
LOF 77 0.31250 0.23917 0.37393 0.30715 0.53659 0.48715 0.87042
LOF 80 0.37500 0.30833 0.37623 0.30969 0.55000 0.50200 0.86667
LOF 95 0.50000 0.44667 0.40850 0.34540 0.53333 0.48356 0.85583
LOF 100 0.50000 0.44667 0.41352 0.35096 0.52381 0.47302 0.85875
SimplifiedLOF 97 0.31250 0.23917 0.28619 0.21005 0.38356 0.31781 0.81208
SimplifiedLOF 99 0.31250 0.23917 0.28935 0.21355 0.40000 0.33600 0.81542
SimplifiedLOF 100 0.31250 0.23917 0.28798 0.21203 0.39437 0.32977 0.81583
LoOP 94 0.25000 0.17000 0.28265 0.20613 0.40000 0.33600 0.81708
LoOP 95 0.25000 0.17000 0.27871 0.20177 0.40678 0.34350 0.80292
LoOP 96 0.31250 0.23917 0.28805 0.21211 0.40678 0.34350 0.80542
LoOP 99 0.31250 0.23917 0.29283 0.21740 0.40678 0.34350 0.80625
LDOF 6 0.25000 0.17000 0.15180 0.06133 0.25000 0.17000 0.51708
LDOF 96 0.25000 0.17000 0.23765 0.15634 0.36111 0.29296 0.78458
LDOF 100 0.25000 0.17000 0.24508 0.16456 0.35294 0.28392 0.78542
ODIN 77 0.35938 0.29104 0.29971 0.22501 0.47273 0.41648 0.84042
ODIN 89 0.31250 0.23917 0.34445 0.27452 0.51163 0.45953 0.85042
ODIN 92 0.31250 0.23917 0.33368 0.26260 0.53333 0.48356 0.84833
FastABOD 67 0.43750 0.37750 0.46447 0.40734 0.50000 0.44667 0.87250
FastABOD 76 0.43750 0.37750 0.48270 0.42753 0.50000 0.44667 0.87792
FastABOD 85 0.50000 0.44667 0.47920 0.42365 0.50000 0.44667 0.88042
FastABOD 100 0.43750 0.37750 0.47574 0.41982 0.50000 0.44667 0.88250
KDEOS 3 0.18750 0.10083 0.18954 0.10309 0.21429 0.13048 0.51792
KDEOS 94 0.12500 0.03167 0.14676 0.05574 0.28571 0.20952 0.66417
KDEOS 100 0.12500 0.03167 0.15609 0.06607 0.28571 0.20952 0.67875
LDF 58 0.62500 0.58500 0.59175 0.54820 0.62500 0.58500 0.90292
LDF 60 0.56250 0.51583 0.59495 0.55175 0.61111 0.56963 0.90583
LDF 71 0.62500 0.58500 0.59099 0.54736 0.64706 0.60941 0.90042
LDF 74 0.62500 0.58500 0.60039 0.55777 0.64706 0.60941 0.90292
INFLO 75 0.31250 0.23917 0.26969 0.19179 0.42623 0.36503 0.83292
INFLO 97 0.31250 0.23917 0.30926 0.23558 0.45455 0.39636 0.82083
INFLO 98 0.31250 0.23917 0.30605 0.23203 0.46154 0.40410 0.84313
COF 57 0.56250 0.51583 0.47806 0.42239 0.57143 0.52571 0.84833
COF 88 0.50000 0.44667 0.55062 0.50269 0.61538 0.57436 0.89417
COF 90 0.50000 0.44667 0.53607 0.48659 0.59259 0.54914 0.89750

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (41.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.31250 0.23917 0.18842 0.10185 0.35294 0.28392 0.58313
KNN 11 0.25000 0.17000 0.20425 0.11937 0.40000 0.33600 0.58458
KNN 12 0.25000 0.17000 0.20550 0.12076 0.36842 0.30105 0.57646
KNN 35 0.25000 0.17000 0.17810 0.09043 0.31818 0.24545 0.59667
KNNW 1 0.31250 0.23917 0.17218 0.08388 0.33333 0.26222 0.59833
KNNW 3 0.31250 0.23917 0.18539 0.09850 0.32432 0.25225 0.60208
KNNW 15 0.31250 0.23917 0.19422 0.10827 0.38889 0.32370 0.58792
KNNW 17 0.31250 0.23917 0.19904 0.11360 0.38889 0.32370 0.58833
LOF 4 0.18750 0.10083 0.13520 0.04295 0.24561 0.16515 0.60500
LOF 24 0.18750 0.10083 0.17465 0.08662 0.34286 0.27276 0.55708
LOF 27 0.31250 0.23917 0.17512 0.08713 0.31579 0.24281 0.55833
LOF 33 0.31250 0.23917 0.17861 0.09099 0.31579 0.24281 0.56583
SimplifiedLOF 8 0.31250 0.23917 0.15881 0.06909 0.31250 0.23917 0.60125
SimplifiedLOF 22 0.25000 0.17000 0.18214 0.09490 0.30189 0.22742 0.61125
SimplifiedLOF 23 0.25000 0.17000 0.18061 0.09321 0.28571 0.20952 0.61167
SimplifiedLOF 41 0.25000 0.17000 0.17661 0.08878 0.35000 0.28067 0.58958
LoOP 8 0.25000 0.17000 0.16057 0.07104 0.29412 0.21882 0.61521
LoOP 22 0.25000 0.17000 0.17629 0.08843 0.29412 0.21882 0.62375
LoOP 32 0.18750 0.10083 0.17393 0.08581 0.35000 0.28067 0.60042
LoOP 52 0.25000 0.17000 0.18557 0.09870 0.32432 0.25225 0.58646
LDOF 4 0.12500 0.03167 0.13987 0.04813 0.25000 0.17000 0.65542
LDOF 49 0.25000 0.17000 0.17457 0.08653 0.27451 0.19712 0.58792
LDOF 63 0.25000 0.17000 0.17958 0.09207 0.29167 0.21611 0.58708
LDOF 89 0.25000 0.17000 0.17112 0.08271 0.31579 0.24281 0.57125
ODIN 1 0.17241 0.08414 0.14590 0.05480 0.27027 0.19243 0.62833
ODIN 8 0.25000 0.17000 0.18218 0.09494 0.29630 0.22123 0.58062
ODIN 60 0.31250 0.23917 0.16772 0.07895 0.31250 0.23917 0.55000
ODIN 62 0.31250 0.23917 0.16652 0.07762 0.32258 0.25032 0.55042
FastABOD 3 0.18750 0.10083 0.18429 0.09728 0.29032 0.21462 0.66667
FastABOD 5 0.31250 0.23917 0.20058 0.11530 0.31250 0.23917 0.62292
FastABOD 11 0.31250 0.23917 0.20361 0.11866 0.34483 0.27494 0.60375
FastABOD 21 0.31250 0.23917 0.21282 0.12885 0.34483 0.27494 0.60833
KDEOS 6 0.18750 0.10083 0.14254 0.05108 0.28571 0.20952 0.59167
KDEOS 17 0.12500 0.03167 0.22080 0.13768 0.22222 0.13926 0.53583
KDEOS 75 0.12500 0.03167 0.12960 0.03675 0.29787 0.22298 0.55250
LDF 5 0.31250 0.23917 0.20677 0.12216 0.32258 0.25032 0.62250
LDF 14 0.25000 0.17000 0.19293 0.10684 0.38889 0.32370 0.60375
INFLO 5 0.12500 0.03167 0.15593 0.06589 0.28571 0.20952 0.67667
INFLO 9 0.25000 0.17000 0.17417 0.08608 0.31818 0.24545 0.63250
INFLO 26 0.18750 0.10083 0.16853 0.07984 0.33333 0.26222 0.62333
INFLO 41 0.25000 0.17000 0.18639 0.09961 0.29268 0.21724 0.63667
COF 15 0.31250 0.23917 0.18791 0.10128 0.34043 0.27007 0.65229
COF 94 0.37500 0.30833 0.22228 0.13932 0.38710 0.32172 0.62958
COF 99 0.37500 0.30833 0.20074 0.11548 0.40000 0.33600 0.62354

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO