Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (44% of outliers)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 270 objects, 120 outliers (44.44%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (50.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 22 0.54167 0.17500 0.57226 0.23006 0.70064 0.46115 0.68011
KNN 81 0.58333 0.25000 0.60064 0.28115 0.67606 0.41690 0.68383
KNN 92 0.60833 0.29500 0.59816 0.27669 0.66460 0.39627 0.67631
KNN 97 0.60833 0.29500 0.60678 0.29221 0.66899 0.40418 0.68214
KNNW 71 0.55000 0.19000 0.56546 0.21783 0.68966 0.44138 0.66056
KNNW 99 0.55000 0.19000 0.57298 0.23137 0.69670 0.45405 0.66861
KNNW 100 0.55000 0.19000 0.57327 0.23188 0.69670 0.45405 0.66883
LOF 71 0.53333 0.16000 0.52436 0.14386 0.67541 0.41574 0.62133
LOF 100 0.63333 0.34000 0.57754 0.23957 0.64348 0.35826 0.65583
SimplifiedLOF 67 0.50833 0.11500 0.49557 0.09202 0.61538 0.30769 0.54978
SimplifiedLOF 96 0.50833 0.11500 0.49866 0.09759 0.63192 0.33746 0.56761
SimplifiedLOF 100 0.50833 0.11500 0.50077 0.10138 0.62987 0.33377 0.56933
LoOP 1 0.43333 -0.02000 0.42839 -0.02889 0.61538 0.30769 0.48764
LoOP 48 0.50833 0.11500 0.49357 0.08843 0.61538 0.30769 0.54483
LoOP 52 0.50833 0.11500 0.49907 0.09832 0.61538 0.30769 0.54806
LoOP 60 0.50833 0.11500 0.49603 0.09286 0.61538 0.30769 0.56139
LDOF 2 0.41667 -0.05000 0.41941 -0.04506 0.63441 0.34194 0.48567
LDOF 11 0.49167 0.08500 0.50770 0.11387 0.62703 0.32865 0.56522
LDOF 14 0.53333 0.16000 0.50206 0.10371 0.62632 0.32737 0.56906
ODIN 45 0.50500 0.10900 0.53283 0.15909 0.62016 0.31628 0.58744
ODIN 82 0.52917 0.15250 0.52812 0.15062 0.63576 0.34437 0.60592
ODIN 83 0.52500 0.14500 0.52894 0.15210 0.63758 0.34765 0.60472
ODIN 96 0.53611 0.16500 0.51958 0.13524 0.62259 0.32066 0.59700
FastABOD 82 0.62500 0.32500 0.66388 0.39499 0.74834 0.54702 0.74978
FastABOD 91 0.61667 0.31000 0.66647 0.39964 0.75083 0.55150 0.75217
FastABOD 100 0.62500 0.32500 0.67035 0.40663 0.75083 0.55150 0.75567
KDEOS 4 0.46667 0.04000 0.44883 0.00790 0.62842 0.33115 0.52011
KDEOS 62 0.46667 0.04000 0.50156 0.10280 0.61856 0.31340 0.54117
KDEOS 94 0.49167 0.08500 0.49475 0.09055 0.61856 0.31340 0.55394
KDEOS 100 0.49167 0.08500 0.49831 0.09697 0.61856 0.31340 0.55694
LDF 67 0.60833 0.29500 0.56123 0.21021 0.68217 0.42791 0.66439
LDF 79 0.65833 0.38500 0.63102 0.33583 0.67722 0.41899 0.71494
LDF 83 0.65000 0.37000 0.65338 0.37609 0.67763 0.41974 0.72056
LDF 100 0.61667 0.31000 0.67523 0.41541 0.64723 0.36501 0.70011
INFLO 4 0.45000 0.01000 0.48777 0.07799 0.62703 0.32865 0.54472
INFLO 15 0.50000 0.10000 0.48332 0.06998 0.62141 0.31854 0.55967
INFLO 45 0.48829 0.07892 0.49774 0.09593 0.62016 0.31628 0.54881
INFLO 60 0.50016 0.10029 0.48754 0.07756 0.62016 0.31628 0.55528
COF 94 0.64167 0.35500 0.60242 0.28436 0.72785 0.51013 0.70978
COF 95 0.65000 0.37000 0.59937 0.27886 0.71827 0.49288 0.71150
COF 97 0.64167 0.35500 0.60459 0.28826 0.71519 0.48734 0.71011
COF 100 0.63333 0.34000 0.60417 0.28751 0.71895 0.49412 0.71683

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 270 objects, 120 outliers (44.44%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (48.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.60000 0.28000 0.56104 0.20987 0.63664 0.34595 0.64803
KNN 17 0.59167 0.26500 0.57210 0.22978 0.65957 0.38723 0.66469
KNN 20 0.58333 0.25000 0.57022 0.22639 0.66207 0.39172 0.66203
KNN 22 0.58333 0.25000 0.57342 0.23216 0.65772 0.38389 0.66264
KNNW 16 0.60000 0.28000 0.56176 0.21117 0.64516 0.36129 0.65317
KNNW 25 0.59167 0.26500 0.56637 0.21947 0.64748 0.36547 0.65761
KNNW 45 0.56667 0.22000 0.56362 0.21452 0.65101 0.37181 0.65378
LOF 25 0.56667 0.22000 0.52947 0.15305 0.64331 0.35796 0.62639
LOF 31 0.58333 0.25000 0.53437 0.16187 0.62893 0.33208 0.62922
LOF 40 0.57500 0.23500 0.54018 0.17233 0.63448 0.34207 0.63278
LOF 46 0.55833 0.20500 0.54159 0.17487 0.63736 0.34725 0.63189
SimplifiedLOF 68 0.53333 0.16000 0.52654 0.14777 0.63158 0.33684 0.61733
SimplifiedLOF 69 0.53333 0.16000 0.52646 0.14762 0.63750 0.34750 0.61756
SimplifiedLOF 83 0.54167 0.17500 0.52225 0.14005 0.64375 0.35875 0.61400
SimplifiedLOF 98 0.55833 0.20500 0.52266 0.14078 0.63804 0.34847 0.61222
LoOP 1 0.49167 0.08500 0.46097 0.02975 0.61538 0.30769 0.52622
LoOP 56 0.55000 0.19000 0.51022 0.11840 0.61538 0.30769 0.59994
LoOP 60 0.53333 0.16000 0.51108 0.11995 0.61538 0.30769 0.60342
LoOP 65 0.54167 0.17500 0.51197 0.12155 0.61538 0.30769 0.59933
LDOF 53 0.51667 0.13000 0.49736 0.09525 0.63760 0.34768 0.58500
LDOF 61 0.53333 0.16000 0.50523 0.10942 0.63187 0.33736 0.59222
LDOF 71 0.52500 0.14500 0.50903 0.11625 0.62466 0.32438 0.59883
LDOF 78 0.52500 0.14500 0.51206 0.12170 0.62953 0.33315 0.59811
ODIN 7 0.45556 0.02000 0.47590 0.05662 0.62992 0.33386 0.53183
ODIN 53 0.52500 0.14500 0.51984 0.13572 0.62248 0.32046 0.59786
ODIN 84 0.54167 0.17500 0.52210 0.13979 0.61579 0.30842 0.58886
ODIN 100 0.55833 0.20500 0.51409 0.12536 0.61856 0.31340 0.57819
FastABOD 8 0.56667 0.22000 0.54631 0.18337 0.62759 0.32966 0.61578
FastABOD 61 0.59167 0.26500 0.55826 0.20486 0.62016 0.31628 0.63206
FastABOD 96 0.59167 0.26500 0.56445 0.21601 0.62176 0.31917 0.63672
KDEOS 34 0.43333 -0.02000 0.53310 0.15959 0.62176 0.31917 0.54889
KDEOS 87 0.50000 0.10000 0.50181 0.10326 0.63636 0.34545 0.58233
KDEOS 96 0.50833 0.11500 0.50594 0.11069 0.63415 0.34146 0.58811
KDEOS 98 0.50833 0.11500 0.50405 0.10728 0.63415 0.34146 0.58889
LDF 16 0.61667 0.31000 0.57308 0.23154 0.64088 0.35359 0.66700
LDF 26 0.60000 0.28000 0.59468 0.27042 0.64452 0.36013 0.67572
LDF 30 0.60833 0.29500 0.59855 0.27739 0.64495 0.36091 0.67489
LDF 39 0.59167 0.26500 0.58431 0.25177 0.64906 0.36830 0.65789
INFLO 44 0.52595 0.14671 0.48346 0.07023 0.62500 0.32500 0.56694
INFLO 49 0.55894 0.20609 0.49569 0.09225 0.62016 0.31628 0.59694
COF 44 0.63333 0.34000 0.62043 0.31677 0.66904 0.40427 0.69794
COF 45 0.60833 0.29500 0.61754 0.31157 0.67586 0.41655 0.69275
COF 66 0.64167 0.35500 0.60002 0.28003 0.65672 0.38209 0.67894

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO