Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (10% of outliers version#06)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (44.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 20 0.56250 0.51583 0.48746 0.43279 0.56250 0.51583 0.85542
KNN 65 0.56250 0.51583 0.53866 0.48945 0.60000 0.55733 0.86333
KNN 68 0.56250 0.51583 0.54728 0.49899 0.58065 0.53591 0.86458
KNN 70 0.56250 0.51583 0.53392 0.48420 0.58824 0.54431 0.86583
KNNW 80 0.50000 0.44667 0.49029 0.43593 0.55172 0.50391 0.85167
KNNW 98 0.56250 0.51583 0.49828 0.44477 0.56250 0.51583 0.85042
LOF 84 0.50000 0.44667 0.51846 0.46709 0.57143 0.52571 0.83458
LOF 85 0.56250 0.51583 0.51976 0.46853 0.57143 0.52571 0.83667
LOF 98 0.56250 0.51583 0.52667 0.47618 0.57143 0.52571 0.85333
SimplifiedLOF 92 0.31250 0.23917 0.40045 0.33650 0.48649 0.43171 0.82000
SimplifiedLOF 93 0.37500 0.30833 0.40098 0.33709 0.45000 0.39133 0.81958
SimplifiedLOF 97 0.37500 0.30833 0.41860 0.35658 0.46154 0.40410 0.82417
SimplifiedLOF 100 0.37500 0.30833 0.41464 0.35220 0.48649 0.43171 0.82583
LoOP 96 0.43750 0.37750 0.41178 0.34904 0.46154 0.40410 0.82771
LoOP 100 0.43750 0.37750 0.42888 0.36796 0.50000 0.44667 0.83375
LDOF 4 0.25000 0.17000 0.15701 0.06710 0.26667 0.18844 0.62417
LDOF 100 0.25000 0.17000 0.34462 0.27471 0.39130 0.32638 0.78750
ODIN 86 0.50000 0.44667 0.45647 0.39849 0.50000 0.44667 0.83042
ODIN 87 0.50000 0.44667 0.46634 0.40942 0.53333 0.48356 0.83229
ODIN 100 0.50000 0.44667 0.50611 0.45342 0.53333 0.48356 0.84417
FastABOD 25 0.43750 0.37750 0.38686 0.32146 0.43750 0.37750 0.81583
FastABOD 75 0.43750 0.37750 0.45735 0.39946 0.46667 0.40978 0.84708
FastABOD 82 0.43750 0.37750 0.43133 0.37067 0.48276 0.42759 0.84792
FastABOD 96 0.43750 0.37750 0.43650 0.37639 0.48276 0.42759 0.85167
KDEOS 4 0.12500 0.03167 0.19700 0.11134 0.25352 0.17390 0.63542
KDEOS 5 0.18750 0.10083 0.16177 0.07236 0.29630 0.22123 0.61083
KDEOS 97 0.06250 -0.03750 0.14179 0.05024 0.31818 0.24545 0.66375
KDEOS 100 0.06250 -0.03750 0.14999 0.05932 0.31818 0.24545 0.67208
LDF 25 0.56250 0.51583 0.46911 0.41248 0.56250 0.51583 0.82208
LDF 47 0.50000 0.44667 0.59068 0.54701 0.61111 0.56963 0.85792
LDF 57 0.50000 0.44667 0.60361 0.56133 0.56410 0.51761 0.86583
LDF 66 0.50000 0.44667 0.57721 0.53211 0.55556 0.50815 0.87292
INFLO 76 0.43750 0.37750 0.43320 0.37275 0.47619 0.42032 0.85229
INFLO 87 0.43750 0.37750 0.44382 0.38450 0.50000 0.44667 0.86313
INFLO 100 0.43750 0.37750 0.47371 0.41757 0.48649 0.43171 0.87562
COF 53 0.50000 0.44667 0.49357 0.43955 0.55000 0.50200 0.85708
COF 57 0.43750 0.37750 0.48672 0.43197 0.57143 0.52571 0.85333
COF 89 0.50000 0.44667 0.57883 0.53391 0.52174 0.47072 0.86250
COF 90 0.50000 0.44667 0.57250 0.52690 0.52174 0.47072 0.87792

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (42.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.31250 0.23917 0.26672 0.18851 0.42105 0.35930 0.80396
KNN 7 0.37500 0.30833 0.27624 0.19904 0.42857 0.36762 0.80167
KNN 8 0.31250 0.23917 0.27251 0.19491 0.43636 0.37624 0.80021
KNNW 1 0.31250 0.23917 0.25524 0.17579 0.40000 0.33600 0.79729
KNNW 4 0.31250 0.23917 0.26359 0.18504 0.44444 0.38519 0.80083
KNNW 8 0.31250 0.23917 0.28033 0.20356 0.41176 0.34902 0.80625
LOF 7 0.25000 0.17000 0.22974 0.14758 0.36735 0.29986 0.76292
LOF 8 0.25000 0.17000 0.24491 0.16437 0.41667 0.35444 0.76083
LOF 9 0.25000 0.17000 0.24988 0.16986 0.41509 0.35270 0.74167
LOF 26 0.37500 0.30833 0.23781 0.15651 0.37500 0.30833 0.72917
SimplifiedLOF 8 0.37500 0.30833 0.24675 0.16640 0.37500 0.30833 0.78250
SimplifiedLOF 10 0.25000 0.17000 0.25517 0.17572 0.41026 0.34735 0.79458
SimplifiedLOF 18 0.18750 0.10083 0.23464 0.15300 0.41860 0.35659 0.74125
LoOP 8 0.31250 0.23917 0.23661 0.15518 0.33803 0.26742 0.78167
LoOP 10 0.18750 0.10083 0.24643 0.16605 0.41026 0.34735 0.78917
LDOF 8 0.18750 0.10083 0.20071 0.11545 0.37931 0.31310 0.74792
LDOF 10 0.18750 0.10083 0.20042 0.11514 0.32836 0.25672 0.75500
LDOF 50 0.31250 0.23917 0.22356 0.14074 0.31579 0.24281 0.72583
LDOF 52 0.31250 0.23917 0.22448 0.14176 0.32787 0.25617 0.72458
ODIN 10 0.22500 0.14233 0.20754 0.12301 0.35556 0.28681 0.72542
ODIN 14 0.31250 0.23917 0.21447 0.13068 0.32432 0.25225 0.70562
ODIN 70 0.31250 0.23917 0.21675 0.13320 0.32258 0.25032 0.71583
ODIN 90 0.25000 0.17000 0.21397 0.13013 0.33333 0.26222 0.72646
FastABOD 4 0.25000 0.17000 0.31656 0.24366 0.43478 0.37449 0.82458
FastABOD 6 0.31250 0.23917 0.31820 0.24548 0.45833 0.40056 0.81125
FastABOD 9 0.31250 0.23917 0.29258 0.21713 0.47826 0.42261 0.80583
FastABOD 15 0.37500 0.30833 0.29632 0.22126 0.45833 0.40056 0.81333
KDEOS 9 0.25000 0.17000 0.19043 0.10408 0.29126 0.21566 0.71958
KDEOS 16 0.18750 0.10083 0.23529 0.15372 0.25600 0.17664 0.67208
KDEOS 63 0.25000 0.17000 0.19922 0.11380 0.35294 0.28392 0.72625
KDEOS 99 0.18750 0.10083 0.19868 0.11320 0.32727 0.25552 0.73875
LDF 5 0.37500 0.30833 0.31230 0.23895 0.51429 0.46248 0.75875
LDF 6 0.43750 0.37750 0.28722 0.21120 0.45455 0.39636 0.73625
LDF 58 0.25000 0.17000 0.23833 0.15708 0.37500 0.30833 0.77458
INFLO 8 0.25000 0.17000 0.26978 0.19189 0.40909 0.34606 0.79292
INFLO 9 0.18750 0.10083 0.26393 0.18542 0.43243 0.37189 0.74542
INFLO 10 0.31250 0.23917 0.24116 0.16021 0.35556 0.28681 0.74875
COF 18 0.37500 0.30833 0.25159 0.17176 0.37500 0.30833 0.77208
COF 66 0.25000 0.17000 0.29330 0.21792 0.39344 0.32874 0.80437
COF 74 0.18750 0.10083 0.25773 0.17856 0.41667 0.35444 0.79042
COF 98 0.31250 0.23917 0.32035 0.24785 0.38095 0.31492 0.79208

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO