Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (10% of outliers version#05)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (43.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 15 0.31250 0.23917 0.44780 0.38889 0.46809 0.41135 0.86625
KNN 23 0.31250 0.23917 0.46240 0.40506 0.48387 0.42882 0.86458
KNN 34 0.43750 0.37750 0.50042 0.44713 0.46154 0.40410 0.85625
KNN 73 0.37500 0.30833 0.50560 0.45287 0.48000 0.42453 0.85542
KNNW 71 0.43750 0.37750 0.45781 0.39997 0.45161 0.39312 0.85667
KNNW 75 0.37500 0.30833 0.45786 0.40003 0.45902 0.40131 0.85625
KNNW 100 0.37500 0.30833 0.46919 0.41257 0.45161 0.39312 0.85542
LOF 48 0.37500 0.30833 0.41713 0.35496 0.42424 0.36283 0.83917
LOF 80 0.37500 0.30833 0.50349 0.45053 0.48889 0.43437 0.86250
LOF 94 0.37500 0.30833 0.51717 0.46567 0.46154 0.40410 0.86167
LOF 98 0.37500 0.30833 0.50539 0.45263 0.46154 0.40410 0.86583
SimplifiedLOF 27 0.31250 0.23917 0.23247 0.15060 0.31250 0.23917 0.64542
SimplifiedLOF 95 0.31250 0.23917 0.41406 0.35156 0.41176 0.34902 0.82625
SimplifiedLOF 96 0.31250 0.23917 0.41377 0.35124 0.41791 0.35582 0.82667
SimplifiedLOF 99 0.31250 0.23917 0.41215 0.34945 0.42254 0.36094 0.82458
LoOP 35 0.37500 0.30833 0.29145 0.21588 0.37500 0.30833 0.72167
LoOP 96 0.31250 0.23917 0.41904 0.35707 0.41176 0.34902 0.82500
LoOP 97 0.31250 0.23917 0.41639 0.35413 0.41667 0.35444 0.82292
LDOF 82 0.31250 0.23917 0.30652 0.23255 0.32727 0.25552 0.76208
LDOF 97 0.31250 0.23917 0.35283 0.28380 0.37037 0.30321 0.78583
LDOF 100 0.31250 0.23917 0.34930 0.27990 0.37037 0.30321 0.78917
ODIN 14 0.38750 0.32217 0.25334 0.17369 0.38889 0.32370 0.70979
ODIN 97 0.37500 0.30833 0.47330 0.41712 0.48889 0.43437 0.83542
ODIN 100 0.37500 0.30833 0.47054 0.41406 0.50000 0.44667 0.83500
FastABOD 26 0.50000 0.44667 0.45678 0.39884 0.50000 0.44667 0.84625
FastABOD 68 0.50000 0.44667 0.48545 0.43057 0.53333 0.48356 0.85542
FastABOD 69 0.50000 0.44667 0.48563 0.43076 0.53333 0.48356 0.85583
FastABOD 75 0.50000 0.44667 0.48545 0.43056 0.53333 0.48356 0.85667
KDEOS 9 0.18750 0.10083 0.19871 0.11324 0.25000 0.17000 0.54167
KDEOS 49 0.25000 0.17000 0.14844 0.05760 0.25532 0.17589 0.63625
KDEOS 96 0.25000 0.17000 0.18407 0.09703 0.30233 0.22791 0.72500
KDEOS 99 0.25000 0.17000 0.18824 0.10166 0.29885 0.22406 0.72667
LDF 54 0.50000 0.44667 0.59305 0.54964 0.53659 0.48715 0.89750
LDF 64 0.56250 0.51583 0.60716 0.56525 0.56250 0.51583 0.89333
LDF 66 0.56250 0.51583 0.59069 0.54703 0.58065 0.53591 0.87958
INFLO 33 0.43750 0.37750 0.31927 0.24666 0.43750 0.37750 0.76750
INFLO 95 0.31250 0.23917 0.43481 0.37452 0.46875 0.41208 0.85833
INFLO 96 0.31250 0.23917 0.43277 0.37226 0.48387 0.42882 0.86125
COF 53 0.37500 0.30833 0.49107 0.43678 0.50000 0.44667 0.88875
COF 60 0.50000 0.44667 0.50423 0.45135 0.52381 0.47302 0.86625
COF 68 0.43750 0.37750 0.54734 0.49906 0.56000 0.51307 0.87167
COF 82 0.43750 0.37750 0.57052 0.52471 0.50000 0.44667 0.88042

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (42.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.31250 0.23917 0.21918 0.13589 0.35294 0.28392 0.72188
KNN 8 0.31250 0.23917 0.25303 0.17335 0.44444 0.38519 0.74333
KNN 9 0.31250 0.23917 0.25618 0.17684 0.44444 0.38519 0.73542
KNN 12 0.31250 0.23917 0.25207 0.17229 0.45714 0.39924 0.73646
KNNW 4 0.31250 0.23917 0.21338 0.12948 0.35556 0.28681 0.71167
KNNW 13 0.31250 0.23917 0.23004 0.14791 0.44444 0.38519 0.72625
KNNW 26 0.31250 0.23917 0.23890 0.15772 0.43243 0.37189 0.73125
KNNW 27 0.31250 0.23917 0.24124 0.16030 0.43243 0.37189 0.73083
LOF 9 0.25000 0.17000 0.21559 0.13191 0.36000 0.29173 0.68750
LOF 38 0.31250 0.23917 0.20825 0.12380 0.37209 0.30512 0.72208
LOF 49 0.25000 0.17000 0.20451 0.11966 0.40000 0.33600 0.70458
SimplifiedLOF 9 0.31250 0.23917 0.18495 0.09802 0.31250 0.23917 0.63458
SimplifiedLOF 72 0.25000 0.17000 0.20667 0.12205 0.40000 0.33600 0.71000
SimplifiedLOF 75 0.25000 0.17000 0.20530 0.12054 0.40000 0.33600 0.71208
LoOP 7 0.31250 0.23917 0.17974 0.09225 0.31250 0.23917 0.60833
LoOP 69 0.25000 0.17000 0.19975 0.11439 0.35294 0.28392 0.71604
LoOP 72 0.25000 0.17000 0.20242 0.11734 0.38889 0.32370 0.71396
LoOP 79 0.25000 0.17000 0.20511 0.12033 0.38889 0.32370 0.71396
LDOF 77 0.25000 0.17000 0.19971 0.11434 0.35000 0.28067 0.70875
LDOF 83 0.25000 0.17000 0.19914 0.11372 0.35897 0.29060 0.70208
LDOF 85 0.31250 0.23917 0.19943 0.11403 0.35000 0.28067 0.69708
LDOF 87 0.31250 0.23917 0.19992 0.11458 0.35000 0.28067 0.69625
ODIN 12 0.32143 0.24905 0.18839 0.10181 0.32258 0.25032 0.63708
ODIN 65 0.25000 0.17000 0.21523 0.13152 0.32558 0.25364 0.69688
ODIN 68 0.29167 0.21611 0.21158 0.12748 0.35294 0.28392 0.69875
ODIN 73 0.25000 0.17000 0.19891 0.11346 0.36842 0.30105 0.69146
FastABOD 5 0.31250 0.23917 0.21803 0.13462 0.33333 0.26222 0.67000
FastABOD 10 0.31250 0.23917 0.23868 0.15747 0.40000 0.33600 0.70292
FastABOD 15 0.31250 0.23917 0.22088 0.13777 0.33333 0.26222 0.70458
KDEOS 7 0.25000 0.17000 0.25804 0.17890 0.29630 0.22123 0.62833
KDEOS 9 0.31250 0.23917 0.15735 0.06747 0.31250 0.23917 0.58958
KDEOS 70 0.12500 0.03167 0.19208 0.10591 0.32558 0.25364 0.67292
KDEOS 100 0.18750 0.10083 0.18209 0.09484 0.32000 0.24747 0.69583
LDF 5 0.18750 0.10083 0.23386 0.15213 0.41026 0.34735 0.70458
LDF 6 0.31250 0.23917 0.23146 0.14948 0.38095 0.31492 0.69750
LDF 8 0.31250 0.23917 0.23581 0.15430 0.37209 0.30512 0.72042
LDF 13 0.31250 0.23917 0.23479 0.15317 0.35556 0.28681 0.75458
INFLO 5 0.31250 0.23917 0.19424 0.10829 0.32258 0.25032 0.59146
INFLO 53 0.31250 0.23917 0.22052 0.13738 0.36364 0.29576 0.75458
INFLO 56 0.25000 0.17000 0.20670 0.12208 0.38889 0.32370 0.70792
INFLO 59 0.25000 0.17000 0.21142 0.12731 0.37838 0.31207 0.76375
COF 51 0.25000 0.17000 0.26538 0.18702 0.40909 0.34606 0.77833
COF 75 0.50000 0.44667 0.29298 0.21757 0.51613 0.46452 0.76542
COF 77 0.50000 0.44667 0.28921 0.21339 0.53333 0.48356 0.74583

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO