Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (5% of outliers version#03)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (37.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 29 0.42857 0.40190 0.53959 0.51810 0.60000 0.58133 0.83810
KNN 50 0.57143 0.55143 0.53002 0.50809 0.57143 0.55143 0.86476
KNN 54 0.57143 0.55143 0.56567 0.54541 0.60000 0.58133 0.86571
KNNW 2 0.42857 0.40190 0.33477 0.30373 0.46154 0.43641 0.75524
KNNW 15 0.42857 0.40190 0.49608 0.47256 0.60000 0.58133 0.79905
KNNW 83 0.42857 0.40190 0.54954 0.52852 0.60000 0.58133 0.84762
LOF 37 0.42857 0.40190 0.41262 0.38521 0.44444 0.41852 0.79429
LOF 69 0.42857 0.40190 0.52899 0.50701 0.60000 0.58133 0.85714
LOF 78 0.42857 0.40190 0.54814 0.52705 0.60000 0.58133 0.86762
LOF 80 0.42857 0.40190 0.55577 0.53504 0.60000 0.58133 0.86667
SimplifiedLOF 70 0.42857 0.40190 0.41495 0.38764 0.46154 0.43641 0.78952
SimplifiedLOF 76 0.42857 0.40190 0.43364 0.40721 0.50000 0.47667 0.79905
SimplifiedLOF 96 0.42857 0.40190 0.44947 0.42378 0.50000 0.47667 0.82095
SimplifiedLOF 97 0.42857 0.40190 0.45129 0.42569 0.50000 0.47667 0.82095
LoOP 60 0.28571 0.25238 0.39119 0.36278 0.44444 0.41852 0.80286
LoOP 70 0.42857 0.40190 0.40133 0.37339 0.44444 0.41852 0.76381
LoOP 79 0.42857 0.40190 0.43861 0.41241 0.50000 0.47667 0.78143
LoOP 91 0.42857 0.40190 0.44940 0.42370 0.50000 0.47667 0.79333
LDOF 70 0.42857 0.40190 0.34771 0.31727 0.42857 0.40190 0.76095
LDOF 98 0.42857 0.40190 0.43608 0.40976 0.50000 0.47667 0.80762
LDOF 100 0.42857 0.40190 0.44222 0.41619 0.50000 0.47667 0.80952
ODIN 66 0.42857 0.40190 0.42588 0.39909 0.44444 0.41852 0.81143
ODIN 84 0.42857 0.40190 0.50937 0.48648 0.54545 0.52424 0.84333
ODIN 87 0.42857 0.40190 0.51125 0.48844 0.54545 0.52424 0.84952
ODIN 100 0.42857 0.40190 0.46189 0.43678 0.50000 0.47667 0.85524
FastABOD 6 0.42857 0.40190 0.46417 0.43917 0.53333 0.51156 0.83429
FastABOD 11 0.42857 0.40190 0.52428 0.50208 0.60000 0.58133 0.84000
FastABOD 61 0.42857 0.40190 0.55650 0.53580 0.60000 0.58133 0.86952
FastABOD 89 0.42857 0.40190 0.55633 0.53563 0.60000 0.58133 0.87429
KDEOS 5 0.14286 0.10286 0.07585 0.03272 0.16667 0.12778 0.53238
KDEOS 11 0.14286 0.10286 0.11424 0.07290 0.28571 0.25238 0.61429
KDEOS 12 0.14286 0.10286 0.12487 0.08403 0.22222 0.18593 0.59619
KDEOS 99 0.00000 -0.04667 0.07405 0.03083 0.16438 0.12539 0.68095
LDF 31 0.57143 0.55143 0.55903 0.53846 0.60000 0.58133 0.80667
LDF 32 0.57143 0.55143 0.59224 0.57321 0.66667 0.65111 0.80952
LDF 69 0.57143 0.55143 0.64601 0.62949 0.66667 0.65111 0.91238
INFLO 32 0.42857 0.40190 0.35737 0.32738 0.42857 0.40190 0.80762
INFLO 89 0.42857 0.40190 0.52108 0.49873 0.60000 0.58133 0.77429
INFLO 90 0.42857 0.40190 0.49498 0.47141 0.54545 0.52424 0.87429
COF 54 0.57143 0.55143 0.58442 0.56503 0.60000 0.58133 0.84190
COF 55 0.57143 0.55143 0.61506 0.59709 0.66667 0.65111 0.84381
COF 87 0.42857 0.40190 0.53189 0.51005 0.60000 0.58133 0.85048

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (37.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.28571 0.25238 0.17532 0.13683 0.30769 0.27538 0.77905
KNN 9 0.14286 0.10286 0.15283 0.11329 0.31579 0.28386 0.76095
KNNW 1 0.28571 0.25238 0.18062 0.14238 0.28571 0.25238 0.68333
KNNW 17 0.28571 0.25238 0.15601 0.11662 0.28571 0.25238 0.76190
LOF 11 0.14286 0.10286 0.14094 0.10085 0.25000 0.21500 0.76952
LOF 12 0.28571 0.25238 0.15379 0.11430 0.28571 0.25238 0.76571
LOF 19 0.28571 0.25238 0.16226 0.12316 0.33333 0.30222 0.74857
LOF 22 0.28571 0.25238 0.16709 0.12822 0.31579 0.28386 0.74952
SimplifiedLOF 7 0.28571 0.25238 0.11026 0.06874 0.28571 0.25238 0.62381
SimplifiedLOF 25 0.28571 0.25238 0.16788 0.12904 0.30769 0.27538 0.76667
SimplifiedLOF 34 0.28571 0.25238 0.16567 0.12673 0.31579 0.28386 0.75333
LoOP 8 0.28571 0.25238 0.10743 0.06577 0.28571 0.25238 0.61048
LoOP 25 0.28571 0.25238 0.16422 0.12521 0.30769 0.27538 0.74857
LoOP 28 0.28571 0.25238 0.16425 0.12524 0.30769 0.27538 0.74714
LoOP 34 0.28571 0.25238 0.16027 0.12108 0.31579 0.28386 0.72810
LDOF 8 0.28571 0.25238 0.11841 0.07726 0.28571 0.25238 0.61333
LDOF 22 0.28571 0.25238 0.16238 0.12329 0.30000 0.26733 0.75524
LDOF 25 0.28571 0.25238 0.16649 0.12759 0.31579 0.28386 0.75238
LDOF 41 0.28571 0.25238 0.15096 0.11134 0.33333 0.30222 0.73429
ODIN 12 0.28571 0.25238 0.14495 0.10505 0.28571 0.25238 0.73238
ODIN 17 0.28571 0.25238 0.18191 0.14373 0.30769 0.27538 0.78476
ODIN 18 0.28571 0.25238 0.18727 0.14934 0.30769 0.27538 0.77381
ODIN 47 0.28571 0.25238 0.17030 0.13158 0.37500 0.34583 0.71810
FastABOD 3 0.28571 0.25238 0.14426 0.10433 0.28571 0.25238 0.73905
FastABOD 5 0.14286 0.10286 0.16660 0.12770 0.26667 0.23244 0.79905
FastABOD 19 0.28571 0.25238 0.18798 0.15009 0.28571 0.25238 0.76381
KDEOS 7 0.28571 0.25238 0.14675 0.10693 0.33333 0.30222 0.61429
KDEOS 15 0.28571 0.25238 0.28979 0.25665 0.40000 0.37200 0.66667
KDEOS 51 0.14286 0.10286 0.17064 0.13193 0.37500 0.34583 0.77905
LDF 9 0.14286 0.10286 0.17363 0.13507 0.30000 0.26733 0.83238
LDF 12 0.14286 0.10286 0.15837 0.11910 0.33333 0.30222 0.77429
LDF 13 0.28571 0.25238 0.15760 0.11829 0.31579 0.28386 0.74095
INFLO 6 0.28571 0.25238 0.12105 0.08003 0.28571 0.25238 0.67048
INFLO 13 0.28571 0.25238 0.16250 0.12341 0.30769 0.27538 0.78952
INFLO 26 0.28571 0.25238 0.17543 0.13695 0.35294 0.32275 0.73714
COF 28 0.14286 0.10286 0.18788 0.14998 0.28571 0.25238 0.80190
COF 31 0.28571 0.25238 0.16492 0.12595 0.30769 0.27538 0.80667
COF 78 0.14286 0.10286 0.15501 0.11557 0.35294 0.32275 0.69619

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO