Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (5% of outliers version#07)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (36.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.71429 0.70095 0.70491 0.69114 0.72727 0.71455 0.89857
KNN 70 0.57143 0.55143 0.55505 0.53428 0.66667 0.65111 0.96952
KNNW 4 0.57143 0.55143 0.54191 0.52053 0.57143 0.55143 0.88000
KNNW 6 0.57143 0.55143 0.59982 0.58114 0.62500 0.60750 0.89619
KNNW 7 0.57143 0.55143 0.57392 0.55403 0.66667 0.65111 0.90381
KNNW 93 0.57143 0.55143 0.55262 0.53175 0.63158 0.61439 0.96190
LOF 27 0.57143 0.55143 0.49813 0.47471 0.57143 0.55143 0.92857
LOF 28 0.57143 0.55143 0.55595 0.53523 0.66667 0.65111 0.93429
LOF 42 0.57143 0.55143 0.60360 0.58510 0.66667 0.65111 0.95905
LOF 57 0.57143 0.55143 0.52394 0.50173 0.61538 0.59744 0.96762
SimplifiedLOF 72 0.42857 0.40190 0.44002 0.41389 0.50000 0.47667 0.93619
SimplifiedLOF 73 0.42857 0.40190 0.39679 0.36864 0.55556 0.53481 0.93905
SimplifiedLOF 99 0.42857 0.40190 0.47357 0.44900 0.55556 0.53481 0.95333
LoOP 45 0.42857 0.40190 0.40048 0.37250 0.46154 0.43641 0.89429
LoOP 73 0.42857 0.40190 0.42328 0.39637 0.55556 0.53481 0.94286
LoOP 99 0.42857 0.40190 0.47815 0.45379 0.55556 0.53481 0.95524
LDOF 82 0.28571 0.25238 0.41606 0.38881 0.50000 0.47667 0.92857
LDOF 87 0.42857 0.40190 0.45664 0.43128 0.47619 0.45175 0.92857
LDOF 96 0.42857 0.40190 0.48313 0.45901 0.47619 0.45175 0.94000
LDOF 97 0.42857 0.40190 0.49528 0.47172 0.50000 0.47667 0.93810
ODIN 52 0.57143 0.55143 0.47468 0.45017 0.57143 0.55143 0.94476
ODIN 55 0.57143 0.55143 0.50523 0.48214 0.66667 0.65111 0.95286
ODIN 82 0.57143 0.55143 0.49158 0.46785 0.57143 0.55143 0.96667
ODIN 85 0.57143 0.55143 0.57754 0.55783 0.57143 0.55143 0.96524
FastABOD 9 0.42857 0.40190 0.56583 0.54556 0.60000 0.58133 0.91905
FastABOD 34 0.57143 0.55143 0.55921 0.53864 0.57143 0.55143 0.94190
FastABOD 71 0.42857 0.40190 0.58636 0.56706 0.55556 0.53481 0.94667
FastABOD 90 0.42857 0.40190 0.57495 0.55511 0.54545 0.52424 0.94762
KDEOS 83 0.14286 0.10286 0.14553 0.10565 0.31250 0.28042 0.81238
KDEOS 97 0.28571 0.25238 0.17106 0.13237 0.28571 0.25238 0.82571
KDEOS 100 0.28571 0.25238 0.18241 0.14425 0.29412 0.26118 0.83524
LDF 22 0.42857 0.40190 0.58305 0.56359 0.57143 0.55143 0.96190
LDF 23 0.42857 0.40190 0.58216 0.56266 0.57143 0.55143 0.96381
LDF 35 0.57143 0.55143 0.51836 0.49589 0.57143 0.55143 0.95619
INFLO 78 0.28571 0.25238 0.43200 0.40549 0.63158 0.61439 0.96286
INFLO 91 0.42857 0.40190 0.52534 0.50319 0.58824 0.56902 0.96286
INFLO 99 0.57143 0.55143 0.52088 0.49852 0.57143 0.55143 0.95143
COF 36 0.57143 0.55143 0.68762 0.67304 0.72727 0.71455 0.95333
COF 79 0.71429 0.70095 0.60760 0.58928 0.71429 0.70095 0.95238

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (38.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.00000 -0.04667 0.16862 0.12982 0.36364 0.33394 0.84714
KNN 6 0.14286 0.10286 0.16605 0.12713 0.34783 0.31739 0.83619
KNN 7 0.14286 0.10286 0.18028 0.14202 0.36364 0.33394 0.85429
KNNW 1 0.00000 -0.04667 0.20224 0.16501 0.38462 0.35590 0.88476
KNNW 19 0.14286 0.10286 0.17049 0.13178 0.36364 0.33394 0.84857
LOF 1 0.00000 -0.04667 0.07483 0.03165 0.18750 0.14958 0.53190
LOF 18 0.00000 -0.04667 0.16337 0.12433 0.33333 0.30222 0.85619
LOF 25 0.00000 -0.04667 0.17288 0.13428 0.40000 0.37200 0.83524
SimplifiedLOF 1 0.00000 -0.04667 0.05045 0.00614 0.10909 0.06752 0.52952
SimplifiedLOF 18 0.00000 -0.04667 0.16016 0.12097 0.36364 0.33394 0.85333
SimplifiedLOF 24 0.00000 -0.04667 0.16170 0.12258 0.30769 0.27538 0.84952
LoOP 12 0.14286 0.10286 0.13193 0.09142 0.27273 0.23879 0.81048
LoOP 18 0.00000 -0.04667 0.16301 0.12395 0.36364 0.33394 0.85333
LoOP 24 0.00000 -0.04667 0.16190 0.12278 0.32000 0.28827 0.85429
LDOF 7 0.14286 0.10286 0.08576 0.04309 0.14286 0.10286 0.66190
LDOF 41 0.00000 -0.04667 0.13895 0.09876 0.28571 0.25238 0.83905
LDOF 73 0.00000 -0.04667 0.13661 0.09632 0.30769 0.27538 0.82381
ODIN 6 0.09524 0.05302 0.12193 0.08096 0.22727 0.19121 0.79619
ODIN 8 0.04762 0.00317 0.16071 0.12154 0.32000 0.28827 0.81286
ODIN 9 0.07143 0.02810 0.15251 0.11296 0.27273 0.23879 0.82762
FastABOD 4 0.14286 0.10286 0.14843 0.10869 0.31250 0.28042 0.81333
FastABOD 33 0.14286 0.10286 0.19140 0.15367 0.40000 0.37200 0.85905
FastABOD 58 0.00000 -0.04667 0.19129 0.15355 0.42105 0.39404 0.84762
FastABOD 68 0.00000 -0.04667 0.19264 0.15496 0.42105 0.39404 0.85048
KDEOS 5 0.28571 0.25238 0.14867 0.10895 0.28571 0.25238 0.70667
KDEOS 7 0.14286 0.10286 0.24793 0.21283 0.30000 0.26733 0.71905
KDEOS 9 0.14286 0.10286 0.18150 0.14330 0.33333 0.30222 0.79619
KDEOS 72 0.00000 -0.04667 0.13694 0.09667 0.29412 0.26118 0.82762
LDF 1 0.28571 0.25238 0.19796 0.16053 0.33333 0.30222 0.59857
LDF 15 0.00000 -0.04667 0.19215 0.15445 0.40000 0.37200 0.86381
LDF 16 0.00000 -0.04667 0.19297 0.15531 0.38095 0.35206 0.86952
INFLO 8 0.14286 0.10286 0.11568 0.07442 0.21429 0.17762 0.77905
INFLO 18 0.00000 -0.04667 0.15682 0.11747 0.37037 0.34099 0.84952
INFLO 25 0.00000 -0.04667 0.15162 0.11203 0.29167 0.25861 0.85810
COF 57 0.28571 0.25238 0.20833 0.17139 0.38462 0.35590 0.87619
COF 70 0.28571 0.25238 0.26063 0.22613 0.47059 0.44588 0.88762

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO