Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (5% of outliers version#01)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (39.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 16 0.57143 0.55143 0.49293 0.46927 0.57143 0.55143 0.89524
KNN 41 0.57143 0.55143 0.51463 0.49198 0.61538 0.59744 0.90476
KNN 46 0.57143 0.55143 0.53092 0.50903 0.66667 0.65111 0.90238
KNN 47 0.57143 0.55143 0.53128 0.50940 0.66667 0.65111 0.90190
KNNW 61 0.57143 0.55143 0.48711 0.46318 0.57143 0.55143 0.88571
KNNW 66 0.57143 0.55143 0.48822 0.46434 0.57143 0.55143 0.88762
LOF 62 0.57143 0.55143 0.47437 0.44984 0.57143 0.55143 0.89810
LOF 73 0.57143 0.55143 0.48295 0.45883 0.61538 0.59744 0.89429
LOF 97 0.57143 0.55143 0.49176 0.46804 0.57143 0.55143 0.89619
SimplifiedLOF 94 0.42857 0.40190 0.35949 0.32960 0.53333 0.51156 0.85905
SimplifiedLOF 97 0.57143 0.55143 0.34613 0.31562 0.57143 0.55143 0.85905
LoOP 92 0.57143 0.55143 0.34538 0.31483 0.57143 0.55143 0.85619
LoOP 96 0.57143 0.55143 0.44086 0.41477 0.57143 0.55143 0.85905
LDOF 71 0.28571 0.25238 0.19513 0.15757 0.30769 0.27538 0.80095
LDOF 98 0.28571 0.25238 0.29188 0.25884 0.44444 0.41852 0.83143
LDOF 99 0.28571 0.25238 0.28864 0.25544 0.50000 0.47667 0.83143
LDOF 100 0.28571 0.25238 0.30305 0.27052 0.50000 0.47667 0.83143
ODIN 87 0.42857 0.40190 0.45027 0.42462 0.53333 0.51156 0.87286
ODIN 93 0.57143 0.55143 0.42362 0.39672 0.57143 0.55143 0.87667
ODIN 100 0.57143 0.55143 0.42558 0.39878 0.57143 0.55143 0.88238
FastABOD 29 0.42857 0.40190 0.42365 0.39675 0.44444 0.41852 0.85619
FastABOD 70 0.42857 0.40190 0.46789 0.44306 0.50000 0.47667 0.88571
FastABOD 99 0.42857 0.40190 0.48123 0.45702 0.50000 0.47667 0.89524
KDEOS 23 0.14286 0.10286 0.08513 0.04243 0.21053 0.17368 0.59524
KDEOS 24 0.00000 -0.04667 0.08390 0.04115 0.22222 0.18593 0.58571
KDEOS 34 0.14286 0.10286 0.09044 0.04800 0.17391 0.13536 0.65143
KDEOS 98 0.00000 -0.04667 0.07702 0.03395 0.17949 0.14120 0.70190
LDF 30 0.57143 0.55143 0.54729 0.52617 0.57143 0.55143 0.90667
LDF 41 0.57143 0.55143 0.62292 0.60532 0.66667 0.65111 0.92095
LDF 69 0.42857 0.40190 0.63291 0.61578 0.60000 0.58133 0.94667
INFLO 87 0.57143 0.55143 0.44713 0.42133 0.57143 0.55143 0.87905
INFLO 96 0.57143 0.55143 0.52012 0.49773 0.66667 0.65111 0.89333
INFLO 98 0.57143 0.55143 0.50555 0.48248 0.61538 0.59744 0.90095
INFLO 100 0.57143 0.55143 0.52353 0.50130 0.66667 0.65111 0.90000
COF 35 0.57143 0.55143 0.54563 0.52442 0.57143 0.55143 0.90762
COF 38 0.57143 0.55143 0.63558 0.61857 0.72727 0.71455 0.90095
COF 40 0.57143 0.55143 0.64161 0.62488 0.72727 0.71455 0.91143
COF 57 0.57143 0.55143 0.61174 0.59363 0.60000 0.58133 0.93524

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 157 objects, 7 outliers (4.46%)

Download raw algorithm results (1.3 MB) Download raw algorithm evaluation table (39.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.04667 0.09787 0.05577 0.26667 0.23244 0.68714
KNN 3 0.00000 -0.04667 0.11338 0.07201 0.28571 0.25238 0.68476
KNNW 1 0.00000 -0.04667 0.10657 0.06488 0.26087 0.22638 0.71619
KNNW 4 0.00000 -0.04667 0.11156 0.07010 0.29630 0.26346 0.69429
KNNW 7 0.00000 -0.04667 0.11513 0.07384 0.28571 0.25238 0.68381
LOF 2 0.14286 0.10286 0.09073 0.04829 0.20000 0.16267 0.59810
LOF 3 0.14286 0.10286 0.11175 0.07030 0.22222 0.18593 0.57143
LOF 9 0.00000 -0.04667 0.09781 0.05571 0.27586 0.24207 0.62857
LOF 11 0.00000 -0.04667 0.10147 0.05953 0.25000 0.21500 0.67429
SimplifiedLOF 2 0.14286 0.10286 0.08825 0.04571 0.16000 0.12080 0.64667
SimplifiedLOF 4 0.14286 0.10286 0.13033 0.08974 0.22222 0.18593 0.64095
SimplifiedLOF 9 0.00000 -0.04667 0.11226 0.07083 0.29630 0.26346 0.69619
SimplifiedLOF 10 0.00000 -0.04667 0.10929 0.06773 0.30769 0.27538 0.69619
LoOP 2 0.14286 0.10286 0.09685 0.05471 0.18750 0.14958 0.65238
LoOP 5 0.14286 0.10286 0.14396 0.10401 0.22222 0.18593 0.69524
LoOP 9 0.00000 -0.04667 0.11554 0.07427 0.29630 0.26346 0.71048
LDOF 3 0.14286 0.10286 0.09445 0.05219 0.20000 0.16267 0.64000
LDOF 4 0.14286 0.10286 0.12257 0.08162 0.21053 0.17368 0.65619
LDOF 5 0.00000 -0.04667 0.10316 0.06131 0.21429 0.17762 0.70095
LDOF 30 0.00000 -0.04667 0.08410 0.04135 0.23077 0.19487 0.62571
ODIN 2 0.10526 0.06351 0.09146 0.04906 0.16438 0.12539 0.76143
ODIN 9 0.00000 -0.04667 0.08885 0.04633 0.21053 0.17368 0.58762
ODIN 13 0.00000 -0.04667 0.09211 0.04974 0.21053 0.17368 0.58476
FastABOD 4 0.00000 -0.04667 0.10878 0.06719 0.24242 0.20707 0.70000
FastABOD 5 0.14286 0.10286 0.12265 0.08170 0.27586 0.24207 0.69333
FastABOD 11 0.00000 -0.04667 0.12343 0.08252 0.31579 0.28386 0.67810
KDEOS 2 0.14286 0.10286 0.12819 0.08750 0.22222 0.18593 0.67048
KDEOS 5 0.14286 0.10286 0.23162 0.19576 0.25000 0.21500 0.71905
KDEOS 6 0.00000 -0.04667 0.12845 0.08778 0.28571 0.25238 0.75143
LDF 1 0.14286 0.10286 0.12054 0.07949 0.22222 0.18593 0.39762
LDF 4 0.00000 -0.04667 0.12183 0.08085 0.25000 0.21500 0.73619
LDF 5 0.14286 0.10286 0.12548 0.08467 0.32000 0.28827 0.69714
INFLO 1 0.14286 0.10286 0.10269 0.06082 0.18605 0.14806 0.73476
INFLO 5 0.14286 0.10286 0.12837 0.08770 0.25641 0.22171 0.73333
INFLO 8 0.00000 -0.04667 0.11827 0.07713 0.28571 0.25238 0.70000
INFLO 9 0.00000 -0.04667 0.11542 0.07414 0.23077 0.19487 0.74190
COF 10 0.00000 -0.04667 0.10600 0.06428 0.22222 0.18593 0.72857
COF 14 0.14286 0.10286 0.10444 0.06264 0.21053 0.17368 0.69714
COF 16 0.14286 0.10286 0.10629 0.06459 0.23810 0.20254 0.69524
COF 61 0.00000 -0.04667 0.08861 0.04607 0.24242 0.20707 0.65238

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO