Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (10% of outliers version#10)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (44.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 23 0.50000 0.44667 0.46180 0.40439 0.51613 0.46452 0.83000
KNN 59 0.43750 0.37750 0.47970 0.42420 0.53846 0.48923 0.82042
KNN 74 0.43750 0.37750 0.50328 0.45030 0.51852 0.46716 0.83625
KNNW 87 0.43750 0.37750 0.46090 0.40340 0.45714 0.39924 0.81667
KNNW 90 0.43750 0.37750 0.46525 0.40821 0.45714 0.39924 0.81750
KNNW 97 0.43750 0.37750 0.46859 0.41191 0.46667 0.40978 0.81708
LOF 74 0.43750 0.37750 0.44060 0.38093 0.46512 0.40806 0.82542
LOF 98 0.43750 0.37750 0.49353 0.43950 0.50000 0.44667 0.84708
LOF 100 0.43750 0.37750 0.49683 0.44316 0.51852 0.46716 0.84708
SimplifiedLOF 87 0.31250 0.23917 0.31707 0.24423 0.37288 0.30599 0.77417
SimplifiedLOF 99 0.31250 0.23917 0.38668 0.32126 0.41667 0.35444 0.79000
SimplifiedLOF 100 0.31250 0.23917 0.38819 0.32294 0.41667 0.35444 0.79083
LoOP 83 0.31250 0.23917 0.30313 0.22880 0.36620 0.29859 0.76667
LoOP 100 0.31250 0.23917 0.38830 0.32305 0.41667 0.35444 0.79125
LDOF 92 0.31250 0.23917 0.28080 0.20408 0.34211 0.27193 0.75125
LDOF 99 0.31250 0.23917 0.28485 0.20857 0.35294 0.28392 0.76042
LDOF 100 0.31250 0.23917 0.31817 0.24544 0.35294 0.28392 0.76042
ODIN 91 0.43750 0.37750 0.40356 0.33994 0.46154 0.40410 0.81354
ODIN 97 0.37500 0.30833 0.42809 0.36708 0.47619 0.42032 0.82750
ODIN 99 0.37500 0.30833 0.42009 0.35823 0.48780 0.43317 0.82604
ODIN 100 0.37500 0.30833 0.41651 0.35427 0.47619 0.42032 0.82812
FastABOD 33 0.43750 0.37750 0.45976 0.40214 0.45455 0.39636 0.80875
FastABOD 85 0.43750 0.37750 0.47009 0.41356 0.48276 0.42759 0.81125
FastABOD 95 0.43750 0.37750 0.47445 0.41839 0.48276 0.42759 0.81333
FastABOD 98 0.43750 0.37750 0.47344 0.41727 0.48276 0.42759 0.81458
KDEOS 7 0.18750 0.10083 0.13337 0.04093 0.21583 0.13218 0.57542
KDEOS 14 0.06250 -0.03750 0.16529 0.07625 0.21053 0.12632 0.53625
KDEOS 98 0.00000 -0.10667 0.13625 0.04412 0.27660 0.19943 0.65083
KDEOS 100 0.00000 -0.10667 0.13866 0.04679 0.27660 0.19943 0.65625
LDF 36 0.50000 0.44667 0.53972 0.49062 0.52381 0.47302 0.82833
LDF 41 0.50000 0.44667 0.56495 0.51854 0.53333 0.48356 0.83458
LDF 61 0.50000 0.44667 0.55977 0.51282 0.55172 0.50391 0.84917
LDF 98 0.43750 0.37750 0.54212 0.49328 0.52632 0.47579 0.85875
INFLO 87 0.37500 0.30833 0.38688 0.32148 0.42623 0.36503 0.77375
INFLO 90 0.31250 0.23917 0.38691 0.32151 0.41667 0.35444 0.81750
INFLO 91 0.31250 0.23917 0.41088 0.34804 0.41096 0.34813 0.81375
INFLO 98 0.37500 0.30833 0.40088 0.33697 0.44444 0.38519 0.78208
COF 53 0.43750 0.37750 0.48964 0.43520 0.48889 0.43437 0.81667
COF 55 0.43750 0.37750 0.49924 0.44582 0.50000 0.44667 0.80958
COF 87 0.43750 0.37750 0.48144 0.42613 0.45455 0.39636 0.82625

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 166 objects, 16 outliers (9.64%)

Download raw algorithm results (1.4 MB) Download raw algorithm evaluation table (43.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.31250 0.23917 0.17748 0.08974 0.31250 0.23917 0.66000
KNN 7 0.25000 0.17000 0.20256 0.11750 0.40000 0.33600 0.68792
KNN 59 0.18750 0.10083 0.19587 0.11009 0.40816 0.34503 0.69396
KNN 67 0.18750 0.10083 0.19253 0.10640 0.39216 0.32732 0.70042
KNNW 8 0.31250 0.23917 0.18406 0.09703 0.35294 0.28392 0.66250
KNNW 52 0.25000 0.17000 0.19401 0.10804 0.38889 0.32370 0.68708
KNNW 66 0.25000 0.17000 0.19536 0.10954 0.37838 0.31207 0.68917
KNNW 98 0.18750 0.10083 0.19290 0.10681 0.37037 0.30321 0.69042
LOF 18 0.18750 0.10083 0.14397 0.05266 0.25926 0.18025 0.62958
LOF 73 0.12500 0.03167 0.18714 0.10043 0.37500 0.30833 0.68042
LOF 95 0.18750 0.10083 0.19171 0.10549 0.39216 0.32732 0.67917
LOF 100 0.18750 0.10083 0.19127 0.10501 0.40000 0.33600 0.67542
SimplifiedLOF 26 0.18750 0.10083 0.14646 0.05542 0.29167 0.21611 0.63333
SimplifiedLOF 97 0.12500 0.03167 0.17222 0.08392 0.32653 0.25469 0.67833
SimplifiedLOF 100 0.12500 0.03167 0.17576 0.08784 0.32558 0.25364 0.68083
LoOP 3 0.18750 0.10083 0.12593 0.03269 0.23529 0.15373 0.54708
LoOP 96 0.12500 0.03167 0.16458 0.07547 0.32000 0.24747 0.66875
LoOP 100 0.12500 0.03167 0.16474 0.07565 0.30986 0.23624 0.67042
LDOF 6 0.18750 0.10083 0.13897 0.04713 0.25000 0.17000 0.59458
LDOF 42 0.18750 0.10083 0.15973 0.07011 0.32000 0.24747 0.65750
LDOF 83 0.12500 0.03167 0.16276 0.07346 0.29333 0.21796 0.67125
LDOF 90 0.12500 0.03167 0.16481 0.07572 0.30435 0.23014 0.67042
ODIN 14 0.25000 0.17000 0.14864 0.05782 0.25000 0.17000 0.62167
ODIN 96 0.12500 0.03167 0.18318 0.09606 0.35484 0.28602 0.66958
ODIN 99 0.12500 0.03167 0.18631 0.09952 0.38596 0.32047 0.66896
FastABOD 3 0.25000 0.17000 0.25363 0.17402 0.32353 0.25137 0.68042
FastABOD 4 0.25000 0.17000 0.20930 0.12496 0.39216 0.32732 0.69208
FastABOD 35 0.31250 0.23917 0.20770 0.12319 0.34286 0.27276 0.70083
FastABOD 87 0.37500 0.30833 0.21480 0.13105 0.37500 0.30833 0.69958
KDEOS 7 0.25000 0.17000 0.28845 0.21255 0.32000 0.24747 0.58000
KDEOS 8 0.25000 0.17000 0.28662 0.21052 0.34783 0.27826 0.59917
KDEOS 9 0.31250 0.23917 0.21554 0.13187 0.33333 0.26222 0.56625
KDEOS 100 0.12500 0.03167 0.13665 0.04455 0.26506 0.18667 0.62583
LDF 9 0.31250 0.23917 0.18869 0.10215 0.34146 0.27122 0.65292
LDF 62 0.25000 0.17000 0.20531 0.12055 0.38462 0.31897 0.69625
LDF 73 0.18750 0.10083 0.20675 0.12213 0.40816 0.34503 0.69542
LDF 95 0.18750 0.10083 0.20341 0.11844 0.40909 0.34606 0.69458
INFLO 4 0.18750 0.10083 0.14402 0.05272 0.27778 0.20074 0.57750
INFLO 50 0.12500 0.03167 0.16976 0.08120 0.35000 0.28067 0.70979
INFLO 98 0.12500 0.03167 0.17867 0.09106 0.39286 0.32810 0.70521
INFLO 100 0.12500 0.03167 0.17765 0.08993 0.40741 0.34420 0.70500
COF 15 0.31250 0.23917 0.17103 0.08261 0.33333 0.26222 0.64062
COF 35 0.06250 -0.03750 0.17788 0.09019 0.32911 0.25755 0.69375
COF 94 0.18750 0.10083 0.20559 0.12086 0.37736 0.31094 0.68250
COF 100 0.31250 0.23917 0.21696 0.13343 0.35294 0.28392 0.69167

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO