Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (20% of outliers version#07)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (51.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 22 0.43243 0.29243 0.42127 0.27852 0.54206 0.42910 0.78072
KNN 49 0.48649 0.35982 0.45579 0.32155 0.52941 0.41333 0.77477
KNN 72 0.43243 0.29243 0.45961 0.32631 0.53846 0.42462 0.78000
KNN 85 0.43243 0.29243 0.45252 0.31747 0.55556 0.44593 0.76739
KNNW 89 0.37838 0.22505 0.42936 0.28860 0.54545 0.43333 0.76432
KNNW 95 0.40541 0.25874 0.43580 0.29663 0.54545 0.43333 0.76468
KNNW 100 0.40541 0.25874 0.43779 0.29911 0.53465 0.41987 0.76577
LOF 80 0.45946 0.32613 0.43291 0.29303 0.54717 0.43547 0.76036
LOF 91 0.45946 0.32613 0.44852 0.31248 0.55238 0.44197 0.76288
LOF 98 0.45946 0.32613 0.45229 0.31719 0.55769 0.44859 0.75495
LOF 99 0.45946 0.32613 0.45432 0.31971 0.55769 0.44859 0.75495
SimplifiedLOF 80 0.32432 0.15766 0.28117 0.10386 0.46617 0.33449 0.67910
SimplifiedLOF 99 0.27027 0.09027 0.31836 0.15022 0.48485 0.35778 0.71333
SimplifiedLOF 100 0.27027 0.09027 0.32045 0.15283 0.48485 0.35778 0.71459
LoOP 67 0.32432 0.15766 0.27826 0.10023 0.44615 0.30954 0.66640
LoOP 100 0.29730 0.12396 0.33038 0.16521 0.48120 0.35323 0.71468
LDOF 4 0.27027 0.09027 0.23612 0.04770 0.38532 0.23370 0.58901
LDOF 93 0.27027 0.09027 0.27183 0.09221 0.44776 0.31154 0.66649
LDOF 100 0.27027 0.09027 0.28729 0.11149 0.44776 0.31154 0.67964
ODIN 75 0.37838 0.22505 0.35774 0.19932 0.50435 0.38209 0.71892
ODIN 98 0.36036 0.20258 0.39410 0.24464 0.52336 0.40579 0.73919
ODIN 100 0.35135 0.19135 0.39925 0.25106 0.51852 0.39975 0.73964
FastABOD 26 0.45946 0.32613 0.45584 0.32162 0.49231 0.36708 0.76811
FastABOD 81 0.45946 0.32613 0.49014 0.36437 0.54386 0.43135 0.79568
FastABOD 97 0.45946 0.32613 0.50353 0.38107 0.54386 0.43135 0.79982
KDEOS 5 0.27027 0.09027 0.27583 0.09720 0.34197 0.17965 0.54162
KDEOS 19 0.29730 0.12396 0.21626 0.02293 0.34742 0.18645 0.51712
KDEOS 80 0.18919 -0.01081 0.22964 0.03962 0.44156 0.30381 0.59622
KDEOS 100 0.24324 0.05658 0.25101 0.06626 0.43871 0.30026 0.63189
LDF 62 0.56757 0.46090 0.56617 0.45915 0.56863 0.46222 0.80541
LDF 74 0.51351 0.39351 0.56883 0.46248 0.56566 0.45852 0.81207
LDF 79 0.54054 0.42721 0.58135 0.47808 0.57944 0.47570 0.80811
LDF 88 0.54054 0.42721 0.57495 0.47010 0.60417 0.50653 0.79766
INFLO 73 0.35135 0.19135 0.34280 0.18069 0.56897 0.46264 0.74739
INFLO 91 0.37838 0.22505 0.38805 0.23710 0.60417 0.50653 0.71225
INFLO 97 0.35135 0.19135 0.40489 0.25810 0.63830 0.54908 0.73703
INFLO 100 0.35135 0.19135 0.41424 0.26975 0.63830 0.54908 0.73937
COF 86 0.48649 0.35982 0.58655 0.48457 0.60000 0.50133 0.81153
COF 87 0.45946 0.32613 0.57756 0.47335 0.60784 0.51111 0.81171
COF 97 0.54054 0.42721 0.57390 0.46879 0.58586 0.48370 0.81586
COF 100 0.51351 0.39351 0.58332 0.48054 0.58000 0.47640 0.81820

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (49.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 9 0.37838 0.22505 0.28449 0.10800 0.42308 0.28077 0.64865
KNN 17 0.29730 0.12396 0.28201 0.10491 0.43200 0.29189 0.65000
KNNW 10 0.32432 0.15766 0.27725 0.09897 0.40426 0.25730 0.63550
KNNW 11 0.35135 0.19135 0.27721 0.09892 0.40426 0.25730 0.63586
KNNW 31 0.29730 0.12396 0.27501 0.09618 0.41860 0.27519 0.64072
KNNW 33 0.29730 0.12396 0.27431 0.09531 0.42520 0.28341 0.63982
LOF 6 0.29730 0.12396 0.27659 0.09815 0.39506 0.24584 0.63532
LOF 7 0.32432 0.15766 0.27140 0.09167 0.40449 0.25760 0.61910
LOF 13 0.29730 0.12396 0.27250 0.09305 0.40580 0.25923 0.64144
LOF 17 0.29730 0.12396 0.26576 0.08465 0.42222 0.27970 0.62919
SimplifiedLOF 19 0.37838 0.22505 0.25747 0.07431 0.38462 0.23282 0.60775
SimplifiedLOF 26 0.35135 0.19135 0.26686 0.08601 0.39080 0.24054 0.62559
SimplifiedLOF 45 0.27027 0.09027 0.26114 0.07889 0.41584 0.27175 0.62559
SimplifiedLOF 55 0.29730 0.12396 0.26245 0.08052 0.40260 0.25524 0.62775
LoOP 19 0.35135 0.19135 0.25322 0.06901 0.36585 0.20943 0.59991
LoOP 27 0.35135 0.19135 0.26243 0.08049 0.39080 0.24054 0.61207
LoOP 46 0.27027 0.09027 0.25186 0.06732 0.40777 0.26168 0.61297
LoOP 51 0.27027 0.09027 0.25360 0.06948 0.40000 0.25200 0.62036
LDOF 27 0.29730 0.12396 0.25553 0.07189 0.39437 0.24498 0.61730
LDOF 48 0.24324 0.05658 0.25234 0.06792 0.41584 0.27175 0.61459
LDOF 55 0.24324 0.05658 0.25829 0.07534 0.38532 0.23370 0.62126
LDOF 58 0.24324 0.05658 0.25945 0.07678 0.38168 0.22916 0.61730
ODIN 14 0.32924 0.16378 0.27526 0.09650 0.39535 0.24620 0.62072
ODIN 15 0.31532 0.14643 0.27624 0.09771 0.39130 0.24116 0.61595
ODIN 47 0.24324 0.05658 0.25265 0.06831 0.39726 0.24858 0.60712
FastABOD 4 0.32432 0.15766 0.27078 0.09090 0.41935 0.27613 0.61459
FastABOD 17 0.32432 0.15766 0.28441 0.10790 0.42017 0.27714 0.62775
FastABOD 65 0.27027 0.09027 0.27895 0.10110 0.46429 0.33214 0.63676
FastABOD 69 0.29730 0.12396 0.27983 0.10219 0.46018 0.32702 0.63784
KDEOS 24 0.24324 0.05658 0.29806 0.12491 0.36957 0.21406 0.55514
KDEOS 74 0.32432 0.15766 0.27485 0.09598 0.39241 0.24253 0.61441
KDEOS 95 0.24324 0.05658 0.26416 0.08265 0.41667 0.27278 0.61856
KDEOS 100 0.27027 0.09027 0.26690 0.08607 0.41176 0.26667 0.62198
LDF 2 0.29730 0.12396 0.28697 0.11109 0.37500 0.22083 0.58865
LDF 6 0.27027 0.09027 0.28041 0.10291 0.43137 0.29111 0.64631
LDF 14 0.27027 0.09027 0.28157 0.10436 0.41935 0.27613 0.64847
LDF 32 0.35135 0.19135 0.27266 0.09325 0.40000 0.25200 0.62378
INFLO 19 0.35135 0.19135 0.26879 0.08842 0.46400 0.33179 0.63964
INFLO 35 0.24324 0.05658 0.28620 0.11012 0.53333 0.41822 0.69081
INFLO 49 0.32432 0.15766 0.29313 0.11877 0.53097 0.41528 0.68613
COF 14 0.32432 0.15766 0.25833 0.07538 0.38217 0.22977 0.60360
COF 55 0.24324 0.05658 0.29684 0.12340 0.45614 0.32199 0.65495
COF 67 0.29730 0.12396 0.31884 0.15082 0.43548 0.29624 0.66144
COF 68 0.27027 0.09027 0.31688 0.14838 0.43836 0.29982 0.67117

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO