Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (20% of outliers version#10)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (51.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 54 0.54054 0.42721 0.55137 0.44070 0.56250 0.45458 0.83027
KNN 79 0.54054 0.42721 0.60614 0.50898 0.58586 0.48370 0.84613
KNN 84 0.54054 0.42721 0.60617 0.50902 0.58140 0.47814 0.84324
KNN 95 0.54054 0.42721 0.58883 0.48741 0.60317 0.50529 0.83495
KNNW 91 0.40541 0.25874 0.47552 0.34615 0.54717 0.43547 0.81009
KNNW 96 0.43243 0.29243 0.48101 0.35300 0.54206 0.42910 0.81189
KNNW 99 0.43243 0.29243 0.48366 0.35630 0.54206 0.42910 0.81297
LOF 81 0.54054 0.42721 0.56662 0.45972 0.58716 0.48532 0.83856
LOF 87 0.54054 0.42721 0.58675 0.48481 0.60550 0.50820 0.84595
LOF 99 0.54054 0.42721 0.60278 0.50480 0.59048 0.48946 0.84919
SimplifiedLOF 74 0.29730 0.12396 0.32115 0.15370 0.46715 0.33572 0.68180
SimplifiedLOF 95 0.27027 0.09027 0.36891 0.21324 0.50000 0.37667 0.73027
SimplifiedLOF 100 0.27027 0.09027 0.37808 0.22467 0.50000 0.37667 0.73892
LoOP 3 0.32432 0.15766 0.27150 0.09181 0.33333 0.16889 0.52333
LoOP 99 0.27027 0.09027 0.39559 0.24650 0.50394 0.38157 0.73721
LoOP 100 0.27027 0.09027 0.39715 0.24845 0.50394 0.38157 0.73811
LDOF 5 0.35135 0.19135 0.30162 0.12936 0.38621 0.23480 0.59459
LDOF 98 0.27027 0.09027 0.34151 0.17909 0.46358 0.33126 0.69315
LDOF 100 0.27027 0.09027 0.34415 0.18238 0.46154 0.32872 0.69586
ODIN 94 0.45946 0.32613 0.48229 0.35459 0.56863 0.46222 0.81279
ODIN 100 0.48108 0.35308 0.51958 0.40107 0.56000 0.45147 0.82207
FastABOD 58 0.37838 0.22505 0.43270 0.29276 0.54098 0.42776 0.79351
FastABOD 72 0.37838 0.22505 0.44189 0.30422 0.56911 0.46282 0.80036
FastABOD 93 0.37838 0.22505 0.45376 0.31903 0.56452 0.45710 0.80739
FastABOD 100 0.37838 0.22505 0.45256 0.31752 0.56667 0.45978 0.80937
KDEOS 5 0.29730 0.12396 0.25475 0.07093 0.33803 0.17474 0.53063
KDEOS 24 0.24324 0.05658 0.29562 0.12187 0.33514 0.17114 0.48955
KDEOS 96 0.18919 -0.01081 0.23809 0.05015 0.45122 0.31585 0.62072
KDEOS 100 0.18919 -0.01081 0.24633 0.06042 0.45122 0.31585 0.62901
LDF 64 0.64865 0.56198 0.68873 0.61195 0.67442 0.59411 0.88559
LDF 72 0.67568 0.59568 0.69053 0.61419 0.67568 0.59568 0.87459
LDF 86 0.59459 0.49459 0.69931 0.62514 0.67742 0.59785 0.87586
LDF 91 0.59459 0.49459 0.70789 0.63584 0.65625 0.57146 0.87477
INFLO 87 0.45946 0.32613 0.48275 0.35516 0.64000 0.55120 0.78180
INFLO 98 0.45946 0.32613 0.52703 0.41037 0.68750 0.61042 0.81550
INFLO 100 0.45946 0.32613 0.52661 0.40984 0.69474 0.61944 0.81640
COF 77 0.59459 0.49459 0.53536 0.42075 0.61856 0.52447 0.84865
COF 86 0.56757 0.46090 0.55395 0.44392 0.64078 0.55217 0.85964
COF 96 0.59459 0.49459 0.62312 0.53015 0.61538 0.52051 0.86973
COF 99 0.54054 0.42721 0.63288 0.54232 0.60215 0.50401 0.86523

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (49.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.43243 0.29243 0.35887 0.20073 0.48077 0.35269 0.72649
KNN 5 0.45946 0.32613 0.35739 0.19888 0.49462 0.36996 0.72333
KNN 7 0.45946 0.32613 0.35605 0.19721 0.52500 0.40783 0.72288
KNNW 5 0.48649 0.35982 0.35501 0.19591 0.49315 0.36813 0.72739
KNNW 7 0.48649 0.35982 0.35962 0.20166 0.50704 0.38545 0.72901
LOF 18 0.35135 0.19135 0.31426 0.14511 0.48276 0.35517 0.69369
LOF 20 0.43243 0.29243 0.32104 0.15356 0.47619 0.34698 0.69892
LOF 23 0.40541 0.25874 0.32013 0.15243 0.46667 0.33511 0.70468
LOF 28 0.43243 0.29243 0.32623 0.16003 0.46154 0.32872 0.69892
SimplifiedLOF 34 0.43243 0.29243 0.30140 0.12908 0.43678 0.29785 0.67243
SimplifiedLOF 38 0.40541 0.25874 0.30310 0.13120 0.47059 0.34000 0.67171
SimplifiedLOF 74 0.37838 0.22505 0.30310 0.13120 0.43750 0.29875 0.67441
LoOP 33 0.40541 0.25874 0.29624 0.12265 0.43373 0.29406 0.66000
LoOP 34 0.43243 0.29243 0.29478 0.12083 0.43243 0.29243 0.66072
LoOP 35 0.40541 0.25874 0.29422 0.12013 0.44186 0.30419 0.65874
LoOP 82 0.40541 0.25874 0.29080 0.11587 0.42353 0.28133 0.66793
LDOF 66 0.35135 0.19135 0.27583 0.09720 0.42254 0.28009 0.64036
LDOF 79 0.37838 0.22505 0.28090 0.10353 0.41379 0.26920 0.64667
LDOF 82 0.35135 0.19135 0.28322 0.10642 0.41096 0.26566 0.65045
ODIN 30 0.35676 0.19809 0.30682 0.13583 0.40426 0.25730 0.63568
ODIN 38 0.39382 0.24430 0.29386 0.11968 0.40909 0.26333 0.64901
ODIN 42 0.37838 0.22505 0.29283 0.11839 0.40559 0.25897 0.65423
ODIN 91 0.37838 0.22505 0.28827 0.11271 0.43636 0.29733 0.63748
FastABOD 3 0.40541 0.25874 0.38212 0.22971 0.44944 0.31363 0.68703
FastABOD 36 0.37838 0.22505 0.33055 0.16541 0.46465 0.33259 0.68432
FastABOD 80 0.40541 0.25874 0.33437 0.17018 0.46000 0.32680 0.68883
KDEOS 3 0.29730 0.12396 0.33919 0.17619 0.37838 0.22505 0.60018
KDEOS 75 0.27027 0.09027 0.29542 0.12162 0.45872 0.32520 0.64973
KDEOS 90 0.32432 0.15766 0.30584 0.13462 0.44211 0.30449 0.66486
KDEOS 93 0.35135 0.19135 0.29364 0.11940 0.44681 0.31035 0.66270
LDF 10 0.43243 0.29243 0.35859 0.20038 0.48739 0.36095 0.73495
LDF 15 0.40541 0.25874 0.36539 0.20885 0.53012 0.41422 0.72577
LDF 20 0.43243 0.29243 0.37140 0.21635 0.51163 0.39116 0.73748
INFLO 32 0.40541 0.25874 0.28849 0.11299 0.46281 0.33030 0.62261
INFLO 46 0.37838 0.22505 0.32061 0.15303 0.54701 0.43527 0.73333
COF 47 0.48649 0.35982 0.39353 0.24394 0.50667 0.38498 0.73712
COF 60 0.48649 0.35982 0.40419 0.25723 0.50602 0.38418 0.75568
COF 67 0.48649 0.35982 0.42327 0.28101 0.50602 0.38418 0.74991
COF 73 0.45946 0.32613 0.38985 0.23934 0.52632 0.40947 0.74234

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO