Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (20% of outliers version#05)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (51.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 67 0.43243 0.29243 0.50542 0.38342 0.55556 0.44593 0.80000
KNN 82 0.43243 0.29243 0.50669 0.38500 0.53211 0.41670 0.79991
KNN 88 0.45946 0.32613 0.49975 0.37635 0.52252 0.40474 0.79045
KNNW 43 0.35135 0.19135 0.41240 0.26745 0.51908 0.40046 0.74505
KNNW 67 0.37838 0.22505 0.42958 0.28888 0.51515 0.39556 0.75802
KNNW 98 0.37838 0.22505 0.45185 0.31664 0.51515 0.39556 0.77225
LOF 84 0.48649 0.35982 0.51548 0.39597 0.58621 0.48414 0.81784
LOF 95 0.48649 0.35982 0.52376 0.40629 0.56637 0.45941 0.82234
LOF 96 0.45946 0.32613 0.52344 0.40588 0.56140 0.45322 0.82360
SimplifiedLOF 81 0.32432 0.15766 0.33242 0.16776 0.46715 0.33572 0.68505
SimplifiedLOF 93 0.29730 0.12396 0.36870 0.21298 0.48276 0.35517 0.70721
SimplifiedLOF 100 0.29730 0.12396 0.37530 0.22121 0.47945 0.35105 0.71532
LoOP 80 0.32432 0.15766 0.33592 0.17212 0.45926 0.32588 0.67910
LoOP 87 0.29730 0.12396 0.36360 0.20662 0.47619 0.34698 0.70703
LoOP 100 0.32432 0.15766 0.37580 0.22183 0.47059 0.34000 0.71117
LDOF 3 0.27027 0.09027 0.23943 0.05182 0.38168 0.22916 0.60108
LDOF 100 0.24324 0.05658 0.34870 0.18805 0.45333 0.31849 0.68126
ODIN 95 0.42162 0.27895 0.45375 0.31901 0.50909 0.38800 0.78189
ODIN 98 0.41441 0.26997 0.45934 0.32597 0.51562 0.39615 0.78559
ODIN 100 0.39189 0.24189 0.45799 0.32429 0.51852 0.39975 0.78658
FastABOD 74 0.45946 0.32613 0.47595 0.34668 0.54867 0.43735 0.79766
FastABOD 97 0.43243 0.29243 0.47989 0.35160 0.55462 0.44476 0.80144
FastABOD 99 0.43243 0.29243 0.48098 0.35296 0.55932 0.45062 0.80144
KDEOS 58 0.32432 0.15766 0.24577 0.05972 0.38636 0.23500 0.58036
KDEOS 100 0.27027 0.09027 0.28346 0.10671 0.46053 0.32746 0.65225
LDF 64 0.54054 0.42721 0.61296 0.51749 0.63529 0.54533 0.85405
LDF 68 0.59459 0.49459 0.61234 0.51672 0.62338 0.53048 0.84721
LDF 72 0.56757 0.46090 0.62128 0.52786 0.59770 0.49847 0.84072
INFLO 96 0.37838 0.22505 0.46687 0.33536 0.61702 0.52255 0.74577
INFLO 98 0.40541 0.25874 0.45569 0.32143 0.60215 0.50401 0.71135
COF 71 0.54054 0.42721 0.51705 0.39792 0.54737 0.43572 0.80234
COF 86 0.43243 0.29243 0.58437 0.48185 0.62626 0.53407 0.83099
COF 88 0.43243 0.29243 0.58599 0.48387 0.60784 0.51111 0.82739

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (48.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.27027 0.09027 0.30223 0.13011 0.43860 0.30012 0.65676
KNN 10 0.37838 0.22505 0.31727 0.14886 0.42000 0.27693 0.67315
KNN 12 0.35135 0.19135 0.31423 0.14508 0.41509 0.27082 0.67459
KNN 39 0.40541 0.25874 0.29169 0.11697 0.40541 0.25874 0.63784
KNNW 1 0.32432 0.15766 0.31969 0.15187 0.42735 0.28610 0.67477
KNNW 5 0.32432 0.15766 0.30321 0.13133 0.43902 0.30065 0.66036
KNNW 95 0.40541 0.25874 0.29211 0.11750 0.41558 0.27143 0.64450
LOF 16 0.29730 0.12396 0.27754 0.09933 0.44231 0.30474 0.65045
LOF 25 0.32432 0.15766 0.28147 0.10423 0.41830 0.27481 0.66162
LOF 60 0.37838 0.22505 0.27699 0.09865 0.40741 0.26123 0.60883
LOF 81 0.37838 0.22505 0.28312 0.10628 0.42105 0.27825 0.61333
SimplifiedLOF 9 0.35135 0.19135 0.28284 0.10595 0.37879 0.22556 0.59081
SimplifiedLOF 39 0.29730 0.12396 0.27546 0.09674 0.43137 0.29111 0.63423
SimplifiedLOF 77 0.32432 0.15766 0.28203 0.10493 0.40708 0.26083 0.64432
SimplifiedLOF 84 0.35135 0.19135 0.28457 0.10810 0.40909 0.26333 0.64432
LoOP 9 0.32432 0.15766 0.29237 0.11782 0.37037 0.21506 0.59090
LoOP 10 0.35135 0.19135 0.27487 0.09601 0.35556 0.19659 0.57153
LoOP 57 0.24324 0.05658 0.26547 0.08429 0.41818 0.27467 0.63477
LoOP 65 0.29730 0.12396 0.26229 0.08032 0.42105 0.27825 0.62288
LDOF 5 0.32432 0.15766 0.29272 0.11826 0.44275 0.30529 0.64468
LDOF 36 0.37838 0.22505 0.27473 0.09583 0.41379 0.26920 0.63099
ODIN 10 0.29730 0.12396 0.30415 0.13251 0.36364 0.20667 0.60099
ODIN 34 0.31081 0.14081 0.28026 0.10273 0.39344 0.24383 0.63432
ODIN 90 0.36937 0.21381 0.27514 0.09634 0.40000 0.25200 0.58892
ODIN 96 0.35135 0.19135 0.27149 0.09179 0.41176 0.26667 0.59333
FastABOD 3 0.35135 0.19135 0.34381 0.18195 0.41176 0.26667 0.66108
FastABOD 6 0.40541 0.25874 0.33981 0.17696 0.42177 0.27914 0.67568
FastABOD 31 0.35135 0.19135 0.31409 0.14490 0.43750 0.29875 0.67027
KDEOS 7 0.35135 0.19135 0.31616 0.14748 0.38462 0.23282 0.61333
KDEOS 10 0.35135 0.19135 0.31059 0.14053 0.40476 0.25794 0.60991
KDEOS 24 0.21622 0.02288 0.32083 0.15330 0.36522 0.20864 0.56649
KDEOS 100 0.32432 0.15766 0.27010 0.09006 0.39604 0.24706 0.63550
LDF 21 0.35135 0.19135 0.31061 0.14056 0.41333 0.26862 0.67207
LDF 23 0.35135 0.19135 0.31108 0.14115 0.40678 0.26045 0.66901
LDF 43 0.40541 0.25874 0.29740 0.12409 0.41558 0.27143 0.62901
LDF 91 0.37838 0.22505 0.28306 0.10622 0.44444 0.30741 0.61477
INFLO 45 0.27027 0.09027 0.29018 0.11509 0.52727 0.41067 0.66685
INFLO 61 0.27027 0.09027 0.30100 0.12858 0.52542 0.40836 0.71640
INFLO 99 0.40541 0.25874 0.28454 0.10807 0.49020 0.36444 0.67550
COF 24 0.29730 0.12396 0.33270 0.16810 0.45802 0.32433 0.70036
COF 31 0.35135 0.19135 0.32897 0.16345 0.47312 0.34315 0.68559
COF 33 0.37838 0.22505 0.33284 0.16827 0.46552 0.33368 0.67730
COF 71 0.40541 0.25874 0.31914 0.15119 0.44944 0.31363 0.67387

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO