Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (20% of outliers version#09)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (51.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 61 0.45946 0.32613 0.48180 0.35398 0.55670 0.44735 0.77658
KNN 78 0.48649 0.35982 0.52978 0.41380 0.54000 0.42653 0.79622
KNN 97 0.51351 0.39351 0.54465 0.43233 0.51948 0.40095 0.78036
KNNW 80 0.40541 0.25874 0.44627 0.30968 0.49206 0.36677 0.75225
KNNW 98 0.40541 0.25874 0.45499 0.32056 0.50909 0.38800 0.75874
KNNW 99 0.40541 0.25874 0.45506 0.32064 0.51376 0.39382 0.75874
LOF 85 0.51351 0.39351 0.50705 0.38546 0.55238 0.44197 0.78072
LOF 86 0.51351 0.39351 0.51017 0.38934 0.55769 0.44859 0.77964
LOF 99 0.51351 0.39351 0.53848 0.42464 0.55769 0.44859 0.79640
LOF 100 0.51351 0.39351 0.53894 0.42521 0.55769 0.44859 0.79171
SimplifiedLOF 89 0.29730 0.12396 0.29747 0.12418 0.47328 0.34336 0.67027
SimplifiedLOF 98 0.27027 0.09027 0.32644 0.16029 0.47619 0.34698 0.68486
SimplifiedLOF 99 0.27027 0.09027 0.32766 0.16182 0.47619 0.34698 0.68631
SimplifiedLOF 100 0.27027 0.09027 0.32639 0.16024 0.47619 0.34698 0.68865
LoOP 90 0.29730 0.12396 0.30841 0.13782 0.46154 0.32872 0.67270
LoOP 95 0.27027 0.09027 0.32616 0.15995 0.48000 0.35173 0.67495
LoOP 99 0.27027 0.09027 0.33786 0.17453 0.46512 0.33318 0.68270
LoOP 100 0.27027 0.09027 0.33701 0.17347 0.46512 0.33318 0.68378
LDOF 4 0.27027 0.09027 0.22968 0.03966 0.35754 0.19907 0.56198
LDOF 91 0.18919 -0.01081 0.27448 0.09551 0.45588 0.32167 0.63081
LDOF 99 0.18919 -0.01081 0.28264 0.10570 0.44604 0.30940 0.64036
LDOF 100 0.18919 -0.01081 0.28155 0.10433 0.44604 0.30940 0.64162
ODIN 98 0.43243 0.29243 0.43748 0.29872 0.51765 0.39867 0.75937
ODIN 100 0.47104 0.34057 0.44927 0.31343 0.51064 0.38993 0.76153
FastABOD 67 0.40541 0.25874 0.45121 0.31585 0.52252 0.40474 0.76360
FastABOD 93 0.40541 0.25874 0.46124 0.32835 0.50769 0.38626 0.76847
FastABOD 95 0.43243 0.29243 0.45723 0.32335 0.50769 0.38626 0.76901
FastABOD 99 0.43243 0.29243 0.45877 0.32527 0.50794 0.38656 0.77063
KDEOS 2 0.27341 0.09419 0.24678 0.06098 0.35052 0.19031 0.56802
KDEOS 5 0.32432 0.15766 0.21657 0.02332 0.33673 0.17313 0.50775
KDEOS 91 0.18919 -0.01081 0.22255 0.03078 0.43137 0.29111 0.58991
KDEOS 100 0.21622 0.02288 0.22990 0.03995 0.43137 0.29111 0.60162
LDF 68 0.64865 0.56198 0.63264 0.54203 0.65753 0.57306 0.81820
LDF 76 0.59459 0.49459 0.64809 0.56129 0.61111 0.51519 0.82288
LDF 99 0.54054 0.42721 0.63627 0.54655 0.57576 0.47111 0.82865
INFLO 75 0.29730 0.12396 0.38446 0.23263 0.58929 0.48798 0.77459
INFLO 98 0.40541 0.25874 0.42663 0.28520 0.60870 0.51217 0.72261
INFLO 100 0.37838 0.22505 0.42688 0.28551 0.61538 0.52051 0.73423
COF 61 0.54054 0.42721 0.46659 0.33502 0.54795 0.43644 0.73009
COF 89 0.43243 0.29243 0.62156 0.52821 0.61856 0.52447 0.81874
COF 96 0.45946 0.32613 0.61407 0.51887 0.64444 0.55674 0.82505

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (49.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.37838 0.22505 0.33656 0.17292 0.45570 0.32143 0.69000
KNN 7 0.41892 0.27559 0.32146 0.15409 0.44156 0.30381 0.67712
KNN 15 0.37838 0.22505 0.32980 0.16448 0.46154 0.32872 0.70730
KNN 18 0.35135 0.19135 0.32723 0.16128 0.46377 0.33150 0.70297
KNNW 11 0.43243 0.29243 0.32940 0.16399 0.44444 0.30741 0.68667
KNNW 20 0.37838 0.22505 0.33346 0.16905 0.43609 0.29699 0.69712
KNNW 26 0.37838 0.22505 0.32856 0.16293 0.44615 0.30954 0.69910
KNNW 59 0.37838 0.22505 0.31948 0.15162 0.46617 0.33449 0.69099
LOF 7 0.35135 0.19135 0.27358 0.09439 0.39490 0.24565 0.61477
LOF 26 0.32432 0.15766 0.30126 0.12891 0.44068 0.30271 0.68396
LOF 28 0.29730 0.12396 0.30389 0.13219 0.43243 0.29243 0.68396
LOF 46 0.32432 0.15766 0.30150 0.12920 0.45802 0.32433 0.67423
SimplifiedLOF 25 0.35135 0.19135 0.28606 0.10996 0.44186 0.30419 0.62324
SimplifiedLOF 30 0.40541 0.25874 0.28714 0.11131 0.41096 0.26566 0.62757
SimplifiedLOF 89 0.35135 0.19135 0.29526 0.12142 0.43796 0.29932 0.66432
SimplifiedLOF 99 0.35135 0.19135 0.29725 0.12390 0.43609 0.29699 0.66252
LoOP 26 0.37838 0.22505 0.27691 0.09855 0.43590 0.29675 0.59505
LoOP 28 0.40541 0.25874 0.27736 0.09911 0.42105 0.27825 0.59622
LoOP 100 0.35135 0.19135 0.28282 0.10591 0.42336 0.28112 0.64928
LDOF 26 0.35135 0.19135 0.25747 0.07432 0.38776 0.23673 0.59279
LDOF 97 0.32432 0.15766 0.28274 0.10581 0.43284 0.29294 0.64793
LDOF 99 0.32432 0.15766 0.28342 0.10666 0.41958 0.27641 0.64595
ODIN 15 0.36293 0.20579 0.28930 0.11400 0.38554 0.23398 0.62631
ODIN 22 0.36937 0.21381 0.27820 0.10016 0.37736 0.22377 0.62018
ODIN 63 0.27027 0.09027 0.27793 0.09982 0.43284 0.29294 0.64216
ODIN 90 0.32432 0.15766 0.28678 0.11085 0.42188 0.27927 0.64495
FastABOD 4 0.35135 0.19135 0.34781 0.18694 0.50909 0.38800 0.72396
FastABOD 68 0.40541 0.25874 0.33517 0.17118 0.45455 0.32000 0.69351
KDEOS 83 0.35135 0.19135 0.26763 0.08697 0.38655 0.23524 0.63532
KDEOS 99 0.32432 0.15766 0.27898 0.10113 0.41441 0.26997 0.64955
KDEOS 100 0.32432 0.15766 0.27829 0.10027 0.41818 0.27467 0.65045
LDF 12 0.37838 0.22505 0.32083 0.15331 0.44944 0.31363 0.66811
LDF 23 0.32432 0.15766 0.33398 0.16969 0.46154 0.32872 0.71351
LDF 32 0.35135 0.19135 0.32668 0.16060 0.48000 0.35173 0.70468
INFLO 16 0.32432 0.15766 0.26919 0.08893 0.44961 0.31385 0.62559
INFLO 100 0.32432 0.15766 0.32140 0.15402 0.57692 0.47256 0.73883
COF 26 0.43243 0.29243 0.34675 0.18561 0.45455 0.32000 0.68432
COF 63 0.37838 0.22505 0.36607 0.20970 0.48214 0.35440 0.73640
COF 84 0.43243 0.29243 0.36426 0.20744 0.50847 0.38723 0.73045
COF 86 0.43243 0.29243 0.36881 0.21312 0.50420 0.38190 0.72991

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO