Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (20% of outliers version#01)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (51.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 69 0.37838 0.22505 0.45267 0.31766 0.51327 0.39322 0.76595
KNN 74 0.37838 0.22505 0.45136 0.31602 0.52174 0.40377 0.76108
KNN 83 0.43243 0.29243 0.47311 0.34315 0.50909 0.38800 0.75712
KNN 94 0.40541 0.25874 0.48312 0.35562 0.48421 0.35698 0.74973
KNNW 34 0.32432 0.15766 0.35631 0.19754 0.49231 0.36708 0.71027
KNNW 93 0.37838 0.22505 0.39482 0.24554 0.49231 0.36708 0.74000
KNNW 100 0.37838 0.22505 0.40679 0.26047 0.49231 0.36708 0.74216
LOF 86 0.43243 0.29243 0.44101 0.30312 0.52830 0.41195 0.73946
LOF 88 0.40541 0.25874 0.44760 0.31134 0.54206 0.42910 0.74126
LOF 94 0.40541 0.25874 0.45666 0.32264 0.53704 0.42284 0.74324
SimplifiedLOF 59 0.27027 0.09027 0.24928 0.06411 0.46259 0.33002 0.63694
SimplifiedLOF 100 0.24324 0.05658 0.30124 0.12888 0.48175 0.35392 0.68937
LoOP 32 0.27027 0.09027 0.21252 0.01828 0.35374 0.19433 0.54054
LoOP 96 0.27027 0.09027 0.29776 0.12454 0.47826 0.34957 0.68811
LoOP 99 0.27027 0.09027 0.31631 0.14767 0.47826 0.34957 0.69225
LoOP 100 0.27027 0.09027 0.31675 0.14821 0.47826 0.34957 0.69189
LDOF 3 0.29730 0.12396 0.25426 0.07031 0.40299 0.25572 0.62811
LDOF 96 0.21622 0.02288 0.27641 0.09792 0.46715 0.33572 0.66000
LDOF 98 0.21622 0.02288 0.27771 0.09955 0.46377 0.33150 0.66162
LDOF 100 0.21622 0.02288 0.27843 0.10044 0.46715 0.33572 0.66090
ODIN 99 0.40541 0.25874 0.38016 0.22726 0.49600 0.37168 0.71937
ODIN 100 0.40541 0.25874 0.38461 0.23281 0.49600 0.37168 0.72171
FastABOD 26 0.37838 0.22505 0.43299 0.29313 0.48889 0.36281 0.74523
FastABOD 89 0.35135 0.19135 0.45364 0.31887 0.52033 0.40201 0.76955
FastABOD 96 0.35135 0.19135 0.45796 0.32425 0.52033 0.40201 0.77099
FastABOD 100 0.35135 0.19135 0.45707 0.32315 0.52033 0.40201 0.77171
KDEOS 4 0.27027 0.09027 0.27358 0.09439 0.35165 0.19172 0.57568
KDEOS 9 0.29730 0.12396 0.25944 0.07676 0.35503 0.19594 0.55676
KDEOS 92 0.24324 0.05658 0.24319 0.05651 0.46053 0.32746 0.61640
KDEOS 100 0.24324 0.05658 0.24709 0.06137 0.45455 0.32000 0.62559
LDF 47 0.51351 0.39351 0.50627 0.38449 0.53333 0.41822 0.76739
LDF 63 0.48649 0.35982 0.56235 0.45440 0.56180 0.45371 0.78324
LDF 64 0.48649 0.35982 0.56731 0.46058 0.57500 0.47017 0.78288
LDF 66 0.48649 0.35982 0.56790 0.46131 0.55696 0.44768 0.78090
INFLO 75 0.29730 0.12396 0.37788 0.22442 0.61404 0.51883 0.76919
INFLO 78 0.37838 0.22505 0.37452 0.22023 0.59813 0.49900 0.74703
INFLO 99 0.35135 0.19135 0.41386 0.26928 0.60870 0.51217 0.72234
COF 68 0.51351 0.39351 0.47041 0.33978 0.52055 0.40228 0.76324
COF 84 0.40541 0.25874 0.50034 0.37710 0.54369 0.43113 0.77297
COF 100 0.48649 0.35982 0.53215 0.41675 0.53465 0.41987 0.79189

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (49.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 6 0.45946 0.32613 0.33493 0.17088 0.48000 0.35173 0.66973
KNN 8 0.45946 0.32613 0.35455 0.19534 0.48718 0.36068 0.68622
KNN 11 0.43243 0.29243 0.35821 0.19990 0.46512 0.33318 0.69126
KNNW 15 0.45946 0.32613 0.34311 0.18108 0.46154 0.32872 0.67784
KNNW 32 0.43243 0.29243 0.35296 0.19335 0.47059 0.34000 0.68432
KNNW 33 0.43243 0.29243 0.35227 0.19250 0.45977 0.32651 0.68450
KNNW 88 0.45946 0.32613 0.32569 0.15936 0.47500 0.34550 0.67297
LOF 25 0.37838 0.22505 0.32202 0.15478 0.43411 0.29452 0.68036
LOF 54 0.37838 0.22505 0.32886 0.16332 0.44444 0.30741 0.67207
LOF 62 0.45946 0.32613 0.32403 0.15730 0.46341 0.33106 0.67009
LOF 71 0.45946 0.32613 0.32037 0.15273 0.47368 0.34386 0.66559
SimplifiedLOF 74 0.37838 0.22505 0.32057 0.15298 0.43902 0.30065 0.66901
SimplifiedLOF 86 0.43243 0.29243 0.31357 0.14424 0.43243 0.29243 0.66649
SimplifiedLOF 87 0.43243 0.29243 0.31577 0.14699 0.43243 0.29243 0.66919
LoOP 70 0.35135 0.19135 0.30910 0.13868 0.41270 0.26783 0.66396
LoOP 84 0.40541 0.25874 0.30792 0.13721 0.42667 0.28524 0.67153
LoOP 94 0.43243 0.29243 0.30314 0.13125 0.43243 0.29243 0.66559
LDOF 44 0.35135 0.19135 0.28111 0.10378 0.40000 0.25200 0.62901
LDOF 93 0.35135 0.19135 0.29742 0.12412 0.43373 0.29406 0.65532
LDOF 97 0.35135 0.19135 0.30116 0.12878 0.42857 0.28762 0.65946
LDOF 99 0.32432 0.15766 0.30182 0.12960 0.42857 0.28762 0.65730
ODIN 44 0.35676 0.19809 0.29697 0.12356 0.41611 0.27208 0.66207
ODIN 92 0.38739 0.23628 0.31389 0.14465 0.44444 0.30741 0.65288
ODIN 100 0.40541 0.25874 0.30530 0.13395 0.42500 0.28317 0.64396
FastABOD 3 0.32432 0.15766 0.37093 0.21575 0.43357 0.29385 0.67568
FastABOD 39 0.43243 0.29243 0.33479 0.17070 0.44444 0.30741 0.66523
FastABOD 73 0.43243 0.29243 0.33822 0.17498 0.45833 0.32472 0.66811
KDEOS 16 0.27027 0.09027 0.32657 0.16045 0.36181 0.20439 0.53243
KDEOS 88 0.35135 0.19135 0.29029 0.11522 0.40336 0.25619 0.64090
KDEOS 93 0.32432 0.15766 0.28926 0.11394 0.41739 0.27368 0.64270
KDEOS 99 0.32432 0.15766 0.29829 0.12521 0.41441 0.26997 0.64703
LDF 22 0.40541 0.25874 0.37190 0.21696 0.46154 0.32872 0.71225
LDF 43 0.45946 0.32613 0.35196 0.19211 0.50000 0.37667 0.68216
LDF 45 0.48649 0.35982 0.34408 0.18228 0.48649 0.35982 0.67964
INFLO 38 0.40541 0.25874 0.30811 0.13745 0.50000 0.37667 0.65072
INFLO 55 0.35135 0.19135 0.33467 0.17055 0.54386 0.43135 0.71360
INFLO 95 0.37838 0.22505 0.32221 0.15502 0.53211 0.41670 0.72919
COF 26 0.45946 0.32613 0.35383 0.19444 0.47368 0.34386 0.65315
COF 41 0.37838 0.22505 0.34753 0.18659 0.52000 0.40160 0.67459
COF 52 0.35135 0.19135 0.37201 0.21710 0.50000 0.37667 0.70432
COF 79 0.35135 0.19135 0.40374 0.25666 0.48000 0.35173 0.68486

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO