Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (20% of outliers version#08)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (51.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 57 0.54054 0.42721 0.50806 0.38671 0.54054 0.42721 0.80414
KNN 64 0.48649 0.35982 0.50268 0.38001 0.54717 0.43547 0.79712
KNN 80 0.45946 0.32613 0.53469 0.41991 0.54054 0.42721 0.79153
KNNW 57 0.43243 0.29243 0.43399 0.29437 0.52083 0.40264 0.77351
KNNW 73 0.45946 0.32613 0.45142 0.31610 0.51546 0.39595 0.78162
KNNW 89 0.45946 0.32613 0.46554 0.33371 0.51685 0.39768 0.78667
KNNW 96 0.45946 0.32613 0.47240 0.34226 0.51685 0.39768 0.78667
LOF 77 0.48649 0.35982 0.46209 0.32940 0.54054 0.42721 0.78018
LOF 86 0.48649 0.35982 0.50401 0.38166 0.56604 0.45899 0.79279
LOF 88 0.48649 0.35982 0.50828 0.38699 0.56604 0.45899 0.79387
LOF 99 0.45946 0.32613 0.52479 0.40757 0.56311 0.45534 0.78541
SimplifiedLOF 87 0.29730 0.12396 0.29641 0.12286 0.47761 0.34876 0.70613
SimplifiedLOF 96 0.35135 0.19135 0.33504 0.17102 0.46809 0.33688 0.72180
SimplifiedLOF 100 0.35135 0.19135 0.34315 0.18113 0.47143 0.34105 0.72685
LoOP 92 0.35135 0.19135 0.32828 0.16259 0.46479 0.33277 0.70937
LoOP 100 0.35135 0.19135 0.36103 0.20341 0.47143 0.34105 0.72234
LDOF 94 0.29730 0.12396 0.28347 0.10672 0.44615 0.30954 0.67694
LDOF 99 0.29730 0.12396 0.30217 0.13004 0.45638 0.32228 0.68937
LDOF 100 0.29730 0.12396 0.30221 0.13009 0.45333 0.31849 0.69009
ODIN 79 0.40541 0.25874 0.37610 0.22221 0.51765 0.39867 0.73261
ODIN 94 0.47297 0.34297 0.42571 0.28406 0.50820 0.38689 0.75189
ODIN 99 0.47297 0.34297 0.44295 0.30555 0.51376 0.39382 0.75387
ODIN 100 0.45946 0.32613 0.45107 0.31566 0.50943 0.38843 0.75306
FastABOD 68 0.43243 0.29243 0.45539 0.32105 0.53097 0.41528 0.79369
FastABOD 99 0.48649 0.35982 0.47608 0.34685 0.53097 0.41528 0.80126
KDEOS 3 0.27027 0.09027 0.24485 0.05858 0.34637 0.18514 0.52757
KDEOS 4 0.24324 0.05658 0.25160 0.06699 0.34419 0.18242 0.51910
KDEOS 97 0.16216 -0.04450 0.24028 0.05288 0.45455 0.32000 0.62919
KDEOS 100 0.16216 -0.04450 0.24342 0.05680 0.45455 0.32000 0.63405
LDF 66 0.62162 0.52829 0.64044 0.55175 0.63014 0.53890 0.82216
LDF 69 0.62162 0.52829 0.64358 0.55566 0.64000 0.55120 0.82631
LDF 77 0.62162 0.52829 0.65047 0.56425 0.62857 0.53695 0.82450
LDF 84 0.51351 0.39351 0.63772 0.54835 0.57426 0.46924 0.82901
INFLO 73 0.37838 0.22505 0.35499 0.19589 0.57391 0.46881 0.74649
INFLO 85 0.43243 0.29243 0.40096 0.25319 0.55769 0.44859 0.69784
INFLO 100 0.43243 0.29243 0.41686 0.27302 0.59574 0.49603 0.69973
COF 68 0.54054 0.42721 0.48685 0.36027 0.62651 0.53438 0.79369
COF 69 0.59459 0.49459 0.48797 0.36167 0.60976 0.51350 0.80198
COF 99 0.48649 0.35982 0.58241 0.47941 0.57732 0.47306 0.81946
COF 100 0.51351 0.39351 0.59204 0.49142 0.56250 0.45458 0.81946

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (49.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.35135 0.19135 0.28972 0.11452 0.39416 0.24472 0.62559
KNN 3 0.31081 0.14081 0.29758 0.12431 0.43548 0.29624 0.67216
KNN 18 0.29730 0.12396 0.29611 0.12249 0.46400 0.33179 0.66577
KNNW 4 0.32432 0.15766 0.29161 0.11687 0.40580 0.25923 0.64991
KNNW 6 0.35135 0.19135 0.28751 0.11176 0.41333 0.26862 0.65135
KNNW 26 0.29730 0.12396 0.29088 0.11596 0.43411 0.29452 0.65928
KNNW 27 0.29730 0.12396 0.29029 0.11522 0.43750 0.29875 0.65856
LOF 30 0.29730 0.12396 0.26894 0.08861 0.43478 0.29536 0.63405
LOF 38 0.29730 0.12396 0.27420 0.09517 0.40764 0.26153 0.64270
LOF 69 0.35135 0.19135 0.27588 0.09726 0.39695 0.24819 0.62847
SimplifiedLOF 15 0.35135 0.19135 0.25549 0.07185 0.36364 0.20667 0.58378
SimplifiedLOF 49 0.29730 0.12396 0.25981 0.07723 0.40252 0.25514 0.61009
SimplifiedLOF 86 0.29730 0.12396 0.26813 0.08761 0.40000 0.25200 0.62432
SimplifiedLOF 91 0.29730 0.12396 0.26726 0.08652 0.40000 0.25200 0.62486
LoOP 5 0.27027 0.09027 0.25094 0.06618 0.40000 0.25200 0.58613
LoOP 16 0.35135 0.19135 0.26241 0.08047 0.38462 0.23282 0.58360
LoOP 90 0.29730 0.12396 0.25629 0.07284 0.38926 0.23861 0.61009
LDOF 39 0.35135 0.19135 0.24877 0.06347 0.36842 0.21263 0.58486
LDOF 92 0.29730 0.12396 0.25643 0.07302 0.40278 0.25546 0.60793
LDOF 94 0.29730 0.12396 0.25587 0.07231 0.40845 0.26254 0.60685
ODIN 10 0.33033 0.16515 0.27304 0.09373 0.35484 0.19570 0.58459
ODIN 13 0.32883 0.16327 0.27492 0.09607 0.35000 0.18967 0.58180
ODIN 43 0.24324 0.05658 0.24816 0.06271 0.40278 0.25546 0.60937
FastABOD 4 0.40541 0.25874 0.32179 0.15450 0.48837 0.36217 0.67622
FastABOD 5 0.35135 0.19135 0.32224 0.15506 0.45161 0.31634 0.67207
FastABOD 21 0.35135 0.19135 0.31281 0.14331 0.44944 0.31363 0.67748
KDEOS 28 0.29730 0.12396 0.33130 0.16635 0.36975 0.21429 0.58919
KDEOS 39 0.35135 0.19135 0.30579 0.13456 0.36649 0.21023 0.59982
KDEOS 83 0.32432 0.15766 0.26411 0.08260 0.40491 0.25812 0.61333
KDEOS 99 0.27027 0.09027 0.26539 0.08418 0.40000 0.25200 0.62541
LDF 1 0.35135 0.19135 0.27274 0.09335 0.38655 0.23524 0.61135
LDF 28 0.29730 0.12396 0.29823 0.12513 0.44660 0.31010 0.66703
INFLO 14 0.35135 0.19135 0.25999 0.07746 0.43333 0.29356 0.58901
INFLO 55 0.29730 0.12396 0.29327 0.11894 0.53448 0.41966 0.69045
COF 57 0.35135 0.19135 0.36592 0.20952 0.49541 0.37095 0.71766
COF 59 0.35135 0.19135 0.37490 0.22071 0.48780 0.36146 0.72667
COF 63 0.40541 0.25874 0.37853 0.22524 0.49485 0.37024 0.71694

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO