Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (5% of outliers version#05)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (41.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.33333 0.30055 0.39848 0.36890 0.44444 0.41712 0.80738
KNN 20 0.33333 0.30055 0.42800 0.39986 0.50000 0.47541 0.81489
KNN 84 0.33333 0.30055 0.43403 0.40620 0.50000 0.47541 0.83948
KNN 97 0.33333 0.30055 0.43311 0.40523 0.50000 0.47541 0.84153
KNNW 1 0.33333 0.30055 0.36515 0.33392 0.40000 0.37049 0.81352
KNNW 53 0.33333 0.30055 0.42922 0.40115 0.50000 0.47541 0.82377
KNNW 58 0.33333 0.30055 0.42949 0.40143 0.50000 0.47541 0.82445
KNNW 100 0.33333 0.30055 0.42940 0.40134 0.50000 0.47541 0.83197
LOF 9 0.33333 0.30055 0.36580 0.33461 0.38095 0.35051 0.86680
LOF 11 0.41667 0.38798 0.37832 0.34775 0.41667 0.38798 0.86510
LOF 96 0.33333 0.30055 0.42569 0.39745 0.50000 0.47541 0.82343
LOF 100 0.33333 0.30055 0.42609 0.39786 0.50000 0.47541 0.82377
SimplifiedLOF 3 0.33333 0.30055 0.35946 0.32796 0.40000 0.37049 0.87671
SimplifiedLOF 20 0.41667 0.38798 0.41652 0.38783 0.42105 0.39258 0.85724
SimplifiedLOF 72 0.33333 0.30055 0.42053 0.39203 0.47059 0.44455 0.84768
SimplifiedLOF 75 0.33333 0.30055 0.42192 0.39349 0.47059 0.44455 0.84939
LoOP 3 0.33333 0.30055 0.35576 0.32408 0.40000 0.37049 0.87056
LoOP 22 0.41667 0.38798 0.39534 0.36560 0.42105 0.39258 0.85280
LoOP 66 0.33333 0.30055 0.42343 0.39507 0.47059 0.44455 0.85178
LDOF 2 0.33333 0.30055 0.16868 0.12780 0.33333 0.30055 0.78928
LDOF 18 0.33333 0.30055 0.42967 0.40162 0.47059 0.44455 0.85827
LDOF 42 0.33333 0.30055 0.43750 0.40984 0.47059 0.44455 0.87295
LDOF 45 0.33333 0.30055 0.42937 0.40130 0.44444 0.41712 0.87602
ODIN 61 0.33333 0.30055 0.22175 0.18348 0.33333 0.30055 0.86066
ODIN 97 0.41667 0.38798 0.34156 0.30918 0.41667 0.38798 0.84973
ODIN 100 0.41667 0.38798 0.34728 0.31518 0.45455 0.42772 0.84682
FastABOD 4 0.33333 0.30055 0.39020 0.36021 0.47059 0.44455 0.80260
FastABOD 10 0.41667 0.38798 0.36720 0.33607 0.43478 0.40699 0.80908
FastABOD 75 0.41667 0.38798 0.39360 0.36377 0.43478 0.40699 0.82411
FastABOD 80 0.41667 0.38798 0.39986 0.37034 0.45455 0.42772 0.81796
KDEOS 5 0.25000 0.21311 0.16415 0.12304 0.25000 0.21311 0.72165
KDEOS 10 0.25000 0.21311 0.19188 0.15214 0.32000 0.28656 0.79064
KDEOS 11 0.25000 0.21311 0.22721 0.18920 0.29630 0.26169 0.78928
KDEOS 78 0.16667 0.12568 0.14713 0.10518 0.24444 0.20729 0.82582
LDF 62 0.50000 0.47541 0.36867 0.33762 0.50000 0.47541 0.88900
LDF 63 0.50000 0.47541 0.36371 0.33241 0.50000 0.47541 0.89447
LDF 72 0.41667 0.38798 0.43677 0.40907 0.50000 0.47541 0.87329
INFLO 2 0.33333 0.30055 0.37095 0.34002 0.35294 0.32112 0.88422
INFLO 71 0.33333 0.30055 0.44747 0.42030 0.50000 0.47541 0.86646
INFLO 78 0.33333 0.30055 0.44923 0.42214 0.50000 0.47541 0.86783
COF 2 0.33333 0.30055 0.37954 0.34902 0.38095 0.35051 0.85724
COF 3 0.33333 0.30055 0.39998 0.37047 0.42105 0.39258 0.86783
COF 5 0.33333 0.30055 0.35678 0.32514 0.44444 0.41712 0.81728

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (41.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.50000 0.47541 0.40313 0.37377 0.52174 0.49822 0.80396
KNN 4 0.41667 0.38798 0.40993 0.38091 0.50000 0.47541 0.79816
KNN 5 0.41667 0.38798 0.40310 0.37375 0.52632 0.50302 0.80669
KNN 92 0.41667 0.38798 0.39269 0.36282 0.52632 0.50302 0.83197
KNNW 1 0.50000 0.47541 0.37767 0.34707 0.50000 0.47541 0.76452
KNNW 12 0.50000 0.47541 0.41839 0.38978 0.50000 0.47541 0.79986
KNNW 21 0.41667 0.38798 0.39976 0.37024 0.52632 0.50302 0.80328
KNNW 63 0.41667 0.38798 0.39847 0.36888 0.52632 0.50302 0.82309
LOF 11 0.50000 0.47541 0.40531 0.37606 0.50000 0.47541 0.78996
LOF 24 0.50000 0.47541 0.42327 0.39491 0.50000 0.47541 0.81694
LOF 30 0.41667 0.38798 0.39540 0.36567 0.50000 0.47541 0.83026
LOF 32 0.41667 0.38798 0.39906 0.36950 0.52632 0.50302 0.82514
SimplifiedLOF 9 0.50000 0.47541 0.36556 0.33436 0.50000 0.47541 0.76195
SimplifiedLOF 12 0.50000 0.47541 0.41883 0.39025 0.57143 0.55035 0.75615
SimplifiedLOF 25 0.50000 0.47541 0.44515 0.41786 0.57143 0.55035 0.80294
SimplifiedLOF 98 0.41667 0.38798 0.39066 0.36069 0.50000 0.47541 0.81967
LoOP 11 0.50000 0.47541 0.37362 0.34282 0.50000 0.47541 0.74710
LoOP 20 0.50000 0.47541 0.44890 0.42179 0.60000 0.58033 0.79474
LoOP 81 0.50000 0.47541 0.40667 0.37749 0.52632 0.50302 0.82138
LDOF 18 0.50000 0.47541 0.40629 0.37709 0.50000 0.47541 0.77869
LDOF 27 0.50000 0.47541 0.45498 0.42818 0.60000 0.58033 0.80328
LDOF 34 0.50000 0.47541 0.45507 0.42827 0.60000 0.58033 0.80020
LDOF 100 0.50000 0.47541 0.41668 0.38799 0.57143 0.55035 0.81865
ODIN 42 0.43056 0.40255 0.29266 0.25787 0.43478 0.40699 0.81831
ODIN 89 0.41667 0.38798 0.30943 0.27547 0.45455 0.42772 0.83999
ODIN 95 0.41667 0.38798 0.31589 0.28224 0.47619 0.45043 0.83504
ODIN 97 0.41667 0.38798 0.31726 0.28368 0.47619 0.45043 0.83385
FastABOD 4 0.25000 0.21311 0.34729 0.31519 0.37500 0.34426 0.78245
FastABOD 7 0.33333 0.30055 0.33157 0.29870 0.35294 0.32112 0.76571
FastABOD 16 0.33333 0.30055 0.33503 0.30233 0.40000 0.37049 0.77186
FastABOD 84 0.33333 0.30055 0.30726 0.27319 0.37037 0.33940 0.80191
KDEOS 6 0.33333 0.30055 0.15750 0.11606 0.33333 0.30055 0.65505
KDEOS 13 0.33333 0.30055 0.30219 0.26787 0.40000 0.37049 0.69023
KDEOS 82 0.25000 0.21311 0.18070 0.14041 0.35714 0.32553 0.79918
LDF 40 0.25000 0.21311 0.24021 0.20284 0.30769 0.27364 0.79440
LDF 45 0.33333 0.30055 0.25354 0.21683 0.34783 0.31575 0.75751
LDF 48 0.33333 0.30055 0.28663 0.25155 0.42105 0.39258 0.75956
LDF 100 0.25000 0.21311 0.34273 0.31040 0.40000 0.37049 0.74590
INFLO 10 0.50000 0.47541 0.36666 0.33551 0.50000 0.47541 0.73531
INFLO 20 0.50000 0.47541 0.44669 0.41948 0.60000 0.58033 0.80977
INFLO 24 0.50000 0.47541 0.44938 0.42230 0.60000 0.58033 0.81421
INFLO 51 0.50000 0.47541 0.43202 0.40409 0.60000 0.58033 0.84939
COF 11 0.25000 0.21311 0.34409 0.31183 0.40000 0.37049 0.74829
COF 12 0.25000 0.21311 0.34767 0.31558 0.40000 0.37049 0.74385
COF 21 0.33333 0.30055 0.30100 0.26663 0.37500 0.34426 0.75342
COF 74 0.16667 0.12568 0.23461 0.19697 0.28571 0.25059 0.76537

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO