Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (5% of outliers version#02)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (41.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.41667 0.38798 0.38871 0.35864 0.41667 0.38798 0.83145
KNN 18 0.41667 0.38798 0.41759 0.38895 0.50000 0.47541 0.81950
KNN 31 0.41667 0.38798 0.42514 0.39687 0.50000 0.47541 0.82565
KNNW 18 0.41667 0.38798 0.38826 0.35817 0.44444 0.41712 0.81933
KNNW 49 0.41667 0.38798 0.40732 0.37817 0.47619 0.45043 0.82309
KNNW 73 0.41667 0.38798 0.41541 0.38666 0.50000 0.47541 0.82275
LOF 51 0.41667 0.38798 0.38052 0.35005 0.44444 0.41712 0.81831
LOF 57 0.41667 0.38798 0.38143 0.35100 0.48000 0.45443 0.82036
LOF 66 0.41667 0.38798 0.38309 0.35274 0.46154 0.43506 0.82275
LOF 92 0.41667 0.38798 0.40147 0.37203 0.47619 0.45043 0.81318
SimplifiedLOF 65 0.41667 0.38798 0.38336 0.35303 0.46154 0.43506 0.82036
SimplifiedLOF 73 0.33333 0.30055 0.38152 0.35110 0.44444 0.41712 0.82582
SimplifiedLOF 79 0.41667 0.38798 0.39728 0.36764 0.48000 0.45443 0.82343
LoOP 76 0.41667 0.38798 0.37687 0.34622 0.42857 0.40047 0.82206
LoOP 79 0.41667 0.38798 0.37842 0.34785 0.46154 0.43506 0.82172
LoOP 88 0.41667 0.38798 0.38878 0.35872 0.46154 0.43506 0.82411
LoOP 99 0.41667 0.38798 0.38237 0.35200 0.43478 0.40699 0.82514
LDOF 7 0.41667 0.38798 0.26590 0.22979 0.43478 0.40699 0.74727
LDOF 81 0.33333 0.30055 0.37365 0.34284 0.40000 0.37049 0.82480
LDOF 84 0.33333 0.30055 0.36743 0.33633 0.40000 0.37049 0.82548
ODIN 80 0.31250 0.27869 0.22531 0.18722 0.35294 0.32112 0.82975
ODIN 81 0.31250 0.27869 0.22694 0.18892 0.36364 0.33234 0.82889
ODIN 82 0.33333 0.30055 0.23142 0.19363 0.36364 0.33234 0.82855
ODIN 99 0.33333 0.30055 0.23399 0.19632 0.34783 0.31575 0.82172
FastABOD 14 0.33333 0.30055 0.38207 0.35168 0.40000 0.37049 0.82548
FastABOD 48 0.41667 0.38798 0.36531 0.33410 0.41667 0.38798 0.80669
FastABOD 54 0.41667 0.38798 0.39814 0.36855 0.45455 0.42772 0.80840
FastABOD 93 0.33333 0.30055 0.41984 0.39130 0.44444 0.41712 0.80977
KDEOS 8 0.25000 0.21311 0.26346 0.22724 0.32258 0.28926 0.75000
KDEOS 12 0.33333 0.30055 0.19532 0.15575 0.36364 0.33234 0.77186
LDF 7 0.41667 0.38798 0.27948 0.24405 0.41667 0.38798 0.84819
LDF 38 0.41667 0.38798 0.34576 0.31359 0.52632 0.50302 0.86031
LDF 45 0.33333 0.30055 0.30436 0.27015 0.40000 0.37049 0.87158
LDF 84 0.33333 0.30055 0.39518 0.36544 0.47059 0.44455 0.75956
INFLO 51 0.41667 0.38798 0.37633 0.34566 0.41667 0.38798 0.83026
INFLO 66 0.41667 0.38798 0.38176 0.35136 0.48000 0.45443 0.83402
INFLO 79 0.41667 0.38798 0.39444 0.36466 0.42857 0.40047 0.83675
INFLO 87 0.41667 0.38798 0.38569 0.35548 0.44444 0.41712 0.84187
COF 11 0.41667 0.38798 0.31659 0.28298 0.43478 0.40699 0.84426
COF 56 0.41667 0.38798 0.40516 0.37591 0.45455 0.42772 0.75307
COF 75 0.41667 0.38798 0.35201 0.32014 0.50000 0.47541 0.75239

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (42.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.25000 0.21311 0.23372 0.19603 0.30000 0.26557 0.75956
KNN 84 0.08333 0.03825 0.16086 0.11959 0.24390 0.20672 0.77766
KNNW 1 0.25000 0.21311 0.21929 0.18089 0.27027 0.23438 0.74761
KNNW 2 0.25000 0.21311 0.22690 0.18888 0.28571 0.25059 0.75444
KNNW 3 0.25000 0.21311 0.22830 0.19035 0.28571 0.25059 0.76332
KNNW 5 0.25000 0.21311 0.22550 0.18741 0.28571 0.25059 0.77049
LOF 1 0.08333 0.03825 0.14314 0.10100 0.26087 0.22452 0.77561
LOF 6 0.25000 0.21311 0.21878 0.18036 0.26087 0.22452 0.76161
LOF 8 0.16667 0.12568 0.23003 0.19216 0.25806 0.22158 0.76264
LOF 21 0.25000 0.21311 0.22240 0.18416 0.35294 0.32112 0.75990
SimplifiedLOF 6 0.25000 0.21311 0.22171 0.18344 0.27273 0.23696 0.74351
SimplifiedLOF 21 0.25000 0.21311 0.22081 0.18249 0.35294 0.32112 0.76503
SimplifiedLOF 37 0.25000 0.21311 0.22285 0.18463 0.35294 0.32112 0.76673
SimplifiedLOF 87 0.25000 0.21311 0.17137 0.13062 0.27273 0.23696 0.77425
LoOP 5 0.25000 0.21311 0.22529 0.18719 0.33333 0.30055 0.74641
LoOP 6 0.25000 0.21311 0.22807 0.19010 0.29412 0.25940 0.74949
LoOP 19 0.25000 0.21311 0.21865 0.18023 0.35294 0.32112 0.75546
LoOP 100 0.25000 0.21311 0.17397 0.13335 0.26667 0.23060 0.77596
LDOF 7 0.25000 0.21311 0.17148 0.13073 0.25000 0.21311 0.73770
LDOF 27 0.25000 0.21311 0.22562 0.18754 0.35294 0.32112 0.77869
LDOF 40 0.25000 0.21311 0.23273 0.19500 0.35294 0.32112 0.79201
LDOF 43 0.25000 0.21311 0.22997 0.19210 0.35294 0.32112 0.79235
ODIN 6 0.20000 0.16066 0.15147 0.10974 0.28571 0.25059 0.79269
ODIN 15 0.33333 0.30055 0.18184 0.14160 0.33333 0.30055 0.75820
ODIN 25 0.33333 0.30055 0.26964 0.23372 0.42105 0.39258 0.78876
ODIN 26 0.33333 0.30055 0.27378 0.23807 0.42105 0.39258 0.78876
FastABOD 6 0.33333 0.30055 0.24377 0.20658 0.33333 0.30055 0.75102
FastABOD 8 0.25000 0.21311 0.26973 0.23382 0.33333 0.30055 0.77459
FastABOD 13 0.25000 0.21311 0.26055 0.22418 0.33333 0.30055 0.78210
KDEOS 7 0.25000 0.21311 0.12798 0.08509 0.25000 0.21311 0.69023
KDEOS 10 0.25000 0.21311 0.31400 0.28026 0.37500 0.34426 0.73327
KDEOS 100 0.08333 0.03825 0.11008 0.06632 0.23077 0.19294 0.76195
LDF 1 0.16667 0.12568 0.10896 0.06514 0.21429 0.17564 0.62278
LDF 3 0.16667 0.12568 0.14946 0.10763 0.30769 0.27364 0.73583
LDF 4 0.16667 0.12568 0.16770 0.12676 0.27586 0.24025 0.77971
LDF 14 0.16667 0.12568 0.21469 0.17607 0.25000 0.21311 0.72336
INFLO 5 0.25000 0.21311 0.21762 0.17914 0.32432 0.29109 0.74317
INFLO 7 0.25000 0.21311 0.23569 0.19810 0.31579 0.28214 0.73566
INFLO 19 0.25000 0.21311 0.22096 0.18265 0.35294 0.32112 0.76537
INFLO 95 0.16667 0.12568 0.17706 0.13659 0.26087 0.22452 0.80362
COF 8 0.33333 0.30055 0.27660 0.24103 0.38710 0.35695 0.74556
COF 13 0.16667 0.12568 0.23516 0.19754 0.36364 0.33234 0.76913

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO