Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (46% of outliers)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 450 objects, 206 outliers (45.78%)

Download raw algorithm results (4.0 MB) Download raw algorithm evaluation table (60.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 21 0.66990 0.39121 0.75254 0.54361 0.67426 0.39925 0.74761
KNN 56 0.66019 0.37331 0.75680 0.55148 0.67890 0.40781 0.75158
KNN 60 0.66505 0.38226 0.75669 0.55128 0.67916 0.40828 0.75205
KNN 86 0.66019 0.37331 0.75642 0.55077 0.68998 0.42824 0.74998
KNNW 4 0.65049 0.35540 0.74437 0.52855 0.68776 0.42415 0.74457
KNNW 34 0.66505 0.38226 0.75101 0.54080 0.67882 0.40765 0.74837
KNNW 74 0.66019 0.37331 0.75446 0.54717 0.68046 0.41068 0.75008
KNNW 83 0.66019 0.37331 0.75471 0.54762 0.68046 0.41068 0.75008
LOF 4 0.63107 0.31959 0.69178 0.43157 0.68750 0.42367 0.71584
LOF 7 0.66019 0.37331 0.70107 0.44870 0.68220 0.41390 0.72428
LOF 94 0.66019 0.37331 0.74522 0.53013 0.67562 0.40175 0.74421
LOF 95 0.66019 0.37331 0.74543 0.53051 0.67556 0.40164 0.74415
SimplifiedLOF 8 0.64563 0.34645 0.69369 0.43509 0.68421 0.41760 0.71930
SimplifiedLOF 65 0.66505 0.38226 0.72728 0.49703 0.67283 0.39661 0.73812
SimplifiedLOF 98 0.65049 0.35540 0.73726 0.51545 0.67757 0.40535 0.74339
SimplifiedLOF 100 0.65049 0.35540 0.73739 0.51568 0.67841 0.40691 0.74320
LoOP 7 0.64078 0.33750 0.69080 0.42975 0.68504 0.41913 0.71847
LoOP 77 0.66019 0.37331 0.72628 0.49518 0.67151 0.39417 0.73836
LoOP 98 0.64563 0.34645 0.73251 0.50668 0.67562 0.40175 0.74111
LoOP 100 0.65049 0.35540 0.73272 0.50706 0.67692 0.40416 0.74097
LDOF 15 0.64563 0.34645 0.68409 0.41737 0.68041 0.41060 0.71986
LDOF 16 0.63107 0.31959 0.68390 0.41703 0.68737 0.42343 0.71871
LDOF 100 0.63592 0.32855 0.71941 0.48253 0.67666 0.40368 0.73452
ODIN 69 0.63471 0.32631 0.66024 0.37340 0.68113 0.41192 0.71863
ODIN 80 0.64563 0.34645 0.67232 0.39568 0.67811 0.40635 0.72276
ODIN 98 0.63731 0.33110 0.67994 0.40973 0.67552 0.40158 0.72669
FastABOD 7 0.66505 0.38226 0.71887 0.48151 0.68966 0.42764 0.73404
FastABOD 98 0.65049 0.35540 0.73063 0.50320 0.67511 0.40081 0.74176
KDEOS 16 0.58252 0.23007 0.53922 0.15021 0.67557 0.40167 0.65053
KDEOS 21 0.62136 0.30169 0.54973 0.16958 0.66916 0.38984 0.66101
LDF 30 0.64078 0.33750 0.66212 0.37687 0.67135 0.39389 0.71103
LDF 41 0.61650 0.29273 0.65186 0.35794 0.67925 0.40844 0.70536
LDF 67 0.63592 0.32855 0.71397 0.47249 0.67213 0.39532 0.72286
LDF 91 0.62136 0.30169 0.72490 0.49264 0.64754 0.34997 0.70822
INFLO 82 0.63592 0.32855 0.72676 0.49607 0.68683 0.42242 0.73038
INFLO 86 0.65049 0.35540 0.72736 0.49719 0.67982 0.40951 0.73038
INFLO 91 0.64563 0.34645 0.72788 0.49814 0.67849 0.40706 0.73154
COF 3 0.60680 0.27483 0.69403 0.43572 0.68451 0.41816 0.71160
COF 7 0.66019 0.37331 0.71332 0.47128 0.67261 0.39620 0.72340
COF 39 0.66019 0.37331 0.74202 0.52421 0.66528 0.38269 0.73385
COF 65 0.62621 0.31064 0.74664 0.53274 0.66667 0.38525 0.72959

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 450 objects, 206 outliers (45.78%)

Download raw algorithm results (4.0 MB) Download raw algorithm evaluation table (60.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.66990 0.39121 0.76047 0.55825 0.70070 0.44801 0.76212
KNN 6 0.67476 0.40017 0.76306 0.56303 0.69907 0.44501 0.76508
KNN 7 0.68447 0.41807 0.76273 0.56242 0.70023 0.44714 0.76660
KNN 8 0.68447 0.41807 0.76179 0.56068 0.69585 0.43907 0.76749
KNNW 13 0.67476 0.40017 0.76066 0.55859 0.70208 0.45055 0.76331
KNNW 19 0.67961 0.40912 0.76171 0.56053 0.69725 0.44165 0.76462
KNNW 37 0.68447 0.41807 0.76013 0.55762 0.69031 0.42885 0.76482
KNNW 42 0.67961 0.40912 0.75989 0.55717 0.69031 0.42885 0.76498
LOF 8 0.68932 0.42703 0.71931 0.48233 0.69953 0.44586 0.74735
LOF 12 0.68932 0.42703 0.73129 0.50443 0.70698 0.45959 0.75668
LOF 84 0.67476 0.40017 0.75468 0.54756 0.68329 0.41591 0.76218
LOF 85 0.67476 0.40017 0.75488 0.54793 0.68329 0.41591 0.76212
SimplifiedLOF 18 0.69903 0.44493 0.73412 0.50966 0.70218 0.45074 0.75736
SimplifiedLOF 23 0.69903 0.44493 0.73945 0.51948 0.70588 0.45757 0.76148
SimplifiedLOF 98 0.68932 0.42703 0.75612 0.55022 0.69100 0.43012 0.76649
LoOP 22 0.70388 0.45388 0.73104 0.50397 0.70531 0.45652 0.75999
LoOP 27 0.69903 0.44493 0.73846 0.51764 0.70309 0.45242 0.76314
LoOP 99 0.68932 0.42703 0.75024 0.53937 0.69082 0.42979 0.76285
LDOF 23 0.69903 0.44493 0.71845 0.48074 0.70588 0.45757 0.75748
LDOF 27 0.69903 0.44493 0.73183 0.50543 0.70073 0.44807 0.76727
LDOF 30 0.68447 0.41807 0.73145 0.50472 0.70616 0.45808 0.76245
LDOF 86 0.68447 0.41807 0.75359 0.54555 0.69212 0.43220 0.76482
ODIN 35 0.68447 0.41807 0.68915 0.42671 0.69630 0.43989 0.74802
ODIN 47 0.69417 0.43598 0.70830 0.46202 0.69417 0.43598 0.75585
ODIN 80 0.68689 0.42255 0.72469 0.49226 0.69031 0.42885 0.75820
ODIN 92 0.67961 0.40912 0.72864 0.49954 0.68868 0.42584 0.75814
FastABOD 36 0.68447 0.41807 0.73415 0.50971 0.69359 0.43489 0.76132
FastABOD 39 0.68447 0.41807 0.73834 0.51743 0.68750 0.42367 0.76319
FastABOD 90 0.67476 0.40017 0.73999 0.52047 0.68736 0.42341 0.76146
KDEOS 94 0.66505 0.38226 0.61968 0.29859 0.68511 0.41925 0.71680
KDEOS 97 0.66990 0.39121 0.62003 0.29923 0.68376 0.41677 0.71680
KDEOS 100 0.66990 0.39121 0.61997 0.29913 0.68444 0.41803 0.71727
LDF 76 0.63107 0.31959 0.64352 0.34255 0.66436 0.38099 0.67854
LDF 87 0.59709 0.25692 0.65003 0.35456 0.67518 0.40095 0.67589
LDF 95 0.59709 0.25692 0.67695 0.40422 0.66545 0.38300 0.69471
INFLO 24 0.68447 0.41807 0.72558 0.49390 0.70862 0.46263 0.75207
INFLO 30 0.69903 0.44493 0.73195 0.50564 0.70309 0.45242 0.75535
INFLO 58 0.68447 0.41807 0.74308 0.52618 0.69458 0.43673 0.76419
INFLO 59 0.68932 0.42703 0.74300 0.52602 0.69380 0.43528 0.76519
COF 18 0.64563 0.34645 0.73068 0.50330 0.68398 0.41718 0.73675
COF 19 0.66505 0.38226 0.73224 0.50618 0.68240 0.41427 0.73812
COF 24 0.66990 0.39121 0.73034 0.50268 0.67943 0.40878 0.72885
COF 53 0.64563 0.34645 0.73647 0.51397 0.65893 0.37098 0.73301

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO