Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (10% of outliers version#09)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (46.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.59259 0.54751 0.64958 0.61081 0.61905 0.57689 0.83303
KNN 27 0.59259 0.54751 0.67440 0.63837 0.65217 0.61368 0.83774
KNN 39 0.59259 0.54751 0.68025 0.64487 0.68182 0.64661 0.83546
KNNW 4 0.59259 0.54751 0.65187 0.61335 0.63415 0.59366 0.82939
KNNW 39 0.59259 0.54751 0.66935 0.63276 0.65217 0.61368 0.83470
KNNW 58 0.59259 0.54751 0.67139 0.63503 0.65217 0.61368 0.83455
LOF 44 0.59259 0.54751 0.65185 0.61333 0.61905 0.57689 0.83485
LOF 48 0.59259 0.54751 0.65297 0.61457 0.61905 0.57689 0.83713
LOF 76 0.59259 0.54751 0.66529 0.62826 0.65306 0.61467 0.83424
LOF 94 0.59259 0.54751 0.66769 0.63091 0.65217 0.61368 0.83379
SimplifiedLOF 56 0.59259 0.54751 0.65035 0.61166 0.61905 0.57689 0.83500
SimplifiedLOF 58 0.59259 0.54751 0.65060 0.61193 0.61905 0.57689 0.83576
SimplifiedLOF 85 0.59259 0.54751 0.65780 0.61993 0.63830 0.59827 0.83242
SimplifiedLOF 100 0.59259 0.54751 0.65854 0.62075 0.63830 0.59827 0.83136
LoOP 54 0.55556 0.50638 0.64236 0.60279 0.60465 0.56090 0.83865
LoOP 56 0.59259 0.54751 0.64759 0.60859 0.60870 0.56540 0.83849
LoOP 95 0.59259 0.54751 0.65783 0.61997 0.63830 0.59827 0.83713
LDOF 47 0.55556 0.50638 0.61911 0.57696 0.58537 0.53948 0.83485
LDOF 57 0.55556 0.50638 0.62706 0.58579 0.60870 0.56540 0.83470
LDOF 58 0.55556 0.50638 0.63199 0.59127 0.60870 0.56540 0.83637
LDOF 99 0.55556 0.50638 0.64123 0.60153 0.60870 0.56540 0.82969
ODIN 83 0.50617 0.45153 0.46163 0.40206 0.57143 0.52400 0.82407
ODIN 98 0.55556 0.50638 0.48838 0.43176 0.56140 0.51287 0.83022
ODIN 99 0.56481 0.51666 0.48784 0.43117 0.56604 0.51802 0.83030
ODIN 100 0.54815 0.49815 0.48708 0.43033 0.56667 0.51872 0.83075
FastABOD 86 0.59259 0.54751 0.62212 0.58030 0.59259 0.54751 0.83819
FastABOD 98 0.59259 0.54751 0.62967 0.58870 0.61224 0.56934 0.84047
FastABOD 99 0.59259 0.54751 0.63222 0.59152 0.61224 0.56934 0.84092
KDEOS 10 0.22222 0.13616 0.18716 0.09721 0.29032 0.21179 0.69308
KDEOS 99 0.18519 0.09502 0.18293 0.09252 0.35772 0.28665 0.72602
KDEOS 100 0.14815 0.05389 0.18467 0.09445 0.35484 0.28345 0.72678
LDF 71 0.59259 0.54751 0.66707 0.63022 0.68182 0.64661 0.83865
LDF 83 0.62963 0.58865 0.61914 0.57699 0.62963 0.58865 0.84760
LDF 87 0.62963 0.58865 0.65307 0.61468 0.63830 0.59827 0.85413
LDF 90 0.59259 0.54751 0.67739 0.64169 0.68182 0.64661 0.84897
INFLO 35 0.55556 0.50638 0.64129 0.60159 0.60465 0.56090 0.84199
INFLO 40 0.59259 0.54751 0.64689 0.60782 0.63636 0.59613 0.83030
INFLO 69 0.59259 0.54751 0.66160 0.62415 0.66667 0.62978 0.82969
INFLO 71 0.59259 0.54751 0.66440 0.62726 0.66667 0.62978 0.82939
COF 25 0.55556 0.50638 0.53971 0.48878 0.58182 0.53554 0.82332
COF 34 0.55556 0.50638 0.58325 0.53714 0.62500 0.58350 0.82665
COF 36 0.51852 0.46524 0.57979 0.53329 0.55814 0.50925 0.83288
COF 52 0.48148 0.42410 0.61924 0.57710 0.61905 0.57689 0.82301

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (46.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.51852 0.46524 0.52797 0.47573 0.57778 0.53106 0.77907
KNN 2 0.51852 0.46524 0.51621 0.46268 0.58333 0.53723 0.77193
KNN 26 0.51852 0.46524 0.48665 0.42984 0.56000 0.51131 0.79007
KNNW 1 0.51852 0.46524 0.54407 0.49362 0.56000 0.51131 0.79159
KNNW 4 0.51852 0.46524 0.53693 0.48569 0.59574 0.55101 0.78021
LOF 10 0.51852 0.46524 0.51411 0.46034 0.54545 0.49516 0.78036
LOF 11 0.51852 0.46524 0.52683 0.47447 0.55814 0.50925 0.78901
LOF 12 0.51852 0.46524 0.51913 0.46592 0.58333 0.53723 0.79356
LOF 19 0.51852 0.46524 0.49604 0.44028 0.59574 0.55101 0.78188
SimplifiedLOF 5 0.51852 0.46524 0.49684 0.44117 0.52830 0.47611 0.76852
SimplifiedLOF 12 0.51852 0.46524 0.52489 0.47232 0.56000 0.51131 0.78734
SimplifiedLOF 22 0.51852 0.46524 0.51768 0.46430 0.55814 0.50925 0.79781
SimplifiedLOF 52 0.51852 0.46524 0.50136 0.44618 0.59574 0.55101 0.78628
LoOP 12 0.51852 0.46524 0.50881 0.45446 0.55814 0.50925 0.78764
LoOP 13 0.51852 0.46524 0.51921 0.46601 0.55814 0.50925 0.78871
LoOP 23 0.51852 0.46524 0.49946 0.44407 0.57143 0.52400 0.80191
LoOP 52 0.51852 0.46524 0.49999 0.44466 0.59574 0.55101 0.78969
LDOF 23 0.51852 0.46524 0.50855 0.45416 0.55319 0.50375 0.81724
LDOF 30 0.51852 0.46524 0.51992 0.46680 0.59574 0.55101 0.81239
LDOF 47 0.55556 0.50638 0.49833 0.44281 0.58537 0.53948 0.79766
ODIN 45 0.52910 0.47699 0.45114 0.39041 0.53846 0.48739 0.78802
ODIN 73 0.51852 0.46524 0.50548 0.45076 0.59091 0.54564 0.78560
ODIN 86 0.51852 0.46524 0.50798 0.45354 0.55814 0.50925 0.78871
ODIN 89 0.51852 0.46524 0.50354 0.44860 0.56000 0.51131 0.79053
FastABOD 31 0.51852 0.46524 0.55122 0.50156 0.60870 0.56540 0.80829
FastABOD 32 0.55556 0.50638 0.55319 0.50375 0.58333 0.53723 0.81072
FastABOD 33 0.55556 0.50638 0.55774 0.50881 0.58333 0.53723 0.81178
FastABOD 36 0.51852 0.46524 0.55559 0.50642 0.59574 0.55101 0.81193
KDEOS 13 0.40741 0.34183 0.29011 0.21156 0.40741 0.34183 0.73270
KDEOS 14 0.29630 0.21843 0.34100 0.26808 0.36066 0.28991 0.72981
KDEOS 23 0.25926 0.17729 0.22869 0.14334 0.37500 0.30584 0.73892
KDEOS 100 0.22222 0.13616 0.22318 0.13722 0.40964 0.34431 0.73330
LDF 3 0.40741 0.34183 0.24060 0.15657 0.41509 0.35037 0.66773
LDF 12 0.37037 0.30070 0.39062 0.32319 0.40000 0.33361 0.72116
LDF 76 0.25926 0.17729 0.36890 0.29906 0.43478 0.37224 0.76199
LDF 100 0.33333 0.25956 0.37489 0.30571 0.39130 0.32395 0.77140
INFLO 11 0.51852 0.46524 0.46379 0.40446 0.52632 0.47390 0.77838
INFLO 49 0.51852 0.46524 0.50770 0.45322 0.62222 0.58042 0.81132
INFLO 50 0.51852 0.46524 0.51012 0.45591 0.62222 0.58042 0.81087
INFLO 70 0.51852 0.46524 0.49655 0.44085 0.56000 0.51131 0.81770
COF 3 0.40741 0.34183 0.44255 0.38087 0.45902 0.39915 0.78689
COF 5 0.51852 0.46524 0.53287 0.48118 0.51852 0.46524 0.77049
COF 14 0.48148 0.42410 0.50955 0.45528 0.54167 0.49095 0.76442

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO