Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (10% of outliers version#10)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (47.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.40741 0.34183 0.47031 0.41169 0.44000 0.37803 0.78582
KNN 19 0.40741 0.34183 0.49249 0.43633 0.50000 0.44467 0.77975
KNN 23 0.40741 0.34183 0.49968 0.44431 0.48889 0.43233 0.78689
KNNW 1 0.40741 0.34183 0.47288 0.41455 0.45455 0.39419 0.77231
KNNW 34 0.40741 0.34183 0.49282 0.43670 0.48889 0.43233 0.78506
KNNW 63 0.40741 0.34183 0.49832 0.44280 0.48780 0.43113 0.78264
LOF 3 0.44444 0.38297 0.43375 0.37109 0.45283 0.39228 0.80146
LOF 95 0.40741 0.34183 0.48413 0.42704 0.50000 0.44467 0.77656
SimplifiedLOF 5 0.44444 0.38297 0.40628 0.34058 0.45283 0.39228 0.78021
SimplifiedLOF 6 0.44444 0.38297 0.42246 0.35855 0.48980 0.43334 0.79114
SimplifiedLOF 91 0.40741 0.34183 0.48327 0.42609 0.47619 0.41823 0.77945
LoOP 5 0.44444 0.38297 0.39994 0.33354 0.44444 0.38297 0.78036
LoOP 6 0.44444 0.38297 0.42592 0.36239 0.47059 0.41201 0.79478
LoOP 82 0.40741 0.34183 0.48092 0.42348 0.48780 0.43113 0.78172
LoOP 89 0.40741 0.34183 0.48248 0.42521 0.48780 0.43113 0.78112
LDOF 9 0.48148 0.42410 0.42680 0.36337 0.49123 0.43493 0.77990
LDOF 11 0.40741 0.34183 0.44156 0.37977 0.47826 0.42053 0.79296
LDOF 96 0.40741 0.34183 0.46989 0.41123 0.47619 0.41823 0.77292
ODIN 18 0.37931 0.31063 0.28746 0.20861 0.39286 0.32567 0.78696
ODIN 74 0.40741 0.34183 0.36221 0.29164 0.45833 0.39839 0.76799
ODIN 80 0.43519 0.37269 0.38134 0.31288 0.45833 0.39839 0.76844
ODIN 81 0.44444 0.38297 0.36857 0.29870 0.45833 0.39839 0.76746
FastABOD 4 0.44444 0.38297 0.42145 0.35743 0.48000 0.42246 0.76154
FastABOD 5 0.44444 0.38297 0.43104 0.36809 0.50000 0.44467 0.76776
FastABOD 52 0.44444 0.38297 0.47653 0.41861 0.45283 0.39228 0.78901
FastABOD 98 0.40741 0.34183 0.48302 0.42581 0.45833 0.39839 0.78446
KDEOS 11 0.29630 0.21843 0.24334 0.15961 0.36145 0.29079 0.74196
KDEOS 12 0.29630 0.21843 0.24431 0.16068 0.33803 0.26478 0.74499
KDEOS 17 0.25926 0.17729 0.23810 0.15379 0.38636 0.31846 0.74575
KDEOS 19 0.25926 0.17729 0.22506 0.13931 0.39474 0.32776 0.74165
LDF 49 0.48148 0.42410 0.35761 0.28653 0.50000 0.44467 0.78613
LDF 75 0.40741 0.34183 0.48378 0.42666 0.50000 0.44467 0.80631
LDF 100 0.44444 0.38297 0.53256 0.48083 0.55814 0.50925 0.77808
INFLO 5 0.44444 0.38297 0.41863 0.35429 0.46809 0.40923 0.78264
INFLO 10 0.44444 0.38297 0.44636 0.38510 0.48980 0.43334 0.78704
INFLO 28 0.44444 0.38297 0.48147 0.42409 0.48000 0.42246 0.79614
INFLO 90 0.40741 0.34183 0.48587 0.42897 0.48889 0.43233 0.77611
COF 3 0.40741 0.34183 0.44206 0.38032 0.45000 0.38914 0.78802
COF 4 0.44444 0.38297 0.44581 0.38449 0.46154 0.40195 0.77489
COF 5 0.40741 0.34183 0.45066 0.38987 0.45000 0.38914 0.76298

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (47.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.37037 0.30070 0.39627 0.32947 0.43243 0.36963 0.76882
KNN 4 0.37037 0.30070 0.40131 0.33507 0.43243 0.36963 0.77171
KNN 11 0.33333 0.25956 0.37247 0.30303 0.45000 0.38914 0.75334
KNNW 1 0.37037 0.30070 0.42102 0.35695 0.43243 0.36963 0.72920
KNNW 7 0.33333 0.25956 0.41297 0.34802 0.43243 0.36963 0.76548
LOF 2 0.40741 0.34183 0.40330 0.33727 0.45902 0.39915 0.76503
LOF 5 0.40741 0.34183 0.42317 0.35934 0.42308 0.35924 0.77034
LOF 7 0.37037 0.30070 0.42141 0.35739 0.41667 0.35212 0.77823
SimplifiedLOF 4 0.40741 0.34183 0.42500 0.36137 0.44828 0.38722 0.78567
SimplifiedLOF 5 0.40741 0.34183 0.43867 0.37656 0.43137 0.36845 0.78643
SimplifiedLOF 6 0.40741 0.34183 0.43720 0.37492 0.44068 0.37879 0.79296
SimplifiedLOF 52 0.37037 0.30070 0.38373 0.31554 0.46154 0.40195 0.75015
LoOP 5 0.40741 0.34183 0.43415 0.37153 0.43636 0.37399 0.78658
LoOP 7 0.40741 0.34183 0.42442 0.36073 0.41509 0.35037 0.79387
LoOP 34 0.37037 0.30070 0.38597 0.31802 0.46154 0.40195 0.75698
LDOF 8 0.37037 0.30070 0.40202 0.33585 0.42857 0.36534 0.81527
LDOF 13 0.48148 0.42410 0.40746 0.34189 0.49123 0.43493 0.78537
LDOF 23 0.40741 0.34183 0.44134 0.37953 0.48889 0.43233 0.77292
LDOF 39 0.44444 0.38297 0.40422 0.33829 0.50000 0.44467 0.75395
ODIN 6 0.27586 0.19573 0.22972 0.14448 0.36036 0.28958 0.77353
ODIN 33 0.45791 0.39793 0.33200 0.25808 0.49123 0.43493 0.74560
ODIN 49 0.47222 0.41382 0.35833 0.28733 0.47273 0.41438 0.75250
ODIN 93 0.39259 0.32538 0.36789 0.29795 0.41935 0.35510 0.75220
FastABOD 4 0.40741 0.34183 0.42616 0.36266 0.44000 0.37803 0.79539
FastABOD 5 0.40741 0.34183 0.42533 0.36174 0.44000 0.37803 0.80965
FastABOD 17 0.40741 0.34183 0.43248 0.36968 0.44444 0.38297 0.79250
FastABOD 29 0.40741 0.34183 0.42887 0.36567 0.47368 0.41544 0.78704
KDEOS 7 0.25926 0.17729 0.19234 0.10297 0.31707 0.24150 0.73634
KDEOS 28 0.14815 0.05389 0.29847 0.22084 0.35789 0.28684 0.72936
KDEOS 44 0.14815 0.05389 0.20614 0.11829 0.39024 0.32277 0.72465
KDEOS 89 0.33333 0.25956 0.22127 0.13510 0.35955 0.28868 0.73194
LDF 2 0.18519 0.09502 0.18674 0.09675 0.30556 0.22871 0.71494
LDF 21 0.25926 0.17729 0.31804 0.24257 0.31250 0.23642 0.63707
LDF 23 0.29630 0.21843 0.29850 0.22087 0.29630 0.21843 0.64876
LDF 48 0.25926 0.17729 0.21912 0.13271 0.33333 0.25956 0.64162
INFLO 2 0.40741 0.34183 0.36462 0.29431 0.45714 0.39707 0.76321
INFLO 4 0.40741 0.34183 0.43905 0.37698 0.44444 0.38297 0.82726
INFLO 52 0.37037 0.30070 0.38785 0.32011 0.46512 0.40593 0.74514
COF 4 0.44444 0.38297 0.47332 0.41504 0.47368 0.41544 0.76396
COF 5 0.48148 0.42410 0.47084 0.41229 0.50000 0.44467 0.75880

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO