Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (10% of outliers version#03)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (46.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.44444 0.38297 0.48091 0.42347 0.45833 0.39839 0.76366
KNN 14 0.44444 0.38297 0.48254 0.42528 0.50000 0.44467 0.75531
KNN 18 0.48148 0.42410 0.48211 0.42481 0.48148 0.42410 0.75273
KNN 28 0.44444 0.38297 0.48765 0.43095 0.50000 0.44467 0.75137
KNNW 1 0.40741 0.34183 0.47476 0.41664 0.45455 0.39419 0.76541
KNNW 6 0.44444 0.38297 0.47907 0.42143 0.46512 0.40593 0.75911
KNNW 18 0.44444 0.38297 0.48518 0.42821 0.48000 0.42246 0.75911
KNNW 21 0.44444 0.38297 0.48577 0.42887 0.48000 0.42246 0.75911
LOF 1 0.44444 0.38297 0.43792 0.37573 0.48000 0.42246 0.78764
LOF 18 0.37037 0.30070 0.45102 0.39027 0.48780 0.43113 0.75622
LOF 74 0.44444 0.38297 0.48656 0.42974 0.46512 0.40593 0.75046
SimplifiedLOF 1 0.48148 0.42410 0.41456 0.34978 0.49057 0.43419 0.76131
SimplifiedLOF 20 0.40741 0.34183 0.44912 0.38817 0.46512 0.40593 0.78370
SimplifiedLOF 30 0.40741 0.34183 0.46550 0.40636 0.50000 0.44467 0.77125
SimplifiedLOF 94 0.44444 0.38297 0.49456 0.43864 0.48980 0.43334 0.76457
LoOP 1 0.48148 0.42410 0.41456 0.34978 0.49057 0.43419 0.76131
LoOP 9 0.44444 0.38297 0.40327 0.33724 0.45833 0.39839 0.78461
LoOP 34 0.40741 0.34183 0.47551 0.41747 0.50000 0.44467 0.77140
LoOP 86 0.44444 0.38297 0.48773 0.43104 0.50000 0.44467 0.75918
LDOF 19 0.44444 0.38297 0.42634 0.36286 0.45833 0.39839 0.78537
LDOF 20 0.44444 0.38297 0.41937 0.35512 0.46154 0.40195 0.78582
LDOF 71 0.44444 0.38297 0.48944 0.43294 0.46809 0.40923 0.77383
LDOF 89 0.44444 0.38297 0.47561 0.41758 0.50000 0.44467 0.76988
ODIN 15 0.28571 0.20667 0.24731 0.16402 0.38889 0.32127 0.76298
ODIN 27 0.37037 0.30070 0.28081 0.20123 0.46667 0.40765 0.75349
ODIN 37 0.40741 0.34183 0.28168 0.20220 0.43038 0.36735 0.75319
ODIN 100 0.40741 0.34183 0.38500 0.31694 0.45455 0.39419 0.74977
FastABOD 18 0.33333 0.25956 0.39566 0.32878 0.41558 0.35092 0.74863
FastABOD 29 0.40741 0.34183 0.38626 0.31835 0.44118 0.37934 0.73755
FastABOD 93 0.40741 0.34183 0.46012 0.40038 0.47368 0.41544 0.74302
FastABOD 97 0.40741 0.34183 0.46188 0.40234 0.47368 0.41544 0.74211
KDEOS 9 0.29630 0.21843 0.28942 0.21080 0.35821 0.28719 0.74879
KDEOS 10 0.33333 0.25956 0.23936 0.15519 0.35088 0.27905 0.74863
KDEOS 11 0.29630 0.21843 0.22328 0.13733 0.33846 0.26526 0.75471
KDEOS 92 0.22222 0.13616 0.20663 0.11884 0.38889 0.32127 0.72131
LDF 35 0.44444 0.38297 0.36498 0.29471 0.45283 0.39228 0.76336
LDF 50 0.18519 0.09502 0.30514 0.22825 0.38636 0.31846 0.77307
LDF 100 0.37037 0.30070 0.44610 0.38481 0.42857 0.36534 0.71949
INFLO 5 0.40741 0.34183 0.39925 0.33277 0.41509 0.35037 0.79098
INFLO 7 0.44444 0.38297 0.39224 0.32499 0.44444 0.38297 0.77216
INFLO 83 0.44444 0.38297 0.49538 0.43954 0.48980 0.43334 0.76670
INFLO 85 0.44444 0.38297 0.49352 0.43747 0.50000 0.44467 0.76563
COF 1 0.48148 0.42410 0.41456 0.34978 0.49057 0.43419 0.76131
COF 2 0.40741 0.34183 0.43263 0.36985 0.46809 0.40923 0.78119
COF 4 0.44444 0.38297 0.39715 0.33044 0.50847 0.45408 0.75812

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (50.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.44444 0.38297 0.44209 0.38036 0.48485 0.42784 0.82195
KNN 3 0.44444 0.38297 0.43643 0.37407 0.47458 0.41644 0.84039
KNN 4 0.44444 0.38297 0.43785 0.37565 0.49180 0.43557 0.83409
KNNW 1 0.44444 0.38297 0.46256 0.40308 0.49123 0.43493 0.83121
KNNW 2 0.44444 0.38297 0.43466 0.37210 0.50794 0.45349 0.82939
KNNW 6 0.44444 0.38297 0.43993 0.37795 0.47619 0.41823 0.83561
LOF 7 0.33333 0.25956 0.45953 0.39973 0.47222 0.41382 0.84699
LOF 9 0.33333 0.25956 0.45730 0.39724 0.50667 0.45208 0.85079
LOF 48 0.40741 0.34183 0.42015 0.35599 0.47222 0.41382 0.81239
SimplifiedLOF 5 0.40741 0.34183 0.43454 0.37197 0.45455 0.39419 0.82362
SimplifiedLOF 7 0.37037 0.30070 0.46274 0.40329 0.48193 0.42460 0.84684
SimplifiedLOF 8 0.33333 0.25956 0.45508 0.39478 0.50549 0.45077 0.85094
SimplifiedLOF 13 0.37037 0.30070 0.44648 0.38523 0.48780 0.43113 0.85610
LoOP 8 0.33333 0.25956 0.43081 0.36782 0.51111 0.45701 0.85041
LoOP 10 0.33333 0.25956 0.44612 0.38483 0.48936 0.43286 0.85701
LoOP 19 0.37037 0.30070 0.43962 0.37761 0.49462 0.43870 0.85906
LoOP 22 0.40741 0.34183 0.43475 0.37221 0.47500 0.41691 0.85246
LDOF 4 0.40741 0.34183 0.34336 0.27070 0.44828 0.38722 0.74787
LDOF 19 0.33333 0.25956 0.42571 0.36216 0.50000 0.44467 0.85443
LDOF 20 0.33333 0.25956 0.41731 0.35284 0.50549 0.45077 0.84821
LDOF 47 0.37037 0.30070 0.43254 0.36975 0.47826 0.42053 0.84366
ODIN 35 0.35802 0.28699 0.37443 0.30521 0.50000 0.44467 0.82248
ODIN 43 0.39815 0.33155 0.39552 0.32863 0.50000 0.44467 0.82559
ODIN 66 0.44444 0.38297 0.40629 0.34059 0.48193 0.42460 0.82081
ODIN 100 0.37037 0.30070 0.43037 0.36734 0.48649 0.42966 0.80798
FastABOD 4 0.37037 0.30070 0.33824 0.26502 0.41270 0.34771 0.77945
FastABOD 17 0.37037 0.30070 0.38002 0.31141 0.42857 0.36534 0.82286
FastABOD 38 0.37037 0.30070 0.36930 0.29951 0.43678 0.37446 0.82362
FastABOD 74 0.33333 0.25956 0.36868 0.29883 0.46512 0.40593 0.81573
KDEOS 12 0.37037 0.30070 0.39317 0.32602 0.45614 0.39596 0.78005
KDEOS 71 0.29630 0.21843 0.27789 0.19798 0.45455 0.39419 0.81557
KDEOS 77 0.22222 0.13616 0.26490 0.18356 0.47312 0.41482 0.81011
LDF 2 0.33333 0.25956 0.25648 0.17420 0.36000 0.28918 0.69854
LDF 15 0.18519 0.09502 0.29322 0.21501 0.31250 0.23642 0.59426
LDF 74 0.18519 0.09502 0.21402 0.12705 0.36364 0.29322 0.70932
LDF 100 0.22222 0.13616 0.27143 0.19081 0.32967 0.25549 0.73194
INFLO 14 0.33333 0.25956 0.42888 0.36568 0.49383 0.43782 0.85155
INFLO 16 0.33333 0.25956 0.42024 0.35608 0.51220 0.45822 0.85216
INFLO 17 0.29630 0.21843 0.42161 0.35760 0.51163 0.45759 0.85367
INFLO 27 0.40741 0.34183 0.40586 0.34012 0.44660 0.38537 0.83470
COF 8 0.48148 0.42410 0.49946 0.44407 0.52000 0.46689 0.81648
COF 9 0.51852 0.46524 0.49381 0.43780 0.53846 0.48739 0.80191

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO