Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (2% of outliers version#06)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (31.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.25000 0.23770 0.30826 0.29692 0.40000 0.39016 0.78996
KNN 18 0.25000 0.23770 0.31079 0.29949 0.40000 0.39016 0.77869
KNNW 1 0.25000 0.23770 0.31563 0.30441 0.40000 0.39016 0.80943
LOF 1 0.25000 0.23770 0.16444 0.15075 0.33333 0.32240 0.77152
LOF 3 0.25000 0.23770 0.27654 0.26468 0.40000 0.39016 0.67111
LOF 12 0.25000 0.23770 0.31775 0.30656 0.40000 0.39016 0.79918
LOF 23 0.25000 0.23770 0.32317 0.31208 0.40000 0.39016 0.79918
SimplifiedLOF 1 0.25000 0.23770 0.28795 0.27628 0.40000 0.39016 0.79098
SimplifiedLOF 7 0.25000 0.23770 0.31670 0.30549 0.40000 0.39016 0.80430
SimplifiedLOF 23 0.25000 0.23770 0.32151 0.31039 0.40000 0.39016 0.80430
LoOP 1 0.25000 0.23770 0.28795 0.27628 0.40000 0.39016 0.79098
LoOP 12 0.25000 0.23770 0.31496 0.30373 0.40000 0.39016 0.80840
LoOP 23 0.25000 0.23770 0.31556 0.30434 0.40000 0.39016 0.80225
LDOF 3 0.25000 0.23770 0.26933 0.25735 0.40000 0.39016 0.65779
LDOF 7 0.25000 0.23770 0.29716 0.28564 0.40000 0.39016 0.79098
LDOF 64 0.25000 0.23770 0.30657 0.29520 0.40000 0.39016 0.78791
ODIN 46 0.09091 0.07601 0.07463 0.05946 0.16216 0.14843 0.79559
ODIN 69 0.25000 0.23770 0.11894 0.10450 0.25000 0.23770 0.78637
ODIN 72 0.25000 0.23770 0.14094 0.12686 0.28571 0.27400 0.79201
FastABOD 4 0.25000 0.23770 0.15176 0.13786 0.33333 0.32240 0.67930
FastABOD 6 0.25000 0.23770 0.28091 0.26912 0.40000 0.39016 0.73258
FastABOD 49 0.25000 0.23770 0.29999 0.28852 0.40000 0.39016 0.76639
FastABOD 57 0.25000 0.23770 0.30124 0.28978 0.40000 0.39016 0.76230
KDEOS 2 0.00000 -0.01639 0.02151 0.00546 0.04211 0.02640 0.54303
KDEOS 19 0.00000 -0.01639 0.07240 0.05719 0.22222 0.20947 0.70082
KDEOS 24 0.00000 -0.01639 0.07774 0.06263 0.22222 0.20947 0.73668
KDEOS 25 0.00000 -0.01639 0.07877 0.06367 0.22222 0.20947 0.72848
LDF 6 0.25000 0.23770 0.29466 0.28310 0.40000 0.39016 0.85451
LDF 77 0.25000 0.23770 0.32433 0.31325 0.40000 0.39016 0.67008
INFLO 1 0.25000 0.23770 0.28592 0.27422 0.40000 0.39016 0.71619
INFLO 9 0.25000 0.23770 0.32431 0.31323 0.40000 0.39016 0.80020
INFLO 24 0.25000 0.23770 0.31790 0.30672 0.40000 0.39016 0.80943
COF 12 0.50000 0.49180 0.27896 0.26714 0.50000 0.49180 0.84119
COF 14 0.25000 0.23770 0.39257 0.38261 0.44444 0.43534 0.86066

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (35.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.25000 0.23770 0.17701 0.16352 0.33333 0.32240 0.83197
KNN 6 0.25000 0.23770 0.13552 0.12134 0.28571 0.27400 0.84221
KNNW 1 0.25000 0.23770 0.18816 0.17485 0.33333 0.32240 0.84939
LOF 2 0.25000 0.23770 0.30661 0.29525 0.40000 0.39016 0.82889
LOF 3 0.25000 0.23770 0.33888 0.32804 0.40000 0.39016 0.85656
LOF 18 0.25000 0.23770 0.13838 0.12425 0.28571 0.27400 0.86578
SimplifiedLOF 2 0.25000 0.23770 0.14007 0.12597 0.28571 0.27400 0.84016
SimplifiedLOF 5 0.25000 0.23770 0.34330 0.33253 0.40000 0.39016 0.88320
SimplifiedLOF 14 0.25000 0.23770 0.17086 0.15726 0.30769 0.29634 0.88627
LoOP 2 0.25000 0.23770 0.11863 0.10418 0.25000 0.23770 0.83094
LoOP 4 0.25000 0.23770 0.21996 0.20717 0.33333 0.32240 0.87500
LoOP 5 0.25000 0.23770 0.22094 0.20817 0.33333 0.32240 0.89037
LDOF 4 0.25000 0.23770 0.17937 0.16591 0.30769 0.29634 0.93340
LDOF 5 0.25000 0.23770 0.23312 0.22055 0.33333 0.32240 0.93648
LDOF 12 0.25000 0.23770 0.26506 0.25302 0.44444 0.43534 0.91393
ODIN 4 0.05556 0.04007 0.05635 0.04088 0.10811 0.09349 0.86270
ODIN 24 0.25000 0.23770 0.10453 0.08985 0.25000 0.23770 0.83863
ODIN 46 0.25000 0.23770 0.12813 0.11384 0.28571 0.27400 0.82941
ODIN 71 0.25000 0.23770 0.13468 0.12049 0.28571 0.27400 0.83607
FastABOD 5 0.25000 0.23770 0.20002 0.18690 0.33333 0.32240 0.84631
FastABOD 10 0.25000 0.23770 0.19949 0.18637 0.33333 0.32240 0.87398
KDEOS 5 0.25000 0.23770 0.11450 0.09998 0.28571 0.27400 0.78893
KDEOS 6 0.25000 0.23770 0.28708 0.27539 0.40000 0.39016 0.81660
KDEOS 10 0.25000 0.23770 0.32992 0.31894 0.40000 0.39016 0.90266
LDF 1 0.00000 -0.01639 0.04056 0.02483 0.11765 0.10318 0.67828
LDF 3 0.00000 -0.01639 0.05744 0.04199 0.12500 0.11066 0.80533
LDF 4 0.00000 -0.01639 0.05491 0.03942 0.10526 0.09060 0.81455
LDF 10 0.00000 -0.01639 0.05637 0.04090 0.15385 0.13997 0.69262
INFLO 2 0.25000 0.23770 0.18643 0.17309 0.33333 0.32240 0.84529
INFLO 5 0.25000 0.23770 0.20675 0.19375 0.33333 0.32240 0.88525
COF 4 0.50000 0.49180 0.39291 0.38295 0.50000 0.49180 0.78381
COF 15 0.25000 0.23770 0.19823 0.18509 0.33333 0.32240 0.86783

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO