Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (2% of outliers version#07)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (39.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.01639 0.03760 0.02182 0.09091 0.07601 0.67777
KNN 73 0.00000 -0.01639 0.05863 0.04320 0.16667 0.15301 0.73258
KNN 84 0.00000 -0.01639 0.06038 0.04498 0.16667 0.15301 0.74898
KNN 93 0.00000 -0.01639 0.06042 0.04501 0.16667 0.15301 0.74898
KNNW 1 0.00000 -0.01639 0.04341 0.02773 0.12500 0.11066 0.72131
KNNW 8 0.00000 -0.01639 0.04162 0.02591 0.08696 0.07199 0.73463
KNNW 100 0.00000 -0.01639 0.05188 0.03634 0.14286 0.12881 0.73258
LOF 1 0.25000 0.23770 0.11032 0.09573 0.28571 0.27400 0.65779
LOF 12 0.00000 -0.01639 0.05772 0.04227 0.12121 0.10681 0.79918
SimplifiedLOF 5 0.25000 0.23770 0.09028 0.07537 0.25000 0.23770 0.73361
SimplifiedLOF 14 0.00000 -0.01639 0.07536 0.06020 0.20000 0.18689 0.80225
LoOP 3 0.25000 0.23770 0.08158 0.06652 0.25000 0.23770 0.55328
LoOP 5 0.25000 0.23770 0.08543 0.07044 0.25000 0.23770 0.69570
LoOP 12 0.00000 -0.01639 0.06513 0.04980 0.16667 0.15301 0.80020
LDOF 2 0.25000 0.23770 0.20674 0.19374 0.36364 0.35320 0.64549
LDOF 3 0.25000 0.23770 0.29924 0.28776 0.40000 0.39016 0.66496
LDOF 47 0.00000 -0.01639 0.05797 0.04253 0.11765 0.10318 0.82582
ODIN 24 0.06250 0.04713 0.04825 0.03264 0.10000 0.08525 0.79559
ODIN 29 0.06667 0.05137 0.04692 0.03129 0.10526 0.09060 0.77357
ODIN 76 0.00000 -0.01639 0.05600 0.04053 0.12903 0.11475 0.76383
FastABOD 3 0.00000 -0.01639 0.04896 0.03337 0.16667 0.15301 0.61783
FastABOD 43 0.00000 -0.01639 0.04336 0.02767 0.09091 0.07601 0.77357
KDEOS 18 0.25000 0.23770 0.09704 0.08224 0.25000 0.23770 0.73156
KDEOS 20 0.25000 0.23770 0.28627 0.27457 0.40000 0.39016 0.74590
KDEOS 57 0.25000 0.23770 0.13459 0.12041 0.26667 0.25464 0.81660
LDF 3 0.40000 0.39016 0.34158 0.33079 0.40000 0.39016 0.57275
LDF 45 0.25000 0.23770 0.10314 0.08844 0.25000 0.23770 0.79713
INFLO 1 0.25000 0.23770 0.08187 0.06681 0.25000 0.23770 0.63678
INFLO 3 0.25000 0.23770 0.10266 0.08795 0.28571 0.27400 0.59221
INFLO 5 0.25000 0.23770 0.10456 0.08988 0.28571 0.27400 0.67418
INFLO 20 0.00000 -0.01639 0.06004 0.04463 0.13333 0.11913 0.81557
COF 10 0.00000 -0.01639 0.04759 0.03198 0.10714 0.09251 0.77357
COF 55 0.25000 0.23770 0.07642 0.06128 0.25000 0.23770 0.51537
COF 86 0.25000 0.23770 0.10089 0.08615 0.28571 0.27400 0.61680

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (42.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.01639 0.02490 0.00892 0.05769 0.04224 0.61783
KNN 26 0.00000 -0.01639 0.02630 0.01034 0.07273 0.05753 0.57889
KNN 68 0.00000 -0.01639 0.02785 0.01191 0.06780 0.05251 0.59426
KNNW 1 0.00000 -0.01639 0.02890 0.01298 0.06723 0.05194 0.67213
KNNW 48 0.00000 -0.01639 0.02605 0.01008 0.07143 0.05621 0.59426
LOF 1 0.00000 -0.01639 0.02758 0.01164 0.05660 0.04114 0.62910
LOF 2 0.00000 -0.01639 0.03550 0.01969 0.08000 0.06492 0.69980
SimplifiedLOF 1 0.00000 -0.01639 0.02406 0.00806 0.05455 0.03905 0.59426
SimplifiedLOF 3 0.00000 -0.01639 0.03350 0.01766 0.07229 0.05708 0.69262
SimplifiedLOF 4 0.00000 -0.01639 0.03828 0.02251 0.10526 0.09060 0.67520
LoOP 1 0.00000 -0.01639 0.02406 0.00806 0.05455 0.03905 0.59426
LoOP 3 0.00000 -0.01639 0.03408 0.01824 0.06977 0.05452 0.69570
LoOP 4 0.00000 -0.01639 0.03957 0.02382 0.11111 0.09654 0.67828
LDOF 2 0.00000 -0.01639 0.03024 0.01434 0.06667 0.05137 0.64242
LDOF 4 0.00000 -0.01639 0.03718 0.02140 0.11111 0.09654 0.63832
LDOF 5 0.00000 -0.01639 0.04557 0.02992 0.11111 0.09654 0.73873
LDOF 6 0.00000 -0.01639 0.04119 0.02547 0.08511 0.07011 0.74795
ODIN 2 0.02817 0.01224 0.03034 0.01445 0.06299 0.04763 0.73668
ODIN 20 0.00000 -0.01639 0.02828 0.01235 0.07692 0.06179 0.64805
FastABOD 3 0.00000 -0.01639 0.02599 0.01002 0.05970 0.04429 0.60656
FastABOD 4 0.00000 -0.01639 0.03924 0.02349 0.08511 0.07011 0.71516
FastABOD 11 0.00000 -0.01639 0.03372 0.01788 0.08696 0.07199 0.71721
KDEOS 7 0.00000 -0.01639 0.10156 0.08683 0.25000 0.23770 0.83197
KDEOS 9 0.25000 0.23770 0.12341 0.10904 0.26667 0.25464 0.76332
KDEOS 11 0.25000 0.23770 0.34530 0.33457 0.40000 0.39016 0.70492
LDF 1 0.00000 -0.01639 0.03494 0.01912 0.10000 0.08525 0.64959
LDF 10 0.25000 0.23770 0.26501 0.25296 0.40000 0.39016 0.52459
LDF 17 0.25000 0.23770 0.26986 0.25789 0.40000 0.39016 0.62295
INFLO 1 0.00000 -0.01639 0.02894 0.01302 0.08000 0.06492 0.59016
INFLO 4 0.00000 -0.01639 0.03819 0.02242 0.09091 0.07601 0.69980
INFLO 9 0.00000 -0.01639 0.03722 0.02143 0.09375 0.07889 0.72848
COF 1 0.00000 -0.01639 0.02406 0.00806 0.05455 0.03905 0.59426
COF 3 0.00000 -0.01639 0.03614 0.02034 0.08511 0.07011 0.73873
COF 4 0.00000 -0.01639 0.03524 0.01942 0.08889 0.07395 0.73463

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO