Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (5% of outliers version#04)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (40.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.50000 0.47541 0.48460 0.45925 0.57143 0.55035 0.77869
KNN 3 0.50000 0.47541 0.49851 0.47384 0.60000 0.58033 0.76793
KNN 68 0.50000 0.47541 0.51260 0.48863 0.57143 0.55035 0.79679
KNN 98 0.50000 0.47541 0.50159 0.47708 0.55556 0.53370 0.79816
KNNW 1 0.50000 0.47541 0.50123 0.47670 0.57143 0.55035 0.77971
KNNW 3 0.50000 0.47541 0.50102 0.47648 0.60000 0.58033 0.77596
KNNW 84 0.50000 0.47541 0.51096 0.48691 0.57143 0.55035 0.78859
KNNW 97 0.50000 0.47541 0.50565 0.48133 0.55556 0.53370 0.79030
LOF 6 0.58333 0.56284 0.43900 0.41141 0.60870 0.58945 0.74829
LOF 8 0.58333 0.56284 0.47314 0.44723 0.63636 0.61848 0.76537
LOF 88 0.50000 0.47541 0.50230 0.47782 0.60000 0.58033 0.79884
LOF 89 0.50000 0.47541 0.51883 0.49517 0.60000 0.58033 0.79884
SimplifiedLOF 8 0.58333 0.56284 0.46339 0.43699 0.60870 0.58945 0.76844
SimplifiedLOF 9 0.58333 0.56284 0.50492 0.48057 0.63636 0.61848 0.77152
SimplifiedLOF 52 0.50000 0.47541 0.51558 0.49175 0.63158 0.61346 0.78723
SimplifiedLOF 81 0.50000 0.47541 0.50629 0.48201 0.60000 0.58033 0.79372
LoOP 8 0.58333 0.56284 0.49797 0.47328 0.63636 0.61848 0.77220
LoOP 52 0.50000 0.47541 0.51720 0.49345 0.63158 0.61346 0.78654
LoOP 60 0.50000 0.47541 0.50659 0.48233 0.60000 0.58033 0.78962
LDOF 11 0.50000 0.47541 0.45787 0.43120 0.50000 0.47541 0.79679
LDOF 19 0.58333 0.56284 0.52652 0.50324 0.63636 0.61848 0.76503
LDOF 21 0.58333 0.56284 0.54631 0.52400 0.66667 0.65027 0.76913
ODIN 86 0.50000 0.47541 0.40463 0.37535 0.52632 0.50302 0.79901
ODIN 90 0.50000 0.47541 0.52422 0.50082 0.58824 0.56798 0.80260
ODIN 94 0.50000 0.47541 0.52330 0.49986 0.58824 0.56798 0.80430
ODIN 95 0.50000 0.47541 0.52549 0.50215 0.58824 0.56798 0.80311
FastABOD 4 0.50000 0.47541 0.42613 0.39791 0.50000 0.47541 0.74624
FastABOD 56 0.50000 0.47541 0.54081 0.51823 0.63158 0.61346 0.78620
FastABOD 86 0.50000 0.47541 0.55033 0.52821 0.66667 0.65027 0.77903
FastABOD 88 0.50000 0.47541 0.55069 0.52859 0.66667 0.65027 0.77971
KDEOS 11 0.33333 0.30055 0.25809 0.22160 0.38095 0.35051 0.75512
KDEOS 12 0.33333 0.30055 0.23311 0.19539 0.37037 0.33940 0.77835
KDEOS 81 0.33333 0.30055 0.18905 0.14917 0.40000 0.37049 0.70492
LDF 40 0.16667 0.12568 0.17763 0.13719 0.36364 0.33234 0.79611
LDF 48 0.41667 0.38798 0.21907 0.18066 0.41667 0.38798 0.77801
LDF 92 0.41667 0.38798 0.43867 0.41106 0.50000 0.47541 0.73702
INFLO 7 0.50000 0.47541 0.45471 0.42789 0.54545 0.52310 0.76878
INFLO 37 0.50000 0.47541 0.51656 0.49278 0.63158 0.61346 0.79235
INFLO 96 0.50000 0.47541 0.52736 0.50412 0.60000 0.58033 0.82992
COF 2 0.41667 0.38798 0.39823 0.36864 0.43478 0.40699 0.79645
COF 3 0.50000 0.47541 0.37405 0.34326 0.50000 0.47541 0.77732
COF 7 0.41667 0.38798 0.44173 0.41427 0.52632 0.50302 0.75751
COF 11 0.50000 0.47541 0.44890 0.42180 0.50000 0.47541 0.70184

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 256 objects, 12 outliers (4.69%)

Download raw algorithm results (2.3 MB) Download raw algorithm evaluation table (39.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.50000 0.47541 0.48308 0.45766 0.60000 0.58033 0.83128
KNN 5 0.58333 0.56284 0.47107 0.44506 0.60000 0.58033 0.81455
KNN 11 0.58333 0.56284 0.48455 0.45920 0.60870 0.58945 0.81079
KNN 14 0.58333 0.56284 0.48697 0.46173 0.60870 0.58945 0.82514
KNNW 1 0.50000 0.47541 0.50876 0.48461 0.56000 0.53836 0.79577
KNNW 2 0.50000 0.47541 0.48590 0.46062 0.60000 0.58033 0.81250
KNNW 30 0.50000 0.47541 0.47891 0.45328 0.60000 0.58033 0.82172
LOF 7 0.50000 0.47541 0.46195 0.43549 0.60000 0.58033 0.77493
LOF 34 0.58333 0.56284 0.45924 0.43264 0.60000 0.58033 0.82275
LOF 41 0.50000 0.47541 0.45602 0.42927 0.60000 0.58033 0.83094
SimplifiedLOF 30 0.58333 0.56284 0.45168 0.42472 0.60870 0.58945 0.82309
SimplifiedLOF 43 0.58333 0.56284 0.47404 0.44818 0.60000 0.58033 0.82684
SimplifiedLOF 47 0.58333 0.56284 0.47343 0.44754 0.60000 0.58033 0.82923
LoOP 27 0.58333 0.56284 0.44777 0.42062 0.60000 0.58033 0.81626
LoOP 34 0.58333 0.56284 0.45770 0.43103 0.63636 0.61848 0.82480
LoOP 44 0.58333 0.56284 0.47359 0.44771 0.60000 0.58033 0.82018
LoOP 47 0.58333 0.56284 0.46153 0.43505 0.60000 0.58033 0.82958
LDOF 24 0.50000 0.47541 0.44506 0.41777 0.60000 0.58033 0.81318
LDOF 47 0.50000 0.47541 0.45913 0.43253 0.60000 0.58033 0.82582
LDOF 48 0.58333 0.56284 0.46234 0.43590 0.60000 0.58033 0.82480
ODIN 34 0.41667 0.38798 0.39180 0.36189 0.51852 0.49484 0.83624
ODIN 56 0.52778 0.50455 0.43379 0.40595 0.56000 0.53836 0.82958
ODIN 96 0.58333 0.56284 0.40101 0.37155 0.58333 0.56284 0.82633
FastABOD 4 0.41667 0.38798 0.46983 0.44376 0.51852 0.49484 0.85143
FastABOD 15 0.50000 0.47541 0.48247 0.45702 0.60000 0.58033 0.79952
FastABOD 16 0.50000 0.47541 0.48316 0.45774 0.57143 0.55035 0.80225
FastABOD 29 0.58333 0.56284 0.45880 0.43218 0.58333 0.56284 0.80943
KDEOS 40 0.25000 0.21311 0.26164 0.22533 0.28571 0.25059 0.77869
KDEOS 66 0.33333 0.30055 0.18699 0.14701 0.34146 0.30908 0.79918
KDEOS 87 0.16667 0.12568 0.19660 0.15709 0.42424 0.39593 0.81113
KDEOS 99 0.16667 0.12568 0.21667 0.17815 0.40000 0.37049 0.81899
LDF 3 0.41667 0.38798 0.25255 0.21579 0.41667 0.38798 0.81404
LDF 58 0.41667 0.38798 0.36049 0.32903 0.55556 0.53370 0.79781
LDF 71 0.33333 0.30055 0.41585 0.38712 0.50000 0.47541 0.76469
INFLO 31 0.50000 0.47541 0.43527 0.40750 0.60000 0.58033 0.84324
INFLO 35 0.58333 0.56284 0.45083 0.42382 0.60870 0.58945 0.81182
INFLO 46 0.58333 0.56284 0.46390 0.43754 0.60870 0.58945 0.81113
COF 5 0.58333 0.56284 0.51480 0.49094 0.63636 0.61848 0.77322
COF 40 0.50000 0.47541 0.43760 0.40994 0.57143 0.55035 0.83846

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO