Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (10% of outliers version#05)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (48.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.37037 0.30070 0.42965 0.36654 0.40816 0.34267 0.78226
KNN 5 0.37037 0.30070 0.44913 0.38818 0.42105 0.35699 0.79690
KNN 34 0.33333 0.25956 0.45985 0.40008 0.41860 0.35427 0.78947
KNN 56 0.33333 0.25956 0.45838 0.39844 0.43902 0.37695 0.78825
KNNW 4 0.37037 0.30070 0.43163 0.36874 0.40580 0.34005 0.78947
KNNW 7 0.37037 0.30070 0.43965 0.37764 0.40964 0.34431 0.79296
KNNW 72 0.33333 0.25956 0.45338 0.39289 0.42857 0.36534 0.78901
KNNW 80 0.33333 0.25956 0.45509 0.39479 0.42857 0.36534 0.78962
LOF 39 0.33333 0.25956 0.40805 0.34254 0.39474 0.32776 0.78886
LOF 48 0.33333 0.25956 0.40470 0.33883 0.41975 0.35555 0.78597
LOF 62 0.37037 0.30070 0.42406 0.36033 0.39130 0.32395 0.78355
LOF 100 0.37037 0.30070 0.44036 0.37844 0.40909 0.34370 0.77672
SimplifiedLOF 85 0.33333 0.25956 0.42337 0.35957 0.42500 0.36137 0.78810
SimplifiedLOF 87 0.37037 0.30070 0.42799 0.36469 0.41026 0.34500 0.78855
SimplifiedLOF 96 0.37037 0.30070 0.43341 0.37071 0.41558 0.35092 0.78871
SimplifiedLOF 99 0.37037 0.30070 0.43320 0.37048 0.41558 0.35092 0.78901
LoOP 83 0.33333 0.25956 0.41490 0.35015 0.41463 0.34986 0.78719
LoOP 95 0.37037 0.30070 0.42556 0.36199 0.41463 0.34986 0.78248
LoOP 98 0.37037 0.30070 0.43150 0.36860 0.41026 0.34500 0.78264
LDOF 11 0.37037 0.30070 0.35725 0.28613 0.37037 0.30070 0.70871
LDOF 81 0.33333 0.25956 0.39419 0.32716 0.38710 0.31928 0.78112
LDOF 96 0.33333 0.25956 0.40009 0.33371 0.40000 0.33361 0.77930
LDOF 98 0.33333 0.25956 0.40681 0.34117 0.40000 0.33361 0.77960
ODIN 83 0.33333 0.25956 0.30460 0.22765 0.40476 0.33890 0.77247
ODIN 93 0.35185 0.28013 0.33997 0.26694 0.39535 0.32844 0.77512
ODIN 98 0.33333 0.25956 0.34103 0.26811 0.39506 0.32812 0.77580
ODIN 100 0.34568 0.27327 0.34354 0.27090 0.40476 0.33890 0.77543
FastABOD 3 0.37037 0.30070 0.35426 0.28281 0.44444 0.38297 0.68807
FastABOD 10 0.40741 0.34183 0.40335 0.33733 0.41667 0.35212 0.77672
FastABOD 44 0.33333 0.25956 0.41532 0.35062 0.40909 0.34370 0.78704
FastABOD 78 0.33333 0.25956 0.42480 0.36115 0.39130 0.32395 0.78218
KDEOS 14 0.22222 0.13616 0.18348 0.09313 0.29268 0.21441 0.67577
KDEOS 31 0.18519 0.09502 0.18682 0.09684 0.34884 0.27678 0.72738
LDF 45 0.40741 0.34183 0.35533 0.28400 0.47761 0.41981 0.79994
LDF 81 0.44444 0.38297 0.46228 0.40278 0.46512 0.40593 0.78036
LDF 86 0.37037 0.30070 0.47829 0.42056 0.47619 0.41823 0.76594
LDF 94 0.40741 0.34183 0.47745 0.41963 0.48780 0.43113 0.73816
INFLO 43 0.37037 0.30070 0.39479 0.32782 0.40000 0.33361 0.78370
INFLO 46 0.33333 0.25956 0.40124 0.33498 0.42254 0.35864 0.78916
INFLO 98 0.37037 0.30070 0.43374 0.37108 0.40816 0.34267 0.80419
INFLO 100 0.37037 0.30070 0.43425 0.37164 0.40541 0.33961 0.78977
COF 4 0.33333 0.25956 0.37674 0.30777 0.40000 0.33361 0.74226
COF 13 0.44444 0.38297 0.35910 0.28818 0.44828 0.38722 0.79706

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (47.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.37037 0.30070 0.34058 0.26761 0.39024 0.32277 0.75121
KNN 7 0.37037 0.30070 0.33750 0.26419 0.40000 0.33361 0.75767
KNNW 1 0.37037 0.30070 0.36569 0.29549 0.41026 0.34500 0.76116
LOF 2 0.33333 0.25956 0.34984 0.27790 0.39024 0.32277 0.74347
LOF 5 0.33333 0.25956 0.37226 0.30280 0.39130 0.32395 0.76336
LOF 11 0.29630 0.21843 0.33233 0.25845 0.41026 0.34500 0.76897
LOF 12 0.29630 0.21843 0.33431 0.26065 0.40000 0.33361 0.77049
SimplifiedLOF 4 0.33333 0.25956 0.35494 0.28356 0.40000 0.33361 0.77672
SimplifiedLOF 5 0.37037 0.30070 0.38836 0.32068 0.40000 0.33361 0.77520
SimplifiedLOF 64 0.33333 0.25956 0.34377 0.27116 0.42105 0.35699 0.75258
LoOP 4 0.33333 0.25956 0.35329 0.28173 0.38298 0.31470 0.77292
LoOP 5 0.37037 0.30070 0.36331 0.29286 0.40816 0.34267 0.77080
LoOP 20 0.33333 0.25956 0.36997 0.30026 0.42105 0.35699 0.75896
LDOF 5 0.37037 0.30070 0.34595 0.27358 0.40580 0.34005 0.77960
LDOF 6 0.37037 0.30070 0.33213 0.25823 0.40860 0.34316 0.78931
LDOF 23 0.33333 0.25956 0.38161 0.31318 0.41026 0.34500 0.76791
LDOF 58 0.33333 0.25956 0.35577 0.28448 0.43243 0.36963 0.76078
ODIN 10 0.27160 0.19100 0.24207 0.15820 0.41463 0.34986 0.74985
ODIN 12 0.30159 0.22430 0.25459 0.17211 0.39560 0.32872 0.76488
ODIN 41 0.37654 0.30755 0.31589 0.24019 0.38095 0.31245 0.75789
ODIN 69 0.30864 0.23214 0.32569 0.25107 0.38095 0.31245 0.75129
FastABOD 4 0.37037 0.30070 0.38931 0.32174 0.43373 0.37107 0.78704
FastABOD 5 0.37037 0.30070 0.38321 0.31496 0.43750 0.37526 0.78689
FastABOD 6 0.37037 0.30070 0.37510 0.30595 0.40000 0.33361 0.79114
KDEOS 11 0.29630 0.21843 0.19917 0.11055 0.32075 0.24559 0.70719
KDEOS 29 0.18519 0.09502 0.24081 0.15680 0.34426 0.27170 0.70993
KDEOS 46 0.22222 0.13616 0.21219 0.12501 0.37975 0.31111 0.73391
KDEOS 47 0.22222 0.13616 0.20653 0.11873 0.38554 0.31755 0.73315
LDF 2 0.22222 0.13616 0.18976 0.10010 0.33898 0.26584 0.67335
LDF 3 0.25926 0.17729 0.17015 0.07832 0.31034 0.23403 0.65369
LDF 15 0.22222 0.13616 0.28187 0.20240 0.26087 0.17908 0.61217
LDF 98 0.18519 0.09502 0.27641 0.19634 0.30769 0.23108 0.70811
INFLO 4 0.33333 0.25956 0.37649 0.30750 0.40476 0.33890 0.79645
INFLO 9 0.37037 0.30070 0.32746 0.25304 0.37037 0.30070 0.74833
INFLO 52 0.29630 0.21843 0.35539 0.28406 0.43243 0.36963 0.76336
COF 2 0.33333 0.25956 0.33800 0.26474 0.36735 0.29734 0.76017
COF 5 0.40741 0.34183 0.40823 0.34274 0.40741 0.34183 0.74636
COF 7 0.33333 0.25956 0.34172 0.26888 0.41860 0.35427 0.71023

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO