Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (20% of outliers version#04)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 305 objects, 61 outliers (20.00%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (53.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 11 0.44262 0.30328 0.48106 0.35133 0.46409 0.33011 0.73011
KNN 15 0.45902 0.32377 0.48262 0.35327 0.45902 0.32377 0.72655
KNN 75 0.42623 0.28279 0.48771 0.35964 0.47312 0.34140 0.72595
KNN 85 0.44262 0.30328 0.49055 0.36318 0.46073 0.32592 0.72719
KNNW 2 0.44262 0.30328 0.46597 0.33247 0.45405 0.31757 0.71809
KNNW 81 0.44262 0.30328 0.48582 0.35728 0.46809 0.33511 0.72816
KNNW 96 0.44262 0.30328 0.48508 0.35636 0.47059 0.33824 0.72743
LOF 7 0.44262 0.30328 0.45881 0.32351 0.45509 0.31886 0.72373
LOF 12 0.42623 0.28279 0.47556 0.34444 0.46835 0.33544 0.72951
LOF 59 0.44262 0.30328 0.48378 0.35473 0.47253 0.34066 0.72648
LOF 69 0.42623 0.28279 0.48258 0.35323 0.47778 0.34722 0.72682
SimplifiedLOF 47 0.45902 0.32377 0.47411 0.34264 0.45902 0.32377 0.72313
SimplifiedLOF 80 0.45902 0.32377 0.48559 0.35698 0.48837 0.36047 0.72904
SimplifiedLOF 85 0.44262 0.30328 0.48591 0.35739 0.48837 0.36047 0.73038
SimplifiedLOF 91 0.44262 0.30328 0.48594 0.35743 0.48276 0.35345 0.72944
LoOP 26 0.42623 0.28279 0.47719 0.34649 0.46897 0.33621 0.73042
LoOP 47 0.45902 0.32377 0.47299 0.34124 0.45902 0.32377 0.72353
LoOP 76 0.45902 0.32377 0.48070 0.35088 0.49123 0.36404 0.72642
LoOP 85 0.45902 0.32377 0.48227 0.35283 0.48837 0.36047 0.72769
LDOF 12 0.45902 0.32377 0.47108 0.33885 0.47154 0.33943 0.73361
LDOF 25 0.39344 0.24180 0.47686 0.34608 0.48780 0.35976 0.73347
LDOF 80 0.44262 0.30328 0.48342 0.35428 0.47727 0.34659 0.73300
LDOF 82 0.44262 0.30328 0.48163 0.35203 0.47674 0.34593 0.73394
ODIN 64 0.44262 0.30328 0.38987 0.23733 0.47126 0.33908 0.72192
ODIN 66 0.44262 0.30328 0.39633 0.24541 0.47674 0.34593 0.72218
ODIN 96 0.42623 0.28279 0.43015 0.28769 0.47337 0.34172 0.72746
ODIN 100 0.42623 0.28279 0.43442 0.29303 0.46809 0.33511 0.72648
FastABOD 14 0.39344 0.24180 0.46505 0.33131 0.47337 0.34172 0.72145
FastABOD 41 0.44262 0.30328 0.46971 0.33713 0.46341 0.32927 0.72736
FastABOD 48 0.44262 0.30328 0.47308 0.34135 0.46316 0.32895 0.72924
KDEOS 13 0.39344 0.24180 0.32387 0.15484 0.45055 0.31319 0.69592
KDEOS 99 0.36066 0.20082 0.31581 0.14476 0.47191 0.33989 0.69773
KDEOS 100 0.36066 0.20082 0.31448 0.14311 0.45902 0.32377 0.69800
LDF 45 0.37705 0.22131 0.44740 0.30924 0.47568 0.34459 0.73549
LDF 47 0.37705 0.22131 0.44340 0.30425 0.49162 0.36453 0.73327
LDF 65 0.44262 0.30328 0.44279 0.30348 0.44628 0.30785 0.71043
LDF 74 0.40984 0.26230 0.47875 0.34844 0.42991 0.28738 0.68275
INFLO 66 0.42623 0.28279 0.49063 0.36328 0.49123 0.36404 0.74513
INFLO 72 0.45902 0.32377 0.48749 0.35937 0.48837 0.36047 0.73095
INFLO 77 0.45902 0.32377 0.49151 0.36439 0.50000 0.37500 0.73619
INFLO 78 0.44262 0.30328 0.49273 0.36591 0.49704 0.37130 0.74395
COF 3 0.39344 0.24180 0.42440 0.28050 0.46575 0.33219 0.72726
COF 4 0.39344 0.24180 0.44646 0.30808 0.46939 0.33673 0.72182
COF 77 0.44262 0.30328 0.41365 0.26706 0.44262 0.30328 0.67388

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 305 objects, 61 outliers (20.00%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (54.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.40984 0.26230 0.44415 0.30519 0.45333 0.31667 0.70515
KNN 2 0.39344 0.24180 0.44481 0.30601 0.45556 0.31944 0.70499
KNN 15 0.39344 0.24180 0.42499 0.28124 0.45333 0.31667 0.71359
KNN 19 0.39344 0.24180 0.42212 0.27766 0.46053 0.32566 0.70932
KNNW 1 0.40984 0.26230 0.44578 0.30723 0.43946 0.29933 0.70495
KNNW 2 0.40984 0.26230 0.44762 0.30952 0.44604 0.30755 0.70640
KNNW 27 0.39344 0.24180 0.43594 0.29492 0.45752 0.32190 0.71191
KNNW 37 0.39344 0.24180 0.43315 0.29144 0.46053 0.32566 0.71103
LOF 11 0.44262 0.30328 0.43341 0.29176 0.45669 0.32087 0.69921
LOF 16 0.39344 0.24180 0.43851 0.29814 0.45802 0.32252 0.70754
LOF 17 0.37705 0.22131 0.43639 0.29549 0.46377 0.32971 0.70902
LOF 20 0.39344 0.24180 0.43160 0.28949 0.47407 0.34259 0.70828
SimplifiedLOF 5 0.34426 0.18033 0.43195 0.28994 0.47853 0.34816 0.71997
SimplifiedLOF 13 0.47541 0.34426 0.44144 0.30180 0.47541 0.34426 0.71520
SimplifiedLOF 23 0.42623 0.28279 0.45321 0.31651 0.46512 0.33140 0.72467
SimplifiedLOF 24 0.42623 0.28279 0.45424 0.31780 0.46715 0.33394 0.72400
LoOP 5 0.34426 0.18033 0.41300 0.26624 0.47205 0.34006 0.71950
LoOP 14 0.45902 0.32377 0.43223 0.29029 0.45902 0.32377 0.71926
LoOP 23 0.42623 0.28279 0.44094 0.30118 0.46809 0.33511 0.72333
LoOP 24 0.42623 0.28279 0.44002 0.30003 0.46897 0.33621 0.72336
LDOF 12 0.39344 0.24180 0.42351 0.27939 0.50340 0.37925 0.74241
LDOF 24 0.44262 0.30328 0.43905 0.29881 0.48120 0.35150 0.72904
LDOF 42 0.42623 0.28279 0.44654 0.30817 0.45926 0.32407 0.71540
ODIN 20 0.46721 0.33402 0.40317 0.25396 0.48322 0.35403 0.72366
ODIN 22 0.44858 0.31073 0.41447 0.26809 0.49635 0.37044 0.72608
ODIN 57 0.39672 0.24590 0.44855 0.31068 0.47482 0.34353 0.71748
FastABOD 6 0.44262 0.30328 0.40039 0.25049 0.48175 0.35219 0.69303
FastABOD 10 0.40984 0.26230 0.41413 0.26766 0.49275 0.36594 0.71856
FastABOD 16 0.42623 0.28279 0.42267 0.27834 0.46667 0.33333 0.72373
FastABOD 72 0.42623 0.28279 0.42551 0.28189 0.45679 0.32099 0.71473
KDEOS 13 0.29508 0.11885 0.37228 0.21535 0.44156 0.30195 0.69007
KDEOS 58 0.44262 0.30328 0.32673 0.15841 0.45528 0.31911 0.70069
KDEOS 88 0.40984 0.26230 0.35657 0.19571 0.46897 0.33621 0.71224
KDEOS 93 0.36066 0.20082 0.36477 0.20596 0.45752 0.32190 0.71352
LDF 8 0.37705 0.22131 0.33207 0.16509 0.41600 0.27000 0.64845
LDF 9 0.32787 0.15984 0.35515 0.19394 0.39604 0.24505 0.65292
LDF 12 0.36066 0.20082 0.39763 0.24704 0.39344 0.24180 0.63780
LDF 80 0.24590 0.05738 0.30232 0.12790 0.43137 0.28922 0.63733
INFLO 4 0.39344 0.24180 0.41563 0.26954 0.46667 0.33333 0.73858
INFLO 17 0.44262 0.30328 0.42988 0.28735 0.46575 0.33219 0.71668
INFLO 30 0.40984 0.26230 0.44531 0.30663 0.46591 0.33239 0.73737
INFLO 98 0.37705 0.22131 0.43008 0.28760 0.50000 0.37500 0.73240
COF 3 0.36066 0.20082 0.39905 0.24881 0.46316 0.32895 0.70952
COF 4 0.37705 0.22131 0.42845 0.28557 0.48052 0.35065 0.70787
COF 5 0.34426 0.18033 0.43519 0.29398 0.47297 0.34122 0.70532
COF 13 0.39344 0.24180 0.41132 0.26415 0.41791 0.27239 0.67173

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO