Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (10% of outliers version#02)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (49.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.40741 0.34183 0.42889 0.36570 0.40741 0.34183 0.78552
KNN 53 0.40741 0.34183 0.48527 0.42831 0.47368 0.41544 0.78977
KNN 59 0.40741 0.34183 0.48601 0.42914 0.47368 0.41544 0.79174
KNN 99 0.40741 0.34183 0.48357 0.42643 0.46154 0.40195 0.79463
KNNW 4 0.40741 0.34183 0.43418 0.37157 0.43077 0.36778 0.78081
KNNW 30 0.37037 0.30070 0.46789 0.40901 0.45455 0.39419 0.78248
KNNW 86 0.40741 0.34183 0.47815 0.42040 0.45000 0.38914 0.78734
KNNW 89 0.40741 0.34183 0.47838 0.42066 0.45000 0.38914 0.78734
LOF 55 0.40741 0.34183 0.44299 0.38136 0.44444 0.38297 0.78081
LOF 75 0.40741 0.34183 0.46079 0.40112 0.46512 0.40593 0.78901
LOF 91 0.40741 0.34183 0.47193 0.41349 0.45455 0.39419 0.79205
LOF 97 0.40741 0.34183 0.47118 0.41266 0.45455 0.39419 0.79341
SimplifiedLOF 3 0.40741 0.34183 0.36568 0.29548 0.40741 0.34183 0.76730
SimplifiedLOF 65 0.37037 0.30070 0.42936 0.36622 0.46575 0.40664 0.78051
SimplifiedLOF 96 0.40741 0.34183 0.46389 0.40457 0.45714 0.39707 0.78825
SimplifiedLOF 100 0.40741 0.34183 0.46349 0.40412 0.45455 0.39419 0.78901
LoOP 3 0.40741 0.34183 0.36044 0.28967 0.40741 0.34183 0.76472
LoOP 67 0.37037 0.30070 0.42557 0.36201 0.47222 0.41382 0.77778
LoOP 100 0.40741 0.34183 0.45499 0.39468 0.45455 0.39419 0.78567
LDOF 11 0.29630 0.21843 0.39131 0.32396 0.41860 0.35427 0.79083
LDOF 24 0.40741 0.34183 0.39646 0.32968 0.40964 0.34431 0.77307
LDOF 71 0.33333 0.25956 0.41969 0.35548 0.47887 0.42121 0.78036
LDOF 100 0.40741 0.34183 0.43406 0.37143 0.45946 0.39965 0.78871
ODIN 74 0.42857 0.36534 0.37779 0.30894 0.46429 0.40501 0.78362
ODIN 77 0.44444 0.38297 0.38413 0.31598 0.45614 0.39596 0.78552
ODIN 96 0.37037 0.30070 0.41144 0.34631 0.42857 0.36534 0.78764
ODIN 100 0.37037 0.30070 0.40755 0.34199 0.42500 0.36137 0.78863
FastABOD 20 0.33333 0.25956 0.39833 0.33175 0.40000 0.33361 0.78157
FastABOD 23 0.37037 0.30070 0.38763 0.31987 0.40000 0.33361 0.77413
FastABOD 54 0.33333 0.25956 0.40102 0.33474 0.42424 0.36053 0.77171
FastABOD 71 0.37037 0.30070 0.41791 0.35350 0.39437 0.32735 0.77247
KDEOS 14 0.25926 0.17729 0.22559 0.13989 0.39130 0.32395 0.77019
KDEOS 15 0.29630 0.21843 0.22465 0.13885 0.39474 0.32776 0.76336
KDEOS 16 0.29630 0.21843 0.22170 0.13557 0.40000 0.33361 0.75987
LDF 31 0.51852 0.46524 0.34228 0.26950 0.52830 0.47611 0.78400
LDF 39 0.40741 0.34183 0.35161 0.27986 0.44068 0.37879 0.80039
LDF 50 0.44444 0.38297 0.38839 0.32071 0.53333 0.48169 0.76063
LDF 95 0.37037 0.30070 0.42130 0.35726 0.43243 0.36963 0.77596
INFLO 45 0.33333 0.25956 0.43570 0.37326 0.47761 0.41981 0.79903
INFLO 80 0.40741 0.34183 0.46129 0.40168 0.44118 0.37934 0.79311
INFLO 97 0.40741 0.34183 0.47356 0.41531 0.44444 0.38297 0.80927
INFLO 100 0.40741 0.34183 0.47566 0.41764 0.44444 0.38297 0.80146
COF 3 0.37037 0.30070 0.35095 0.27913 0.39216 0.32490 0.77110
COF 4 0.33333 0.25956 0.39316 0.32601 0.40000 0.33361 0.76503
COF 33 0.33333 0.25956 0.32836 0.25403 0.41860 0.35427 0.73512

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (48.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.33333 0.25956 0.42115 0.35710 0.43590 0.37348 0.79561
KNN 4 0.33333 0.25956 0.40756 0.34201 0.44776 0.38665 0.79250
KNN 20 0.37037 0.30070 0.38117 0.31270 0.40449 0.33860 0.79554
KNN 51 0.37037 0.30070 0.38822 0.32052 0.40816 0.34267 0.80404
KNNW 5 0.33333 0.25956 0.41222 0.34717 0.44444 0.38297 0.79129
KNNW 6 0.33333 0.25956 0.41213 0.34708 0.45161 0.39093 0.79114
KNNW 32 0.37037 0.30070 0.38875 0.32111 0.40000 0.33361 0.79569
KNNW 96 0.37037 0.30070 0.38841 0.32074 0.40816 0.34267 0.80176
LOF 18 0.40741 0.34183 0.36486 0.29458 0.41176 0.34667 0.78446
LOF 26 0.37037 0.30070 0.37405 0.30478 0.44776 0.38665 0.77869
LOF 76 0.37037 0.30070 0.38001 0.31141 0.40000 0.33361 0.79751
LOF 92 0.37037 0.30070 0.37908 0.31037 0.42105 0.35699 0.80009
SimplifiedLOF 5 0.29630 0.21843 0.39907 0.33257 0.38596 0.31802 0.76260
SimplifiedLOF 17 0.40741 0.34183 0.38221 0.31385 0.41509 0.35037 0.80206
SimplifiedLOF 53 0.37037 0.30070 0.38227 0.31391 0.44776 0.38665 0.79614
SimplifiedLOF 77 0.37037 0.30070 0.39133 0.32398 0.41791 0.35350 0.80358
LoOP 17 0.40741 0.34183 0.37962 0.31097 0.41509 0.35037 0.80191
LoOP 22 0.40741 0.34183 0.39423 0.32720 0.43137 0.36845 0.80601
LoOP 23 0.40741 0.34183 0.39502 0.32807 0.41791 0.35350 0.80540
LoOP 58 0.37037 0.30070 0.38785 0.32011 0.45455 0.39419 0.80070
LDOF 12 0.40741 0.34183 0.38488 0.31681 0.41667 0.35212 0.79174
LDOF 17 0.37037 0.30070 0.39978 0.33336 0.43478 0.37224 0.82605
LDOF 20 0.37037 0.30070 0.40491 0.33906 0.46512 0.40593 0.82165
LDOF 22 0.37037 0.30070 0.41218 0.34713 0.46512 0.40593 0.82225
ODIN 29 0.39506 0.32812 0.40844 0.34298 0.47619 0.41823 0.80760
ODIN 41 0.43827 0.37611 0.38961 0.32207 0.44898 0.38801 0.80594
ODIN 52 0.38889 0.32127 0.41413 0.34930 0.47619 0.41823 0.80829
ODIN 66 0.37037 0.30070 0.40703 0.34141 0.45455 0.39419 0.81299
FastABOD 4 0.48148 0.42410 0.44694 0.38574 0.50000 0.44467 0.79766
FastABOD 16 0.40741 0.34183 0.41505 0.35033 0.47619 0.41823 0.80707
KDEOS 10 0.33333 0.25956 0.36864 0.29878 0.36842 0.29853 0.75683
KDEOS 81 0.29630 0.21843 0.27151 0.19090 0.44156 0.37976 0.78886
KDEOS 90 0.37037 0.30070 0.27473 0.19447 0.43836 0.37621 0.79098
KDEOS 94 0.33333 0.25956 0.28028 0.20064 0.43836 0.37621 0.79296
LDF 4 0.25926 0.17729 0.25312 0.17047 0.35052 0.27865 0.74287
LDF 6 0.29630 0.21843 0.30445 0.22748 0.38095 0.31245 0.73824
LDF 9 0.33333 0.25956 0.32480 0.25009 0.38095 0.31245 0.67896
LDF 11 0.25926 0.17729 0.34647 0.27415 0.36842 0.29853 0.68291
INFLO 32 0.40741 0.34183 0.39282 0.32563 0.44898 0.38801 0.81527
INFLO 48 0.44444 0.38297 0.39453 0.32754 0.44444 0.38297 0.81740
INFLO 55 0.40741 0.34183 0.40098 0.33469 0.41860 0.35427 0.82256
INFLO 77 0.40741 0.34183 0.39313 0.32598 0.40741 0.34183 0.82483
COF 5 0.37037 0.30070 0.42848 0.36523 0.45455 0.39419 0.74879
COF 8 0.40741 0.34183 0.37351 0.30418 0.42105 0.35699 0.74605
COF 11 0.33333 0.25956 0.36494 0.29467 0.46154 0.40195 0.75956
COF 12 0.33333 0.25956 0.38033 0.31176 0.42857 0.36534 0.76472

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO