Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (2% of outliers version#10)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (38.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.01639 0.04427 0.02860 0.10714 0.09251 0.76025
KNN 26 0.00000 -0.01639 0.04965 0.03408 0.12766 0.11336 0.77971
KNN 95 0.00000 -0.01639 0.06372 0.04837 0.16667 0.15301 0.76230
KNN 99 0.00000 -0.01639 0.06395 0.04860 0.16667 0.15301 0.76230
KNNW 1 0.00000 -0.01639 0.03461 0.01879 0.08511 0.07011 0.66701
KNNW 20 0.00000 -0.01639 0.04311 0.02742 0.12245 0.10806 0.74795
KNNW 85 0.00000 -0.01639 0.04894 0.03335 0.12000 0.10557 0.76434
LOF 1 0.00000 -0.01639 0.02551 0.00953 0.08696 0.07199 0.43084
LOF 38 0.00000 -0.01639 0.04769 0.03208 0.12121 0.10681 0.76844
LOF 44 0.00000 -0.01639 0.04831 0.03271 0.11429 0.09977 0.77152
LOF 96 0.00000 -0.01639 0.04935 0.03376 0.11765 0.10318 0.76639
SimplifiedLOF 1 0.00000 -0.01639 0.02628 0.01032 0.07143 0.05621 0.54098
SimplifiedLOF 45 0.00000 -0.01639 0.05024 0.03467 0.11765 0.10318 0.76332
SimplifiedLOF 49 0.00000 -0.01639 0.05113 0.03557 0.13333 0.11913 0.76025
LoOP 1 0.00000 -0.01639 0.02628 0.01032 0.07143 0.05621 0.54098
LoOP 37 0.00000 -0.01639 0.05099 0.03544 0.10526 0.09060 0.76947
LoOP 44 0.00000 -0.01639 0.05291 0.03738 0.12121 0.10681 0.76947
LoOP 50 0.00000 -0.01639 0.05154 0.03599 0.13793 0.12380 0.76127
LDOF 2 0.00000 -0.01639 0.02118 0.00514 0.04372 0.02804 0.52357
LDOF 44 0.00000 -0.01639 0.06409 0.04875 0.16000 0.14623 0.77357
LDOF 50 0.00000 -0.01639 0.06662 0.05132 0.18182 0.16841 0.76537
ODIN 3 0.03896 0.02321 0.03735 0.02157 0.07407 0.05889 0.78740
ODIN 25 0.06250 0.04713 0.05529 0.03980 0.12245 0.10806 0.75666
ODIN 30 0.06250 0.04713 0.05919 0.04376 0.13636 0.12221 0.76076
FastABOD 3 0.00000 -0.01639 0.04447 0.02880 0.12500 0.11066 0.66086
FastABOD 5 0.00000 -0.01639 0.05202 0.03648 0.11765 0.10318 0.80020
FastABOD 19 0.00000 -0.01639 0.05162 0.03607 0.14286 0.12881 0.77459
KDEOS 3 0.00000 -0.01639 0.07410 0.05893 0.18182 0.16841 0.82172
KDEOS 48 0.25000 0.23770 0.08855 0.07361 0.25000 0.23770 0.72439
KDEOS 84 0.25000 0.23770 0.10807 0.09345 0.28571 0.27400 0.70697
LDF 2 0.25000 0.23770 0.08309 0.06806 0.25000 0.23770 0.55328
LDF 16 0.00000 -0.01639 0.10733 0.09270 0.26087 0.24875 0.89959
LDF 62 0.00000 -0.01639 0.09586 0.08104 0.28571 0.27400 0.78176
LDF 90 0.25000 0.23770 0.12559 0.11126 0.28571 0.27400 0.81557
INFLO 1 0.00000 -0.01639 0.05628 0.04081 0.15385 0.13997 0.70594
INFLO 26 0.00000 -0.01639 0.06073 0.04533 0.12500 0.11066 0.79508
INFLO 37 0.00000 -0.01639 0.05657 0.04111 0.15385 0.13997 0.77664
INFLO 100 0.00000 -0.01639 0.06629 0.05098 0.15385 0.13997 0.78279
COF 1 0.00000 -0.01639 0.02628 0.01032 0.07143 0.05621 0.54098
COF 69 0.00000 -0.01639 0.04349 0.02781 0.12500 0.11066 0.71209
COF 82 0.00000 -0.01639 0.04544 0.02980 0.10811 0.09349 0.73975

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (40.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 33 0.00000 -0.01639 0.05716 0.04171 0.18182 0.16841 0.67828
KNN 36 0.25000 0.23770 0.08314 0.06811 0.25000 0.23770 0.67213
KNN 43 0.25000 0.23770 0.10285 0.08814 0.28571 0.27400 0.65471
KNNW 56 0.00000 -0.01639 0.05140 0.03585 0.16667 0.15301 0.66086
KNNW 92 0.25000 0.23770 0.08163 0.06657 0.25000 0.23770 0.64754
LOF 1 0.00000 -0.01639 0.01966 0.00359 0.05000 0.03443 0.38115
LOF 41 0.00000 -0.01639 0.03356 0.01772 0.09091 0.07601 0.64139
LOF 100 0.00000 -0.01639 0.06777 0.05249 0.22222 0.20947 0.61475
SimplifiedLOF 1 0.00000 -0.01639 0.02182 0.00578 0.05479 0.03930 0.53637
SimplifiedLOF 55 0.00000 -0.01639 0.03648 0.02068 0.10526 0.09060 0.64447
SimplifiedLOF 89 0.00000 -0.01639 0.04120 0.02549 0.13333 0.11913 0.62602
LoOP 1 0.00000 -0.01639 0.02182 0.00578 0.05479 0.03930 0.53637
LoOP 55 0.00000 -0.01639 0.03793 0.02216 0.11111 0.09654 0.64857
LoOP 99 0.00000 -0.01639 0.04320 0.02751 0.14286 0.12881 0.61988
LDOF 2 0.00000 -0.01639 0.05048 0.03492 0.15385 0.13997 0.67418
LDOF 5 0.25000 0.23770 0.08188 0.06683 0.25000 0.23770 0.64652
ODIN 9 0.04762 0.03201 0.02884 0.01292 0.08000 0.06492 0.60092
ODIN 26 0.00000 -0.01639 0.03346 0.01762 0.09091 0.07601 0.63986
ODIN 76 0.00000 -0.01639 0.03394 0.01810 0.08696 0.07199 0.66291
FastABOD 3 0.00000 -0.01639 0.02675 0.01080 0.07273 0.05753 0.54303
FastABOD 5 0.00000 -0.01639 0.04543 0.02978 0.15385 0.13997 0.61578
FastABOD 10 0.00000 -0.01639 0.03824 0.02247 0.10000 0.08525 0.68238
KDEOS 4 0.25000 0.23770 0.14074 0.12666 0.33333 0.32240 0.55635
KDEOS 11 0.25000 0.23770 0.26859 0.25660 0.40000 0.39016 0.59631
KDEOS 75 0.00000 -0.01639 0.02983 0.01392 0.06667 0.05137 0.64242
LDF 18 0.25000 0.23770 0.09525 0.08042 0.25000 0.23770 0.72336
LDF 26 0.25000 0.23770 0.17484 0.16132 0.33333 0.32240 0.82582
LDF 68 0.25000 0.23770 0.27895 0.26713 0.40000 0.39016 0.74795
LDF 77 0.25000 0.23770 0.28699 0.27530 0.40000 0.39016 0.74283
INFLO 34 0.00000 -0.01639 0.04175 0.02604 0.11765 0.10318 0.68750
INFLO 80 0.25000 0.23770 0.08233 0.06729 0.25000 0.23770 0.66291
COF 1 0.00000 -0.01639 0.02182 0.00578 0.05479 0.03930 0.53637
COF 79 0.00000 -0.01639 0.05432 0.03882 0.12903 0.11475 0.69570
COF 96 0.00000 -0.01639 0.07755 0.06243 0.20000 0.18689 0.67008

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO