Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (10% of outliers version#06)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (47.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.44444 0.38297 0.48918 0.43265 0.48387 0.42676 0.77573
KNN 31 0.40741 0.34183 0.51033 0.45614 0.47619 0.41823 0.78597
KNN 69 0.40741 0.34183 0.52095 0.46794 0.52459 0.47198 0.78279
KNNW 1 0.33333 0.25956 0.48937 0.43286 0.45902 0.39915 0.78620
KNNW 2 0.40741 0.34183 0.48720 0.43046 0.48387 0.42676 0.78112
KNNW 87 0.40741 0.34183 0.51102 0.45692 0.47619 0.41823 0.78309
LOF 2 0.44444 0.38297 0.43366 0.37099 0.44444 0.38297 0.75804
LOF 35 0.40741 0.34183 0.49539 0.43955 0.48387 0.42676 0.78855
LOF 37 0.40741 0.34183 0.49696 0.44129 0.49180 0.43557 0.78810
LOF 99 0.40741 0.34183 0.50501 0.45024 0.48387 0.42676 0.77626
SimplifiedLOF 3 0.48148 0.42410 0.49660 0.44090 0.50000 0.44467 0.77838
SimplifiedLOF 38 0.40741 0.34183 0.50437 0.44953 0.46875 0.40996 0.79432
LoOP 3 0.48148 0.42410 0.49074 0.43439 0.48148 0.42410 0.77527
LoOP 45 0.44444 0.38297 0.50223 0.44715 0.46875 0.40996 0.78916
LoOP 94 0.44444 0.38297 0.49489 0.43900 0.50000 0.44467 0.78613
LDOF 6 0.48148 0.42410 0.44409 0.38257 0.49057 0.43419 0.77535
LDOF 43 0.37037 0.30070 0.49744 0.44183 0.45946 0.39965 0.79432
ODIN 76 0.40741 0.34183 0.38534 0.31733 0.47826 0.42053 0.78430
ODIN 81 0.43210 0.36926 0.40376 0.33778 0.46667 0.40765 0.78461
ODIN 96 0.46296 0.40354 0.38400 0.31584 0.47273 0.41438 0.78575
FastABOD 10 0.44444 0.38297 0.48283 0.42561 0.47273 0.41438 0.77854
FastABOD 20 0.44444 0.38297 0.48716 0.43041 0.50000 0.44467 0.78931
FastABOD 23 0.40741 0.34183 0.49551 0.43968 0.48780 0.43113 0.79463
FastABOD 33 0.44444 0.38297 0.48267 0.42542 0.45455 0.39419 0.79797
KDEOS 10 0.33333 0.25956 0.25016 0.16719 0.37895 0.31022 0.74681
KDEOS 11 0.37037 0.30070 0.24606 0.16263 0.38462 0.31652 0.73209
KDEOS 15 0.37037 0.30070 0.25146 0.16863 0.37975 0.31111 0.74241
KDEOS 22 0.22222 0.13616 0.21666 0.12998 0.41176 0.34667 0.72556
LDF 19 0.29630 0.21843 0.27986 0.20017 0.42667 0.36322 0.77064
LDF 90 0.44444 0.38297 0.48829 0.43167 0.51163 0.45759 0.74423
LDF 98 0.44444 0.38297 0.50143 0.44626 0.48780 0.43113 0.75288
LDF 100 0.48148 0.42410 0.49275 0.43662 0.48148 0.42410 0.75698
INFLO 3 0.48148 0.42410 0.46924 0.41051 0.50000 0.44467 0.77292
INFLO 41 0.44444 0.38297 0.51609 0.46254 0.48649 0.42966 0.80874
INFLO 46 0.44444 0.38297 0.51699 0.46354 0.50000 0.44467 0.79675
INFLO 49 0.40741 0.34183 0.50898 0.45465 0.50847 0.45408 0.79766
COF 2 0.44444 0.38297 0.46615 0.40708 0.45455 0.39419 0.75395
COF 3 0.44444 0.38297 0.48387 0.42676 0.50000 0.44467 0.77998

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 271 objects, 27 outliers (9.96%)

Download raw algorithm results (2.4 MB) Download raw algorithm evaluation table (47.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.40741 0.34183 0.46967 0.41098 0.48780 0.43113 0.77527
KNN 3 0.48148 0.42410 0.46761 0.40870 0.52459 0.47198 0.77876
KNN 58 0.48148 0.42410 0.44820 0.38714 0.54237 0.49173 0.79539
KNN 65 0.44444 0.38297 0.44547 0.38411 0.53333 0.48169 0.79584
KNNW 1 0.44444 0.38297 0.47371 0.41548 0.50847 0.45408 0.78430
KNNW 6 0.44444 0.38297 0.47718 0.41933 0.52941 0.47734 0.78172
KNNW 40 0.44444 0.38297 0.44307 0.38144 0.53968 0.48875 0.79144
KNNW 66 0.44444 0.38297 0.44385 0.38231 0.52459 0.47198 0.79250
LOF 1 0.48148 0.42410 0.47115 0.41263 0.48148 0.42410 0.70264
LOF 6 0.48148 0.42410 0.47441 0.41625 0.49231 0.43613 0.77110
LOF 86 0.44444 0.38297 0.44311 0.38149 0.54839 0.49841 0.78825
LOF 90 0.44444 0.38297 0.44239 0.38068 0.54839 0.49841 0.79053
SimplifiedLOF 3 0.51852 0.46524 0.45724 0.39718 0.52632 0.47390 0.74545
SimplifiedLOF 5 0.51852 0.46524 0.48740 0.43068 0.56140 0.51287 0.78522
SimplifiedLOF 95 0.44444 0.38297 0.44203 0.38029 0.52174 0.46882 0.79098
LoOP 5 0.51852 0.46524 0.46493 0.40572 0.51852 0.46524 0.78400
LoOP 12 0.44444 0.38297 0.47237 0.41398 0.49315 0.43706 0.76920
LoOP 81 0.40741 0.34183 0.43880 0.37670 0.52174 0.46882 0.79265
LoOP 85 0.40741 0.34183 0.43769 0.37547 0.51429 0.46054 0.79349
LDOF 5 0.44444 0.38297 0.43513 0.37263 0.51064 0.45649 0.80480
LDOF 10 0.48148 0.42410 0.46177 0.40221 0.48148 0.42410 0.78021
LDOF 13 0.48148 0.42410 0.47987 0.42231 0.49123 0.43493 0.78127
LDOF 62 0.44444 0.38297 0.45771 0.39770 0.54237 0.49173 0.78597
ODIN 40 0.44444 0.38297 0.43001 0.36693 0.48571 0.42881 0.78954
ODIN 63 0.43210 0.36926 0.46359 0.40423 0.49275 0.43662 0.79311
ODIN 68 0.44444 0.38297 0.45455 0.39419 0.50746 0.45296 0.79394
ODIN 99 0.40741 0.34183 0.42621 0.36272 0.51515 0.46150 0.78734
FastABOD 4 0.44444 0.38297 0.42492 0.36129 0.46154 0.40195 0.80237
FastABOD 16 0.48148 0.42410 0.44261 0.38093 0.49123 0.43493 0.78597
FastABOD 35 0.48148 0.42410 0.45518 0.39489 0.50909 0.45477 0.79174
FastABOD 58 0.44444 0.38297 0.44433 0.38284 0.51064 0.45649 0.79022
KDEOS 11 0.37037 0.30070 0.36552 0.29531 0.39130 0.32395 0.72511
KDEOS 14 0.40741 0.34183 0.34789 0.27573 0.40741 0.34183 0.71448
KDEOS 76 0.18519 0.09502 0.22036 0.13409 0.42000 0.35582 0.74165
KDEOS 100 0.25926 0.17729 0.24827 0.16508 0.41379 0.34893 0.75030
LDF 15 0.33333 0.25956 0.38293 0.31464 0.40678 0.34114 0.73816
LDF 42 0.29630 0.21843 0.34312 0.27043 0.36111 0.29041 0.74909
INFLO 4 0.48148 0.42410 0.41991 0.35572 0.49057 0.43419 0.79417
INFLO 5 0.48148 0.42410 0.43636 0.37399 0.49123 0.43493 0.81390
INFLO 66 0.48148 0.42410 0.46241 0.40293 0.51613 0.46259 0.80578
INFLO 91 0.40741 0.34183 0.44602 0.38472 0.54237 0.49173 0.79136
COF 3 0.48148 0.42410 0.46704 0.40807 0.54167 0.49095 0.76609
COF 5 0.44444 0.38297 0.44208 0.38034 0.49180 0.43557 0.77034

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO