Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (20% of outliers version#01)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 305 objects, 61 outliers (20.00%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (54.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.44262 0.30328 0.50661 0.38326 0.47244 0.34055 0.72837
KNN 19 0.42623 0.28279 0.51625 0.39531 0.46479 0.33099 0.72171
KNN 31 0.42623 0.28279 0.51443 0.39303 0.47945 0.34932 0.71325
KNNW 2 0.45902 0.32377 0.50711 0.38388 0.47154 0.33943 0.72601
KNNW 3 0.47541 0.34426 0.51008 0.38760 0.47541 0.34426 0.72481
KNNW 22 0.42623 0.28279 0.51602 0.39502 0.46377 0.32971 0.72407
LOF 3 0.45902 0.32377 0.45235 0.31543 0.51095 0.38869 0.73031
LOF 4 0.49180 0.36475 0.47374 0.34217 0.50407 0.38008 0.73381
LOF 5 0.47541 0.34426 0.48462 0.35578 0.50000 0.37500 0.74194
LOF 46 0.44262 0.30328 0.50700 0.38374 0.47482 0.34353 0.71936
SimplifiedLOF 5 0.50820 0.38525 0.47825 0.34781 0.52033 0.40041 0.74207
SimplifiedLOF 6 0.49180 0.36475 0.48799 0.35999 0.51200 0.39000 0.74308
SimplifiedLOF 97 0.44262 0.30328 0.51449 0.39312 0.49231 0.36538 0.72185
LoOP 5 0.52459 0.40574 0.48075 0.35094 0.53333 0.41667 0.74325
LoOP 97 0.44262 0.30328 0.51169 0.38961 0.50394 0.37992 0.71923
LDOF 5 0.45902 0.32377 0.44786 0.30983 0.46429 0.33036 0.73871
LDOF 83 0.49180 0.36475 0.50159 0.37699 0.50382 0.37977 0.72218
LDOF 97 0.49180 0.36475 0.50695 0.38369 0.50407 0.38008 0.72151
LDOF 99 0.49180 0.36475 0.50570 0.38212 0.51200 0.39000 0.72091
ODIN 26 0.46370 0.32963 0.39479 0.24349 0.46707 0.33383 0.71610
ODIN 60 0.50000 0.37500 0.39853 0.24816 0.51613 0.39516 0.71254
ODIN 68 0.51366 0.39208 0.41008 0.26260 0.51613 0.39516 0.71577
ODIN 97 0.45355 0.31694 0.45592 0.31990 0.48951 0.36189 0.71446
FastABOD 3 0.45902 0.32377 0.46358 0.32948 0.46667 0.33333 0.71883
FastABOD 48 0.40984 0.26230 0.48398 0.35498 0.47458 0.34322 0.71634
FastABOD 87 0.40984 0.26230 0.49436 0.36795 0.45714 0.32143 0.71345
KDEOS 14 0.34426 0.18033 0.33828 0.17285 0.40323 0.25403 0.66797
KDEOS 15 0.34426 0.18033 0.33502 0.16878 0.42857 0.28571 0.67992
KDEOS 25 0.34426 0.18033 0.29893 0.12366 0.44706 0.30882 0.67287
KDEOS 93 0.37705 0.22131 0.30481 0.13102 0.41958 0.27448 0.66629
LDF 53 0.37705 0.22131 0.42818 0.28523 0.46835 0.33544 0.71157
LDF 79 0.47541 0.34426 0.50442 0.38052 0.48739 0.35924 0.70707
LDF 89 0.45902 0.32377 0.47659 0.34574 0.50485 0.38107 0.69585
INFLO 4 0.52459 0.40574 0.46726 0.33408 0.52459 0.40574 0.73579
INFLO 5 0.52459 0.40574 0.47217 0.34022 0.53226 0.41532 0.73482
INFLO 62 0.47541 0.34426 0.51452 0.39316 0.50394 0.37992 0.73831
INFLO 100 0.44262 0.30328 0.51988 0.39984 0.47945 0.34932 0.73206
COF 2 0.44262 0.30328 0.43028 0.28785 0.45113 0.31391 0.70116
COF 3 0.44262 0.30328 0.47244 0.34055 0.47945 0.34932 0.73401
COF 4 0.42623 0.28279 0.49538 0.36923 0.46377 0.32971 0.73562

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 305 objects, 61 outliers (20.00%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (52.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.49180 0.36475 0.54347 0.42934 0.51701 0.39626 0.73189
KNN 3 0.49180 0.36475 0.54111 0.42639 0.52336 0.40421 0.73697
KNN 8 0.50820 0.38525 0.53785 0.42231 0.52336 0.40421 0.72924
KNN 10 0.47541 0.34426 0.53398 0.41748 0.53211 0.41514 0.72863
KNNW 1 0.50820 0.38525 0.54315 0.42894 0.53226 0.41532 0.72376
KNNW 6 0.49180 0.36475 0.54228 0.42785 0.52336 0.40421 0.73414
LOF 8 0.49180 0.36475 0.53122 0.41403 0.52023 0.40029 0.74194
LOF 12 0.52459 0.40574 0.54116 0.42645 0.54264 0.42829 0.73912
LOF 13 0.52459 0.40574 0.54235 0.42793 0.53968 0.42460 0.74066
SimplifiedLOF 10 0.49180 0.36475 0.54779 0.43473 0.52632 0.40789 0.74711
SimplifiedLOF 22 0.50820 0.38525 0.55073 0.43842 0.54400 0.43000 0.74348
SimplifiedLOF 48 0.55738 0.44672 0.54035 0.42544 0.55738 0.44672 0.73273
LoOP 10 0.49180 0.36475 0.52650 0.40812 0.51969 0.39961 0.74772
LoOP 13 0.54098 0.42623 0.53607 0.42009 0.54839 0.43548 0.74254
LoOP 22 0.50820 0.38525 0.54736 0.43420 0.54264 0.42829 0.74200
LoOP 28 0.54098 0.42623 0.54244 0.42805 0.55462 0.44328 0.73555
LDOF 22 0.49180 0.36475 0.54354 0.42942 0.53846 0.42308 0.74409
LDOF 25 0.52459 0.40574 0.55177 0.43972 0.53846 0.42308 0.73945
LDOF 33 0.54098 0.42623 0.54144 0.42680 0.55932 0.44915 0.73623
LDOF 34 0.54098 0.42623 0.54029 0.42537 0.56410 0.45513 0.73535
ODIN 19 0.49180 0.36475 0.44274 0.30342 0.51563 0.39453 0.73690
ODIN 27 0.51366 0.39208 0.45760 0.32200 0.51667 0.39583 0.72514
ODIN 54 0.50820 0.38525 0.49079 0.36349 0.53125 0.41406 0.72417
ODIN 90 0.50000 0.37500 0.50796 0.38495 0.50435 0.38043 0.71705
FastABOD 6 0.54098 0.42623 0.50634 0.38292 0.55118 0.43898 0.72662
FastABOD 10 0.49180 0.36475 0.52105 0.40131 0.52252 0.40315 0.73502
FastABOD 28 0.49180 0.36475 0.52720 0.40901 0.53731 0.42164 0.73011
KDEOS 85 0.44262 0.30328 0.36211 0.20264 0.48235 0.35294 0.70297
KDEOS 99 0.40984 0.26230 0.38976 0.23720 0.49664 0.37081 0.70572
LDF 2 0.40984 0.26230 0.38337 0.22922 0.42276 0.27846 0.68826
LDF 4 0.39344 0.24180 0.36873 0.21091 0.43396 0.29245 0.63276
LDF 100 0.29508 0.11885 0.41404 0.26755 0.41916 0.27395 0.64734
INFLO 12 0.54098 0.42623 0.52204 0.40254 0.55285 0.44106 0.74651
INFLO 14 0.54098 0.42623 0.52694 0.40867 0.56452 0.45565 0.75191
INFLO 24 0.50820 0.38525 0.54789 0.43486 0.54687 0.43359 0.73838
COF 5 0.45902 0.32377 0.49745 0.37181 0.48951 0.36189 0.72548
COF 6 0.44262 0.30328 0.50653 0.38316 0.48611 0.35764 0.71634
COF 31 0.49180 0.36475 0.48388 0.35485 0.49180 0.36475 0.70660
COF 37 0.47541 0.34426 0.49235 0.36544 0.51200 0.39000 0.69679

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO