Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (2% of outliers version#03)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (30.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 15 0.50000 0.49180 0.47439 0.46577 0.50000 0.49180 0.95287
KNN 32 0.50000 0.49180 0.55278 0.54545 0.66667 0.66120 0.96311
KNN 62 0.50000 0.49180 0.55441 0.54711 0.66667 0.66120 0.96516
KNNW 43 0.50000 0.49180 0.47500 0.46639 0.50000 0.49180 0.95389
KNNW 75 0.50000 0.49180 0.52564 0.51786 0.60000 0.59344 0.95902
KNNW 91 0.50000 0.49180 0.52703 0.51927 0.60000 0.59344 0.96107
LOF 1 0.25000 0.23770 0.15083 0.13691 0.33333 0.32240 0.63525
LOF 94 0.25000 0.23770 0.47316 0.46453 0.50000 0.49180 0.96107
SimplifiedLOF 1 0.25000 0.23770 0.30021 0.28874 0.40000 0.39016 0.82275
SimplifiedLOF 95 0.25000 0.23770 0.41389 0.40428 0.42857 0.41920 0.95902
SimplifiedLOF 97 0.25000 0.23770 0.41627 0.40670 0.42857 0.41920 0.96107
LoOP 1 0.25000 0.23770 0.30021 0.28874 0.40000 0.39016 0.82275
LoOP 95 0.25000 0.23770 0.40377 0.39400 0.40000 0.39016 0.95902
LDOF 2 0.50000 0.49180 0.45229 0.44331 0.57143 0.56440 0.90574
LDOF 94 0.25000 0.23770 0.36706 0.35669 0.40000 0.39016 0.94877
ODIN 65 0.25000 0.23770 0.14155 0.12748 0.25000 0.23770 0.91445
ODIN 68 0.25000 0.23770 0.16131 0.14756 0.28571 0.27400 0.91803
ODIN 94 0.25000 0.23770 0.16684 0.15318 0.28571 0.27400 0.92572
ODIN 100 0.25000 0.23770 0.16389 0.15018 0.28571 0.27400 0.92623
FastABOD 3 0.25000 0.23770 0.30590 0.29452 0.40000 0.39016 0.64857
FastABOD 99 0.25000 0.23770 0.40474 0.39498 0.42857 0.41920 0.93135
FastABOD 100 0.25000 0.23770 0.40504 0.39529 0.42857 0.41920 0.93238
KDEOS 14 0.00000 -0.01639 0.12493 0.11058 0.27273 0.26080 0.91393
KDEOS 19 0.00000 -0.01639 0.13113 0.11689 0.28571 0.27400 0.88012
KDEOS 21 0.25000 0.23770 0.12520 0.11086 0.25000 0.23770 0.82787
LDF 7 0.75000 0.74590 0.61417 0.60784 0.75000 0.74590 0.89959
LDF 24 0.75000 0.74590 0.79545 0.79210 0.85714 0.85480 0.98156
LDF 25 0.75000 0.74590 0.80000 0.79672 0.85714 0.85480 0.98361
LDF 43 0.50000 0.49180 0.45833 0.44945 0.66667 0.66120 0.98873
INFLO 1 0.25000 0.23770 0.29106 0.27944 0.40000 0.39016 0.77818
INFLO 95 0.25000 0.23770 0.41726 0.40771 0.40000 0.39016 0.95902
COF 26 0.25000 0.23770 0.38289 0.37277 0.40000 0.39016 0.91803
COF 48 0.75000 0.74590 0.69590 0.69092 0.75000 0.74590 0.88115
COF 49 0.75000 0.74590 0.75800 0.75403 0.85714 0.85480 0.87602
COF 62 0.75000 0.74590 0.75877 0.75482 0.85714 0.85480 0.88730

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 248 objects, 4 outliers (1.61%)

Download raw algorithm results (2.2 MB) Download raw algorithm evaluation table (28.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.25000 0.23770 0.22198 0.20923 0.33333 0.32240 0.93750
KNN 3 0.25000 0.23770 0.22768 0.21502 0.33333 0.32240 0.94160
KNN 4 0.25000 0.23770 0.19215 0.17891 0.29630 0.28476 0.94467
KNNW 1 0.25000 0.23770 0.18430 0.17093 0.33333 0.32240 0.87295
KNNW 11 0.25000 0.23770 0.22886 0.21622 0.33333 0.32240 0.94262
KNNW 40 0.25000 0.23770 0.19013 0.17685 0.28571 0.27400 0.94365
LOF 1 0.25000 0.23770 0.27083 0.25887 0.40000 0.39016 0.67213
LOF 5 0.25000 0.23770 0.30777 0.29642 0.40000 0.39016 0.88627
LOF 61 0.25000 0.23770 0.18950 0.17622 0.30769 0.29634 0.94262
SimplifiedLOF 1 0.25000 0.23770 0.28529 0.27357 0.40000 0.39016 0.80328
SimplifiedLOF 7 0.25000 0.23770 0.31566 0.30444 0.40000 0.39016 0.89652
SimplifiedLOF 59 0.25000 0.23770 0.19030 0.17702 0.28571 0.27400 0.94365
LoOP 1 0.25000 0.23770 0.28529 0.27357 0.40000 0.39016 0.80328
LoOP 7 0.25000 0.23770 0.32092 0.30979 0.40000 0.39016 0.90779
LoOP 58 0.25000 0.23770 0.19196 0.17872 0.28571 0.27400 0.94467
LDOF 3 0.25000 0.23770 0.14089 0.12681 0.28571 0.27400 0.88730
LDOF 4 0.25000 0.23770 0.30751 0.29615 0.40000 0.39016 0.88934
LDOF 6 0.25000 0.23770 0.32945 0.31846 0.40000 0.39016 0.92213
LDOF 93 0.25000 0.23770 0.19082 0.17755 0.30769 0.29634 0.94365
ODIN 24 0.25000 0.23770 0.18299 0.16960 0.28571 0.27400 0.95850
ODIN 27 0.25000 0.23770 0.24821 0.23589 0.42857 0.41920 0.96773
ODIN 32 0.25000 0.23770 0.21511 0.20224 0.33333 0.32240 0.96875
FastABOD 3 0.25000 0.23770 0.17074 0.15715 0.36364 0.35320 0.77561
FastABOD 8 0.25000 0.23770 0.24188 0.22945 0.33333 0.32240 0.94980
FastABOD 9 0.25000 0.23770 0.24265 0.23024 0.33333 0.32240 0.94980
KDEOS 7 0.25000 0.23770 0.12720 0.11290 0.28571 0.27400 0.81762
KDEOS 10 0.25000 0.23770 0.31153 0.30025 0.40000 0.39016 0.83094
KDEOS 11 0.25000 0.23770 0.33836 0.32751 0.40000 0.39016 0.84939
KDEOS 96 0.00000 -0.01639 0.10951 0.09491 0.20690 0.19389 0.91189
LDF 6 0.25000 0.23770 0.12038 0.10596 0.28571 0.27400 0.81352
LDF 9 0.25000 0.23770 0.28924 0.27759 0.40000 0.39016 0.81967
LDF 10 0.25000 0.23770 0.32683 0.31580 0.40000 0.39016 0.85451
LDF 14 0.25000 0.23770 0.30706 0.29570 0.40000 0.39016 0.85553
INFLO 1 0.25000 0.23770 0.32051 0.30937 0.40000 0.39016 0.91291
INFLO 4 0.25000 0.23770 0.32078 0.30965 0.40000 0.39016 0.90881
INFLO 33 0.25000 0.23770 0.20000 0.18689 0.28571 0.27400 0.94877
COF 1 0.25000 0.23770 0.28529 0.27357 0.40000 0.39016 0.80328
COF 5 0.25000 0.23770 0.29635 0.28482 0.40000 0.39016 0.81045
COF 28 0.25000 0.23770 0.23956 0.22709 0.33333 0.32240 0.94877

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO