Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Waveform (version#07)

This dataset represents 3 classes of waves. Class 0 was defined here as an outlier class and downsampled to 100 objects. After preprocessing, this database has 21 numeric attributes and 3443 instances, divided into 100 outliers (2.9%) and 3343 inliers (97.1%) [1].

References:

[1] A. Zimek, M. Gaudet, R. J. G. B. Campello, and J. Sander. Subsampling for efficient and effective unsupervised outlier detection ensembles. In Proc. KDD, pages 428-436, 2013.

Download all data set variants used (5.1 MB). You can also access the original data. (waveform.data.Z)

Normalized, without duplicates

This version contains 21 attributes, 3443 objects, 100 outliers (2.90%)

Download raw algorithm results (30.1 MB) Download raw algorithm evaluation table (66.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 44 0.24000 0.21727 0.16432 0.13932 0.26335 0.24131 0.76178
KNN 84 0.24000 0.21727 0.16959 0.14475 0.26909 0.24723 0.76991
KNN 88 0.23000 0.20697 0.16909 0.14423 0.27306 0.25132 0.77028
KNN 94 0.24000 0.21727 0.16914 0.14428 0.27206 0.25028 0.77144
KNNW 36 0.21000 0.18637 0.14352 0.11790 0.23048 0.20746 0.75313
KNNW 100 0.21000 0.18637 0.16132 0.13623 0.24549 0.22292 0.76266
LOF 88 0.20000 0.17607 0.13110 0.10511 0.23592 0.21307 0.75514
LOF 97 0.20000 0.17607 0.13358 0.10767 0.24090 0.21819 0.75633
LOF 99 0.20000 0.17607 0.13363 0.10772 0.23889 0.21612 0.75677
LOF 100 0.20000 0.17607 0.13443 0.10854 0.23955 0.21681 0.75657
SimplifiedLOF 82 0.18000 0.15547 0.10027 0.07335 0.18182 0.15734 0.72187
SimplifiedLOF 94 0.17000 0.14517 0.10434 0.07755 0.19048 0.16626 0.72341
SimplifiedLOF 99 0.17000 0.14517 0.10589 0.07914 0.18799 0.16370 0.72398
SimplifiedLOF 100 0.17000 0.14517 0.10136 0.07447 0.18701 0.16269 0.72402
LoOP 81 0.17000 0.14517 0.08803 0.06075 0.17259 0.14784 0.71406
LoOP 96 0.17000 0.14517 0.09775 0.07076 0.17716 0.15254 0.71565
LoOP 99 0.17000 0.14517 0.09271 0.06557 0.17436 0.14966 0.71599
LoOP 100 0.17000 0.14517 0.09387 0.06677 0.17801 0.15342 0.71592
LDOF 59 0.11000 0.08338 0.05412 0.02582 0.11304 0.08651 0.66756
LDOF 98 0.08000 0.05248 0.05688 0.02866 0.12593 0.09978 0.68351
LDOF 100 0.09000 0.06278 0.05722 0.02902 0.12319 0.09696 0.68442
ODIN 97 0.05600 0.02776 0.05567 0.02742 0.12336 0.09714 0.69174
ODIN 98 0.05143 0.02305 0.05590 0.02766 0.12371 0.09750 0.69182
ODIN 100 0.05500 0.02673 0.05626 0.02802 0.12147 0.09519 0.69263
FastABOD 15 0.07000 0.04218 0.04709 0.01859 0.09281 0.06568 0.65462
FastABOD 30 0.05000 0.02158 0.04997 0.02155 0.10412 0.07732 0.67299
FastABOD 31 0.04000 0.01128 0.04983 0.02140 0.10314 0.07631 0.67357
FastABOD 34 0.05000 0.02158 0.04956 0.02113 0.10565 0.07890 0.67180
KDEOS 2 0.07000 0.04218 0.03317 0.00425 0.07407 0.04638 0.51214
KDEOS 5 0.03000 0.00098 0.03260 0.00367 0.08511 0.05774 0.50824
KDEOS 8 0.04000 0.01128 0.03587 0.00703 0.06384 0.03584 0.53194
KDEOS 100 0.03000 0.00098 0.03292 0.00399 0.07911 0.05157 0.57912
LDF 26 0.30000 0.27906 0.26422 0.24222 0.31579 0.29532 0.77849
LDF 28 0.29000 0.26876 0.26612 0.24416 0.32335 0.30311 0.78369
LDF 100 0.25000 0.22757 0.22123 0.19793 0.27615 0.25450 0.78608
INFLO 75 0.13000 0.10398 0.07910 0.05156 0.15132 0.12593 0.68777
INFLO 82 0.12000 0.09368 0.08141 0.05393 0.15385 0.12853 0.69301
INFLO 88 0.11000 0.08338 0.07582 0.04818 0.15743 0.13223 0.69485
INFLO 97 0.11000 0.08338 0.07826 0.05069 0.16514 0.14016 0.69364
COF 76 0.29000 0.26876 0.24481 0.22222 0.30052 0.27959 0.69973
COF 96 0.29000 0.26876 0.24304 0.22039 0.31638 0.29594 0.71737
COF 99 0.28000 0.25846 0.25039 0.22796 0.32941 0.30935 0.71554

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 3443 objects, 100 outliers (2.90%)

Download raw algorithm results (30.2 MB) Download raw algorithm evaluation table (65.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 84 0.31000 0.28936 0.23750 0.21469 0.33155 0.31156 0.77383
KNN 88 0.31000 0.28936 0.23728 0.21446 0.32632 0.30616 0.77434
KNN 95 0.32000 0.29966 0.23825 0.21547 0.32836 0.30827 0.77399
KNN 100 0.32000 0.29966 0.23908 0.21632 0.32461 0.30440 0.77432
KNNW 97 0.27000 0.24816 0.21618 0.19273 0.29524 0.27416 0.76605
KNNW 98 0.28000 0.25846 0.21636 0.19292 0.29524 0.27416 0.76613
KNNW 100 0.28000 0.25846 0.21667 0.19324 0.29524 0.27416 0.76630
LOF 92 0.25000 0.22757 0.18298 0.15854 0.28085 0.25934 0.76063
LOF 93 0.25000 0.22757 0.18526 0.16089 0.27848 0.25690 0.76086
LOF 97 0.26000 0.23786 0.18949 0.16525 0.27500 0.25331 0.76067
LOF 100 0.25000 0.22757 0.19150 0.16732 0.27731 0.25569 0.76038
SimplifiedLOF 98 0.16000 0.13487 0.12648 0.10035 0.21579 0.19233 0.73144
SimplifiedLOF 99 0.16000 0.13487 0.12719 0.10108 0.21348 0.18996 0.73151
LoOP 94 0.15000 0.12457 0.11224 0.08568 0.19895 0.17499 0.72229
LoOP 98 0.15000 0.12457 0.11466 0.08818 0.20159 0.17771 0.72359
LoOP 100 0.15000 0.12457 0.11529 0.08883 0.20000 0.17607 0.72322
LDOF 99 0.13000 0.10398 0.06254 0.03450 0.13514 0.10926 0.69154
LDOF 100 0.13000 0.10398 0.06232 0.03427 0.13761 0.11182 0.69149
ODIN 61 0.05000 0.02158 0.04488 0.01631 0.09653 0.06951 0.65652
ODIN 99 0.02652 -0.00260 0.05370 0.02539 0.11966 0.09332 0.68714
ODIN 100 0.02636 -0.00276 0.05387 0.02557 0.12153 0.09525 0.68710
FastABOD 3 0.04000 0.01128 0.03058 0.00158 0.06420 0.03620 0.49245
FastABOD 6 0.01000 -0.01961 0.03007 0.00105 0.06542 0.03746 0.52868
KDEOS 11 0.02000 -0.00931 0.03831 0.00954 0.07619 0.04856 0.53791
KDEOS 13 0.05000 0.02158 0.03662 0.00780 0.07222 0.04447 0.53393
KDEOS 16 0.03000 0.00098 0.03535 0.00649 0.08815 0.06088 0.53346
KDEOS 99 0.02000 -0.00931 0.03181 0.00285 0.07501 0.04734 0.56684
LDF 49 0.37000 0.35115 0.32695 0.30682 0.37000 0.35115 0.78921
LDF 52 0.36000 0.34086 0.32928 0.30922 0.38298 0.36452 0.78515
LDF 92 0.35000 0.33056 0.31097 0.29036 0.38674 0.36840 0.78712
INFLO 84 0.15000 0.12457 0.09118 0.06399 0.16887 0.14400 0.69816
INFLO 90 0.14000 0.11427 0.09308 0.06595 0.17067 0.14586 0.70026
INFLO 98 0.14000 0.11427 0.09649 0.06946 0.17500 0.15032 0.69540
INFLO 100 0.14000 0.11427 0.09724 0.07024 0.17436 0.14966 0.69547
COF 82 0.27000 0.24816 0.26568 0.24371 0.30058 0.27966 0.73994
COF 88 0.30000 0.27906 0.27276 0.25101 0.31138 0.29078 0.73931
COF 100 0.28000 0.25846 0.28207 0.26059 0.34146 0.32176 0.73183

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO