Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Waveform (version#02)

This dataset represents 3 classes of waves. Class 0 was defined here as an outlier class and downsampled to 100 objects. After preprocessing, this database has 21 numeric attributes and 3443 instances, divided into 100 outliers (2.9%) and 3343 inliers (97.1%) [1].

References:

[1] A. Zimek, M. Gaudet, R. J. G. B. Campello, and J. Sander. Subsampling for efficient and effective unsupervised outlier detection ensembles. In Proc. KDD, pages 428-436, 2013.

Download all data set variants used (5.1 MB). You can also access the original data. (waveform.data.Z)

Normalized, without duplicates

This version contains 21 attributes, 3443 objects, 100 outliers (2.90%)

Download raw algorithm results (30.2 MB) Download raw algorithm evaluation table (64.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 65 0.26000 0.23786 0.17915 0.15459 0.26131 0.23921 0.78164
KNN 71 0.26000 0.23786 0.18116 0.15667 0.27083 0.24902 0.78301
KNN 81 0.26000 0.23786 0.17896 0.15440 0.26943 0.24758 0.78422
KNNW 57 0.24000 0.21727 0.16359 0.13857 0.24000 0.21727 0.77296
KNNW 96 0.24000 0.21727 0.17233 0.14757 0.24908 0.22662 0.77725
KNNW 97 0.24000 0.21727 0.17243 0.14767 0.24908 0.22662 0.77733
KNNW 100 0.24000 0.21727 0.17208 0.14732 0.24731 0.22480 0.77753
LOF 96 0.23000 0.20697 0.14482 0.11924 0.23656 0.21372 0.76568
LOF 100 0.23000 0.20697 0.14554 0.11998 0.23834 0.21556 0.76623
SimplifiedLOF 90 0.20000 0.17607 0.10719 0.08048 0.20388 0.18007 0.75094
SimplifiedLOF 98 0.20000 0.17607 0.11115 0.08456 0.21858 0.19520 0.75221
SimplifiedLOF 100 0.20000 0.17607 0.11165 0.08507 0.22099 0.19769 0.75176
LoOP 96 0.19000 0.16577 0.10186 0.07500 0.20388 0.18007 0.74885
LoOP 98 0.20000 0.17607 0.10217 0.07531 0.20290 0.17905 0.74977
LoOP 100 0.20000 0.17607 0.10300 0.07616 0.20290 0.17905 0.74829
LDOF 63 0.11000 0.08338 0.06251 0.03447 0.12560 0.09945 0.69730
LDOF 92 0.09000 0.06278 0.06590 0.03796 0.13986 0.11413 0.70802
LDOF 98 0.10000 0.07308 0.06683 0.03892 0.13665 0.11082 0.70890
LDOF 99 0.10000 0.07308 0.06717 0.03926 0.13665 0.11082 0.70887
ODIN 55 0.05000 0.02158 0.05137 0.02299 0.10877 0.08211 0.69023
ODIN 100 0.04857 0.02011 0.06076 0.03266 0.13592 0.11007 0.71372
FastABOD 18 0.07000 0.04218 0.05057 0.02217 0.10889 0.08224 0.66585
FastABOD 52 0.03000 0.00098 0.05331 0.02499 0.10776 0.08107 0.68509
FastABOD 78 0.06000 0.03188 0.05431 0.02602 0.11576 0.08931 0.68468
FastABOD 99 0.04000 0.01128 0.05333 0.02501 0.12186 0.09560 0.68229
KDEOS 7 0.07000 0.04218 0.04256 0.01392 0.09006 0.06284 0.57476
KDEOS 8 0.06000 0.03188 0.04390 0.01530 0.08581 0.05846 0.57373
KDEOS 25 0.08000 0.05248 0.03793 0.00916 0.08824 0.06096 0.57332
KDEOS 99 0.01000 -0.01961 0.03834 0.00958 0.08289 0.05546 0.61914
LDF 9 0.26000 0.23786 0.23688 0.21405 0.27317 0.25143 0.77620
LDF 18 0.33000 0.30996 0.26352 0.24149 0.33166 0.31167 0.75971
LDF 29 0.31000 0.28936 0.27505 0.25337 0.36025 0.34111 0.76793
LDF 37 0.31000 0.28936 0.27848 0.25689 0.34320 0.32355 0.77081
INFLO 78 0.12000 0.09368 0.07985 0.05233 0.16327 0.13824 0.73929
INFLO 91 0.14000 0.11427 0.08372 0.05631 0.16958 0.14474 0.73260
INFLO 98 0.13000 0.10398 0.08659 0.05927 0.17544 0.15077 0.73430
INFLO 100 0.13000 0.10398 0.08700 0.05969 0.17544 0.15077 0.73469
COF 71 0.27000 0.24816 0.25807 0.23588 0.33987 0.32012 0.73262
COF 82 0.30000 0.27906 0.27507 0.25338 0.33577 0.31590 0.74390

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 3443 objects, 100 outliers (2.90%)

Download raw algorithm results (30.2 MB) Download raw algorithm evaluation table (65.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 33 0.31000 0.28936 0.23543 0.21256 0.31963 0.29928 0.78851
KNN 35 0.29000 0.26876 0.23729 0.21448 0.32075 0.30044 0.78997
KNN 100 0.29000 0.26876 0.26283 0.24078 0.31356 0.29303 0.79939
KNNW 35 0.27000 0.24816 0.20024 0.17632 0.27972 0.25817 0.78199
KNNW 77 0.25000 0.22757 0.22865 0.20557 0.30837 0.28768 0.78935
KNNW 100 0.27000 0.24816 0.24030 0.21757 0.30435 0.28354 0.79155
LOF 90 0.27000 0.24816 0.18691 0.16259 0.27966 0.25811 0.78257
LOF 93 0.26000 0.23786 0.18973 0.16549 0.28448 0.26308 0.78293
LOF 98 0.26000 0.23786 0.19279 0.16864 0.28085 0.25934 0.78302
LOF 100 0.26000 0.23786 0.19392 0.16980 0.28085 0.25934 0.78283
SimplifiedLOF 93 0.20000 0.17607 0.12753 0.10143 0.22137 0.19808 0.76424
SimplifiedLOF 99 0.20000 0.17607 0.13241 0.10646 0.22936 0.20631 0.76504
SimplifiedLOF 100 0.20000 0.17607 0.13373 0.10782 0.22222 0.19896 0.76540
LoOP 93 0.18000 0.15547 0.11378 0.08727 0.20896 0.18529 0.76090
LoOP 100 0.18000 0.15547 0.11777 0.09138 0.20690 0.18317 0.76213
LDOF 73 0.12000 0.09368 0.06831 0.04044 0.13499 0.10912 0.71311
LDOF 98 0.11000 0.08338 0.07247 0.04473 0.14453 0.11894 0.72402
LDOF 100 0.11000 0.08338 0.07305 0.04532 0.14286 0.11722 0.72526
ODIN 9 0.05078 0.02239 0.04054 0.01184 0.07773 0.05015 0.61620
ODIN 98 0.02000 -0.00931 0.05808 0.02991 0.13075 0.10475 0.71490
ODIN 99 0.02273 -0.00651 0.05812 0.02994 0.12883 0.10277 0.71504
FastABOD 3 0.06000 0.03188 0.03267 0.00373 0.06357 0.03555 0.51033
FastABOD 8 0.02000 -0.00931 0.03254 0.00360 0.06699 0.03908 0.53650
FastABOD 11 0.02000 -0.00931 0.03246 0.00351 0.07159 0.04382 0.53131
FastABOD 24 0.02000 -0.00931 0.03311 0.00418 0.06952 0.04169 0.53645
KDEOS 13 0.06000 0.03188 0.04330 0.01468 0.07652 0.04889 0.56739
KDEOS 14 0.08000 0.05248 0.04203 0.01337 0.08743 0.06013 0.57002
KDEOS 25 0.06000 0.03188 0.04083 0.01214 0.09486 0.06779 0.57386
KDEOS 100 0.03000 0.00098 0.03718 0.00838 0.07867 0.05111 0.60680
LDF 18 0.34000 0.32026 0.32451 0.30430 0.37975 0.36119 0.80006
LDF 20 0.35000 0.33056 0.32538 0.30520 0.36585 0.34688 0.79693
LDF 25 0.31000 0.28936 0.33678 0.31694 0.37086 0.35204 0.79501
LDF 26 0.33000 0.30996 0.33368 0.31374 0.38356 0.36512 0.79302
INFLO 91 0.14000 0.11427 0.09007 0.06285 0.17391 0.14920 0.74769
INFLO 99 0.14000 0.11427 0.09602 0.06898 0.18280 0.15835 0.75104
INFLO 100 0.14000 0.11427 0.09674 0.06972 0.18133 0.15684 0.75178
COF 58 0.32000 0.29966 0.28759 0.26627 0.32804 0.30794 0.74623
COF 89 0.29000 0.26876 0.30898 0.28831 0.34286 0.32320 0.76670
COF 94 0.28000 0.25846 0.30284 0.28198 0.35762 0.33840 0.77177
COF 97 0.27000 0.24816 0.30531 0.28453 0.36765 0.34873 0.76731

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO