Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Waveform (version#09)

This dataset represents 3 classes of waves. Class 0 was defined here as an outlier class and downsampled to 100 objects. After preprocessing, this database has 21 numeric attributes and 3443 instances, divided into 100 outliers (2.9%) and 3343 inliers (97.1%) [1].

References:

[1] A. Zimek, M. Gaudet, R. J. G. B. Campello, and J. Sander. Subsampling for efficient and effective unsupervised outlier detection ensembles. In Proc. KDD, pages 428-436, 2013.

Download all data set variants used (5.1 MB). You can also access the original data. (waveform.data.Z)

Normalized, without duplicates

This version contains 21 attributes, 3443 objects, 100 outliers (2.90%)

Download raw algorithm results (30.2 MB) Download raw algorithm evaluation table (65.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 22 0.24000 0.21727 0.15074 0.12534 0.24000 0.21727 0.76956
KNN 72 0.24000 0.21727 0.17911 0.15456 0.26282 0.24077 0.77910
KNN 76 0.24000 0.21727 0.18106 0.15656 0.25705 0.23483 0.77948
KNN 92 0.24000 0.21727 0.17975 0.15522 0.25545 0.23318 0.78221
KNNW 79 0.25000 0.22757 0.16142 0.13634 0.25253 0.23017 0.77343
KNNW 92 0.25000 0.22757 0.16339 0.13836 0.25381 0.23149 0.77462
KNNW 100 0.25000 0.22757 0.16531 0.14034 0.25381 0.23149 0.77530
LOF 89 0.19000 0.16577 0.13025 0.10423 0.23270 0.20975 0.76672
LOF 99 0.19000 0.16577 0.13613 0.11029 0.23429 0.21138 0.76823
LOF 100 0.19000 0.16577 0.13629 0.11045 0.23496 0.21207 0.76789
SimplifiedLOF 73 0.14000 0.11427 0.08925 0.06201 0.19595 0.17189 0.73722
SimplifiedLOF 94 0.16000 0.13487 0.09797 0.07098 0.18803 0.16375 0.74468
SimplifiedLOF 100 0.16000 0.13487 0.10181 0.07494 0.19409 0.16999 0.74627
LoOP 92 0.14000 0.11427 0.09044 0.06324 0.18773 0.16343 0.73876
LoOP 97 0.15000 0.12457 0.09189 0.06472 0.18243 0.15798 0.73919
LoOP 100 0.14000 0.11427 0.09326 0.06614 0.18367 0.15925 0.74030
LDOF 6 0.09000 0.06278 0.04325 0.01463 0.09462 0.06754 0.60407
LDOF 26 0.07000 0.04218 0.05460 0.02632 0.12698 0.10087 0.66234
LDOF 98 0.09000 0.06278 0.05783 0.02965 0.10964 0.08301 0.69104
LDOF 99 0.09000 0.06278 0.05770 0.02951 0.11060 0.08399 0.69157
ODIN 14 0.08527 0.05791 0.04598 0.01744 0.09607 0.06903 0.62573
ODIN 85 0.03737 0.00857 0.05326 0.02494 0.11881 0.09245 0.69757
ODIN 100 0.03000 0.00098 0.05613 0.02790 0.11705 0.09064 0.70405
FastABOD 7 0.08000 0.05248 0.04772 0.01923 0.10554 0.07878 0.63976
FastABOD 14 0.05000 0.02158 0.05106 0.02267 0.11144 0.08486 0.67045
FastABOD 16 0.05000 0.02158 0.05065 0.02225 0.11615 0.08971 0.66794
KDEOS 7 0.06000 0.03188 0.04397 0.01537 0.07650 0.04888 0.56526
KDEOS 17 0.08000 0.05248 0.04200 0.01334 0.09150 0.06433 0.57473
KDEOS 18 0.07000 0.04218 0.04235 0.01371 0.09211 0.06495 0.57137
KDEOS 100 0.03000 0.00098 0.03526 0.00640 0.07754 0.04994 0.60191
LDF 16 0.30000 0.27906 0.23988 0.21714 0.30392 0.28310 0.76805
LDF 24 0.27000 0.24816 0.25400 0.23168 0.33803 0.31823 0.77467
LDF 33 0.28000 0.25846 0.26331 0.24128 0.31447 0.29396 0.78339
INFLO 76 0.13000 0.10398 0.07361 0.04590 0.16216 0.13710 0.71908
INFLO 77 0.12000 0.09368 0.07397 0.04627 0.16615 0.14121 0.71972
INFLO 99 0.12000 0.09368 0.08141 0.05394 0.16099 0.13589 0.73192
INFLO 100 0.12000 0.09368 0.08180 0.05434 0.15864 0.13347 0.73060
COF 64 0.29000 0.26876 0.21769 0.19429 0.31206 0.29148 0.70862
COF 82 0.27000 0.24816 0.25459 0.23229 0.35461 0.33530 0.71281
COF 84 0.27000 0.24816 0.25928 0.23712 0.34848 0.32900 0.71781
COF 87 0.27000 0.24816 0.25452 0.23222 0.35115 0.33174 0.72245

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 3443 objects, 100 outliers (2.90%)

Download raw algorithm results (30.2 MB) Download raw algorithm evaluation table (66.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 35 0.29000 0.26876 0.22005 0.19672 0.29000 0.26876 0.78103
KNN 96 0.26000 0.23786 0.25402 0.23171 0.31799 0.29759 0.79127
KNN 99 0.26000 0.23786 0.25609 0.23383 0.31304 0.29249 0.79139
KNN 100 0.26000 0.23786 0.25532 0.23304 0.31304 0.29249 0.79177
KNNW 31 0.26000 0.23786 0.18261 0.15816 0.26906 0.24719 0.77666
KNNW 94 0.25000 0.22757 0.22400 0.20079 0.28235 0.26089 0.78401
KNNW 100 0.25000 0.22757 0.22500 0.20182 0.28030 0.25877 0.78438
LOF 95 0.25000 0.22757 0.17211 0.14735 0.26056 0.23844 0.77819
LOF 97 0.26000 0.23786 0.17377 0.14905 0.26148 0.23939 0.77813
LOF 99 0.26000 0.23786 0.17472 0.15003 0.26531 0.24333 0.77807
LOF 100 0.26000 0.23786 0.17552 0.15085 0.26531 0.24333 0.77803
SimplifiedLOF 96 0.20000 0.17607 0.11241 0.08586 0.21359 0.19007 0.75202
SimplifiedLOF 97 0.21000 0.18637 0.11271 0.08617 0.21359 0.19007 0.75222
SimplifiedLOF 100 0.20000 0.17607 0.11444 0.08795 0.21253 0.18898 0.75324
LoOP 97 0.17000 0.14517 0.10172 0.07485 0.20000 0.17607 0.74614
LoOP 98 0.18000 0.15547 0.10199 0.07513 0.20000 0.17607 0.74649
LoOP 100 0.18000 0.15547 0.10308 0.07625 0.19900 0.17504 0.74736
LDOF 13 0.10000 0.07308 0.05072 0.02232 0.10870 0.08203 0.63570
LDOF 96 0.09000 0.06278 0.05957 0.03144 0.12044 0.09413 0.69751
LDOF 100 0.08000 0.05248 0.05896 0.03081 0.12500 0.09883 0.69896
ODIN 10 0.05000 0.02158 0.03855 0.00979 0.07297 0.04524 0.59957
ODIN 100 0.03000 0.00098 0.05186 0.02350 0.12208 0.09582 0.69733
FastABOD 8 0.04000 0.01128 0.03148 0.00251 0.06708 0.03917 0.50515
FastABOD 10 0.05000 0.02158 0.03223 0.00328 0.06132 0.03324 0.51804
KDEOS 12 0.06000 0.03188 0.04701 0.01851 0.08556 0.05821 0.57863
KDEOS 16 0.08000 0.05248 0.04370 0.01509 0.08609 0.05875 0.57361
KDEOS 21 0.08000 0.05248 0.04092 0.01223 0.08889 0.06163 0.56886
KDEOS 100 0.05000 0.02158 0.04125 0.01258 0.07499 0.04732 0.58894
LDF 31 0.32000 0.29966 0.32786 0.30775 0.36478 0.34578 0.79732
LDF 52 0.38000 0.36145 0.34703 0.32750 0.38000 0.36145 0.79052
LDF 94 0.37000 0.35115 0.32818 0.30808 0.38947 0.37121 0.79167
INFLO 79 0.13000 0.10398 0.07653 0.04891 0.15789 0.13270 0.72678
INFLO 100 0.13000 0.10398 0.08497 0.05759 0.17128 0.14650 0.73765
COF 80 0.29000 0.26876 0.28459 0.26319 0.31183 0.29124 0.74280
COF 83 0.31000 0.28936 0.28650 0.26516 0.33333 0.31339 0.73085
COF 98 0.27000 0.24816 0.30386 0.28303 0.35036 0.33093 0.73853

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO