Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Stamps (2% of outliers version#09)

A data set representing forged (photocopied or scanned+printed) stamps and genuine (ink) stamps. The features are based on color and printing properties of the stamps. Forged stamps are considered to be outliers. The stamps data set is not taken from the UCI repository, but was used in [1].

References:

[1] B. Micenkova, J. van Beusekom, and F. Shafait. Stamp verification for automated document authentication. In 5th Int. Workshop on Computational Forensics, 2012.

Download all data set variants used (371.2 kB).

Normalized, without duplicates

This version contains 9 attributes, 315 objects, 6 outliers (1.90%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (39.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.16667 0.15049 0.21308 0.19780 0.33333 0.32039 0.94822
KNNW 1 0.16667 0.15049 0.20385 0.18839 0.35294 0.34038 0.95280
KNNW 2 0.16667 0.15049 0.22387 0.20880 0.36364 0.35128 0.95631
LOF 3 0.16667 0.15049 0.26417 0.24988 0.40000 0.38835 0.93797
LOF 4 0.16667 0.15049 0.22119 0.20607 0.29412 0.28041 0.94498
LOF 5 0.16667 0.15049 0.27212 0.25799 0.28571 0.27184 0.92341
SimplifiedLOF 4 0.16667 0.15049 0.14257 0.12592 0.28571 0.27184 0.87864
SimplifiedLOF 11 0.16667 0.15049 0.26350 0.24920 0.28571 0.27184 0.92017
SimplifiedLOF 13 0.16667 0.15049 0.14848 0.13194 0.27027 0.25610 0.92665
LoOP 5 0.16667 0.15049 0.09477 0.07719 0.19048 0.17476 0.81931
LoOP 13 0.16667 0.15049 0.17152 0.15543 0.28571 0.27184 0.92503
LoOP 42 0.16667 0.15049 0.25021 0.23565 0.28571 0.27184 0.89374
LDOF 3 0.16667 0.15049 0.04765 0.02916 0.16667 0.15049 0.56796
LDOF 13 0.16667 0.15049 0.13571 0.11893 0.22857 0.21359 0.91909
LDOF 20 0.16667 0.15049 0.13042 0.11353 0.27027 0.25610 0.91694
LDOF 72 0.16667 0.15049 0.15093 0.13444 0.25000 0.23544 0.86300
ODIN 12 0.06667 0.04854 0.10015 0.08268 0.23077 0.21583 0.86084
ODIN 16 0.16667 0.15049 0.14106 0.12439 0.31579 0.30250 0.84493
ODIN 19 0.06667 0.04854 0.14332 0.12669 0.35294 0.34038 0.85248
FastABOD 3 0.00000 -0.01942 0.10573 0.08837 0.21429 0.19903 0.88781
FastABOD 13 0.00000 -0.01942 0.16043 0.14412 0.34783 0.33516 0.93797
FastABOD 14 0.00000 -0.01942 0.16056 0.14426 0.33333 0.32039 0.93797
KDEOS 60 0.00000 -0.01942 0.12962 0.11271 0.27778 0.26375 0.89806
KDEOS 83 0.33333 0.32039 0.14046 0.12377 0.33333 0.32039 0.85653
LDF 3 0.16667 0.15049 0.26111 0.24676 0.38710 0.37520 0.96063
INFLO 1 0.16667 0.15049 0.04469 0.02614 0.16667 0.15049 0.47923
INFLO 9 0.16667 0.15049 0.24785 0.23324 0.28571 0.27184 0.85761
INFLO 13 0.16667 0.15049 0.26884 0.25464 0.28571 0.27184 0.91640
COF 7 0.33333 0.32039 0.35949 0.34705 0.38095 0.36893 0.95200
COF 8 0.16667 0.15049 0.37096 0.35874 0.41667 0.40534 0.95955
COF 10 0.33333 0.32039 0.37619 0.36408 0.40000 0.38835 0.95631

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 9 attributes, 315 objects, 6 outliers (1.90%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (39.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.16667 0.15049 0.22431 0.20924 0.36364 0.35128 0.95523
KNN 2 0.16667 0.15049 0.22994 0.21499 0.30769 0.29425 0.94121
KNN 7 0.33333 0.32039 0.20435 0.18890 0.33333 0.32039 0.93312
KNNW 1 0.16667 0.15049 0.22302 0.20793 0.37500 0.36286 0.95712
KNNW 2 0.16667 0.15049 0.24501 0.23035 0.42105 0.40981 0.96117
LOF 2 0.00000 -0.01942 0.17835 0.16239 0.35714 0.34466 0.93851
LOF 3 0.16667 0.15049 0.24451 0.22984 0.35714 0.34466 0.93635
LOF 7 0.16667 0.15049 0.28493 0.27105 0.28571 0.27184 0.93096
SimplifiedLOF 3 0.16667 0.15049 0.11396 0.09675 0.22222 0.20712 0.82578
SimplifiedLOF 13 0.16667 0.15049 0.28925 0.27545 0.31250 0.29915 0.93689
LoOP 5 0.16667 0.15049 0.10171 0.08426 0.18182 0.16593 0.82848
LoOP 13 0.16667 0.15049 0.29552 0.28184 0.31250 0.29915 0.93420
LDOF 3 0.16667 0.15049 0.05437 0.03600 0.18182 0.16593 0.59223
LDOF 13 0.16667 0.15049 0.14375 0.12712 0.27027 0.25610 0.92665
LDOF 17 0.16667 0.15049 0.12570 0.10872 0.27778 0.26375 0.91478
LDOF 100 0.16667 0.15049 0.15566 0.13926 0.25000 0.23544 0.86839
ODIN 16 0.16667 0.15049 0.09431 0.07673 0.19355 0.17789 0.81419
ODIN 19 0.02778 0.00890 0.09983 0.08235 0.24000 0.22524 0.84142
ODIN 99 0.16667 0.15049 0.11833 0.10121 0.20000 0.18447 0.87325
FastABOD 11 0.00000 -0.01942 0.18952 0.17378 0.36364 0.35128 0.94984
FastABOD 12 0.00000 -0.01942 0.18821 0.17244 0.38095 0.36893 0.94660
FastABOD 14 0.00000 -0.01942 0.19037 0.17465 0.38095 0.36893 0.94876
FastABOD 63 0.16667 0.15049 0.18385 0.16800 0.31579 0.30250 0.94337
KDEOS 47 0.00000 -0.01942 0.10503 0.08765 0.22857 0.21359 0.89213
KDEOS 54 0.00000 -0.01942 0.11150 0.09425 0.27027 0.25610 0.88889
KDEOS 65 0.16667 0.15049 0.09735 0.07983 0.18605 0.17024 0.85814
LDF 2 0.16667 0.15049 0.27143 0.25728 0.37500 0.36286 0.95523
LDF 4 0.16667 0.15049 0.27754 0.26351 0.28571 0.27184 0.92665
INFLO 1 0.16667 0.15049 0.04576 0.02723 0.16667 0.15049 0.50701
INFLO 7 0.16667 0.15049 0.15103 0.13454 0.28571 0.27184 0.86300
INFLO 13 0.16667 0.15049 0.29114 0.27737 0.28571 0.27184 0.92611
COF 7 0.33333 0.32039 0.38796 0.37608 0.41667 0.40534 0.96009
COF 8 0.33333 0.32039 0.47029 0.46000 0.62500 0.61772 0.96548
COF 10 0.33333 0.32039 0.42248 0.41126 0.46154 0.45108 0.96872

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO