Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Stamps (5% of outliers version#04)

A data set representing forged (photocopied or scanned+printed) stamps and genuine (ink) stamps. The features are based on color and printing properties of the stamps. Forged stamps are considered to be outliers. The stamps data set is not taken from the UCI repository, but was used in [1].

References:

[1] B. Micenkova, J. van Beusekom, and F. Shafait. Stamp verification for automated document authentication. In 5th Int. Workshop on Computational Forensics, 2012.

Download all data set variants used (371.2 kB).

Normalized, without duplicates

This version contains 9 attributes, 325 objects, 16 outliers (4.92%)

Download raw algorithm results (2.8 MB) Download raw algorithm evaluation table (47.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.12500 0.07969 0.29273 0.25611 0.49123 0.46488 0.93103
KNN 13 0.25000 0.21117 0.26689 0.22892 0.47619 0.44907 0.92253
KNNW 1 0.12500 0.07969 0.18651 0.14439 0.33333 0.29881 0.79167
KNNW 26 0.12500 0.07969 0.26356 0.22542 0.48387 0.45715 0.92375
KNNW 40 0.12500 0.07969 0.26695 0.22899 0.48387 0.45715 0.92536
KNNW 83 0.12500 0.07969 0.26685 0.22889 0.47761 0.45056 0.92577
LOF 3 0.18750 0.14543 0.14901 0.10495 0.24561 0.20655 0.64422
LOF 99 0.12500 0.07969 0.25293 0.21425 0.34568 0.31180 0.89138
LOF 100 0.12500 0.07969 0.25393 0.21530 0.34568 0.31180 0.89219
SimplifiedLOF 4 0.18750 0.14543 0.10523 0.05890 0.24390 0.20475 0.51922
SimplifiedLOF 97 0.18750 0.14543 0.21653 0.17596 0.28889 0.25207 0.84466
SimplifiedLOF 100 0.18750 0.14543 0.21818 0.17770 0.28889 0.25207 0.84668
LoOP 6 0.18750 0.14543 0.10731 0.06109 0.20000 0.15858 0.57100
LoOP 84 0.18750 0.14543 0.20917 0.16822 0.28571 0.24873 0.83617
LoOP 99 0.18750 0.14543 0.21498 0.17433 0.27957 0.24227 0.84284
LDOF 5 0.18750 0.14543 0.07474 0.02683 0.20000 0.15858 0.46784
LDOF 50 0.18750 0.14543 0.20906 0.16810 0.28571 0.24873 0.85053
LDOF 99 0.18750 0.14543 0.21620 0.17561 0.30000 0.26375 0.84587
LDOF 100 0.18750 0.14543 0.21584 0.17524 0.30380 0.26775 0.84608
ODIN 35 0.18750 0.14543 0.18222 0.13988 0.27778 0.24038 0.83960
ODIN 52 0.18750 0.14543 0.22809 0.18812 0.32432 0.28934 0.85427
ODIN 55 0.18750 0.14543 0.23068 0.19085 0.32432 0.28934 0.85407
FastABOD 15 0.18750 0.14543 0.19835 0.15684 0.29412 0.25757 0.87116
FastABOD 93 0.12500 0.07969 0.21407 0.17337 0.37143 0.33888 0.89057
KDEOS 53 0.12500 0.07969 0.11807 0.07241 0.22917 0.18925 0.79207
KDEOS 77 0.12500 0.07969 0.16335 0.12003 0.27451 0.23694 0.82747
KDEOS 93 0.12500 0.07969 0.17099 0.12807 0.31461 0.27912 0.82423
KDEOS 95 0.12500 0.07969 0.18864 0.14663 0.30769 0.27184 0.82039
LDF 2 0.25000 0.21117 0.12276 0.07734 0.28571 0.24873 0.59456
LDF 88 0.12500 0.07969 0.28954 0.25276 0.43478 0.40552 0.91586
LDF 96 0.12500 0.07969 0.26241 0.22422 0.44776 0.41917 0.91707
LDF 98 0.12500 0.07969 0.26539 0.22735 0.44118 0.41224 0.91828
INFLO 6 0.18750 0.14543 0.12468 0.07935 0.20000 0.15858 0.58222
INFLO 99 0.12500 0.07969 0.22392 0.18373 0.31169 0.27605 0.84061
INFLO 100 0.12500 0.07969 0.22511 0.18499 0.31169 0.27605 0.84183
COF 23 0.18750 0.14543 0.30578 0.26983 0.40541 0.37462 0.90311
COF 26 0.25000 0.21117 0.31477 0.27929 0.41270 0.38229 0.90170
COF 33 0.31250 0.27690 0.33806 0.30378 0.39216 0.36068 0.88511
COF 35 0.37500 0.34264 0.32698 0.29213 0.39130 0.35979 0.88451

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 9 attributes, 325 objects, 16 outliers (4.92%)

Download raw algorithm results (2.8 MB) Download raw algorithm evaluation table (47.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.12500 0.07969 0.27937 0.24206 0.44776 0.41917 0.92638
KNN 13 0.25000 0.21117 0.27019 0.23240 0.47619 0.44907 0.92476
KNN 37 0.12500 0.07969 0.27018 0.23238 0.50000 0.47411 0.92759
KNN 45 0.12500 0.07969 0.27512 0.23758 0.48485 0.45817 0.92961
KNNW 1 0.12500 0.07969 0.18665 0.14454 0.32653 0.29166 0.79895
KNNW 86 0.12500 0.07969 0.27143 0.23370 0.48485 0.45817 0.92799
LOF 2 0.18750 0.14543 0.11325 0.06733 0.24242 0.20320 0.58141
LOF 96 0.12500 0.07969 0.26911 0.23127 0.38961 0.35800 0.90392
LOF 100 0.12500 0.07969 0.27179 0.23408 0.38710 0.35536 0.90615
SimplifiedLOF 4 0.25000 0.21117 0.11294 0.06701 0.25000 0.21117 0.52346
SimplifiedLOF 99 0.18750 0.14543 0.23141 0.19161 0.30000 0.26375 0.85518
LoOP 3 0.18750 0.14543 0.10303 0.05659 0.18750 0.14543 0.49788
LoOP 98 0.18750 0.14543 0.21902 0.17858 0.28916 0.25235 0.84810
LoOP 99 0.18750 0.14543 0.22018 0.17980 0.28916 0.25235 0.84992
LDOF 5 0.18750 0.14543 0.08560 0.03825 0.21429 0.17360 0.49737
LDOF 95 0.18750 0.14543 0.21696 0.17641 0.31169 0.27605 0.84749
LDOF 99 0.18750 0.14543 0.21801 0.17752 0.30380 0.26775 0.85032
ODIN 39 0.18750 0.14543 0.20896 0.16800 0.28235 0.24519 0.83839
ODIN 57 0.18750 0.14543 0.22655 0.18650 0.30303 0.26694 0.85133
ODIN 81 0.18750 0.14543 0.22209 0.18181 0.30769 0.27184 0.84678
ODIN 99 0.18750 0.14543 0.22530 0.18519 0.28916 0.25235 0.85700
FastABOD 7 0.18750 0.14543 0.18483 0.14262 0.27119 0.23345 0.85558
FastABOD 78 0.12500 0.07969 0.21307 0.17232 0.35294 0.31944 0.89017
FastABOD 94 0.12500 0.07969 0.21562 0.17501 0.35294 0.31944 0.89159
KDEOS 53 0.12500 0.07969 0.10861 0.06245 0.21505 0.17441 0.77549
KDEOS 90 0.12500 0.07969 0.17439 0.13163 0.29167 0.25499 0.81270
KDEOS 97 0.12500 0.07969 0.14897 0.10490 0.27723 0.23980 0.81452
LDF 2 0.25000 0.21117 0.13669 0.09199 0.28571 0.24873 0.57979
LDF 89 0.12500 0.07969 0.30848 0.27268 0.46875 0.44124 0.92638
LDF 91 0.12500 0.07969 0.27829 0.24091 0.46875 0.44124 0.92658
LDF 100 0.12500 0.07969 0.27824 0.24087 0.48387 0.45715 0.92577
INFLO 2 0.18750 0.14543 0.07968 0.03202 0.18750 0.14543 0.43780
INFLO 97 0.12500 0.07969 0.22624 0.18617 0.32877 0.29401 0.84163
INFLO 99 0.18750 0.14543 0.23085 0.19102 0.32877 0.29401 0.84628
COF 24 0.18750 0.14543 0.29149 0.25480 0.38462 0.35275 0.89320
COF 33 0.31250 0.27690 0.32469 0.28972 0.41667 0.38646 0.88046
COF 34 0.25000 0.21117 0.31828 0.28298 0.42553 0.39579 0.87480
COF 40 0.37500 0.34264 0.28771 0.25082 0.37500 0.34264 0.84830

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO