Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Stamps (5% of outliers version#10)

A data set representing forged (photocopied or scanned+printed) stamps and genuine (ink) stamps. The features are based on color and printing properties of the stamps. Forged stamps are considered to be outliers. The stamps data set is not taken from the UCI repository, but was used in [1].

References:

[1] B. Micenkova, J. van Beusekom, and F. Shafait. Stamp verification for automated document authentication. In 5th Int. Workshop on Computational Forensics, 2012.

Download all data set variants used (371.2 kB).

Normalized, without duplicates

This version contains 9 attributes, 325 objects, 16 outliers (4.92%)

Download raw algorithm results (2.8 MB) Download raw algorithm evaluation table (51.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.12500 0.07969 0.11445 0.06859 0.22000 0.17961 0.75890
KNN 72 0.06250 0.01396 0.19457 0.15287 0.42857 0.39898 0.89745
KNNW 1 0.12500 0.07969 0.10442 0.05805 0.17544 0.13274 0.69711
KNNW 87 0.06250 0.01396 0.19020 0.14826 0.41667 0.38646 0.89502
KNNW 92 0.06250 0.01396 0.19100 0.14911 0.41667 0.38646 0.89563
LOF 2 0.06250 0.01396 0.04872 -0.00054 0.10000 0.05340 0.45024
LOF 94 0.06250 0.01396 0.15422 0.11042 0.34483 0.31090 0.86307
LOF 100 0.06250 0.01396 0.15761 0.11399 0.34043 0.30627 0.86711
SimplifiedLOF 1 0.06250 0.01396 0.05478 0.00584 0.12121 0.07571 0.47886
SimplifiedLOF 95 0.06250 0.01396 0.11951 0.07392 0.28261 0.24546 0.80583
SimplifiedLOF 100 0.06250 0.01396 0.12117 0.07566 0.27368 0.23608 0.80987
LoOP 1 0.06250 0.01396 0.05555 0.00665 0.12121 0.07571 0.48888
LoOP 94 0.06250 0.01396 0.11543 0.06963 0.25743 0.21898 0.79713
LoOP 100 0.06250 0.01396 0.11703 0.07131 0.24528 0.20620 0.80178
LDOF 2 0.06250 0.01396 0.07149 0.02341 0.15909 0.11555 0.58070
LDOF 50 0.06250 0.01396 0.11118 0.06516 0.24060 0.20128 0.79652
LDOF 82 0.06250 0.01396 0.11143 0.06541 0.25000 0.21117 0.78985
LDOF 100 0.06250 0.01396 0.11487 0.06904 0.24444 0.20532 0.79632
ODIN 19 0.07143 0.02335 0.09045 0.04336 0.16867 0.12563 0.70631
ODIN 47 0.06250 0.01396 0.13050 0.08548 0.29268 0.25606 0.82251
ODIN 48 0.06250 0.01396 0.13265 0.08774 0.28205 0.24488 0.82342
ODIN 55 0.06250 0.01396 0.13407 0.08923 0.26316 0.22500 0.82191
FastABOD 3 0.12500 0.07969 0.09761 0.05088 0.17391 0.13114 0.65817
FastABOD 78 0.12500 0.07969 0.12359 0.07821 0.26374 0.22561 0.80340
FastABOD 99 0.12500 0.07969 0.12440 0.07906 0.26374 0.22561 0.80583
KDEOS 5 0.12500 0.07969 0.07000 0.02184 0.17391 0.13114 0.48362
KDEOS 92 0.06250 0.01396 0.12191 0.07645 0.21239 0.17161 0.75546
KDEOS 93 0.06250 0.01396 0.10429 0.05791 0.21705 0.17651 0.75688
KDEOS 99 0.06250 0.01396 0.10587 0.05958 0.22400 0.18382 0.75324
LDF 1 0.06250 0.01396 0.05903 0.01031 0.13333 0.08846 0.46541
LDF 96 0.06250 0.01396 0.18529 0.14310 0.41096 0.38046 0.89118
LDF 100 0.06250 0.01396 0.18759 0.14553 0.40000 0.36893 0.89260
INFLO 12 0.06250 0.01396 0.05436 0.00539 0.11364 0.06774 0.52134
INFLO 82 0.06250 0.01396 0.10736 0.06114 0.26000 0.22168 0.75405
INFLO 99 0.06250 0.01396 0.11633 0.07058 0.25926 0.22090 0.77124
INFLO 100 0.06250 0.01396 0.11665 0.07091 0.25455 0.21595 0.77104
COF 12 0.12500 0.07969 0.07053 0.02240 0.15873 0.11517 0.55886
COF 36 0.06250 0.01396 0.16933 0.12632 0.34483 0.31090 0.84931
COF 37 0.06250 0.01396 0.16918 0.12616 0.34483 0.31090 0.85356

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 9 attributes, 325 objects, 16 outliers (4.92%)

Download raw algorithm results (2.8 MB) Download raw algorithm evaluation table (51.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.12500 0.07969 0.12482 0.07951 0.22917 0.18925 0.78155
KNN 47 0.00000 -0.05178 0.22536 0.18525 0.45614 0.42798 0.91505
KNN 67 0.00000 -0.05178 0.21995 0.17955 0.47458 0.44737 0.91121
KNNW 1 0.12500 0.07969 0.10968 0.06358 0.19048 0.14856 0.72522
KNNW 49 0.00000 -0.05178 0.21180 0.17099 0.45902 0.43100 0.90858
KNNW 83 0.00000 -0.05178 0.21930 0.17888 0.45714 0.42903 0.91282
LOF 1 0.06250 0.01396 0.06956 0.02138 0.12698 0.08178 0.45914
LOF 96 0.00000 -0.05178 0.16545 0.12223 0.36842 0.33572 0.87358
LOF 100 0.00000 -0.05178 0.17023 0.12727 0.36842 0.33572 0.87803
SimplifiedLOF 1 0.06250 0.01396 0.06170 0.01312 0.15000 0.10599 0.48756
SimplifiedLOF 97 0.00000 -0.05178 0.11939 0.07379 0.27368 0.23608 0.80967
SimplifiedLOF 100 0.00000 -0.05178 0.12015 0.07459 0.26923 0.23139 0.81169
LoOP 1 0.06250 0.01396 0.06247 0.01393 0.15000 0.10599 0.49707
LoOP 100 0.00000 -0.05178 0.11871 0.07307 0.26923 0.23139 0.80926
LDOF 2 0.06250 0.01396 0.06006 0.01139 0.12987 0.08481 0.53034
LDOF 96 0.00000 -0.05178 0.11515 0.06933 0.27160 0.23389 0.80016
LDOF 100 0.00000 -0.05178 0.11626 0.07050 0.26190 0.22369 0.80400
ODIN 11 0.08929 0.04213 0.06661 0.01828 0.13095 0.08595 0.59618
ODIN 41 0.00000 -0.05178 0.12917 0.08408 0.29545 0.25897 0.81119
ODIN 53 0.00000 -0.05178 0.13078 0.08578 0.27957 0.24227 0.82575
ODIN 55 0.00000 -0.05178 0.13523 0.09045 0.28571 0.24873 0.82423
FastABOD 3 0.12500 0.07969 0.11013 0.06405 0.18182 0.13945 0.70125
FastABOD 93 0.12500 0.07969 0.12877 0.08365 0.26966 0.23185 0.82201
FastABOD 94 0.12500 0.07969 0.12890 0.08379 0.26966 0.23185 0.82221
FastABOD 96 0.12500 0.07969 0.12906 0.08397 0.26966 0.23185 0.82221
KDEOS 6 0.12500 0.07969 0.05393 0.00494 0.13333 0.08846 0.43244
KDEOS 57 0.06250 0.01396 0.13653 0.09182 0.17045 0.12750 0.69701
KDEOS 95 0.06250 0.01396 0.09805 0.05135 0.22222 0.18195 0.75344
LDF 1 0.12500 0.07969 0.09660 0.04983 0.17391 0.13114 0.45206
LDF 94 0.00000 -0.05178 0.21815 0.17767 0.46667 0.43905 0.90959
LDF 100 0.06250 0.01396 0.22001 0.17962 0.46667 0.43905 0.91060
INFLO 1 0.12500 0.07969 0.08376 0.03632 0.17143 0.12853 0.53085
INFLO 97 0.00000 -0.05178 0.12496 0.07965 0.27027 0.23248 0.80400
INFLO 98 0.00000 -0.05178 0.12369 0.07832 0.27778 0.24038 0.80097
COF 13 0.18750 0.14543 0.07651 0.02869 0.19355 0.15179 0.54308
COF 34 0.00000 -0.05178 0.14057 0.09607 0.30612 0.27019 0.82909
COF 36 0.00000 -0.05178 0.14044 0.09593 0.30233 0.26620 0.83374

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO