Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (2% of outliers version#08)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1630 objects, 32 outliers (1.96%)

Download raw algorithm results (10.0 MB) Download raw algorithm evaluation table (57.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.41346 0.40172 0.45510 0.44418 0.52174 0.51216 0.85889
KNN 2 0.48958 0.47936 0.48248 0.47212 0.53846 0.52922 0.85246
KNN 4 0.46875 0.45811 0.45014 0.43913 0.56000 0.55119 0.81841
KNNW 2 0.46875 0.45811 0.45298 0.44203 0.50000 0.48999 0.86972
KNNW 3 0.53125 0.52186 0.48927 0.47904 0.53125 0.52186 0.86825
KNNW 5 0.50000 0.48999 0.50584 0.49595 0.56604 0.55735 0.85172
KNNW 14 0.46875 0.45811 0.47023 0.45962 0.58824 0.57999 0.79235
LOF 4 0.43750 0.42624 0.35038 0.33737 0.45455 0.44362 0.81858
LOF 9 0.37500 0.36248 0.38960 0.37737 0.42308 0.41152 0.89510
LOF 27 0.43750 0.42624 0.44316 0.43201 0.49123 0.48104 0.85521
LOF 74 0.43750 0.42624 0.41306 0.40131 0.51852 0.50888 0.77049
SimplifiedLOF 7 0.43750 0.42624 0.41179 0.40002 0.44444 0.43332 0.88822
SimplifiedLOF 9 0.40625 0.39436 0.41855 0.40691 0.42857 0.41713 0.89925
SimplifiedLOF 30 0.43750 0.42624 0.47217 0.46161 0.50909 0.49926 0.85777
SimplifiedLOF 50 0.43750 0.42624 0.45206 0.44109 0.52000 0.51039 0.82175
LoOP 12 0.46875 0.45811 0.50172 0.49174 0.54321 0.53406 0.91767
LoOP 14 0.56250 0.55374 0.52062 0.51102 0.57143 0.56285 0.91626
LoOP 16 0.56250 0.55374 0.53121 0.52183 0.60000 0.59199 0.91659
LoOP 19 0.56250 0.55374 0.53172 0.52234 0.58182 0.57344 0.91312
LDOF 13 0.37500 0.36248 0.37189 0.35931 0.37500 0.36248 0.90185
LDOF 51 0.53125 0.52186 0.49325 0.48310 0.53125 0.52186 0.86383
LDOF 53 0.53125 0.52186 0.49779 0.48773 0.53125 0.52186 0.85675
ODIN 29 0.32857 0.31513 0.21762 0.20196 0.36364 0.35089 0.86039
ODIN 97 0.38393 0.37159 0.27248 0.25791 0.38806 0.37581 0.84838
ODIN 100 0.38393 0.37159 0.27268 0.25812 0.38806 0.37581 0.84854
FastABOD 15 0.07143 0.05283 0.11300 0.09524 0.28571 0.27141 0.86983
FastABOD 16 0.40625 0.39436 0.38775 0.37549 0.48148 0.47110 0.83574
FastABOD 17 0.43750 0.42624 0.38244 0.37007 0.46154 0.45076 0.82624
FastABOD 20 0.40625 0.39436 0.42108 0.40948 0.47059 0.45999 0.84548
KDEOS 7 0.15625 0.13935 0.16355 0.14680 0.26316 0.24840 0.80292
KDEOS 64 0.21875 0.20311 0.14432 0.12718 0.27160 0.25702 0.83448
KDEOS 67 0.25000 0.23498 0.13417 0.11684 0.26087 0.24607 0.82410
KDEOS 68 0.25000 0.23498 0.14689 0.12980 0.28947 0.27525 0.82681
LDF 5 0.04071 0.02150 0.03905 0.01981 0.08764 0.06937 0.73570
LDF 100 0.01923 -0.00041 0.05467 0.03574 0.18987 0.17365 0.68499
INFLO 13 0.37500 0.36248 0.44330 0.43216 0.41860 0.40696 0.91252
INFLO 23 0.50000 0.48999 0.47569 0.46519 0.50794 0.49808 0.88327
INFLO 29 0.50000 0.48999 0.49047 0.48027 0.50909 0.49926 0.87185
INFLO 50 0.43750 0.42624 0.46447 0.45375 0.52830 0.51886 0.83415
COF 4 0.18750 0.17123 0.07666 0.05817 0.21538 0.19967 0.62593
COF 5 0.25000 0.23498 0.13075 0.11335 0.27692 0.26244 0.60246
COF 7 0.12500 0.10748 0.13858 0.12133 0.22472 0.20919 0.50049

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2867 objects, 57 outliers (1.99%)

Download raw algorithm results (12.0 MB) Download raw algorithm evaluation table (67.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.43021 0.41865 0.42045 0.40869 0.48276 0.47227 0.88763
KNN 2 0.42982 0.41826 0.47282 0.46213 0.53488 0.52545 0.88222
KNNW 3 0.41228 0.40036 0.42818 0.41659 0.46154 0.45062 0.88810
KNNW 4 0.42105 0.40931 0.45079 0.43965 0.50000 0.48986 0.88393
KNNW 6 0.40351 0.39141 0.46372 0.45284 0.54762 0.53844 0.87865
KNNW 7 0.40351 0.39141 0.46333 0.45244 0.55422 0.54517 0.87552
LOF 7 0.04902 0.02973 0.05196 0.03273 0.12339 0.10560 0.80681
LOF 8 0.04603 0.02668 0.05115 0.03190 0.12092 0.10309 0.80991
SimplifiedLOF 8 0.04797 0.02865 0.04728 0.02795 0.10022 0.08197 0.78146
SimplifiedLOF 9 0.04438 0.02499 0.04553 0.02617 0.10033 0.08209 0.77746
LoOP 2 0.26706 0.25219 0.17803 0.16136 0.27907 0.26445 0.76928
LoOP 12 0.08025 0.06159 0.12846 0.11078 0.16185 0.14485 0.83418
LDOF 10 0.08772 0.06921 0.11222 0.09421 0.18487 0.16834 0.81016
LDOF 73 0.14035 0.12291 0.12670 0.10898 0.21429 0.19835 0.79515
LDOF 74 0.14035 0.12291 0.12977 0.11212 0.22642 0.21072 0.79744
ODIN 82 0.36842 0.35561 0.29383 0.27950 0.47826 0.46768 0.88948
ODIN 97 0.36204 0.34910 0.29828 0.28405 0.47826 0.46768 0.88267
FastABOD 29 0.03597 0.01642 0.06295 0.04394 0.16444 0.14750 0.79472
FastABOD 31 0.03200 0.01236 0.06496 0.04599 0.16895 0.15209 0.79691
FastABOD 73 0.03509 0.01551 0.06915 0.05026 0.17209 0.15530 0.79377
FastABOD 75 0.03509 0.01551 0.06802 0.04912 0.17290 0.15612 0.79249
KDEOS 9 0.04918 0.02989 0.04445 0.02506 0.09877 0.08048 0.75656
KDEOS 73 0.07018 0.05131 0.04000 0.02052 0.09497 0.07661 0.68210
KDEOS 79 0.07018 0.05131 0.03975 0.02028 0.10458 0.08641 0.65756
LDF 1 0.03371 0.01411 0.02218 0.00234 0.09217 0.07375 0.41273
LDF 3 0.04237 0.02295 0.02557 0.00581 0.07968 0.06101 0.52986
LDF 6 0.00870 -0.01141 0.02652 0.00678 0.05283 0.03361 0.59387
INFLO 7 0.04910 0.02981 0.04976 0.03048 0.12251 0.10471 0.78469
INFLO 8 0.04610 0.02676 0.04982 0.03055 0.12138 0.10356 0.80100
COF 73 0.12281 0.10501 0.07591 0.05716 0.18685 0.17036 0.76685
COF 77 0.12281 0.10501 0.07997 0.06131 0.20661 0.19052 0.77622
COF 87 0.07018 0.05131 0.07222 0.05340 0.19008 0.17365 0.77634

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO