Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (5% of outliers version#07)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1682 objects, 84 outliers (4.99%)

Download raw algorithm results (10.5 MB) Download raw algorithm evaluation table (66.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.44286 0.41357 0.44534 0.41618 0.45312 0.42438 0.82362
KNN 3 0.45982 0.43143 0.48435 0.45725 0.48951 0.46268 0.82254
KNN 11 0.44335 0.41409 0.43708 0.40749 0.52174 0.49660 0.73457
KNNW 4 0.45238 0.42359 0.45818 0.42970 0.46914 0.44123 0.83262
KNNW 7 0.48810 0.46119 0.49050 0.46372 0.50617 0.48021 0.81861
KNNW 8 0.48810 0.46119 0.49123 0.46448 0.50000 0.47372 0.81202
KNNW 26 0.44048 0.41106 0.46634 0.43829 0.52857 0.50379 0.74655
LOF 21 0.41667 0.38600 0.42383 0.39354 0.47761 0.45015 0.81826
LOF 26 0.45238 0.42359 0.44648 0.41738 0.50000 0.47372 0.81214
LOF 28 0.46429 0.43613 0.43630 0.40667 0.49333 0.46670 0.81490
SimplifiedLOF 26 0.50000 0.47372 0.46612 0.43805 0.51389 0.48834 0.82709
SimplifiedLOF 28 0.48810 0.46119 0.46859 0.44065 0.51656 0.49114 0.83292
LoOP 14 0.46429 0.43613 0.43271 0.40289 0.51562 0.49016 0.81315
LoOP 25 0.40476 0.37347 0.42515 0.39494 0.45255 0.42378 0.83791
LoOP 92 0.50000 0.47372 0.45451 0.42583 0.50314 0.47703 0.79500
LoOP 93 0.50000 0.47372 0.45458 0.42591 0.50633 0.48038 0.79459
LDOF 28 0.39286 0.36094 0.39069 0.35866 0.41573 0.38502 0.84250
LDOF 78 0.48810 0.46119 0.45159 0.42276 0.50955 0.48377 0.79702
ODIN 30 0.33280 0.29773 0.24297 0.20318 0.42241 0.39205 0.81479
ODIN 35 0.34021 0.30553 0.24695 0.20736 0.42982 0.39985 0.81133
ODIN 39 0.33825 0.30347 0.24439 0.20467 0.43049 0.40056 0.80912
ODIN 94 0.36685 0.33357 0.23664 0.19652 0.40212 0.37069 0.77415
FastABOD 17 0.46429 0.43613 0.30505 0.26852 0.46707 0.43905 0.78241
FastABOD 21 0.46429 0.43613 0.30666 0.27021 0.46857 0.44064 0.78966
FastABOD 23 0.44048 0.41106 0.30879 0.27245 0.45810 0.42962 0.78995
FastABOD 99 0.44048 0.41106 0.31401 0.27795 0.44048 0.41106 0.75217
KDEOS 53 0.11905 0.07274 0.14980 0.10511 0.16616 0.12233 0.70331
KDEOS 64 0.20238 0.16045 0.11555 0.06906 0.22093 0.17998 0.73007
KDEOS 67 0.22619 0.18551 0.12295 0.07685 0.22619 0.18551 0.72413
KDEOS 83 0.16667 0.12286 0.12384 0.07779 0.23232 0.19197 0.71165
LDF 3 0.09510 0.04753 0.07924 0.03084 0.17077 0.12718 0.58612
LDF 6 0.03745 -0.01314 0.07697 0.02845 0.19134 0.14883 0.68335
LDF 99 0.08333 0.03515 0.08590 0.03785 0.20984 0.16830 0.64304
LDF 100 0.08333 0.03515 0.08649 0.03847 0.20984 0.16830 0.64567
INFLO 26 0.48810 0.46119 0.45126 0.42241 0.51899 0.49370 0.82515
INFLO 28 0.47619 0.44866 0.44216 0.41284 0.50000 0.47372 0.83350
INFLO 94 0.50000 0.47372 0.46138 0.43306 0.50000 0.47372 0.77460
INFLO 95 0.50000 0.47372 0.46164 0.43334 0.50314 0.47703 0.77352
COF 2 0.13095 0.08527 0.09508 0.04751 0.16265 0.11863 0.62408
COF 6 0.20238 0.16045 0.12088 0.07466 0.21302 0.17165 0.57526

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2957 objects, 147 outliers (4.97%)

Download raw algorithm results (12.6 MB) Download raw algorithm evaluation table (72.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.36147 0.32807 0.41852 0.38810 0.39149 0.35966 0.87929
KNN 6 0.47096 0.44328 0.49914 0.47294 0.50644 0.48062 0.86795
KNN 7 0.46272 0.43462 0.50066 0.47453 0.50584 0.47999 0.86197
KNN 8 0.46712 0.43924 0.50003 0.47388 0.51556 0.49021 0.85117
KNNW 7 0.38095 0.34857 0.44485 0.41581 0.42017 0.38984 0.87789
KNNW 14 0.46259 0.43447 0.48959 0.46289 0.48000 0.45280 0.86131
KNNW 16 0.45578 0.42731 0.49380 0.46732 0.49091 0.46428 0.85431
KNNW 32 0.44898 0.42015 0.48331 0.45628 0.53456 0.51021 0.81010
LOF 7 0.08833 0.04064 0.09428 0.04690 0.19394 0.15177 0.73524
LOF 8 0.08742 0.03968 0.09431 0.04693 0.20053 0.15870 0.73708
SimplifiedLOF 8 0.08848 0.04080 0.08300 0.03503 0.15020 0.10574 0.68829
SimplifiedLOF 9 0.08628 0.03848 0.08342 0.03547 0.16029 0.11636 0.69117
SimplifiedLOF 11 0.08596 0.03814 0.08278 0.03480 0.16493 0.12125 0.68895
LoOP 12 0.08163 0.03359 0.10356 0.05666 0.21574 0.17472 0.73083
LoOP 71 0.19048 0.14813 0.12174 0.07579 0.24201 0.20236 0.71235
LDOF 74 0.17007 0.12665 0.11261 0.06618 0.22122 0.18048 0.70601
LDOF 77 0.17007 0.12665 0.11675 0.07054 0.23445 0.19440 0.71093
LDOF 78 0.17007 0.12665 0.11727 0.07109 0.23445 0.19440 0.71490
LDOF 93 0.17007 0.12665 0.11411 0.06777 0.21308 0.17191 0.71854
ODIN 16 0.20426 0.16263 0.17214 0.12883 0.32168 0.28619 0.78415
ODIN 84 0.37332 0.34053 0.24495 0.20545 0.43597 0.40646 0.75720
ODIN 91 0.37279 0.33998 0.24137 0.20169 0.44199 0.41280 0.75631
ODIN 99 0.38184 0.34950 0.24256 0.20294 0.43575 0.40624 0.75388
FastABOD 32 0.04082 -0.00936 0.11280 0.06639 0.25179 0.21265 0.76504
FastABOD 72 0.10204 0.05507 0.11518 0.06889 0.25126 0.21209 0.76077
FastABOD 82 0.10204 0.05507 0.11591 0.06966 0.25503 0.21606 0.76262
FastABOD 95 0.10204 0.05507 0.11543 0.06915 0.26377 0.22526 0.75963
KDEOS 2 0.04466 -0.00532 0.06936 0.02067 0.17754 0.13451 0.61445
KDEOS 10 0.08163 0.03359 0.08037 0.03226 0.15724 0.11315 0.68041
KDEOS 11 0.08163 0.03359 0.08016 0.03204 0.15601 0.11186 0.68263
KDEOS 73 0.10884 0.06222 0.07046 0.02183 0.13318 0.08784 0.62422
LDF 1 0.17036 0.12696 0.07558 0.02722 0.21277 0.17158 0.36781
LDF 20 0.12500 0.07923 0.07004 0.02139 0.15504 0.11084 0.58561
INFLO 7 0.08847 0.04078 0.08466 0.03678 0.17986 0.13695 0.69294
INFLO 9 0.08841 0.04072 0.08833 0.04064 0.18595 0.14336 0.70892
INFLO 10 0.08803 0.04033 0.08840 0.04072 0.18389 0.14119 0.71701
COF 74 0.17007 0.12665 0.10836 0.06172 0.23773 0.19785 0.69074
COF 77 0.17007 0.12665 0.11604 0.06980 0.25121 0.21204 0.70206

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO