Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (2% of outliers version#09)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1630 objects, 32 outliers (1.96%)

Download raw algorithm results (10.1 MB) Download raw algorithm evaluation table (56.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.50000 0.48999 0.48664 0.47636 0.59259 0.58443 0.85842
KNN 3 0.50000 0.48999 0.48497 0.47466 0.61538 0.60768 0.83273
KNNW 3 0.34375 0.33061 0.38268 0.37032 0.44898 0.43795 0.86440
KNNW 5 0.50000 0.48999 0.45332 0.44238 0.51613 0.50644 0.85522
KNNW 12 0.50000 0.48999 0.47177 0.46119 0.57143 0.56285 0.79642
KNNW 13 0.50000 0.48999 0.47233 0.46177 0.57143 0.56285 0.79055
LOF 4 0.46875 0.45811 0.43178 0.42040 0.50000 0.48999 0.90442
LOF 5 0.40625 0.39436 0.42925 0.41782 0.48649 0.47620 0.91574
LOF 20 0.46875 0.45811 0.47959 0.46917 0.55556 0.54666 0.86350
SimplifiedLOF 6 0.40625 0.39436 0.35506 0.34214 0.43333 0.42199 0.91408
SimplifiedLOF 13 0.46875 0.45811 0.42743 0.41597 0.48387 0.47354 0.89199
SimplifiedLOF 20 0.46875 0.45811 0.50732 0.49746 0.56522 0.55651 0.87973
LoOP 9 0.56250 0.55374 0.46441 0.45369 0.59016 0.58196 0.93022
LoOP 11 0.59375 0.58561 0.47752 0.46706 0.62295 0.61540 0.92582
LoOP 17 0.59375 0.58561 0.51084 0.50104 0.62295 0.61540 0.91943
LDOF 9 0.34375 0.33061 0.34172 0.32854 0.40000 0.38798 0.90634
LDOF 44 0.46875 0.45811 0.45954 0.44872 0.56000 0.55119 0.85973
LDOF 54 0.46875 0.45811 0.51332 0.50358 0.61224 0.60448 0.84310
LDOF 55 0.46875 0.45811 0.51346 0.50372 0.61224 0.60448 0.84260
ODIN 27 0.30966 0.29583 0.24026 0.22505 0.36800 0.35534 0.90242
ODIN 93 0.37946 0.36704 0.29020 0.27599 0.42424 0.41271 0.85663
ODIN 96 0.36719 0.35452 0.29565 0.28155 0.42424 0.41271 0.85584
FastABOD 21 0.34375 0.33061 0.30970 0.29588 0.37500 0.36248 0.82883
FastABOD 26 0.37500 0.36248 0.30163 0.28764 0.38095 0.36856 0.81369
FastABOD 97 0.37500 0.36248 0.32847 0.31503 0.41463 0.40291 0.77783
FastABOD 98 0.37500 0.36248 0.33371 0.32037 0.41463 0.40291 0.77791
KDEOS 67 0.21875 0.20311 0.15176 0.13477 0.24691 0.23183 0.82556
KDEOS 68 0.21875 0.20311 0.16132 0.14452 0.25641 0.24152 0.82597
KDEOS 69 0.21875 0.20311 0.16104 0.14424 0.25974 0.24492 0.82644
LDF 3 0.03916 0.01992 0.03903 0.01979 0.09015 0.07193 0.71727
LDF 4 0.01681 -0.00288 0.04156 0.02237 0.11738 0.09971 0.77971
LDF 100 0.00000 -0.02003 0.03884 0.01959 0.13397 0.11663 0.63785
INFLO 4 0.46875 0.45811 0.36059 0.34778 0.48000 0.46959 0.87308
INFLO 6 0.34375 0.33061 0.35255 0.33958 0.43836 0.42711 0.91315
INFLO 20 0.46875 0.45811 0.52097 0.51138 0.56604 0.55735 0.88534
INFLO 52 0.46875 0.45811 0.49602 0.48592 0.60000 0.59199 0.82551
COF 2 0.09375 0.07560 0.09980 0.08177 0.14286 0.12569 0.64028
COF 9 0.06250 0.04373 0.07555 0.05704 0.19178 0.17560 0.51339
COF 16 0.18750 0.17123 0.05085 0.03184 0.18750 0.17123 0.47818
COF 77 0.09375 0.07560 0.11173 0.09394 0.17143 0.15484 0.50810

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2867 objects, 57 outliers (1.99%)

Download raw algorithm results (12.0 MB) Download raw algorithm evaluation table (68.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.44928 0.43810 0.44139 0.43006 0.48780 0.47742 0.89825
KNN 2 0.49769 0.48750 0.50096 0.49084 0.56471 0.55588 0.89708
KNNW 4 0.49123 0.48091 0.47351 0.46283 0.52830 0.51873 0.89595
KNNW 8 0.49123 0.48091 0.49553 0.48529 0.55556 0.54654 0.88852
KNNW 11 0.47368 0.46301 0.49218 0.48188 0.57471 0.56609 0.87945
LOF 7 0.04746 0.02814 0.04700 0.02767 0.11714 0.09923 0.76203
LOF 10 0.04292 0.02350 0.04563 0.02628 0.10363 0.08544 0.78443
SimplifiedLOF 7 0.04397 0.02458 0.04178 0.02234 0.09292 0.07452 0.73649
SimplifiedLOF 8 0.04651 0.02717 0.04320 0.02379 0.09061 0.07217 0.74139
SimplifiedLOF 10 0.04184 0.02240 0.04143 0.02199 0.08875 0.07027 0.75076
LoOP 1 0.18583 0.16932 0.08172 0.06310 0.19231 0.17592 0.65855
LoOP 2 0.16667 0.14976 0.08335 0.06476 0.19858 0.18233 0.73707
LoOP 15 0.05263 0.03341 0.08324 0.06464 0.17037 0.15354 0.83013
LoOP 16 0.05263 0.03341 0.08639 0.06786 0.16858 0.15172 0.82951
LDOF 5 0.08772 0.06921 0.04822 0.02891 0.11077 0.09273 0.72093
LDOF 18 0.08772 0.06921 0.06061 0.04155 0.13968 0.12223 0.79767
LDOF 77 0.08772 0.06921 0.06959 0.05072 0.17692 0.16023 0.77952
ODIN 15 0.10526 0.08711 0.10823 0.09014 0.22727 0.21160 0.85084
ODIN 90 0.33825 0.32482 0.21973 0.20390 0.40625 0.39421 0.83961
ODIN 100 0.33825 0.32482 0.21992 0.20409 0.40625 0.39421 0.83896
FastABOD 15 0.01340 -0.00661 0.04311 0.02370 0.12553 0.10780 0.77191
FastABOD 30 0.00000 -0.02028 0.05929 0.04020 0.16738 0.15049 0.78129
FastABOD 73 0.00000 -0.02028 0.06188 0.04285 0.15258 0.13539 0.78498
FastABOD 97 0.00000 -0.02028 0.06317 0.04417 0.15966 0.14262 0.78383
KDEOS 2 0.00000 -0.02028 0.03065 0.01099 0.08470 0.06613 0.64572
KDEOS 10 0.00000 -0.02028 0.03909 0.01960 0.10087 0.08263 0.73385
LDF 1 0.07527 0.05651 0.03073 0.01107 0.12335 0.10557 0.40433
LDF 6 0.04800 0.02869 0.02864 0.00893 0.06593 0.04699 0.59630
INFLO 7 0.04754 0.02822 0.04582 0.02647 0.11429 0.09632 0.75739
COF 74 0.07018 0.05131 0.06553 0.04658 0.16901 0.15216 0.74709
COF 76 0.07018 0.05131 0.06734 0.04842 0.17561 0.15889 0.75166
COF 77 0.07018 0.05131 0.06624 0.04730 0.17822 0.16155 0.72971

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO