Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (2% of outliers version#10)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1630 objects, 32 outliers (1.96%)

Download raw algorithm results (10.2 MB) Download raw algorithm evaluation table (55.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.35227 0.33930 0.38360 0.37126 0.39216 0.37998 0.91107
KNN 3 0.50000 0.48999 0.47447 0.46394 0.55556 0.54666 0.89589
KNNW 3 0.40625 0.39436 0.40937 0.39754 0.46154 0.45076 0.92002
KNNW 6 0.50000 0.48999 0.47211 0.46154 0.53571 0.52642 0.90562
KNNW 11 0.50000 0.48999 0.48470 0.47439 0.54902 0.53999 0.87386
KNNW 12 0.50000 0.48999 0.48170 0.47132 0.55172 0.54275 0.86772
LOF 10 0.28125 0.26686 0.28248 0.26811 0.36364 0.35089 0.91180
LOF 22 0.50000 0.48999 0.45267 0.44171 0.50746 0.49760 0.88267
LOF 27 0.50000 0.48999 0.49381 0.48367 0.51724 0.50757 0.87627
LOF 71 0.46875 0.45811 0.48054 0.47013 0.52632 0.51683 0.83251
SimplifiedLOF 10 0.28125 0.26686 0.30687 0.29299 0.38298 0.37062 0.92516
SimplifiedLOF 22 0.50000 0.48999 0.48480 0.47448 0.52174 0.51216 0.90031
SimplifiedLOF 27 0.50000 0.48999 0.52833 0.51889 0.53333 0.52399 0.89203
LoOP 14 0.40625 0.39436 0.35261 0.33965 0.43333 0.42199 0.94452
LoOP 45 0.50000 0.48999 0.44508 0.43397 0.50000 0.48999 0.90232
LoOP 84 0.50000 0.48999 0.51092 0.50113 0.53333 0.52399 0.86956
LoOP 93 0.50000 0.48999 0.51429 0.50456 0.52174 0.51216 0.86379
LDOF 27 0.43750 0.42624 0.40823 0.39638 0.47222 0.46165 0.91884
LDOF 56 0.50000 0.48999 0.48512 0.47481 0.50794 0.49808 0.89602
LDOF 73 0.50000 0.48999 0.50154 0.49155 0.52459 0.51507 0.88136
ODIN 33 0.33705 0.32378 0.20350 0.18755 0.34146 0.32828 0.87489
ODIN 55 0.34000 0.32678 0.20639 0.19050 0.36364 0.35089 0.87384
ODIN 96 0.35938 0.34655 0.21284 0.19707 0.36364 0.35089 0.84327
ODIN 100 0.35938 0.34655 0.21319 0.19743 0.36364 0.35089 0.84379
FastABOD 22 0.37500 0.36248 0.33933 0.32610 0.45946 0.44864 0.89201
FastABOD 30 0.40625 0.39436 0.35494 0.34203 0.45000 0.43899 0.88531
FastABOD 73 0.43750 0.42624 0.33486 0.32154 0.44444 0.43332 0.86909
FastABOD 98 0.40625 0.39436 0.33234 0.31897 0.46154 0.45076 0.84062
KDEOS 4 0.15625 0.13935 0.11118 0.09338 0.20833 0.19248 0.67530
KDEOS 5 0.15625 0.13935 0.11035 0.09254 0.21739 0.20172 0.72870
KDEOS 7 0.15625 0.13935 0.11344 0.09569 0.16949 0.15286 0.72256
KDEOS 69 0.12500 0.10748 0.09245 0.07428 0.17021 0.15360 0.81889
LDF 4 0.05306 0.03410 0.05604 0.03714 0.12109 0.10349 0.83165
INFLO 10 0.34375 0.33061 0.28498 0.27066 0.34375 0.33061 0.91886
INFLO 27 0.50000 0.48999 0.51669 0.50701 0.53333 0.52399 0.90025
INFLO 56 0.53125 0.52186 0.49284 0.48269 0.53125 0.52186 0.86874
INFLO 96 0.50000 0.48999 0.51057 0.50077 0.54167 0.53249 0.84371
COF 7 0.12500 0.10748 0.08845 0.07019 0.15217 0.13520 0.67788
COF 14 0.18750 0.17123 0.07378 0.05523 0.21212 0.19634 0.53106
COF 70 0.18750 0.17123 0.15116 0.13416 0.24390 0.22876 0.50219

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2867 objects, 57 outliers (1.99%)

Download raw algorithm results (12.0 MB) Download raw algorithm evaluation table (66.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.54386 0.53461 0.59130 0.58301 0.65934 0.65243 0.90834
KNN 4 0.54386 0.53461 0.59063 0.58233 0.65934 0.65243 0.91041
KNN 6 0.54094 0.53162 0.57307 0.56441 0.66667 0.65991 0.89833
KNNW 7 0.49123 0.48091 0.56457 0.55573 0.60465 0.59663 0.91157
KNNW 13 0.52632 0.51671 0.57458 0.56595 0.60674 0.59876 0.89566
KNNW 14 0.54386 0.53461 0.57407 0.56543 0.61364 0.60580 0.89240
KNNW 17 0.54386 0.53461 0.57070 0.56199 0.62626 0.61868 0.88361
LOF 7 0.05401 0.03482 0.05703 0.03790 0.13296 0.11538 0.82679
LOF 9 0.05405 0.03487 0.05692 0.03779 0.12783 0.11014 0.83440
LOF 10 0.05421 0.03502 0.05626 0.03712 0.12276 0.10497 0.83173
SimplifiedLOF 9 0.05279 0.03357 0.05143 0.03218 0.10289 0.08470 0.80051
SimplifiedLOF 11 0.05237 0.03315 0.05132 0.03207 0.10352 0.08534 0.80171
LoOP 2 0.26316 0.24821 0.20630 0.19020 0.27273 0.25797 0.73362
LoOP 28 0.07018 0.05131 0.12339 0.10561 0.20151 0.18531 0.87764
LDOF 2 0.05263 0.03341 0.06383 0.04484 0.09146 0.07303 0.57766
LDOF 77 0.05263 0.03341 0.12492 0.10717 0.20328 0.18712 0.85688
ODIN 82 0.32225 0.30851 0.26931 0.25449 0.43617 0.42473 0.90751
ODIN 90 0.32456 0.31086 0.27364 0.25891 0.45304 0.44194 0.90671
ODIN 98 0.32456 0.31086 0.27760 0.26295 0.46328 0.45239 0.90697
ODIN 100 0.32456 0.31086 0.27857 0.26394 0.46328 0.45239 0.90712
FastABOD 73 0.02655 0.00680 0.07945 0.06078 0.18970 0.17327 0.83333
FastABOD 88 0.02655 0.00680 0.08131 0.06267 0.19580 0.17949 0.83766
FastABOD 97 0.02655 0.00680 0.08187 0.06325 0.19580 0.17949 0.83792
KDEOS 10 0.03390 0.01430 0.04292 0.02351 0.10101 0.08277 0.76285
KDEOS 11 0.03390 0.01430 0.04243 0.02300 0.09887 0.08059 0.76420
KDEOS 73 0.05263 0.03341 0.03624 0.01669 0.08401 0.06543 0.69999
LDF 1 0.02273 0.00290 0.02465 0.00486 0.11060 0.09256 0.39418
LDF 4 0.03774 0.01822 0.02230 0.00247 0.05714 0.03802 0.46930
LDF 6 0.00000 -0.02028 0.02713 0.00740 0.05283 0.03361 0.55160
INFLO 8 0.05263 0.03341 0.05354 0.03434 0.12737 0.10967 0.80797
INFLO 9 0.05414 0.03495 0.05579 0.03664 0.12598 0.10826 0.82529
INFLO 10 0.05421 0.03502 0.05535 0.03619 0.12189 0.10408 0.82368
COF 74 0.12281 0.10501 0.09697 0.07865 0.24199 0.22662 0.81923
COF 77 0.12281 0.10501 0.10104 0.08281 0.26459 0.24967 0.82308

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO