Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (10% of outliers version#02)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1775 objects, 177 outliers (9.97%)

Download raw algorithm results (13.2 MB) Download raw algorithm evaluation table (73.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.44950 0.38852 0.49133 0.43499 0.46022 0.40043 0.83034
KNN 5 0.46546 0.40626 0.51543 0.46176 0.48525 0.42823 0.82692
KNN 7 0.48535 0.42834 0.50806 0.45357 0.48930 0.43273 0.79732
KNN 20 0.47232 0.41387 0.49548 0.43959 0.52769 0.47537 0.73581
KNNW 7 0.47458 0.41638 0.48641 0.42952 0.50122 0.44597 0.83717
KNNW 15 0.50847 0.45403 0.53939 0.48837 0.51039 0.45615 0.81214
KNNW 18 0.48588 0.42893 0.54034 0.48943 0.51282 0.45886 0.80335
KNNW 44 0.46893 0.41010 0.52579 0.47326 0.54962 0.49973 0.75376
LOF 53 0.45763 0.39755 0.51698 0.46348 0.48366 0.42647 0.79439
LOF 92 0.50282 0.44776 0.55096 0.50122 0.55749 0.50848 0.78433
LOF 93 0.50282 0.44776 0.55116 0.50145 0.55749 0.50848 0.78435
LOF 97 0.50282 0.44776 0.55114 0.50143 0.56250 0.51404 0.78179
SimplifiedLOF 53 0.45763 0.39755 0.52009 0.46694 0.48505 0.42801 0.80962
SimplifiedLOF 89 0.50847 0.45403 0.55460 0.50526 0.55663 0.50753 0.79924
SimplifiedLOF 97 0.50847 0.45403 0.55762 0.50862 0.56026 0.51155 0.79542
SimplifiedLOF 99 0.50847 0.45403 0.55635 0.50721 0.56209 0.51359 0.79445
LoOP 17 0.45763 0.39755 0.38920 0.32155 0.46334 0.40390 0.73430
LoOP 18 0.45763 0.39755 0.39009 0.32254 0.46884 0.41001 0.73554
LoOP 97 0.42373 0.35990 0.46717 0.40815 0.44586 0.38448 0.80155
LoOP 100 0.42938 0.36617 0.46736 0.40836 0.44667 0.38538 0.80120
LDOF 69 0.42373 0.35990 0.44141 0.37954 0.44248 0.38072 0.79317
LDOF 93 0.43503 0.37245 0.45132 0.39055 0.43626 0.37382 0.79966
LDOF 100 0.41808 0.35362 0.45150 0.39075 0.43333 0.37057 0.79928
ODIN 10 0.21815 0.13155 0.19786 0.10901 0.32734 0.25283 0.70196
ODIN 19 0.28362 0.20427 0.22402 0.13807 0.36211 0.29145 0.70001
ODIN 20 0.29434 0.21618 0.22541 0.13961 0.36170 0.29100 0.69725
ODIN 100 0.32768 0.25322 0.22393 0.13796 0.33256 0.25864 0.68748
FastABOD 24 0.48588 0.42893 0.42429 0.36052 0.49673 0.44099 0.81907
FastABOD 25 0.49153 0.43521 0.42239 0.35841 0.50898 0.45460 0.81787
FastABOD 27 0.49718 0.44148 0.41917 0.35484 0.50292 0.44787 0.81531
KDEOS 27 0.16384 0.07123 0.13856 0.04315 0.23930 0.15504 0.62130
KDEOS 56 0.14689 0.05240 0.16583 0.07343 0.23449 0.14970 0.63765
KDEOS 61 0.18079 0.09005 0.16154 0.06867 0.23886 0.15455 0.64960
KDEOS 68 0.20339 0.11515 0.15375 0.06002 0.23359 0.14870 0.64592
LDF 99 0.20904 0.12143 0.17151 0.07974 0.33469 0.26100 0.67522
LDF 100 0.21469 0.12771 0.17179 0.08006 0.33401 0.26025 0.67464
INFLO 69 0.46328 0.40383 0.51871 0.46540 0.49446 0.43847 0.80651
INFLO 100 0.49153 0.43521 0.53642 0.48507 0.53376 0.48212 0.80102
COF 4 0.19209 0.10260 0.15929 0.06617 0.26366 0.18210 0.64514
COF 7 0.23164 0.14653 0.16971 0.07775 0.24345 0.15965 0.58570
COF 8 0.22599 0.14026 0.17382 0.08231 0.24652 0.16306 0.57719
COF 51 0.16949 0.07750 0.16367 0.07103 0.26496 0.18354 0.59681

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 3122 objects, 312 outliers (9.99%)

Download raw algorithm results (13.6 MB) Download raw algorithm evaluation table (74.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.44712 0.38573 0.49796 0.44221 0.45939 0.39937 0.84031
KNN 13 0.49564 0.43964 0.54615 0.49576 0.54480 0.49426 0.80372
KNN 14 0.49673 0.44085 0.54287 0.49211 0.53805 0.48676 0.79844
KNNW 15 0.46474 0.40531 0.49931 0.44371 0.47333 0.41486 0.83911
KNNW 27 0.50000 0.44448 0.53838 0.48713 0.50575 0.45087 0.82072
KNNW 56 0.49359 0.43736 0.53141 0.47938 0.51866 0.46521 0.78261
LOF 9 0.12412 0.02687 0.14224 0.04700 0.27839 0.19827 0.65946
LOF 11 0.13548 0.03949 0.13787 0.04215 0.26108 0.17903 0.63246
SimplifiedLOF 10 0.13514 0.03911 0.13179 0.03540 0.23756 0.15290 0.61977
SimplifiedLOF 16 0.12197 0.02448 0.12419 0.02694 0.23925 0.15478 0.60420
LoOP 74 0.20192 0.11331 0.16182 0.06876 0.27056 0.18957 0.67227
LoOP 76 0.19872 0.10975 0.16226 0.06924 0.28358 0.20404 0.67402
LoOP 86 0.19551 0.10619 0.16223 0.06921 0.29706 0.21902 0.67720
LoOP 95 0.19551 0.10619 0.15994 0.06666 0.28087 0.20103 0.67777
LDOF 78 0.19551 0.10619 0.16121 0.06808 0.28933 0.21042 0.66203
LDOF 81 0.19231 0.10263 0.16150 0.06841 0.29071 0.21195 0.66367
LDOF 91 0.19231 0.10263 0.16240 0.06940 0.28732 0.20819 0.67044
ODIN 16 0.21624 0.12922 0.18940 0.09939 0.33465 0.26078 0.70462
ODIN 44 0.36959 0.29959 0.22825 0.14256 0.38381 0.31539 0.69266
FastABOD 37 0.11538 0.01716 0.18574 0.09533 0.35207 0.28012 0.73925
FastABOD 74 0.15705 0.06346 0.18697 0.09670 0.34712 0.27463 0.73612
FastABOD 91 0.13462 0.03853 0.18500 0.09451 0.35743 0.28608 0.73459
FastABOD 100 0.13782 0.04209 0.18710 0.09684 0.35517 0.28357 0.73629
KDEOS 5 0.09295 -0.00776 0.10714 0.00801 0.24084 0.15655 0.54370
KDEOS 7 0.10577 0.00648 0.10923 0.01033 0.23356 0.14846 0.55323
KDEOS 11 0.09615 -0.00420 0.12064 0.02301 0.23417 0.14913 0.59519
LDF 1 0.22051 0.13396 0.11884 0.02100 0.22364 0.13744 0.42553
LDF 27 0.14773 0.05310 0.11290 0.01441 0.18851 0.09841 0.54023
INFLO 10 0.13514 0.03911 0.13454 0.03844 0.25540 0.17273 0.63814
INFLO 11 0.13548 0.03949 0.13256 0.03624 0.25226 0.16924 0.62720
COF 38 0.13462 0.03853 0.14288 0.04771 0.25155 0.16845 0.65658
COF 81 0.22756 0.14180 0.15358 0.05960 0.25970 0.17750 0.63931
COF 87 0.21154 0.12399 0.15515 0.06135 0.26351 0.18173 0.65162
COF 96 0.21795 0.13112 0.15287 0.05881 0.26743 0.18610 0.65241

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO