Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (10% of outliers version#10)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1775 objects, 177 outliers (9.97%)

Download raw algorithm results (13.0 MB) Download raw algorithm evaluation table (73.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.42665 0.36314 0.43541 0.37288 0.43573 0.37323 0.78370
KNN 8 0.44963 0.38867 0.49483 0.43887 0.49042 0.43398 0.76932
KNN 23 0.45858 0.39862 0.47061 0.41197 0.50859 0.45416 0.71460
KNN 24 0.46917 0.41037 0.47130 0.41274 0.50676 0.45212 0.71334
KNNW 10 0.42938 0.36617 0.45097 0.39016 0.45146 0.39070 0.79351
KNNW 19 0.45763 0.39755 0.49880 0.44329 0.47887 0.42115 0.77724
KNNW 30 0.44633 0.38500 0.50516 0.45035 0.49231 0.43607 0.75519
KNNW 44 0.45763 0.39755 0.50355 0.44856 0.52330 0.47050 0.73711
LOF 54 0.45198 0.39128 0.50838 0.45392 0.48829 0.43162 0.78010
LOF 76 0.48023 0.42265 0.53001 0.47795 0.52396 0.47123 0.77210
LOF 98 0.48023 0.42265 0.53910 0.48804 0.56089 0.51225 0.76375
SimplifiedLOF 53 0.49153 0.43521 0.50064 0.44533 0.49292 0.43675 0.79057
SimplifiedLOF 54 0.48588 0.42893 0.50707 0.45247 0.49300 0.43684 0.79248
SimplifiedLOF 98 0.48023 0.42265 0.53972 0.48874 0.53846 0.48734 0.77538
SimplifiedLOF 99 0.48588 0.42893 0.53945 0.48843 0.54167 0.49090 0.77453
LoOP 33 0.46893 0.41010 0.38506 0.31694 0.47429 0.41606 0.72241
LoOP 42 0.45763 0.39755 0.39602 0.32913 0.47734 0.41945 0.74349
LoOP 98 0.42373 0.35990 0.44968 0.38872 0.44242 0.38067 0.78258
LoOP 100 0.42373 0.35990 0.45144 0.39068 0.43713 0.37478 0.78228
LDOF 61 0.44633 0.38500 0.42648 0.36296 0.45533 0.39500 0.76052
LDOF 69 0.45198 0.39128 0.42117 0.35705 0.45198 0.39128 0.76085
LDOF 97 0.42938 0.36617 0.43923 0.37712 0.43871 0.37654 0.77742
ODIN 19 0.26718 0.18601 0.20992 0.12241 0.35021 0.27824 0.68605
ODIN 25 0.29317 0.21487 0.21539 0.12849 0.36239 0.29176 0.67836
ODIN 26 0.29180 0.21335 0.21341 0.12629 0.36322 0.29269 0.67791
ODIN 91 0.33466 0.26097 0.20740 0.11961 0.33708 0.26365 0.67617
FastABOD 15 0.37853 0.30970 0.34940 0.27734 0.39609 0.32920 0.76996
FastABOD 25 0.40678 0.34107 0.36354 0.29304 0.43564 0.37313 0.76960
FastABOD 26 0.40678 0.34107 0.35681 0.28557 0.43791 0.37565 0.76589
FastABOD 30 0.42373 0.35990 0.35241 0.28068 0.43149 0.36852 0.76286
KDEOS 60 0.16949 0.07750 0.17819 0.08716 0.24105 0.15699 0.64152
KDEOS 64 0.20339 0.11515 0.16188 0.06904 0.26271 0.18105 0.65094
KDEOS 67 0.23164 0.14653 0.16359 0.07094 0.25710 0.17481 0.65194
KDEOS 72 0.23164 0.14653 0.17504 0.08366 0.25272 0.16995 0.65371
LDF 95 0.39548 0.32852 0.21660 0.12982 0.41237 0.34728 0.69005
LDF 100 0.38418 0.31597 0.21720 0.13049 0.41730 0.35276 0.69601
INFLO 52 0.43503 0.37245 0.45974 0.39990 0.45317 0.39260 0.78591
INFLO 91 0.46328 0.40383 0.49849 0.44294 0.46617 0.40704 0.78346
INFLO 97 0.45763 0.39755 0.51011 0.45585 0.48175 0.42435 0.78324
INFLO 98 0.45198 0.39128 0.51029 0.45605 0.48175 0.42435 0.78269
COF 4 0.14124 0.04612 0.14747 0.05304 0.23864 0.15431 0.60365
COF 5 0.15819 0.06495 0.15190 0.05796 0.24467 0.16100 0.59495
COF 6 0.21469 0.12771 0.16200 0.06918 0.23266 0.14767 0.58475
COF 8 0.21469 0.12771 0.18338 0.09293 0.23490 0.15015 0.57056

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 3122 objects, 312 outliers (9.99%)

Download raw algorithm results (13.7 MB) Download raw algorithm evaluation table (74.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 6 0.44524 0.38364 0.49793 0.44218 0.46623 0.40696 0.85586
KNN 9 0.46763 0.40851 0.51695 0.46331 0.46901 0.41006 0.83780
KNN 10 0.45199 0.39115 0.53092 0.47884 0.49699 0.44114 0.82678
KNNW 13 0.43910 0.37682 0.49498 0.43891 0.46012 0.40018 0.84828
KNNW 22 0.46795 0.40887 0.52680 0.47426 0.48557 0.42845 0.83373
KNNW 24 0.46474 0.40531 0.52688 0.47435 0.48227 0.42478 0.82974
KNNW 78 0.48077 0.42312 0.49821 0.44250 0.48220 0.42471 0.76595
LOF 9 0.14188 0.04660 0.14408 0.04905 0.26063 0.17853 0.64629
LOF 11 0.13379 0.03762 0.14245 0.04723 0.27349 0.19283 0.66175
SimplifiedLOF 2 0.04533 -0.06067 0.09832 -0.00179 0.23646 0.15169 0.52054
SimplifiedLOF 9 0.13885 0.04324 0.12887 0.03215 0.22480 0.13873 0.59457
SimplifiedLOF 12 0.11951 0.02175 0.12592 0.02887 0.23254 0.14732 0.61506
LoOP 2 0.11842 0.02053 0.13612 0.04021 0.29377 0.21535 0.65072
LoOP 14 0.17628 0.08482 0.15798 0.06448 0.25786 0.17545 0.67115
LoOP 57 0.20833 0.12043 0.15249 0.05839 0.26941 0.18829 0.64279
LoOP 75 0.20192 0.11331 0.16330 0.07040 0.27544 0.19499 0.66311
LDOF 78 0.22436 0.13824 0.16107 0.06792 0.28681 0.20762 0.65355
LDOF 79 0.22756 0.14180 0.16057 0.06736 0.28599 0.20671 0.65209
LDOF 100 0.21154 0.12399 0.16106 0.06791 0.28453 0.20509 0.66529
ODIN 23 0.26422 0.18253 0.21971 0.13308 0.37333 0.30375 0.72623
ODIN 36 0.34701 0.27451 0.23890 0.15440 0.39779 0.33093 0.71963
ODIN 37 0.35611 0.28462 0.24111 0.15685 0.39667 0.32968 0.71854
ODIN 43 0.36603 0.29563 0.23906 0.15457 0.38841 0.32050 0.70972
FastABOD 73 0.15705 0.06346 0.18440 0.09384 0.33414 0.26021 0.72954
FastABOD 74 0.15705 0.06346 0.18517 0.09470 0.33468 0.26081 0.73015
FastABOD 83 0.14103 0.04565 0.18513 0.09466 0.33896 0.26556 0.73031
FastABOD 89 0.14103 0.04565 0.18288 0.09216 0.34131 0.26817 0.72827
KDEOS 2 0.10256 0.00292 0.11854 0.02067 0.25850 0.17617 0.58300
KDEOS 10 0.10577 0.00648 0.12367 0.02637 0.22675 0.14089 0.59496
KDEOS 12 0.10256 0.00292 0.12153 0.02399 0.22876 0.14313 0.60347
KDEOS 77 0.11859 0.02072 0.11548 0.01726 0.21454 0.12733 0.56977
LDF 2 0.19872 0.10975 0.11337 0.01493 0.22843 0.14276 0.44561
LDF 6 0.21154 0.12399 0.12784 0.03100 0.22067 0.13413 0.49984
LDF 33 0.12644 0.02944 0.10534 0.00600 0.18321 0.09252 0.52379
INFLO 9 0.14188 0.04660 0.13215 0.03579 0.24808 0.16459 0.61524
INFLO 11 0.13379 0.03762 0.13271 0.03642 0.25395 0.17111 0.63110
INFLO 12 0.11951 0.02175 0.12614 0.02912 0.25608 0.17348 0.61966
COF 36 0.15385 0.05990 0.14548 0.05060 0.25280 0.16984 0.65634
COF 77 0.23077 0.14536 0.15588 0.06216 0.25519 0.17250 0.63784
COF 78 0.22756 0.14180 0.15714 0.06355 0.25664 0.17410 0.63982
COF 87 0.22436 0.13824 0.15263 0.05854 0.26875 0.18755 0.63491

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO