Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (2% of outliers version#03)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1630 objects, 32 outliers (1.96%)

Download raw algorithm results (10.1 MB) Download raw algorithm evaluation table (55.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.40909 0.39726 0.36615 0.35346 0.43137 0.41999 0.79729
KNN 2 0.43750 0.42624 0.37050 0.35789 0.43750 0.42624 0.78453
KNN 3 0.39375 0.38161 0.36732 0.35465 0.46809 0.45743 0.77340
KNNW 1 0.33125 0.31786 0.31808 0.30443 0.37736 0.36489 0.81141
KNNW 3 0.40625 0.39436 0.38911 0.37687 0.46429 0.45356 0.80160
KNNW 6 0.43750 0.42624 0.38802 0.37576 0.44444 0.43332 0.78056
KNNW 7 0.43750 0.42624 0.39182 0.37965 0.45455 0.44362 0.77164
LOF 4 0.40625 0.39436 0.42949 0.41807 0.45833 0.44749 0.89471
LOF 5 0.40625 0.39436 0.40471 0.39279 0.41935 0.40773 0.89732
LOF 16 0.43750 0.42624 0.35384 0.34090 0.43750 0.42624 0.83331
SimplifiedLOF 6 0.34375 0.33061 0.40528 0.39337 0.44000 0.42879 0.87141
SimplifiedLOF 16 0.43750 0.42624 0.40328 0.39133 0.45161 0.44063 0.83384
SimplifiedLOF 52 0.37500 0.36248 0.38111 0.36872 0.45833 0.44749 0.73623
LoOP 7 0.40625 0.39436 0.43522 0.42391 0.47059 0.45999 0.88798
LoOP 9 0.46875 0.45811 0.45612 0.44522 0.50847 0.49863 0.88501
LoOP 15 0.46875 0.45811 0.46402 0.45328 0.50000 0.48999 0.87205
LDOF 7 0.31250 0.29873 0.34808 0.33502 0.39130 0.37912 0.85638
LDOF 29 0.40625 0.39436 0.42034 0.40873 0.48980 0.47958 0.84033
LDOF 30 0.40625 0.39436 0.42656 0.41508 0.48980 0.47958 0.83825
ODIN 16 0.27232 0.25775 0.18359 0.16724 0.34951 0.33649 0.83235
ODIN 99 0.42708 0.41561 0.27370 0.25915 0.44118 0.42999 0.80280
ODIN 100 0.42708 0.41561 0.27377 0.25923 0.44118 0.42999 0.80269
FastABOD 8 0.40625 0.39436 0.41384 0.40210 0.44898 0.43795 0.83537
FastABOD 12 0.40625 0.39436 0.41664 0.40496 0.45283 0.44187 0.82804
FastABOD 24 0.43750 0.42624 0.39136 0.37917 0.45455 0.44362 0.76497
FastABOD 100 0.43750 0.42624 0.35553 0.34262 0.47458 0.46405 0.71552
KDEOS 7 0.28125 0.26686 0.14641 0.12932 0.28571 0.27141 0.79122
KDEOS 30 0.15625 0.13935 0.16435 0.14761 0.21429 0.19855 0.79363
KDEOS 64 0.15625 0.13935 0.11209 0.09431 0.25287 0.23791 0.81283
LDF 1 0.04301 0.02385 0.03340 0.01405 0.09091 0.07270 0.60483
LDF 2 0.02509 0.00557 0.03646 0.01716 0.09843 0.08037 0.71366
LDF 4 0.02981 0.01038 0.03580 0.01649 0.09064 0.07243 0.73278
INFLO 5 0.37500 0.36248 0.39291 0.38075 0.40678 0.39490 0.88023
INFLO 6 0.37500 0.36248 0.40677 0.39489 0.41860 0.40696 0.87383
INFLO 16 0.43750 0.42624 0.40665 0.39477 0.44828 0.43723 0.85082
INFLO 50 0.43750 0.42624 0.39950 0.38747 0.47826 0.46781 0.76398
COF 2 0.12500 0.10748 0.08068 0.06227 0.15385 0.13690 0.62406
COF 8 0.18750 0.17123 0.07250 0.05393 0.19048 0.17427 0.57216
COF 10 0.15625 0.13935 0.12939 0.11196 0.21429 0.19855 0.56722

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2867 objects, 57 outliers (1.99%)

Download raw algorithm results (12.0 MB) Download raw algorithm evaluation table (67.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.44211 0.43079 0.46344 0.45256 0.50000 0.48986 0.90760
KNN 4 0.50627 0.49625 0.50323 0.49316 0.55319 0.54413 0.90226
KNN 5 0.52834 0.51877 0.50217 0.49207 0.56522 0.55640 0.89389
KNNW 5 0.45614 0.44511 0.46225 0.45134 0.50602 0.49600 0.90598
KNNW 11 0.50877 0.49881 0.49465 0.48440 0.57143 0.56274 0.89332
KNNW 13 0.50877 0.49881 0.49822 0.48804 0.58586 0.57746 0.88584
LOF 7 0.05556 0.03640 0.05214 0.03291 0.12139 0.10356 0.76913
LOF 10 0.05007 0.03080 0.05154 0.03230 0.10784 0.08975 0.81158
SimplifiedLOF 8 0.05394 0.03475 0.05036 0.03110 0.10250 0.08430 0.77989
SimplifiedLOF 9 0.04860 0.02930 0.04660 0.02726 0.10336 0.08517 0.76689
SimplifiedLOF 11 0.04666 0.02732 0.04650 0.02716 0.09967 0.08140 0.78502
LoOP 2 0.22807 0.21241 0.14593 0.12860 0.24348 0.22813 0.78375
LoOP 15 0.05263 0.03341 0.10098 0.08275 0.17391 0.15716 0.83596
LDOF 11 0.12281 0.10501 0.09986 0.08161 0.18785 0.17137 0.82662
LDOF 73 0.17544 0.15871 0.11411 0.09615 0.20238 0.18620 0.81768
LDOF 75 0.17544 0.15871 0.11927 0.10140 0.21519 0.19927 0.82272
LDOF 77 0.17544 0.15871 0.12115 0.10333 0.21519 0.19927 0.82450
ODIN 47 0.30175 0.28759 0.23151 0.21592 0.38043 0.36787 0.89912
ODIN 79 0.33459 0.32109 0.26119 0.24620 0.43478 0.42332 0.89611
ODIN 83 0.32982 0.31623 0.26132 0.24634 0.43038 0.41883 0.89381
ODIN 89 0.33468 0.32119 0.25914 0.24412 0.42038 0.40862 0.89258
FastABOD 32 0.02419 0.00440 0.06245 0.04344 0.15054 0.13331 0.80261
FastABOD 73 0.02655 0.00680 0.06591 0.04696 0.15084 0.13361 0.80079
FastABOD 86 0.02655 0.00680 0.06640 0.04746 0.15748 0.14039 0.80064
FastABOD 99 0.02655 0.00680 0.06676 0.04783 0.15748 0.14039 0.80027
KDEOS 2 0.03509 0.01551 0.03689 0.01735 0.10179 0.08357 0.69301
KDEOS 12 0.01667 -0.00328 0.04377 0.02437 0.09059 0.07215 0.75694
KDEOS 78 0.03509 0.01551 0.04669 0.02735 0.11872 0.10084 0.72315
KDEOS 88 0.03509 0.01551 0.04201 0.02258 0.11927 0.10140 0.65695
LDF 1 0.04348 0.02408 0.03602 0.01646 0.15534 0.13821 0.46581
LDF 3 0.05217 0.03295 0.02674 0.00699 0.08295 0.06435 0.53338
LDF 9 0.00461 -0.01558 0.02526 0.00549 0.05239 0.03317 0.57136
INFLO 7 0.05565 0.03649 0.05101 0.03176 0.12062 0.10278 0.76789
INFLO 10 0.05007 0.03080 0.04900 0.02971 0.11206 0.09405 0.78325
COF 73 0.10526 0.08711 0.07507 0.05631 0.20000 0.18377 0.75310
COF 77 0.10526 0.08711 0.08057 0.06192 0.22564 0.20993 0.76541
COF 86 0.05263 0.03341 0.06944 0.05056 0.17333 0.15656 0.76733

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO