Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

InternetAds (5% of outliers version#10)

The data set consists of images from web pages, classified as ads or not. The goal is to learn to remove ads automatically from web pages while retaining regular images. Ads are considered outliers.

Download all data set variants used (6.0 MB). You can also access the original data. (ad.data)

Normalized, without duplicates

This version contains 1555 attributes, 1682 objects, 84 outliers (4.99%)

Download raw algorithm results (10.4 MB) Download raw algorithm evaluation table (67.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.48058 0.45327 0.51459 0.48907 0.52101 0.49583 0.85940
KNN 5 0.53968 0.51549 0.55270 0.52919 0.56489 0.54201 0.85546
KNN 10 0.54762 0.52384 0.54234 0.51829 0.61765 0.59755 0.80535
KNN 12 0.54762 0.52384 0.53611 0.51172 0.62319 0.60338 0.79485
KNNW 7 0.47619 0.44866 0.50150 0.47530 0.49246 0.46578 0.86824
KNNW 14 0.57143 0.54890 0.56714 0.54438 0.59574 0.57449 0.84518
KNNW 18 0.57143 0.54890 0.57012 0.54752 0.60993 0.58942 0.83385
KNNW 20 0.57143 0.54890 0.56651 0.54373 0.61429 0.59401 0.82827
LOF 19 0.50000 0.47372 0.48811 0.46121 0.51799 0.49265 0.88280
LOF 44 0.54762 0.52384 0.54553 0.52164 0.60403 0.58321 0.84748
LOF 52 0.54762 0.52384 0.56020 0.53708 0.60140 0.58045 0.82974
LOF 95 0.53571 0.51131 0.55403 0.53059 0.61429 0.59401 0.81221
SimplifiedLOF 18 0.53571 0.51131 0.51320 0.48761 0.54545 0.52156 0.89225
SimplifiedLOF 31 0.57143 0.54890 0.58114 0.55912 0.60811 0.58751 0.87951
SimplifiedLOF 37 0.55952 0.53637 0.57957 0.55747 0.61111 0.59067 0.87132
LoOP 32 0.52381 0.49878 0.51507 0.48958 0.55172 0.52816 0.89510
LoOP 83 0.57143 0.54890 0.57537 0.55305 0.57534 0.55302 0.86352
LoOP 88 0.55952 0.53637 0.57597 0.55368 0.57692 0.55468 0.86093
LoOP 97 0.55952 0.53637 0.57442 0.55205 0.59060 0.56908 0.85621
LDOF 31 0.50000 0.47372 0.47455 0.44693 0.52174 0.49660 0.88889
LDOF 87 0.54762 0.52384 0.55616 0.53283 0.57143 0.54890 0.86155
ODIN 51 0.27381 0.23564 0.22747 0.18686 0.35880 0.32510 0.82824
ODIN 94 0.31338 0.27729 0.23757 0.19749 0.36981 0.33669 0.81566
ODIN 100 0.31338 0.27729 0.24537 0.20570 0.37262 0.33965 0.81607
FastABOD 18 0.46429 0.43613 0.36590 0.33256 0.48555 0.45851 0.84144
FastABOD 25 0.44048 0.41106 0.38003 0.34744 0.46541 0.43731 0.84864
KDEOS 11 0.23810 0.19805 0.13899 0.09373 0.26950 0.23110 0.68706
KDEOS 12 0.22619 0.18551 0.14722 0.10239 0.27143 0.23313 0.68736
KDEOS 65 0.22619 0.18551 0.15310 0.10858 0.25000 0.21058 0.76254
KDEOS 69 0.21429 0.17298 0.16758 0.12383 0.25778 0.21876 0.75872
LDF 99 0.33333 0.29829 0.16713 0.12335 0.36667 0.33338 0.73802
INFLO 26 0.55952 0.53637 0.54028 0.51612 0.57317 0.55073 0.88590
INFLO 52 0.55952 0.53637 0.57811 0.55594 0.56287 0.53990 0.86109
INFLO 84 0.57143 0.54890 0.57125 0.54871 0.58974 0.56818 0.84858
INFLO 91 0.58333 0.56143 0.57432 0.55194 0.58750 0.56582 0.84410
COF 4 0.22619 0.18551 0.14841 0.10365 0.23776 0.19769 0.66682

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 1555 attributes, 2957 objects, 147 outliers (4.97%)

Download raw algorithm results (12.6 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 4 0.50894 0.48325 0.54100 0.51699 0.52444 0.49957 0.88898
KNN 5 0.50149 0.47541 0.54337 0.51949 0.55319 0.52982 0.88246
KNN 21 0.48591 0.45901 0.48829 0.46152 0.56452 0.54173 0.78829
KNNW 8 0.48299 0.45595 0.51145 0.48589 0.51383 0.48840 0.88578
KNNW 12 0.50340 0.47742 0.54233 0.51839 0.53571 0.51143 0.87904
KNNW 15 0.48980 0.46311 0.54454 0.52071 0.56221 0.53931 0.87016
KNNW 19 0.48980 0.46311 0.54222 0.51827 0.57534 0.55313 0.85887
LOF 8 0.08756 0.03982 0.09387 0.04647 0.20408 0.16244 0.73073
LOF 9 0.09353 0.04610 0.09544 0.04812 0.20000 0.15815 0.73416
SimplifiedLOF 9 0.09142 0.04389 0.08606 0.03825 0.16427 0.12055 0.69295
LoOP 1 0.20862 0.16722 0.11734 0.07116 0.20979 0.16845 0.61307
LoOP 12 0.12557 0.07982 0.13372 0.08840 0.22457 0.18401 0.74164
LoOP 14 0.12796 0.08234 0.13560 0.09038 0.23192 0.19174 0.73754
LoOP 73 0.15646 0.11233 0.11750 0.07133 0.24242 0.20279 0.72134
LDOF 76 0.16327 0.11949 0.11956 0.07350 0.23690 0.19698 0.72759
LDOF 78 0.16327 0.11949 0.12217 0.07625 0.24356 0.20399 0.73174
LDOF 100 0.14966 0.10518 0.11572 0.06946 0.21429 0.17318 0.73243
ODIN 83 0.43760 0.40818 0.26474 0.22627 0.45545 0.42696 0.78195
ODIN 100 0.46399 0.43595 0.26573 0.22732 0.46735 0.43949 0.78022
FastABOD 30 0.03401 -0.01652 0.12465 0.07886 0.28081 0.24319 0.78866
FastABOD 73 0.12925 0.08370 0.12792 0.08229 0.26973 0.23152 0.78578
FastABOD 100 0.12925 0.08370 0.13031 0.08481 0.28389 0.24643 0.78842
KDEOS 2 0.03401 -0.01652 0.06677 0.01795 0.16895 0.12548 0.61848
KDEOS 10 0.06803 0.01927 0.07736 0.02909 0.16571 0.12207 0.66917
KDEOS 76 0.08844 0.04075 0.06840 0.01966 0.13132 0.08588 0.62227
LDF 2 0.09652 0.04926 0.06281 0.01378 0.17866 0.13569 0.46245
LDF 22 0.14570 0.10100 0.06932 0.02064 0.14765 0.10306 0.57305
INFLO 8 0.08769 0.03997 0.08790 0.04019 0.19575 0.15368 0.70495
INFLO 9 0.09366 0.04625 0.09112 0.04357 0.19435 0.15220 0.71439
COF 78 0.18367 0.14097 0.11806 0.07192 0.22312 0.18248 0.71563
COF 79 0.19048 0.14813 0.11563 0.06936 0.22041 0.17963 0.71112

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO