Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Ionosphere

This dataset differentiates good radars which show evidence of some kind of structure in the ionosphere, and bad radars for which signals pass through the radar. In this version (HiCS, [1]), the authors use the class b (minority) as outliers and class g as inliers. They removed attributes 1 and 2 from their dataset. Therefore, after the preprocessing, this dataset has 32 numeric attributes and 351 instances, 126 outliers (35.9%) and 225 inliers (64.1%). This database contains only 1 duplicate, so we did not create a version without duplicates.

References:

[1] F. Keller, E. Mueller, and K. Boehm. HiCS: high contrast subspaces for density-based outlier ranking. In Proc. ICDE, 2012.

Download all data set variants used (28.9 kB). You can also access the original data. (real world datasets)

Normalized, without duplicates

This version contains 32 attributes, 351 objects, 126 outliers (35.90%)

Download raw algorithm results (3.1 MB) Download raw algorithm evaluation table (52.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.84921 0.76476 0.92988 0.89061 0.87764 0.80911 0.92737
KNNW 2 0.84921 0.76476 0.92806 0.88778 0.88235 0.81647 0.92374
KNNW 4 0.85714 0.77714 0.92961 0.89018 0.88235 0.81647 0.92698
LOF 9 0.83333 0.74000 0.86670 0.79205 0.83665 0.74518 0.89898
LOF 83 0.72222 0.56667 0.85730 0.77739 0.75949 0.62481 0.90427
SimplifiedLOF 9 0.80952 0.70286 0.87282 0.80160 0.83459 0.74195 0.90409
SimplifiedLOF 10 0.82540 0.72762 0.87644 0.80724 0.83333 0.74000 0.90504
SimplifiedLOF 11 0.82540 0.72762 0.87748 0.80887 0.82890 0.73308 0.90416
LoOP 11 0.80159 0.69048 0.86177 0.78436 0.82090 0.72060 0.90210
LoOP 12 0.80952 0.70286 0.86116 0.78340 0.81955 0.71850 0.89926
LoOP 14 0.80159 0.69048 0.85278 0.77034 0.82759 0.73103 0.89291
LDOF 14 0.79365 0.67810 0.84845 0.76358 0.80755 0.69977 0.89608
LDOF 15 0.80159 0.69048 0.84828 0.76332 0.80899 0.70202 0.89340
LDOF 16 0.80159 0.69048 0.85465 0.77325 0.82129 0.72122 0.89295
ODIN 13 0.78042 0.65746 0.84768 0.76239 0.78088 0.65817 0.85224
ODIN 18 0.79206 0.67562 0.85499 0.77379 0.79528 0.68063 0.84843
ODIN 19 0.77438 0.64803 0.85596 0.77530 0.78226 0.66032 0.84616
FastABOD 3 0.81746 0.71524 0.88811 0.82546 0.82353 0.72471 0.91333
FastABOD 11 0.85714 0.77714 0.90893 0.85793 0.85714 0.77714 0.90808
KDEOS 57 0.75397 0.61619 0.72529 0.57145 0.76364 0.63127 0.82748
KDEOS 59 0.76984 0.64095 0.71910 0.56180 0.77154 0.64360 0.82790
KDEOS 67 0.75397 0.61619 0.71020 0.54792 0.79699 0.68331 0.83242
KDEOS 71 0.76190 0.62857 0.71100 0.54916 0.78327 0.66190 0.83400
LDF 3 0.82540 0.72762 0.88445 0.81974 0.83761 0.74667 0.90328
LDF 50 0.80952 0.70286 0.88006 0.81289 0.81890 0.71748 0.91668
INFLO 9 0.80952 0.70286 0.86613 0.79117 0.81600 0.71296 0.90342
INFLO 10 0.81746 0.71524 0.86422 0.78819 0.82129 0.72122 0.90379
COF 89 0.84921 0.76476 0.92927 0.88967 0.84921 0.76476 0.95425
COF 100 0.84921 0.76476 0.93965 0.90585 0.85609 0.77550 0.96032

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO