Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

WBC (version#05)

This dataset consists of examples of different cancer types, benign or malignant. Examples of benign cancer are considered inliers, examples of malignant cancer are considered outliers. After downsampling the outliers, following Schubert et al. [1], 10 outliers remain. 234 instances are duplicates (231 inliers and 3 outliers), therefore 229 outliers were removed from the data set with duplicates and 226 outliers from the dataset without duplicates. Furthermore, we removed 16 instances with missing values, two of them being outliers and 14 inliers. The processed data set has 9 numeric attributes and 454 instances, namely 10 outliers (2.2%) and 444 inliers (97.8%). The same pre-processing has also been applied in [2] and [3].

References:

[1] E. Schubert, R. Wojdanowski, A. Zimek, and H.-P. Kriegel. On evaluation of outlier rankings and outlier scores. In Proc. SDM, pages 1047-1058, 2012.
[2] A. Zimek, M. Gaudet, R. J. G. B. Campello, and J. Sander. Subsampling for efficient and effective unsupervised outlier detection ensembles. In Proc. KDD, pages 428-436, 2013.
[3] H.-P. Kriegel, P. Kroeger, E. Schubert, and A. Zimek. Interpreting and unifying outlier scores. In Proc. SDM, pages 13-24, 2011.

Download all data set variants used (57.1 kB). You can also access the original data. (breast-cancer-wisconsin.data)

Normalized, without duplicates

This version contains 9 attributes, 223 objects, 10 outliers (4.48%)

Download raw algorithm results (1.7 MB) Download raw algorithm evaluation table (36.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 15 0.90000 0.89531 0.93152 0.92831 0.90000 0.89531 0.99484
KNN 23 0.90000 0.89531 0.95263 0.95041 0.94737 0.94490 0.99577
KNN 97 0.90000 0.89531 0.95556 0.95347 0.94737 0.94490 0.99624
KNNW 23 0.90000 0.89531 0.93152 0.92831 0.90000 0.89531 0.99484
KNNW 37 0.90000 0.89531 0.95263 0.95041 0.94737 0.94490 0.99577
LOF 73 0.90000 0.89531 0.88306 0.87757 0.90000 0.89531 0.99249
LOF 98 0.90000 0.89531 0.94545 0.94289 0.94737 0.94490 0.99437
SimplifiedLOF 93 0.80000 0.79061 0.82845 0.82040 0.80000 0.79061 0.98967
SimplifiedLOF 96 0.80000 0.79061 0.84598 0.83875 0.85714 0.85044 0.99108
SimplifiedLOF 98 0.80000 0.79061 0.86806 0.86186 0.84211 0.83469 0.99155
LoOP 66 0.40000 0.37183 0.37157 0.34207 0.60000 0.58122 0.95962
LoOP 83 0.40000 0.37183 0.47036 0.44549 0.69231 0.67786 0.97136
LoOP 96 0.40000 0.37183 0.57151 0.55139 0.66667 0.65102 0.97559
LoOP 100 0.40000 0.37183 0.57239 0.55231 0.66667 0.65102 0.97465
LDOF 90 0.40000 0.37183 0.33652 0.30537 0.52941 0.50732 0.95023
LDOF 93 0.30000 0.26714 0.33652 0.30537 0.55556 0.53469 0.95164
LDOF 100 0.40000 0.37183 0.37318 0.34375 0.54545 0.52411 0.95681
ODIN 88 0.20000 0.16244 0.31272 0.28046 0.51613 0.49341 0.94648
ODIN 97 0.30000 0.26714 0.31972 0.28778 0.48276 0.45847 0.94836
ODIN 99 0.30000 0.26714 0.32555 0.29388 0.50000 0.47653 0.94977
FastABOD 15 0.90000 0.89531 0.95032 0.94798 0.90000 0.89531 0.99718
FastABOD 30 0.90000 0.89531 0.96980 0.96838 0.95238 0.95015 0.99859
FastABOD 45 0.90000 0.89531 0.97333 0.97208 0.90909 0.90482 0.99859
KDEOS 13 0.10000 0.05775 0.10509 0.06307 0.25000 0.21479 0.61784
KDEOS 14 0.30000 0.26714 0.12653 0.08553 0.30000 0.26714 0.60469
LDF 42 0.90000 0.89531 0.88807 0.88281 0.90000 0.89531 0.99343
LDF 53 0.90000 0.89531 0.95263 0.95041 0.94737 0.94490 0.99577
INFLO 68 0.40000 0.37183 0.64241 0.62562 0.74074 0.72857 0.98028
INFLO 95 0.60000 0.58122 0.77593 0.76541 0.70588 0.69207 0.98545
INFLO 97 0.70000 0.68592 0.78031 0.77000 0.70588 0.69207 0.98545
INFLO 98 0.70000 0.68592 0.79808 0.78860 0.73684 0.72449 0.98498
COF 58 0.80000 0.79061 0.82183 0.81346 0.81818 0.80965 0.98826

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 9 attributes, 454 objects, 10 outliers (2.20%)

Download raw algorithm results (1.9 MB) Download raw algorithm evaluation table (40.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 12 0.90000 0.89775 0.93152 0.92998 0.90000 0.89775 0.99764
KNN 14 0.90000 0.89775 0.95263 0.95156 0.94737 0.94618 0.99797
KNN 15 0.90000 0.89775 0.95556 0.95455 0.94737 0.94618 0.99820
KNNW 28 0.90000 0.89775 0.94000 0.93865 0.90000 0.89775 0.99752
KNNW 34 0.90000 0.89775 0.93444 0.93297 0.90000 0.89775 0.99775
LOF 88 0.60000 0.59099 0.62975 0.62141 0.66667 0.65916 0.98986
LOF 91 0.60000 0.59099 0.64323 0.63519 0.72727 0.72113 0.99032
LOF 100 0.60000 0.59099 0.77177 0.76663 0.71429 0.70785 0.99324
SimplifiedLOF 99 0.50000 0.48874 0.58752 0.57823 0.66667 0.65916 0.98784
LoOP 49 0.20000 0.18198 0.12312 0.10337 0.23529 0.21807 0.90856
LoOP 98 0.20000 0.18198 0.33732 0.32240 0.56250 0.55265 0.97680
LoOP 100 0.20000 0.18198 0.35334 0.33878 0.56250 0.55265 0.97815
LDOF 57 0.20000 0.18198 0.10248 0.08226 0.20225 0.18428 0.88018
LDOF 95 0.20000 0.18198 0.22078 0.20323 0.39130 0.37759 0.95360
LDOF 99 0.20000 0.18198 0.24818 0.23125 0.39130 0.37759 0.96081
ODIN 9 0.20000 0.18198 0.10580 0.08566 0.23529 0.21807 0.82556
ODIN 92 0.20000 0.18198 0.30476 0.28910 0.50000 0.48874 0.96926
ODIN 97 0.20000 0.18198 0.31831 0.30296 0.50000 0.48874 0.97151
ODIN 100 0.20000 0.18198 0.31831 0.30296 0.50000 0.48874 0.97173
FastABOD 28 0.80000 0.79550 0.87401 0.87117 0.84211 0.83855 0.99572
FastABOD 67 0.80000 0.79550 0.91368 0.91173 0.84211 0.83855 0.99707
KDEOS 15 0.30000 0.28423 0.12218 0.10241 0.33333 0.31832 0.67928
KDEOS 18 0.20000 0.18198 0.13882 0.11943 0.28571 0.26963 0.77275
KDEOS 20 0.20000 0.18198 0.20931 0.19150 0.30769 0.29210 0.76779
LDF 99 0.80000 0.79550 0.89685 0.89453 0.82353 0.81955 0.99617
LDF 100 0.80000 0.79550 0.91685 0.91498 0.88889 0.88639 0.99662
INFLO 85 0.40000 0.38649 0.46556 0.45352 0.64000 0.63189 0.98446
INFLO 87 0.30000 0.28423 0.55080 0.54069 0.69231 0.68538 0.98514
INFLO 99 0.40000 0.38649 0.60000 0.59099 0.69231 0.68538 0.98874
COF 86 0.20000 0.18198 0.33252 0.31748 0.56250 0.55265 0.97770
COF 90 0.40000 0.38649 0.35353 0.33897 0.54545 0.53522 0.97748
COF 96 0.40000 0.38649 0.37929 0.36531 0.51613 0.50523 0.97838

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 9 attributes, 223 objects, 10 outliers (4.48%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (35.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 12 0.90000 0.89531 0.93000 0.92671 0.90000 0.89531 0.99437
KNN 25 0.90000 0.89531 0.94762 0.94516 0.94737 0.94490 0.99484
KNN 28 0.90000 0.89531 0.94762 0.94516 0.94737 0.94490 0.99507
KNNW 25 0.90000 0.89531 0.91639 0.91246 0.90000 0.89531 0.99390
KNNW 34 0.90000 0.89531 0.94000 0.93718 0.90000 0.89531 0.99484
LOF 71 0.90000 0.89531 0.94000 0.93718 0.90000 0.89531 0.99484
LOF 77 0.90000 0.89531 0.94762 0.94516 0.94737 0.94490 0.99484
SimplifiedLOF 98 0.80000 0.79061 0.87821 0.87250 0.81818 0.80965 0.99202
LoOP 88 0.50000 0.47653 0.55306 0.53208 0.66667 0.65102 0.97418
LoOP 100 0.50000 0.47653 0.64442 0.62772 0.72000 0.70685 0.98028
LDOF 91 0.40000 0.37183 0.39515 0.36676 0.58824 0.56890 0.96197
LDOF 95 0.40000 0.37183 0.40810 0.38031 0.64516 0.62850 0.96479
LDOF 99 0.40000 0.37183 0.43987 0.41357 0.64516 0.62850 0.96808
ODIN 6 0.30000 0.26714 0.19319 0.15531 0.30000 0.26714 0.86737
ODIN 98 0.26667 0.23224 0.33743 0.30632 0.53333 0.51142 0.95376
ODIN 100 0.30000 0.26714 0.34716 0.31651 0.53333 0.51142 0.95493
FastABOD 8 0.80000 0.79061 0.86171 0.85522 0.80000 0.79061 0.99061
FastABOD 13 0.80000 0.79061 0.91837 0.91453 0.90909 0.90482 0.99624
FastABOD 18 0.80000 0.79061 0.94515 0.94258 0.90909 0.90482 0.99718
KDEOS 11 0.30000 0.26714 0.30017 0.26731 0.33333 0.30203 0.62770
KDEOS 12 0.30000 0.26714 0.13958 0.09918 0.32000 0.28808 0.63146
LDF 32 0.90000 0.89531 0.90473 0.90026 0.90000 0.89531 0.99390
LDF 57 0.90000 0.89531 0.94556 0.94300 0.90000 0.89531 0.99577
INFLO 85 0.60000 0.58122 0.75586 0.74440 0.72727 0.71447 0.98498
INFLO 98 0.60000 0.58122 0.81620 0.80757 0.78261 0.77240 0.98826
COF 61 0.80000 0.79061 0.70958 0.69594 0.80000 0.79061 0.98357
COF 67 0.70000 0.68592 0.78772 0.77776 0.78261 0.77240 0.98779

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 9 attributes, 454 objects, 10 outliers (2.20%)

Download raw algorithm results (1.9 MB) Download raw algorithm evaluation table (40.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 12 0.90000 0.89775 0.93152 0.92998 0.90000 0.89775 0.99764
KNN 14 0.90000 0.89775 0.95263 0.95156 0.94737 0.94618 0.99797
KNN 15 0.90000 0.89775 0.95556 0.95455 0.94737 0.94618 0.99820
KNNW 28 0.90000 0.89775 0.94000 0.93865 0.90000 0.89775 0.99752
KNNW 34 0.90000 0.89775 0.93444 0.93297 0.90000 0.89775 0.99775
LOF 88 0.60000 0.59099 0.62975 0.62141 0.66667 0.65916 0.98986
LOF 91 0.60000 0.59099 0.64323 0.63519 0.72727 0.72113 0.99032
LOF 100 0.60000 0.59099 0.77690 0.77187 0.72727 0.72113 0.99347
SimplifiedLOF 99 0.50000 0.48874 0.58752 0.57823 0.66667 0.65916 0.98784
SimplifiedLOF 100 0.50000 0.48874 0.58885 0.57959 0.69231 0.68538 0.98784
LoOP 50 0.20000 0.18198 0.12564 0.10594 0.24324 0.22620 0.91149
LoOP 100 0.20000 0.18198 0.35432 0.33978 0.56250 0.55265 0.97838
LDOF 57 0.20000 0.18198 0.10252 0.08231 0.20225 0.18428 0.88018
LDOF 99 0.20000 0.18198 0.24818 0.23125 0.39130 0.37759 0.96081
LDOF 100 0.20000 0.18198 0.24721 0.23026 0.40000 0.38649 0.96081
ODIN 9 0.20000 0.18198 0.10875 0.08867 0.23529 0.21807 0.83660
ODIN 92 0.20000 0.18198 0.30476 0.28910 0.50000 0.48874 0.96926
ODIN 97 0.20000 0.18198 0.31831 0.30296 0.50000 0.48874 0.97151
ODIN 100 0.20000 0.18198 0.31831 0.30296 0.50000 0.48874 0.97196
FastABOD 28 0.80000 0.79550 0.89825 0.89596 0.84211 0.83855 0.99662
FastABOD 33 0.80000 0.79550 0.91021 0.90819 0.84211 0.83855 0.99730
FastABOD 57 0.80000 0.79550 0.91652 0.91464 0.84211 0.83855 0.99707
KDEOS 2 0.00000 -0.02252 0.03734 0.01566 0.10638 0.08626 0.56329
KDEOS 18 0.00000 -0.02252 0.04273 0.02117 0.11429 0.09434 0.67635
KDEOS 22 0.00000 -0.02252 0.03648 0.01478 0.13333 0.11381 0.57162
LDF 99 0.80000 0.79550 0.89685 0.89453 0.82353 0.81955 0.99617
LDF 100 0.80000 0.79550 0.91685 0.91498 0.88889 0.88639 0.99662
INFLO 84 0.40000 0.38649 0.45757 0.44535 0.62069 0.61215 0.98378
INFLO 87 0.30000 0.28423 0.55080 0.54069 0.69231 0.68538 0.98514
INFLO 99 0.40000 0.38649 0.60000 0.59099 0.69231 0.68538 0.98874
COF 89 0.40000 0.38649 0.34116 0.32632 0.52174 0.51097 0.97410
COF 90 0.40000 0.38649 0.34739 0.33269 0.54545 0.53522 0.97523

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO