Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

WBC (version#07)

This dataset consists of examples of different cancer types, benign or malignant. Examples of benign cancer are considered inliers, examples of malignant cancer are considered outliers. After downsampling the outliers, following Schubert et al. [1], 10 outliers remain. 234 instances are duplicates (231 inliers and 3 outliers), therefore 229 outliers were removed from the data set with duplicates and 226 outliers from the dataset without duplicates. Furthermore, we removed 16 instances with missing values, two of them being outliers and 14 inliers. The processed data set has 9 numeric attributes and 454 instances, namely 10 outliers (2.2%) and 444 inliers (97.8%). The same pre-processing has also been applied in [2] and [3].

References:

[1] E. Schubert, R. Wojdanowski, A. Zimek, and H.-P. Kriegel. On evaluation of outlier rankings and outlier scores. In Proc. SDM, pages 1047-1058, 2012.
[2] A. Zimek, M. Gaudet, R. J. G. B. Campello, and J. Sander. Subsampling for efficient and effective unsupervised outlier detection ensembles. In Proc. KDD, pages 428-436, 2013.
[3] H.-P. Kriegel, P. Kroeger, E. Schubert, and A. Zimek. Interpreting and unifying outlier scores. In Proc. SDM, pages 13-24, 2011.

Download all data set variants used (57.1 kB). You can also access the original data. (breast-cancer-wisconsin.data)

Normalized, without duplicates

This version contains 9 attributes, 223 objects, 10 outliers (4.48%)

Download raw algorithm results (1.7 MB) Download raw algorithm evaluation table (33.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.90000 0.89531 0.92434 0.92079 0.90000 0.89531 0.99366
KNNW 2 0.80000 0.79061 0.91151 0.90735 0.84211 0.83469 0.99296
KNNW 3 0.80000 0.79061 0.92727 0.92386 0.88889 0.88367 0.99343
KNNW 4 0.80000 0.79061 0.92944 0.92612 0.88889 0.88367 0.99390
LOF 56 0.70000 0.68592 0.85958 0.85299 0.82353 0.81524 0.98873
LOF 69 0.70000 0.68592 0.87443 0.86854 0.82353 0.81524 0.98967
LOF 77 0.80000 0.79061 0.87545 0.86961 0.82353 0.81524 0.98920
LOF 84 0.80000 0.79061 0.87973 0.87408 0.82353 0.81524 0.98967
SimplifiedLOF 73 0.70000 0.68592 0.87042 0.86434 0.82353 0.81524 0.99014
SimplifiedLOF 98 0.80000 0.79061 0.88170 0.87615 0.82353 0.81524 0.99014
LoOP 94 0.70000 0.68592 0.73088 0.71825 0.70000 0.68592 0.98498
LoOP 99 0.70000 0.68592 0.78765 0.77768 0.73684 0.72449 0.98732
LDOF 75 0.40000 0.37183 0.45901 0.43361 0.61538 0.59733 0.96244
LDOF 92 0.40000 0.37183 0.49938 0.47588 0.66667 0.65102 0.96808
LDOF 100 0.40000 0.37183 0.52260 0.50019 0.66667 0.65102 0.97042
ODIN 84 0.45000 0.42418 0.40265 0.37461 0.58065 0.56096 0.96432
ODIN 92 0.45000 0.42418 0.41658 0.38919 0.60000 0.58122 0.96573
ODIN 98 0.40000 0.37183 0.41426 0.38676 0.62069 0.60288 0.96549
ODIN 100 0.40000 0.37183 0.41541 0.38796 0.62069 0.60288 0.96596
FastABOD 6 0.80000 0.79061 0.71379 0.70035 0.80000 0.79061 0.98779
FastABOD 11 0.80000 0.79061 0.90444 0.89996 0.84211 0.83469 0.99296
FastABOD 12 0.80000 0.79061 0.91263 0.90853 0.88889 0.88367 0.99296
KDEOS 11 0.10000 0.05775 0.06384 0.01989 0.13245 0.09172 0.61549
KDEOS 15 0.10000 0.05775 0.15580 0.11617 0.18182 0.14341 0.63005
KDEOS 18 0.10000 0.05775 0.06778 0.02401 0.14414 0.10396 0.64413
LDF 28 0.70000 0.68592 0.73187 0.71928 0.70000 0.68592 0.98357
LDF 33 0.70000 0.68592 0.84616 0.83893 0.82353 0.81524 0.98638
LDF 38 0.70000 0.68592 0.87938 0.87371 0.82353 0.81524 0.99014
INFLO 66 0.70000 0.68592 0.75608 0.74462 0.75000 0.73826 0.98779
INFLO 71 0.70000 0.68592 0.86571 0.85940 0.80000 0.79061 0.99202
INFLO 74 0.70000 0.68592 0.87846 0.87275 0.82353 0.81524 0.99108
INFLO 75 0.70000 0.68592 0.88138 0.87581 0.82353 0.81524 0.99155
COF 40 0.60000 0.58122 0.79407 0.78440 0.70588 0.69207 0.98451
COF 42 0.60000 0.58122 0.78914 0.77924 0.75000 0.73826 0.98169
COF 60 0.60000 0.58122 0.76952 0.75870 0.72000 0.70685 0.98498
COF 99 0.70000 0.68592 0.73389 0.72140 0.70000 0.68592 0.98028

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 9 attributes, 454 objects, 10 outliers (2.20%)

Download raw algorithm results (2.0 MB) Download raw algorithm evaluation table (40.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.80000 0.79550 0.86878 0.86583 0.84211 0.83855 0.99336
KNN 6 0.80000 0.79550 0.89329 0.89088 0.88889 0.88639 0.99471
KNN 11 0.80000 0.79550 0.93056 0.92899 0.88889 0.88639 0.99752
KNN 76 0.80000 0.79550 0.92064 0.91885 0.85714 0.85393 0.99764
KNNW 7 0.80000 0.79550 0.86286 0.85977 0.82353 0.81955 0.99347
KNNW 15 0.80000 0.79550 0.89840 0.89611 0.88889 0.88639 0.99550
KNNW 41 0.80000 0.79550 0.91056 0.90854 0.82353 0.81955 0.99707
LOF 93 0.70000 0.69324 0.74141 0.73558 0.70000 0.69324 0.98986
LOF 96 0.70000 0.69324 0.81101 0.80675 0.75000 0.74437 0.99144
LOF 99 0.70000 0.69324 0.81361 0.80941 0.75000 0.74437 0.99189
SimplifiedLOF 85 0.50000 0.48874 0.34755 0.33285 0.57143 0.56178 0.97252
SimplifiedLOF 93 0.50000 0.48874 0.42347 0.41049 0.63636 0.62817 0.97770
SimplifiedLOF 100 0.50000 0.48874 0.48632 0.47475 0.63636 0.62817 0.97973
LoOP 88 0.30000 0.28423 0.25390 0.23709 0.44444 0.43193 0.96014
LoOP 98 0.30000 0.28423 0.29817 0.28236 0.48276 0.47111 0.96554
LoOP 100 0.30000 0.28423 0.30785 0.29227 0.48276 0.47111 0.96667
LDOF 58 0.10000 0.07973 0.09463 0.07424 0.22535 0.20791 0.86847
LDOF 100 0.10000 0.07973 0.20011 0.18210 0.35897 0.34454 0.94167
ODIN 98 0.20000 0.18198 0.31623 0.30083 0.50000 0.48874 0.97061
ODIN 99 0.25000 0.23311 0.31794 0.30258 0.50000 0.48874 0.97140
ODIN 100 0.25000 0.23311 0.31910 0.30376 0.50000 0.48874 0.97185
FastABOD 28 0.80000 0.79550 0.86070 0.85756 0.82353 0.81955 0.99279
FastABOD 83 0.80000 0.79550 0.86998 0.86705 0.82353 0.81955 0.99392
KDEOS 2 0.00000 -0.02252 0.10385 0.08366 0.21176 0.19401 0.83491
KDEOS 5 0.20000 0.18198 0.10734 0.08724 0.25000 0.23311 0.76014
KDEOS 7 0.20000 0.18198 0.14736 0.12816 0.32432 0.30911 0.78311
LDF 52 0.70000 0.69324 0.55659 0.54660 0.70000 0.69324 0.98761
LDF 86 0.70000 0.69324 0.85779 0.85459 0.82353 0.81955 0.99392
LDF 97 0.70000 0.69324 0.87641 0.87362 0.82353 0.81955 0.99527
INFLO 84 0.50000 0.48874 0.47701 0.46523 0.64000 0.63189 0.97883
INFLO 96 0.60000 0.59099 0.56885 0.55914 0.61538 0.60672 0.98311
INFLO 100 0.60000 0.59099 0.57485 0.56527 0.61538 0.60672 0.98423
COF 76 0.50000 0.48874 0.39640 0.38280 0.58065 0.57120 0.98176
COF 94 0.40000 0.38649 0.54553 0.53529 0.74074 0.73490 0.98896

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 9 attributes, 223 objects, 10 outliers (4.48%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (33.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.83333 0.82551 0.91126 0.90709 0.85714 0.85044 0.99366
KNNW 3 0.90000 0.89531 0.93545 0.93242 0.90000 0.89531 0.99390
KNNW 5 0.80000 0.79061 0.92334 0.91974 0.85714 0.85044 0.99437
LOF 49 0.70000 0.68592 0.65192 0.63558 0.71429 0.70087 0.98310
LOF 57 0.70000 0.68592 0.85720 0.85050 0.82353 0.81524 0.98826
LOF 87 0.70000 0.68592 0.87443 0.86854 0.82353 0.81524 0.98967
SimplifiedLOF 66 0.70000 0.68592 0.75588 0.74442 0.70000 0.68592 0.98545
SimplifiedLOF 75 0.70000 0.68592 0.85891 0.85228 0.82353 0.81524 0.98826
SimplifiedLOF 89 0.70000 0.68592 0.86723 0.86099 0.82353 0.81524 0.98920
SimplifiedLOF 99 0.70000 0.68592 0.87112 0.86507 0.82353 0.81524 0.98920
LoOP 89 0.50000 0.47653 0.62243 0.60471 0.75000 0.73826 0.97981
LoOP 97 0.70000 0.68592 0.74416 0.73215 0.72000 0.70685 0.98545
LoOP 99 0.70000 0.68592 0.82194 0.81358 0.73684 0.72449 0.98779
LDOF 77 0.40000 0.37183 0.45907 0.43367 0.61538 0.59733 0.96197
LDOF 89 0.40000 0.37183 0.49220 0.46836 0.66667 0.65102 0.96761
LDOF 100 0.40000 0.37183 0.50712 0.48398 0.66667 0.65102 0.97089
ODIN 79 0.42000 0.39277 0.39420 0.36576 0.56250 0.54196 0.96385
ODIN 96 0.40000 0.37183 0.42041 0.39320 0.62069 0.60288 0.96596
ODIN 97 0.40000 0.37183 0.43195 0.40528 0.62069 0.60288 0.96667
ODIN 98 0.40000 0.37183 0.44028 0.41400 0.62069 0.60288 0.96596
FastABOD 6 0.80000 0.79061 0.89263 0.88759 0.82353 0.81524 0.99202
FastABOD 19 0.80000 0.79061 0.90888 0.90460 0.88889 0.88367 0.99249
FastABOD 26 0.80000 0.79061 0.91882 0.91501 0.88889 0.88367 0.99390
KDEOS 12 0.10000 0.05775 0.06497 0.02107 0.13245 0.09172 0.61455
KDEOS 15 0.10000 0.05775 0.07900 0.03576 0.14286 0.10262 0.62066
KDEOS 17 0.10000 0.05775 0.06986 0.02619 0.14184 0.10155 0.64131
KDEOS 100 0.00000 -0.04695 0.05667 0.01238 0.16667 0.12754 0.60376
LDF 29 0.70000 0.68592 0.77790 0.76747 0.70588 0.69207 0.98545
LDF 33 0.70000 0.68592 0.84616 0.83893 0.82353 0.81524 0.98638
LDF 71 0.70000 0.68592 0.86930 0.86317 0.82353 0.81524 0.98920
INFLO 70 0.70000 0.68592 0.82999 0.82201 0.77778 0.76734 0.98873
INFLO 75 0.70000 0.68592 0.87582 0.86999 0.82353 0.81524 0.99061
COF 61 0.70000 0.68592 0.78750 0.77752 0.77778 0.76734 0.98545

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 9 attributes, 454 objects, 10 outliers (2.20%)

Download raw algorithm results (1.9 MB) Download raw algorithm evaluation table (40.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.80000 0.79550 0.88626 0.88370 0.84211 0.83855 0.99538
KNN 10 0.80000 0.79550 0.92186 0.92010 0.88889 0.88639 0.99730
KNN 14 0.80000 0.79550 0.93445 0.93297 0.88889 0.88639 0.99764
KNNW 1 0.80000 0.79550 0.86000 0.85685 0.82353 0.81955 0.99302
KNNW 16 0.80000 0.79550 0.90056 0.89832 0.88889 0.88639 0.99572
KNNW 40 0.80000 0.79550 0.91368 0.91173 0.84211 0.83855 0.99707
LOF 94 0.70000 0.69324 0.71322 0.70676 0.70000 0.69324 0.99009
LOF 99 0.70000 0.69324 0.72024 0.71394 0.73684 0.73092 0.98986
SimplifiedLOF 92 0.50000 0.48874 0.38056 0.36661 0.57143 0.56178 0.97500
SimplifiedLOF 94 0.50000 0.48874 0.41583 0.40267 0.60870 0.59988 0.97748
SimplifiedLOF 99 0.50000 0.48874 0.42662 0.41370 0.60870 0.59988 0.97928
LoOP 89 0.30000 0.28423 0.25149 0.23463 0.42424 0.41127 0.95743
LoOP 98 0.30000 0.28423 0.29904 0.28326 0.51852 0.50767 0.96509
LoOP 100 0.30000 0.28423 0.31359 0.29813 0.51852 0.50767 0.96622
LDOF 96 0.20000 0.18198 0.19242 0.17423 0.32558 0.31039 0.93604
LDOF 99 0.20000 0.18198 0.20192 0.18394 0.35000 0.33536 0.93806
ODIN 94 0.20000 0.18198 0.28696 0.27090 0.45000 0.43761 0.97050
ODIN 98 0.23333 0.21607 0.28696 0.27090 0.45000 0.43761 0.97083
ODIN 100 0.23333 0.21607 0.29053 0.27455 0.45000 0.43761 0.97106
FastABOD 28 0.80000 0.79550 0.87472 0.87190 0.84211 0.83855 0.99392
FastABOD 42 0.80000 0.79550 0.89294 0.89053 0.88889 0.88639 0.99482
KDEOS 2 0.00000 -0.02252 0.08707 0.06651 0.19355 0.17539 0.80957
LDF 73 0.70000 0.69324 0.67087 0.66346 0.73684 0.73092 0.99032
LDF 88 0.70000 0.69324 0.86458 0.86153 0.82353 0.81955 0.99437
LDF 98 0.70000 0.69324 0.86699 0.86400 0.82353 0.81955 0.99482
INFLO 90 0.50000 0.48874 0.48232 0.47066 0.58333 0.57395 0.98018
INFLO 93 0.50000 0.48874 0.49883 0.48754 0.63636 0.62817 0.98086
INFLO 98 0.40000 0.38649 0.54780 0.53762 0.60870 0.59988 0.98266
INFLO 100 0.40000 0.38649 0.53075 0.52018 0.59259 0.58342 0.98288
COF 85 0.40000 0.38649 0.34190 0.32708 0.52941 0.51881 0.97703
COF 97 0.30000 0.28423 0.35795 0.34349 0.60000 0.59099 0.97973
COF 100 0.30000 0.28423 0.36626 0.35198 0.53846 0.52807 0.97680

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO