Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (2% of outliers version#08)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 2579 objects, 51 outliers (1.98%)

Download raw algorithm results (23.1 MB) Download raw algorithm evaluation table (66.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.07843 0.05984 0.07107 0.05233 0.16216 0.14526 0.78096
KNNW 1 0.11765 0.09985 0.07301 0.05431 0.18779 0.17141 0.71614
KNNW 4 0.07843 0.05984 0.06899 0.05021 0.16271 0.14582 0.76977
LOF 46 0.05882 0.03984 0.04283 0.02352 0.10017 0.08202 0.73702
LOF 56 0.03922 0.01983 0.05830 0.03930 0.11940 0.10164 0.75744
LOF 69 0.03922 0.01983 0.05914 0.04016 0.12444 0.10678 0.75556
LOF 71 0.03922 0.01983 0.05873 0.03974 0.12785 0.11026 0.75545
SimplifiedLOF 38 0.00000 -0.02017 0.03764 0.01823 0.09205 0.07373 0.73390
SimplifiedLOF 69 0.05882 0.03984 0.03969 0.02032 0.08517 0.06671 0.71227
SimplifiedLOF 99 0.05882 0.03984 0.04618 0.02694 0.10309 0.08500 0.71888
SimplifiedLOF 100 0.05882 0.03984 0.04622 0.02698 0.10101 0.08287 0.71884
LoOP 68 0.07843 0.05984 0.05289 0.03379 0.10543 0.08738 0.75775
LoOP 86 0.07843 0.05984 0.05899 0.04000 0.12712 0.10951 0.76221
LoOP 89 0.07843 0.05984 0.05933 0.04036 0.12821 0.11062 0.76201
LoOP 97 0.05882 0.03984 0.06024 0.04128 0.12587 0.10824 0.76129
LDOF 56 0.01961 -0.00017 0.04240 0.02308 0.09476 0.07650 0.75255
LDOF 79 0.07843 0.05984 0.04787 0.02866 0.09687 0.07865 0.75070
LDOF 94 0.07843 0.05984 0.05261 0.03350 0.10638 0.08836 0.75129
LDOF 99 0.07843 0.05984 0.05301 0.03391 0.10526 0.08721 0.75000
ODIN 7 0.04651 0.02728 0.03128 0.01173 0.06667 0.04784 0.64671
ODIN 57 0.03922 0.01983 0.05122 0.03208 0.13551 0.11807 0.74602
ODIN 92 0.03922 0.01983 0.05342 0.03432 0.12714 0.10953 0.74733
FastABOD 5 0.09804 0.07984 0.06050 0.04155 0.15534 0.13830 0.74377
FastABOD 25 0.07843 0.05984 0.06250 0.04359 0.16667 0.14985 0.74754
FastABOD 98 0.09804 0.07984 0.06566 0.04681 0.16129 0.14437 0.75221
KDEOS 21 0.01961 -0.00017 0.03033 0.01077 0.06537 0.04652 0.66727
KDEOS 80 0.05882 0.03984 0.03340 0.01390 0.07668 0.05805 0.65714
KDEOS 86 0.01961 -0.00017 0.03333 0.01382 0.09569 0.07745 0.65702
KDEOS 100 0.01961 -0.00017 0.03720 0.01778 0.09272 0.07441 0.66592
LDF 5 0.03922 0.01983 0.04547 0.02622 0.10702 0.08901 0.74274
LDF 6 0.03922 0.01983 0.04281 0.02350 0.08879 0.07041 0.75324
LDF 49 0.09459 0.07633 0.04547 0.02621 0.13978 0.12243 0.60750
LDF 52 0.08696 0.06854 0.04346 0.02416 0.14689 0.12968 0.61318
INFLO 54 0.07843 0.05984 0.06135 0.04241 0.12665 0.10903 0.77263
INFLO 56 0.07843 0.05984 0.06221 0.04329 0.12973 0.11217 0.77445
INFLO 75 0.05882 0.03984 0.06221 0.04329 0.13761 0.12022 0.77337
INFLO 78 0.05882 0.03984 0.06242 0.04350 0.13514 0.11769 0.77311
COF 27 0.00000 -0.02017 0.04559 0.02634 0.12618 0.10855 0.74609
COF 28 0.00000 -0.02017 0.04564 0.02639 0.12500 0.10735 0.74768
COF 81 0.05882 0.03984 0.03903 0.01964 0.09562 0.07737 0.67756
COF 100 0.03922 0.01983 0.04856 0.02937 0.10490 0.08684 0.69337

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 2844 objects, 56 outliers (1.97%)

Download raw algorithm results (23.4 MB) Download raw algorithm evaluation table (67.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.12500 0.10742 0.10353 0.08552 0.22564 0.21009 0.81370
KNN 4 0.16071 0.14386 0.09311 0.07490 0.17467 0.15809 0.78999
KNNW 1 0.17857 0.16207 0.11850 0.10079 0.24161 0.22638 0.83931
LOF 1 0.03571 0.01635 0.02913 0.00963 0.06577 0.04701 0.61868
LOF 10 0.00000 -0.02009 0.04898 0.02988 0.12043 0.10276 0.75908
LOF 12 0.01786 -0.00187 0.05026 0.03119 0.11969 0.10201 0.76216
LOF 15 0.01786 -0.00187 0.04863 0.02952 0.11538 0.09762 0.76627
SimplifiedLOF 11 0.01786 -0.00187 0.04075 0.02149 0.11052 0.09265 0.72425
SimplifiedLOF 14 0.01786 -0.00187 0.04150 0.02225 0.10420 0.08621 0.73592
SimplifiedLOF 23 0.01786 -0.00187 0.04100 0.02174 0.10078 0.08271 0.74182
LoOP 6 0.05357 0.03456 0.04287 0.02365 0.10490 0.08692 0.71003
LoOP 14 0.05357 0.03456 0.05059 0.03152 0.12551 0.10794 0.74826
LoOP 19 0.05357 0.03456 0.05227 0.03323 0.12000 0.10232 0.75667
LoOP 25 0.05357 0.03456 0.05161 0.03256 0.12148 0.10383 0.76681
LDOF 8 0.05357 0.03456 0.03900 0.01970 0.08636 0.06800 0.69052
LDOF 19 0.03571 0.01635 0.04486 0.02567 0.10526 0.08729 0.73563
LDOF 38 0.03571 0.01635 0.04361 0.02440 0.10552 0.08755 0.73738
LDOF 54 0.01786 -0.00187 0.04229 0.02306 0.10114 0.08309 0.74142
ODIN 73 0.05601 0.03705 0.05711 0.03817 0.14388 0.12669 0.74964
ODIN 98 0.06667 0.04792 0.06016 0.04129 0.14070 0.12344 0.75724
ODIN 99 0.06250 0.04367 0.06011 0.04123 0.14141 0.12417 0.75795
ODIN 100 0.06429 0.04549 0.06026 0.04139 0.14141 0.12417 0.75760
FastABOD 10 0.01786 -0.00187 0.06077 0.04190 0.15331 0.13630 0.78010
FastABOD 23 0.07143 0.05278 0.06410 0.04530 0.16279 0.14597 0.77618
FastABOD 48 0.08929 0.07099 0.06383 0.04503 0.15966 0.14278 0.77558
FastABOD 71 0.08929 0.07099 0.06614 0.04738 0.16034 0.14347 0.77577
KDEOS 5 0.05357 0.03456 0.03815 0.01883 0.11659 0.09885 0.63948
KDEOS 8 0.08929 0.07099 0.04194 0.02270 0.09524 0.07706 0.67586
KDEOS 24 0.07143 0.05278 0.04217 0.02293 0.09322 0.07501 0.70747
KDEOS 37 0.01786 -0.00187 0.04024 0.02096 0.09709 0.07895 0.70917
LDF 5 0.05357 0.03456 0.04365 0.02445 0.09645 0.07830 0.74094
LDF 6 0.05357 0.03456 0.04960 0.03051 0.11499 0.09721 0.76198
INFLO 1 0.03571 0.01635 0.02910 0.00960 0.06667 0.04792 0.62350
INFLO 14 0.01786 -0.00187 0.04900 0.02990 0.12888 0.11138 0.75825
INFLO 16 0.01786 -0.00187 0.04853 0.02942 0.12473 0.10715 0.76194
COF 3 0.01786 -0.00187 0.02912 0.00962 0.06192 0.04308 0.65668
COF 26 0.01786 -0.00187 0.04737 0.02824 0.10984 0.09196 0.76650
COF 30 0.01786 -0.00187 0.04662 0.02747 0.12069 0.10303 0.76142

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 2579 objects, 51 outliers (1.98%)

Download raw algorithm results (22.4 MB) Download raw algorithm evaluation table (63.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.17647 0.15986 0.07433 0.05565 0.18750 0.17111 0.71738
KNN 51 0.03922 0.01983 0.05285 0.03374 0.12739 0.10978 0.72117
KNNW 1 0.09804 0.07984 0.06979 0.05103 0.14130 0.12398 0.73748
KNNW 2 0.15686 0.13985 0.07321 0.05452 0.16000 0.14305 0.72841
KNNW 3 0.15686 0.13985 0.07146 0.05272 0.16667 0.14985 0.71943
LOF 1 0.07843 0.05984 0.02472 0.00504 0.08000 0.06144 0.50261
LOF 2 0.05882 0.03984 0.02914 0.00956 0.08054 0.06199 0.54541
LOF 100 0.03922 0.01983 0.03043 0.01086 0.07583 0.05719 0.60462
SimplifiedLOF 2 0.05882 0.03984 0.03408 0.01459 0.08989 0.07153 0.56832
SimplifiedLOF 3 0.09804 0.07984 0.03325 0.01375 0.10753 0.08952 0.57646
SimplifiedLOF 5 0.03922 0.01983 0.03057 0.01102 0.06897 0.05018 0.59758
LoOP 1 0.03922 0.01983 0.03162 0.01208 0.10638 0.08836 0.55809
LoOP 2 0.05882 0.03984 0.03236 0.01284 0.09424 0.07597 0.56900
LoOP 3 0.09804 0.07984 0.03194 0.01241 0.10101 0.08287 0.55709
LoOP 13 0.01961 -0.00017 0.02465 0.00497 0.05336 0.03427 0.57874
LDOF 2 0.05882 0.03984 0.03317 0.01367 0.08824 0.06984 0.57174
LDOF 4 0.07843 0.05984 0.03263 0.01311 0.09615 0.07792 0.51924
ODIN 4 0.02778 0.00816 0.02504 0.00537 0.05199 0.03287 0.58436
ODIN 6 0.02532 0.00565 0.02539 0.00573 0.05588 0.03683 0.57810
ODIN 7 0.01418 -0.00570 0.02469 0.00501 0.05910 0.04012 0.57951
ODIN 12 0.01333 -0.00657 0.02463 0.00495 0.05463 0.03556 0.59660
FastABOD 3 0.11765 0.09985 0.07057 0.05182 0.14679 0.12958 0.74422
FastABOD 6 0.09804 0.07984 0.06527 0.04641 0.15094 0.13381 0.75708
KDEOS 4 0.07843 0.05984 0.04875 0.02956 0.09412 0.07584 0.54514
KDEOS 27 0.03922 0.01983 0.03033 0.01077 0.07055 0.05180 0.60816
LDF 14 0.05882 0.03984 0.02540 0.00574 0.06452 0.04564 0.51918
LDF 84 0.03922 0.01983 0.05082 0.03167 0.14054 0.12320 0.69236
LDF 97 0.03922 0.01983 0.05314 0.03403 0.12438 0.10671 0.69768
LDF 99 0.03922 0.01983 0.05280 0.03370 0.12853 0.11095 0.69950
INFLO 2 0.05882 0.03984 0.02970 0.01013 0.07018 0.05142 0.54332
INFLO 21 0.01961 -0.00017 0.02504 0.00537 0.06528 0.04642 0.60097
INFLO 56 0.03922 0.01983 0.02594 0.00628 0.07063 0.05188 0.58744
COF 3 0.09804 0.07984 0.03193 0.01240 0.10204 0.08393 0.56630
COF 96 0.05882 0.03984 0.04204 0.02271 0.08547 0.06702 0.62149
COF 99 0.05882 0.03984 0.04232 0.02300 0.08929 0.07091 0.61868

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 2844 objects, 56 outliers (1.97%)

Download raw algorithm results (23.2 MB) Download raw algorithm evaluation table (63.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.12500 0.10742 0.09585 0.07769 0.15789 0.14098 0.77210
KNN 2 0.12500 0.10742 0.09840 0.08029 0.16901 0.15232 0.77102
KNN 3 0.12500 0.10742 0.09770 0.07957 0.17021 0.15355 0.77317
KNN 6 0.12500 0.10742 0.09197 0.07373 0.17530 0.15873 0.76770
KNNW 1 0.14286 0.12564 0.11658 0.09884 0.18251 0.16609 0.79300
LOF 1 0.10714 0.08921 0.04110 0.02184 0.11009 0.09222 0.59963
LOF 2 0.07143 0.05278 0.03086 0.01140 0.11111 0.09326 0.51458
LOF 100 0.08929 0.07099 0.04634 0.02718 0.10390 0.08590 0.66061
SimplifiedLOF 1 0.00000 -0.02009 0.03386 0.01445 0.11159 0.09374 0.60550
SimplifiedLOF 22 0.00000 -0.02009 0.03208 0.01264 0.06623 0.04747 0.67103
SimplifiedLOF 100 0.08929 0.07099 0.03609 0.01673 0.09091 0.07265 0.64459
LoOP 1 0.12500 0.10742 0.05525 0.03627 0.14773 0.13061 0.61769
LoOP 2 0.08929 0.07099 0.06545 0.04668 0.10294 0.08492 0.61333
LoOP 4 0.10714 0.08921 0.05853 0.03962 0.12174 0.10410 0.66032
LDOF 2 0.07143 0.05278 0.04162 0.02237 0.10435 0.08636 0.58281
LDOF 4 0.05357 0.03456 0.05525 0.03628 0.09170 0.07346 0.62012
ODIN 13 0.01389 -0.00592 0.02834 0.00882 0.06192 0.04308 0.64462
ODIN 18 0.00000 -0.02009 0.02864 0.00913 0.05612 0.03716 0.63645
ODIN 31 0.03307 0.01365 0.02529 0.00571 0.05138 0.03232 0.59780
FastABOD 8 0.12500 0.10742 0.07132 0.05267 0.14634 0.12919 0.77242
FastABOD 9 0.12500 0.10742 0.07248 0.05385 0.15000 0.13293 0.77242
FastABOD 14 0.12500 0.10742 0.07432 0.05572 0.13514 0.11776 0.77431
FastABOD 79 0.12500 0.10742 0.09236 0.07413 0.14493 0.12775 0.77296
KDEOS 11 0.07143 0.05278 0.02873 0.00922 0.07619 0.05763 0.62232
KDEOS 56 0.05357 0.03456 0.03488 0.01550 0.08824 0.06992 0.67821
KDEOS 57 0.03571 0.01635 0.03505 0.01567 0.08902 0.07072 0.67819
KDEOS 98 0.03571 0.01635 0.03605 0.01669 0.07407 0.05548 0.66530
LDF 66 0.12500 0.10742 0.05738 0.03845 0.13793 0.12062 0.67493
LDF 85 0.10714 0.08921 0.07432 0.05573 0.13514 0.11776 0.69407
LDF 100 0.10714 0.08921 0.06160 0.04275 0.15873 0.14183 0.72074
INFLO 1 0.10714 0.08921 0.03725 0.01791 0.10909 0.09120 0.57669
INFLO 6 0.07143 0.05278 0.03832 0.01901 0.09655 0.07840 0.67713
INFLO 100 0.08929 0.07099 0.04055 0.02128 0.10256 0.08454 0.59581
COF 5 0.10714 0.08921 0.03966 0.02037 0.11009 0.09222 0.62649
COF 87 0.01786 -0.00187 0.03576 0.01639 0.08133 0.06287 0.68510

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO