Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (2% of outliers version#10)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 2579 objects, 51 outliers (1.98%)

Download raw algorithm results (23.1 MB) Download raw algorithm evaluation table (67.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05882 0.03984 0.08158 0.06305 0.19883 0.18267 0.80406
KNN 2 0.07843 0.05984 0.07700 0.05837 0.18653 0.17012 0.79488
KNNW 1 0.07843 0.05984 0.09068 0.07233 0.23313 0.21766 0.79848
KNNW 2 0.07843 0.05984 0.08647 0.06804 0.21348 0.19762 0.80272
KNNW 3 0.09804 0.07984 0.08496 0.06650 0.20732 0.19133 0.80149
LOF 8 0.03922 0.01983 0.04967 0.03050 0.11765 0.09985 0.75596
LOF 14 0.00000 -0.02017 0.05630 0.03727 0.14793 0.13074 0.78955
LOF 77 0.01961 -0.00017 0.05901 0.04003 0.15075 0.13362 0.75680
LOF 91 0.03922 0.01983 0.05744 0.03843 0.15574 0.13871 0.75660
SimplifiedLOF 17 0.00000 -0.02017 0.04487 0.02560 0.12235 0.10465 0.75130
SimplifiedLOF 82 0.03922 0.01983 0.04009 0.02073 0.09383 0.07555 0.71198
SimplifiedLOF 100 0.01961 -0.00017 0.04660 0.02737 0.11765 0.09985 0.72807
LoOP 80 0.07843 0.05984 0.05872 0.03974 0.14070 0.12337 0.76018
LoOP 97 0.07843 0.05984 0.06096 0.04201 0.15166 0.13454 0.76619
LoOP 98 0.07843 0.05984 0.06114 0.04220 0.14884 0.13167 0.76637
LoOP 100 0.07843 0.05984 0.06109 0.04215 0.14575 0.12852 0.76734
LDOF 8 0.03922 0.01983 0.04065 0.02130 0.09492 0.07666 0.69961
LDOF 49 0.01961 -0.00017 0.04662 0.02739 0.11401 0.09614 0.75480
LDOF 98 0.01961 -0.00017 0.05469 0.03562 0.12552 0.10788 0.75364
LDOF 100 0.03922 0.01983 0.05490 0.03584 0.12308 0.10539 0.75445
ODIN 76 0.03922 0.01983 0.05241 0.03329 0.12245 0.10475 0.75344
ODIN 98 0.03922 0.01983 0.05636 0.03732 0.13502 0.11757 0.76274
ODIN 100 0.03922 0.01983 0.05649 0.03746 0.13265 0.11516 0.76314
FastABOD 5 0.07843 0.05984 0.07300 0.05430 0.17526 0.15862 0.78173
FastABOD 12 0.09804 0.07984 0.07408 0.05540 0.17778 0.16119 0.77733
FastABOD 62 0.07843 0.05984 0.07513 0.05647 0.19753 0.18134 0.76487
FastABOD 71 0.07843 0.05984 0.07455 0.05588 0.19917 0.18301 0.76386
KDEOS 5 0.05882 0.03984 0.03405 0.01457 0.07932 0.06075 0.62416
KDEOS 8 0.03922 0.01983 0.03482 0.01535 0.10152 0.08340 0.63604
KDEOS 99 0.05882 0.03984 0.05144 0.03230 0.08980 0.07143 0.67082
KDEOS 100 0.05882 0.03984 0.04917 0.02999 0.09167 0.07334 0.67369
LDF 3 0.05882 0.03984 0.04028 0.02092 0.08134 0.06281 0.70088
LDF 5 0.05882 0.03984 0.04261 0.02329 0.08669 0.06826 0.73395
LDF 6 0.03922 0.01983 0.04572 0.02647 0.10909 0.09112 0.73071
LDF 54 0.03704 0.01761 0.03768 0.01827 0.11518 0.09733 0.64255
INFLO 58 0.07843 0.05984 0.06044 0.04148 0.14706 0.12985 0.76964
INFLO 81 0.05882 0.03984 0.06218 0.04326 0.15444 0.13738 0.77483
INFLO 82 0.05882 0.03984 0.06192 0.04300 0.15625 0.13923 0.77484
INFLO 83 0.05882 0.03984 0.06180 0.04287 0.15873 0.14176 0.77466
COF 23 0.00000 -0.02017 0.05681 0.03779 0.13291 0.11542 0.77485
COF 31 0.00000 -0.02017 0.05387 0.03478 0.14337 0.12609 0.75950
COF 98 0.09804 0.07984 0.04506 0.02580 0.11268 0.09478 0.67373

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 2844 objects, 56 outliers (1.97%)

Download raw algorithm results (23.4 MB) Download raw algorithm evaluation table (69.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.08929 0.07099 0.07953 0.06104 0.16138 0.14454 0.84818
KNN 5 0.10714 0.08921 0.07054 0.05187 0.14536 0.12820 0.82528
KNNW 1 0.10714 0.08921 0.08644 0.06809 0.17600 0.15945 0.81458
KNNW 4 0.08929 0.07099 0.08127 0.06282 0.15385 0.13685 0.84291
LOF 12 0.01786 -0.00187 0.05113 0.03207 0.11982 0.10214 0.80938
LOF 14 0.01786 -0.00187 0.05020 0.03113 0.12371 0.10611 0.80414
LOF 75 0.03571 0.01635 0.02904 0.00954 0.07165 0.05300 0.67547
SimplifiedLOF 13 0.01786 -0.00187 0.03949 0.02019 0.09249 0.07426 0.75941
SimplifiedLOF 16 0.01786 -0.00187 0.03943 0.02013 0.08929 0.07099 0.76123
SimplifiedLOF 38 0.00000 -0.02009 0.03640 0.01704 0.09942 0.08133 0.74396
SimplifiedLOF 92 0.03571 0.01635 0.02826 0.00874 0.07005 0.05137 0.66367
LoOP 11 0.01786 -0.00187 0.04118 0.02192 0.09113 0.07288 0.75103
LoOP 20 0.01786 -0.00187 0.04571 0.02655 0.10049 0.08242 0.77884
LoOP 22 0.01786 -0.00187 0.04569 0.02652 0.10249 0.08447 0.78096
LoOP 52 0.01786 -0.00187 0.03998 0.02070 0.10366 0.08565 0.75974
LDOF 4 0.03571 0.01635 0.03032 0.01084 0.06757 0.04884 0.63003
LDOF 36 0.01786 -0.00187 0.04066 0.02139 0.09565 0.07749 0.76107
LDOF 43 0.01786 -0.00187 0.03984 0.02056 0.09639 0.07824 0.76248
LDOF 49 0.01786 -0.00187 0.03968 0.02039 0.09672 0.07858 0.76163
ODIN 47 0.06331 0.04450 0.04656 0.02741 0.09477 0.07659 0.75610
ODIN 99 0.05357 0.03456 0.05413 0.03513 0.11230 0.09447 0.77669
ODIN 100 0.05357 0.03456 0.05423 0.03523 0.11260 0.09478 0.77529
FastABOD 9 0.01786 -0.00187 0.04951 0.03042 0.11765 0.09992 0.79348
FastABOD 20 0.05357 0.03456 0.05397 0.03497 0.11694 0.09920 0.79943
FastABOD 100 0.05357 0.03456 0.05791 0.03899 0.11739 0.09966 0.79795
KDEOS 5 0.05357 0.03456 0.02857 0.00905 0.05949 0.04060 0.61612
KDEOS 8 0.00000 -0.02009 0.02931 0.00981 0.08088 0.06242 0.63381
KDEOS 36 0.00000 -0.02009 0.03453 0.01514 0.07513 0.05655 0.69254
KDEOS 60 0.01786 -0.00187 0.03307 0.01365 0.07595 0.05739 0.70497
LDF 4 0.01786 -0.00187 0.04559 0.02642 0.11026 0.09239 0.78205
LDF 5 0.03571 0.01635 0.04660 0.02745 0.10707 0.08913 0.77767
INFLO 13 0.01786 -0.00187 0.04687 0.02772 0.10585 0.08789 0.78862
INFLO 18 0.01786 -0.00187 0.04465 0.02547 0.11111 0.09326 0.78318
INFLO 77 0.03571 0.01635 0.03154 0.01209 0.07423 0.05564 0.69284
COF 14 0.01786 -0.00187 0.03994 0.02065 0.08211 0.06367 0.75316
COF 26 0.01786 -0.00187 0.04586 0.02669 0.10753 0.08960 0.77994

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 2579 objects, 51 outliers (1.98%)

Download raw algorithm results (22.4 MB) Download raw algorithm evaluation table (60.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.19608 0.17986 0.10313 0.08503 0.21687 0.20107 0.75686
KNN 2 0.19608 0.17986 0.09980 0.08164 0.21739 0.20160 0.75769
KNN 6 0.11765 0.09985 0.08633 0.06789 0.17699 0.16039 0.76182
KNNW 1 0.19608 0.17986 0.13173 0.11421 0.21053 0.19460 0.75441
KNNW 2 0.17647 0.15986 0.10947 0.09151 0.22500 0.20937 0.76094
KNNW 3 0.19608 0.17986 0.10649 0.08847 0.22222 0.20653 0.76115
LOF 1 0.11765 0.09985 0.06616 0.04732 0.12389 0.10622 0.54744
LOF 4 0.11765 0.09985 0.04254 0.02322 0.13043 0.11289 0.57147
LOF 100 0.05882 0.03984 0.03999 0.02062 0.09174 0.07342 0.67358
SimplifiedLOF 1 0.09804 0.07984 0.05054 0.03138 0.13158 0.11406 0.58647
SimplifiedLOF 10 0.07843 0.05984 0.04616 0.02692 0.13889 0.12152 0.60503
SimplifiedLOF 100 0.07843 0.05984 0.04301 0.02371 0.10256 0.08446 0.64956
LoOP 1 0.09804 0.07984 0.05060 0.03145 0.13158 0.11406 0.59532
LoOP 5 0.11765 0.09985 0.04226 0.02294 0.12500 0.10735 0.58583
LoOP 100 0.07843 0.05984 0.03404 0.01455 0.08889 0.07051 0.61565
LDOF 3 0.03922 0.01983 0.05195 0.03283 0.08333 0.06484 0.60267
LDOF 14 0.09804 0.07984 0.03276 0.01325 0.10000 0.08184 0.55950
LDOF 94 0.07843 0.05984 0.03036 0.01080 0.10256 0.08446 0.55661
ODIN 1 0.03046 0.01090 0.02716 0.00753 0.05792 0.03891 0.61757
ODIN 13 0.01538 -0.00448 0.02741 0.00779 0.06275 0.04384 0.60235
ODIN 45 0.03922 0.01983 0.02398 0.00429 0.05052 0.03137 0.57605
FastABOD 3 0.15686 0.13985 0.10234 0.08423 0.20513 0.18909 0.76672
FastABOD 4 0.15686 0.13985 0.11248 0.09458 0.20253 0.18644 0.77266
FastABOD 6 0.15686 0.13985 0.09946 0.08130 0.20513 0.18909 0.77448
FastABOD 9 0.17647 0.15986 0.09718 0.07896 0.18349 0.16701 0.77221
KDEOS 6 0.01961 -0.00017 0.04201 0.02269 0.04775 0.02854 0.54865
KDEOS 41 0.01961 -0.00017 0.03212 0.01259 0.07590 0.05726 0.62666
KDEOS 83 0.07843 0.05984 0.03171 0.01218 0.07843 0.05984 0.61276
KDEOS 88 0.07843 0.05984 0.03194 0.01242 0.08511 0.06665 0.61378
LDF 68 0.11765 0.09985 0.06400 0.04512 0.15190 0.13479 0.72104
LDF 95 0.09804 0.07984 0.06785 0.04905 0.14286 0.12557 0.74498
INFLO 1 0.09804 0.07984 0.06770 0.04889 0.10811 0.09012 0.58419
INFLO 3 0.11765 0.09985 0.04038 0.02102 0.12000 0.10225 0.59343
INFLO 5 0.11765 0.09985 0.04041 0.02105 0.12500 0.10735 0.59976
INFLO 62 0.07843 0.05984 0.03686 0.01743 0.08511 0.06665 0.66190
COF 7 0.13725 0.11985 0.05384 0.03475 0.15584 0.13881 0.58009
COF 8 0.13725 0.11985 0.05309 0.03399 0.16279 0.14590 0.59643
COF 73 0.07843 0.05984 0.06099 0.04204 0.12500 0.10735 0.67061
COF 94 0.09804 0.07984 0.05926 0.04028 0.11579 0.09795 0.68846

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 2844 objects, 56 outliers (1.97%)

Download raw algorithm results (23.2 MB) Download raw algorithm evaluation table (64.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.19643 0.18029 0.10110 0.08305 0.20561 0.18965 0.80596
KNNW 1 0.17857 0.16207 0.10288 0.08486 0.17978 0.16330 0.81439
KNNW 3 0.19643 0.18029 0.10417 0.08618 0.20755 0.19163 0.80841
LOF 4 0.00000 -0.02009 0.03092 0.01146 0.08333 0.06492 0.62499
LOF 8 0.05357 0.03456 0.02773 0.00820 0.07237 0.05374 0.58125
LOF 100 0.01786 -0.00187 0.03237 0.01293 0.07077 0.05211 0.67151
SimplifiedLOF 2 0.05357 0.03456 0.03270 0.01327 0.07595 0.05739 0.62778
SimplifiedLOF 3 0.03571 0.01635 0.03506 0.01568 0.07921 0.06071 0.66285
LoOP 3 0.05357 0.03456 0.03557 0.01620 0.07708 0.05854 0.67154
LoOP 4 0.07143 0.05278 0.03616 0.01680 0.08081 0.06235 0.65091
LoOP 9 0.07143 0.05278 0.03532 0.01594 0.08850 0.07019 0.65177
LDOF 4 0.03571 0.01635 0.03217 0.01273 0.08491 0.06652 0.58330
LDOF 5 0.05357 0.03456 0.03101 0.01155 0.08032 0.06185 0.59139
LDOF 8 0.01786 -0.00187 0.02995 0.01046 0.09574 0.07758 0.58524
ODIN 2 0.03327 0.01385 0.03178 0.01234 0.06533 0.04655 0.68471
ODIN 6 0.02516 0.00558 0.03047 0.01100 0.06884 0.05013 0.67663
FastABOD 14 0.12500 0.10742 0.07359 0.05498 0.15642 0.13948 0.80159
FastABOD 76 0.12500 0.10742 0.08558 0.06722 0.16766 0.15095 0.80296
FastABOD 96 0.12500 0.10742 0.08582 0.06746 0.16766 0.15095 0.80364
FastABOD 99 0.12500 0.10742 0.08581 0.06745 0.16766 0.15095 0.80367
KDEOS 2 0.05357 0.03456 0.02592 0.00635 0.05455 0.03555 0.57772
KDEOS 21 0.03571 0.01635 0.03378 0.01437 0.08219 0.06376 0.66372
LDF 74 0.10714 0.08921 0.04916 0.03006 0.11667 0.09892 0.71787
LDF 89 0.10714 0.08921 0.05416 0.03516 0.13146 0.11401 0.74880
LDF 90 0.10714 0.08921 0.05454 0.03555 0.13084 0.11338 0.74890
LDF 99 0.10714 0.08921 0.05367 0.03466 0.11429 0.09650 0.75774
INFLO 1 0.03571 0.01635 0.02705 0.00751 0.06014 0.04126 0.57962
INFLO 3 0.00000 -0.02009 0.03531 0.01593 0.09444 0.07625 0.61778
INFLO 9 0.01786 -0.00187 0.03136 0.01191 0.06553 0.04676 0.65709
COF 3 0.01786 -0.00187 0.03317 0.01375 0.08633 0.06798 0.64870
COF 9 0.05357 0.03456 0.02708 0.00754 0.05970 0.04081 0.58385
COF 100 0.05357 0.03456 0.03498 0.01560 0.07168 0.05304 0.68367

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO