Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Wilt (2% of outliers version#06)

The data set differentiates diseased trees from other land covers. The former are considered outliers here.

Download all data set variants used (7.2 MB). You can also access the original data. (merge train and test [wilt.zip])

Normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (39.6 MB) Download raw algorithm evaluation table (73.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.02263 0.00270 0.05915 0.03998 0.58935
KNNW 1 0.00000 -0.02039 0.02469 0.00481 0.06208 0.04296 0.61496
LOF 1 0.07527 0.05642 0.03815 0.01854 0.13230 0.11461 0.58535
LOF 3 0.05376 0.03447 0.04051 0.02095 0.10072 0.08239 0.67541
LOF 7 0.03226 0.01253 0.03633 0.01668 0.08511 0.06646 0.69668
SimplifiedLOF 2 0.05376 0.03447 0.03905 0.01946 0.08104 0.06231 0.69647
SimplifiedLOF 6 0.04301 0.02350 0.05249 0.03317 0.13247 0.11479 0.76181
SimplifiedLOF 7 0.05376 0.03447 0.05284 0.03353 0.12672 0.10892 0.76349
LoOP 7 0.04301 0.02350 0.05447 0.03520 0.13543 0.11781 0.76497
LoOP 14 0.06452 0.04545 0.04347 0.02397 0.09562 0.07718 0.73974
LDOF 5 0.07527 0.05642 0.05128 0.03194 0.12346 0.10559 0.74360
LDOF 6 0.03226 0.01253 0.05430 0.03502 0.13453 0.11689 0.76235
LDOF 10 0.05376 0.03447 0.05805 0.03885 0.13029 0.11256 0.79257
LDOF 11 0.05376 0.03447 0.05623 0.03699 0.12000 0.10206 0.79408
ODIN 4 0.11111 0.09299 0.05659 0.03736 0.12381 0.10595 0.75748
ODIN 5 0.08040 0.06165 0.05282 0.03351 0.12059 0.10266 0.76132
ODIN 8 0.08309 0.06440 0.05838 0.03918 0.13365 0.11599 0.75411
ODIN 9 0.06319 0.04409 0.05520 0.03594 0.13492 0.11729 0.75402
FastABOD 3 0.01075 -0.00941 0.02589 0.00604 0.05834 0.03915 0.61782
FastABOD 4 0.00000 -0.02039 0.02539 0.00552 0.06293 0.04382 0.62544
KDEOS 12 0.13978 0.12225 0.06994 0.05098 0.15707 0.13988 0.77559
KDEOS 22 0.11828 0.10030 0.08548 0.06683 0.17367 0.15682 0.79160
KDEOS 23 0.10753 0.08933 0.08524 0.06659 0.18354 0.16690 0.79312
KDEOS 24 0.10753 0.08933 0.08349 0.06481 0.17204 0.15516 0.79377
LDF 1 0.04301 0.02350 0.03148 0.01174 0.09333 0.07485 0.57265
LDF 2 0.05376 0.03447 0.03478 0.01510 0.08407 0.06540 0.61979
LDF 3 0.03226 0.01253 0.03527 0.01560 0.08902 0.07045 0.65025
LDF 4 0.04301 0.02350 0.03483 0.01515 0.07764 0.05884 0.65991
INFLO 4 0.07527 0.05642 0.04411 0.02462 0.09979 0.08144 0.70073
INFLO 6 0.03226 0.01253 0.04427 0.02479 0.11429 0.09623 0.70053
COF 2 0.05376 0.03447 0.04063 0.02107 0.09677 0.07836 0.69273
COF 4 0.02151 0.00156 0.04402 0.02453 0.10857 0.09040 0.71995

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (39.7 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.02122 0.00134 0.05532 0.03613 0.56645
KNNW 1 0.00000 -0.02031 0.02274 0.00288 0.05758 0.03843 0.58550
LOF 2 0.05376 0.03454 0.02938 0.00966 0.06849 0.04957 0.59504
LOF 4 0.02151 0.00163 0.03087 0.01118 0.06866 0.04974 0.63259
LOF 6 0.02151 0.00163 0.03074 0.01105 0.06707 0.04812 0.64945
LOF 7 0.02151 0.00163 0.02971 0.01000 0.07055 0.05167 0.63590
SimplifiedLOF 3 0.04301 0.02357 0.03430 0.01468 0.08392 0.06531 0.65492
SimplifiedLOF 10 0.03226 0.01260 0.03791 0.01837 0.08629 0.06772 0.70267
SimplifiedLOF 12 0.03226 0.01260 0.03674 0.01717 0.08807 0.06954 0.70035
LoOP 2 0.04301 0.02357 0.03111 0.01143 0.06551 0.04652 0.64488
LoOP 8 0.02151 0.00163 0.04034 0.02085 0.09320 0.07478 0.71354
LoOP 10 0.03226 0.01260 0.04064 0.02116 0.09091 0.07244 0.71971
LDOF 7 0.02151 0.00163 0.04939 0.03008 0.12132 0.10347 0.75104
LDOF 9 0.06452 0.04551 0.05187 0.03261 0.11336 0.09535 0.76882
ODIN 5 0.03949 0.01998 0.04338 0.02394 0.09337 0.07496 0.73546
ODIN 7 0.05560 0.03642 0.04597 0.02659 0.10563 0.08747 0.72977
ODIN 16 0.06272 0.04368 0.04059 0.02110 0.08223 0.06358 0.71495
FastABOD 3 0.00000 -0.02031 0.02146 0.00158 0.05053 0.03124 0.56156
FastABOD 6 0.00000 -0.02031 0.02243 0.00258 0.05577 0.03659 0.58804
FastABOD 8 0.00000 -0.02031 0.02231 0.00245 0.05744 0.03829 0.58788
KDEOS 13 0.11828 0.10037 0.06532 0.04633 0.11892 0.10102 0.73749
KDEOS 21 0.09677 0.07843 0.06842 0.04949 0.14796 0.13065 0.76561
KDEOS 22 0.10753 0.08940 0.06910 0.05019 0.14173 0.12430 0.76573
KDEOS 29 0.08602 0.06745 0.06172 0.04266 0.13708 0.11955 0.76606
LDF 2 0.04301 0.02357 0.02787 0.00812 0.06134 0.04227 0.59638
LDF 4 0.01075 -0.00934 0.02824 0.00850 0.06109 0.04201 0.61559
LDF 7 0.02151 0.00163 0.02515 0.00534 0.06265 0.04360 0.59724
INFLO 2 0.05376 0.03454 0.03237 0.01272 0.07143 0.05257 0.62605
INFLO 4 0.03226 0.01260 0.03196 0.01230 0.07213 0.05328 0.63788
INFLO 7 0.02151 0.00163 0.03360 0.01397 0.09068 0.07221 0.61923
COF 2 0.04301 0.02357 0.03442 0.01480 0.06545 0.04646 0.63265
COF 3 0.02151 0.00163 0.03232 0.01267 0.07579 0.05702 0.63433
COF 4 0.05376 0.03454 0.03408 0.01446 0.07560 0.05682 0.64044
COF 7 0.03226 0.01260 0.03245 0.01279 0.07347 0.05465 0.64525

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (40.1 MB) Download raw algorithm evaluation table (72.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.04013 0.02057 0.09513 0.07668 0.75820
KNN 3 0.00000 -0.02039 0.04188 0.02234 0.10415 0.08588 0.77517
KNNW 1 0.00000 -0.02039 0.03167 0.01193 0.07510 0.05625 0.67652
KNNW 6 0.00000 -0.02039 0.04046 0.02090 0.09740 0.07900 0.76793
KNNW 7 0.00000 -0.02039 0.04042 0.02086 0.09818 0.07979 0.76832
KNNW 13 0.00000 -0.02039 0.03967 0.02009 0.10050 0.08217 0.76460
LOF 10 0.08602 0.06739 0.07939 0.06063 0.16060 0.14349 0.86193
LOF 14 0.10753 0.08933 0.08436 0.06569 0.16960 0.15267 0.86034
LOF 20 0.09677 0.07836 0.08692 0.06831 0.18250 0.16583 0.85906
SimplifiedLOF 22 0.09677 0.07836 0.08209 0.06338 0.15761 0.14044 0.86638
SimplifiedLOF 27 0.08602 0.06739 0.08481 0.06615 0.16796 0.15100 0.86837
SimplifiedLOF 28 0.09677 0.07836 0.08507 0.06641 0.16386 0.14681 0.86721
SimplifiedLOF 30 0.09677 0.07836 0.08435 0.06568 0.17312 0.15626 0.86595
LoOP 23 0.09677 0.07836 0.08011 0.06135 0.15512 0.13790 0.86352
LoOP 27 0.08602 0.06739 0.08262 0.06392 0.16490 0.14788 0.86609
LoOP 28 0.08602 0.06739 0.08279 0.06409 0.16256 0.14549 0.86342
LoOP 35 0.06452 0.04545 0.08271 0.06401 0.17091 0.15401 0.86346
LDOF 25 0.08602 0.06739 0.06358 0.04449 0.13029 0.11256 0.83098
LDOF 46 0.03226 0.01253 0.07544 0.05659 0.16667 0.14968 0.86298
LDOF 52 0.04301 0.02350 0.07563 0.05679 0.16925 0.15232 0.85939
LDOF 59 0.03226 0.01253 0.07686 0.05804 0.16180 0.14472 0.85922
ODIN 20 0.10145 0.08313 0.09102 0.07249 0.16584 0.14883 0.86779
ODIN 26 0.09580 0.07736 0.09793 0.07954 0.18333 0.16668 0.86529
ODIN 28 0.09339 0.07491 0.09588 0.07745 0.19886 0.18253 0.86153
ODIN 45 0.13364 0.11598 0.09384 0.07537 0.18647 0.16989 0.85232
FastABOD 3 0.00000 -0.02039 0.02475 0.00487 0.05498 0.03571 0.60187
FastABOD 25 0.00000 -0.02039 0.02772 0.00790 0.06619 0.04716 0.64353
FastABOD 42 0.00000 -0.02039 0.02768 0.00786 0.06720 0.04818 0.64377
FastABOD 56 0.00000 -0.02039 0.02745 0.00762 0.06744 0.04843 0.64097
KDEOS 96 0.16129 0.14419 0.11468 0.09663 0.20000 0.18369 0.84329
KDEOS 99 0.18280 0.16614 0.11512 0.09709 0.19048 0.17397 0.84307
KDEOS 100 0.19355 0.17711 0.11501 0.09697 0.19780 0.18145 0.84364
LDF 6 0.09677 0.07836 0.07251 0.05360 0.14184 0.12435 0.84633
LDF 9 0.06452 0.04545 0.08447 0.06580 0.17452 0.15769 0.86710
LDF 12 0.09677 0.07836 0.09092 0.07238 0.19238 0.17592 0.85605
LDF 13 0.09677 0.07836 0.09039 0.07185 0.19424 0.17782 0.85457
INFLO 18 0.09677 0.07836 0.07129 0.05236 0.15431 0.13707 0.82197
INFLO 23 0.09677 0.07836 0.07550 0.05665 0.15859 0.14144 0.83930
INFLO 24 0.09677 0.07836 0.07473 0.05587 0.16514 0.14812 0.83737
INFLO 27 0.07527 0.05642 0.07527 0.05642 0.15694 0.13976 0.84143
COF 23 0.08602 0.06739 0.09019 0.07164 0.19465 0.17823 0.85984
COF 26 0.09677 0.07836 0.09269 0.07419 0.21446 0.19845 0.85028
COF 29 0.10753 0.08933 0.09311 0.07463 0.20000 0.18369 0.84518
COF 34 0.13978 0.12225 0.09046 0.07192 0.19286 0.17640 0.83913

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (40.2 MB) Download raw algorithm evaluation table (71.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.04040 0.02090 0.09368 0.07527 0.76596
KNN 4 0.00000 -0.02031 0.04255 0.02310 0.10766 0.08954 0.78036
KNNW 1 0.00000 -0.02031 0.03903 0.01950 0.09686 0.07851 0.72322
KNNW 7 0.00000 -0.02031 0.04197 0.02251 0.10260 0.08437 0.77837
LOF 2 0.09677 0.07843 0.04634 0.02696 0.12903 0.11134 0.63865
LOF 11 0.05376 0.03454 0.08126 0.06259 0.17957 0.16290 0.87226
LOF 14 0.06452 0.04551 0.08400 0.06539 0.17729 0.16058 0.87411
SimplifiedLOF 2 0.08602 0.06745 0.03881 0.01928 0.10050 0.08223 0.59028
SimplifiedLOF 22 0.04301 0.02357 0.07986 0.06117 0.18336 0.16677 0.86802
SimplifiedLOF 23 0.04301 0.02357 0.08069 0.06201 0.17705 0.16033 0.86943
LoOP 2 0.08602 0.06745 0.03814 0.01860 0.11111 0.09305 0.59328
LoOP 25 0.05376 0.03454 0.07934 0.06064 0.17864 0.16196 0.86495
LoOP 28 0.05376 0.03454 0.08002 0.06133 0.17100 0.15416 0.86707
LoOP 29 0.05376 0.03454 0.07970 0.06101 0.17363 0.15685 0.86738
LDOF 7 0.06452 0.04551 0.03880 0.01927 0.08840 0.06988 0.67216
LDOF 58 0.03226 0.01260 0.07540 0.05662 0.15675 0.13962 0.86346
ODIN 17 0.08481 0.06622 0.08973 0.07124 0.17544 0.15869 0.87456
ODIN 27 0.07920 0.06050 0.09712 0.07877 0.20690 0.19079 0.86831
ODIN 74 0.12473 0.10695 0.07308 0.05426 0.14187 0.12444 0.80674
FastABOD 4 0.01075 -0.00934 0.03015 0.01045 0.06818 0.04925 0.65860
FastABOD 6 0.00000 -0.02031 0.03071 0.01102 0.07569 0.05692 0.66924
FastABOD 7 0.00000 -0.02031 0.03096 0.01128 0.07370 0.05488 0.67053
FastABOD 10 0.00000 -0.02031 0.03091 0.01122 0.07155 0.05269 0.67130
KDEOS 76 0.20430 0.18814 0.10245 0.08422 0.20765 0.19155 0.85158
KDEOS 87 0.18280 0.16619 0.10588 0.08771 0.19355 0.17717 0.85349
KDEOS 100 0.19355 0.17717 0.12084 0.10298 0.19913 0.18286 0.85209
LDF 2 0.08602 0.06745 0.04573 0.02634 0.12325 0.10544 0.65932
LDF 7 0.04301 0.02357 0.08342 0.06480 0.18375 0.16716 0.87668
LDF 8 0.05376 0.03454 0.08229 0.06365 0.18623 0.16970 0.87373
LDF 10 0.05376 0.03454 0.08451 0.06591 0.18003 0.16338 0.87113
INFLO 2 0.09677 0.07843 0.04175 0.02228 0.11705 0.09911 0.60420
INFLO 19 0.06452 0.04551 0.07025 0.05136 0.16352 0.14653 0.82165
INFLO 26 0.05376 0.03454 0.07093 0.05206 0.15957 0.14250 0.83529
INFLO 27 0.05376 0.03454 0.07057 0.05168 0.15798 0.14088 0.83609
COF 24 0.13978 0.12231 0.10459 0.08640 0.20178 0.18556 0.85024
COF 26 0.15054 0.13328 0.10280 0.08457 0.19763 0.18133 0.85477
COF 34 0.17204 0.15522 0.09770 0.07937 0.20067 0.18443 0.84004
COF 35 0.15054 0.13328 0.09830 0.07998 0.21459 0.19864 0.84112

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO