Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Wilt (2% of outliers version#03)

The data set differentiates diseased trees from other land covers. The former are considered outliers here.

Download all data set variants used (7.2 MB). You can also access the original data. (merge train and test [wilt.zip])

Normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (39.6 MB) Download raw algorithm evaluation table (73.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.02219 0.00225 0.05888 0.03969 0.58317
KNNW 1 0.00000 -0.02039 0.02438 0.00449 0.06275 0.04365 0.60955
LOF 1 0.05376 0.03447 0.03425 0.01456 0.11765 0.09966 0.56833
LOF 2 0.06452 0.04545 0.03382 0.01412 0.07987 0.06111 0.59831
LOF 6 0.01075 -0.00941 0.03240 0.01268 0.08251 0.06380 0.67687
SimplifiedLOF 2 0.08602 0.06739 0.03980 0.02022 0.08939 0.07082 0.68652
SimplifiedLOF 5 0.01075 -0.00941 0.04380 0.02431 0.10476 0.08651 0.72972
SimplifiedLOF 6 0.01075 -0.00941 0.04304 0.02353 0.10573 0.08750 0.73485
LoOP 2 0.07527 0.05642 0.03919 0.01960 0.08951 0.07095 0.67949
LoOP 5 0.02151 0.00156 0.04369 0.02419 0.10726 0.08906 0.73134
LoOP 6 0.02151 0.00156 0.04361 0.02411 0.10617 0.08795 0.74058
LoOP 8 0.01075 -0.00941 0.04323 0.02372 0.10853 0.09035 0.73912
LDOF 2 0.03226 0.01253 0.03160 0.01186 0.06422 0.04514 0.65885
LDOF 8 0.02151 0.00156 0.04418 0.02470 0.10409 0.08583 0.74700
LDOF 15 0.01075 -0.00941 0.04232 0.02279 0.09845 0.08007 0.76350
ODIN 3 0.06329 0.04420 0.04255 0.02304 0.09247 0.07397 0.72407
ODIN 5 0.05223 0.03291 0.05012 0.03076 0.12331 0.10544 0.74113
ODIN 7 0.04358 0.02409 0.04713 0.02771 0.10526 0.08702 0.74937
FastABOD 3 0.00000 -0.02039 0.02302 0.00310 0.05385 0.03456 0.58940
FastABOD 5 0.00000 -0.02039 0.02360 0.00370 0.06258 0.04347 0.60801
KDEOS 16 0.10753 0.08933 0.05728 0.03806 0.11409 0.09603 0.75191
KDEOS 22 0.09677 0.07836 0.06603 0.04699 0.14384 0.12638 0.77712
KDEOS 25 0.04301 0.02350 0.06336 0.04426 0.15254 0.13527 0.77604
KDEOS 32 0.05376 0.03447 0.06365 0.04457 0.14567 0.12825 0.77968
LDF 1 0.04301 0.02350 0.03146 0.01171 0.09934 0.08098 0.56580
LDF 2 0.06452 0.04545 0.03118 0.01143 0.08756 0.06896 0.58671
LDF 6 0.00000 -0.02039 0.02818 0.00836 0.06592 0.04688 0.64595
INFLO 1 0.07527 0.05642 0.03268 0.01296 0.08434 0.06567 0.59855
INFLO 2 0.05376 0.03447 0.03644 0.01679 0.07654 0.05772 0.64865
INFLO 3 0.04301 0.02350 0.03274 0.01303 0.08562 0.06698 0.63086
INFLO 4 0.04301 0.02350 0.03269 0.01298 0.07567 0.05682 0.64888
COF 2 0.08602 0.06739 0.04184 0.02230 0.08949 0.07092 0.69092
COF 5 0.02151 0.00156 0.04148 0.02193 0.10373 0.08546 0.70664

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (39.7 MB) Download raw algorithm evaluation table (72.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.02309 0.00324 0.05760 0.03845 0.59822
KNNW 1 0.00000 -0.02031 0.02737 0.00761 0.06671 0.04775 0.65569
LOF 2 0.05376 0.03454 0.04002 0.02052 0.09955 0.08126 0.66294
LOF 3 0.03226 0.01260 0.03805 0.01850 0.09171 0.07326 0.67597
SimplifiedLOF 3 0.04301 0.02357 0.04374 0.02431 0.10541 0.08724 0.73162
SimplifiedLOF 5 0.03226 0.01260 0.04990 0.03060 0.12191 0.10407 0.73552
SimplifiedLOF 7 0.04301 0.02357 0.04875 0.02942 0.11940 0.10151 0.74447
LoOP 2 0.06452 0.04551 0.04199 0.02253 0.08333 0.06471 0.72025
LoOP 5 0.05376 0.03454 0.05131 0.03204 0.12773 0.11001 0.73778
LoOP 6 0.05376 0.03454 0.04958 0.03027 0.12991 0.11224 0.73781
LoOP 7 0.04301 0.02357 0.05036 0.03107 0.12361 0.10581 0.74618
LDOF 6 0.06452 0.04551 0.05071 0.03142 0.11098 0.09292 0.76969
LDOF 9 0.05376 0.03454 0.05928 0.04017 0.12773 0.11001 0.79989
ODIN 3 0.06867 0.04975 0.04858 0.02925 0.10817 0.09005 0.74287
ODIN 7 0.07527 0.05648 0.05118 0.03190 0.11806 0.10014 0.73119
ODIN 8 0.08095 0.06228 0.05110 0.03183 0.12411 0.10631 0.72153
FastABOD 3 0.00000 -0.02031 0.02692 0.00715 0.05920 0.04009 0.63097
FastABOD 4 0.00000 -0.02031 0.02707 0.00731 0.06551 0.04653 0.64661
KDEOS 23 0.08602 0.06745 0.08410 0.06550 0.19171 0.17529 0.79633
KDEOS 29 0.11828 0.10037 0.08659 0.06803 0.16514 0.14818 0.79447
KDEOS 52 0.15054 0.13328 0.07534 0.05655 0.15123 0.13399 0.78037
LDF 2 0.05376 0.03454 0.03605 0.01646 0.08403 0.06543 0.64494
LDF 3 0.02151 0.00163 0.03443 0.01481 0.08271 0.06407 0.66108
INFLO 2 0.03226 0.01260 0.03854 0.01901 0.08706 0.06852 0.69585
INFLO 4 0.07527 0.05648 0.04152 0.02205 0.09212 0.07368 0.69304
INFLO 5 0.03226 0.01260 0.03629 0.01671 0.10497 0.08679 0.63255
COF 2 0.06452 0.04551 0.04513 0.02574 0.08868 0.07017 0.72311
COF 3 0.02151 0.00163 0.04279 0.02334 0.10117 0.08291 0.72136

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (40.1 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.04154 0.02200 0.09827 0.07989 0.77672
KNN 3 0.00000 -0.02039 0.04174 0.02220 0.10411 0.08584 0.78359
KNNW 1 0.00000 -0.02039 0.03746 0.01784 0.08990 0.07135 0.74083
KNNW 4 0.00000 -0.02039 0.04190 0.02236 0.09964 0.08128 0.78308
KNNW 5 0.00000 -0.02039 0.04181 0.02228 0.09948 0.08112 0.78314
KNNW 9 0.00000 -0.02039 0.04145 0.02191 0.10369 0.08542 0.78128
LOF 6 0.09677 0.07836 0.06131 0.04218 0.15119 0.13388 0.77747
LOF 20 0.07527 0.05642 0.08992 0.07137 0.18333 0.16668 0.86689
LOF 23 0.05376 0.03447 0.09048 0.07194 0.18901 0.17248 0.86653
LOF 24 0.05376 0.03447 0.08936 0.07079 0.19473 0.17831 0.86455
SimplifiedLOF 14 0.09677 0.07836 0.06972 0.05075 0.15385 0.13660 0.81719
SimplifiedLOF 27 0.07527 0.05642 0.08423 0.06556 0.17871 0.16196 0.86565
SimplifiedLOF 28 0.06452 0.04545 0.08455 0.06589 0.17358 0.15674 0.86566
SimplifiedLOF 36 0.02151 0.00156 0.08467 0.06601 0.17565 0.15884 0.86336
LoOP 15 0.10753 0.08933 0.06878 0.04980 0.14815 0.13078 0.81565
LoOP 28 0.07527 0.05642 0.08244 0.06373 0.16724 0.15026 0.86189
LoOP 35 0.04301 0.02350 0.08258 0.06388 0.17284 0.15598 0.85951
LoOP 36 0.04301 0.02350 0.08331 0.06462 0.17284 0.15598 0.86030
LDOF 15 0.07527 0.05642 0.04492 0.02545 0.11053 0.09239 0.70893
LDOF 57 0.05376 0.03447 0.07484 0.05598 0.15852 0.14136 0.85245
LDOF 58 0.05376 0.03447 0.07465 0.05579 0.15972 0.14259 0.85218
ODIN 28 0.12821 0.11043 0.10377 0.08550 0.19020 0.17369 0.87555
ODIN 31 0.09053 0.07199 0.10410 0.08583 0.21370 0.19767 0.87256
ODIN 38 0.12108 0.10317 0.10481 0.08656 0.21081 0.19472 0.86987
ODIN 56 0.14640 0.12900 0.09574 0.07731 0.19409 0.17766 0.85336
FastABOD 3 0.01075 -0.00941 0.02690 0.00706 0.05791 0.03870 0.63112
FastABOD 17 0.00000 -0.02039 0.02816 0.00835 0.06862 0.04964 0.66248
FastABOD 48 0.00000 -0.02039 0.02785 0.00803 0.07049 0.05154 0.65933
KDEOS 87 0.17204 0.15516 0.10540 0.08716 0.17486 0.15804 0.84204
KDEOS 100 0.15054 0.13322 0.12526 0.10743 0.18433 0.16770 0.84771
LDF 11 0.08602 0.06739 0.09613 0.07771 0.20347 0.18724 0.87027
LDF 14 0.08602 0.06739 0.09864 0.08027 0.21027 0.19417 0.87026
LDF 15 0.09677 0.07836 0.09756 0.07916 0.20723 0.19107 0.86825
LDF 16 0.08602 0.06739 0.09592 0.07749 0.21364 0.19761 0.86645
INFLO 16 0.10753 0.08933 0.06478 0.04572 0.14149 0.12399 0.78538
INFLO 28 0.06452 0.04545 0.07244 0.05354 0.15789 0.14073 0.82662
INFLO 29 0.05376 0.03447 0.07247 0.05356 0.15267 0.13540 0.83133
COF 25 0.10753 0.08933 0.11308 0.09500 0.21677 0.20080 0.88411
COF 30 0.18280 0.16614 0.11790 0.09992 0.23368 0.21805 0.87413
COF 36 0.18280 0.16614 0.11491 0.09687 0.23776 0.22222 0.87033
COF 38 0.20430 0.18808 0.11562 0.09759 0.23599 0.22041 0.86603

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (40.2 MB) Download raw algorithm evaluation table (71.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.04381 0.02439 0.10019 0.08191 0.78818
KNN 2 0.00000 -0.02031 0.04442 0.02501 0.10360 0.08539 0.79500
KNN 4 0.00000 -0.02031 0.04380 0.02437 0.10568 0.08752 0.79487
KNNW 1 0.00000 -0.02031 0.04576 0.02637 0.10838 0.09027 0.79077
KNNW 5 0.00000 -0.02031 0.04499 0.02559 0.10457 0.08638 0.79860
LOF 6 0.09677 0.07843 0.08255 0.06392 0.17172 0.15489 0.85528
LOF 11 0.08602 0.06745 0.09644 0.07808 0.20628 0.19015 0.88462
LOF 12 0.07527 0.05648 0.09781 0.07948 0.19841 0.18213 0.88710
LOF 14 0.07527 0.05648 0.09746 0.07913 0.19772 0.18142 0.88866
SimplifiedLOF 8 0.10753 0.08940 0.07560 0.05682 0.16822 0.15133 0.80988
SimplifiedLOF 21 0.08602 0.06745 0.09344 0.07502 0.20642 0.19030 0.88058
SimplifiedLOF 27 0.06452 0.04551 0.09524 0.07686 0.18771 0.17121 0.88717
LoOP 9 0.08602 0.06745 0.07614 0.05737 0.16667 0.14974 0.81619
LoOP 18 0.08602 0.06745 0.08960 0.07111 0.18935 0.17288 0.86804
LoOP 27 0.06452 0.04551 0.09294 0.07451 0.18526 0.16870 0.88285
LDOF 19 0.09677 0.07843 0.06910 0.05019 0.13659 0.11905 0.82183
LDOF 50 0.06452 0.04551 0.08861 0.07010 0.18209 0.16547 0.88089
LDOF 51 0.06452 0.04551 0.08895 0.07045 0.17778 0.16107 0.88041
LDOF 53 0.06452 0.04551 0.08825 0.06972 0.18278 0.16618 0.87975
ODIN 28 0.13763 0.12012 0.11082 0.09276 0.20282 0.18662 0.88494
ODIN 31 0.13671 0.11918 0.11118 0.09313 0.21053 0.19449 0.88255
ODIN 33 0.14908 0.13180 0.11285 0.09483 0.20619 0.19006 0.88258
ODIN 39 0.15570 0.13855 0.10665 0.08850 0.18889 0.17241 0.87508
FastABOD 4 0.01075 -0.00934 0.02941 0.00969 0.06320 0.04417 0.65718
FastABOD 21 0.00000 -0.02031 0.03055 0.01086 0.07310 0.05427 0.67125
FastABOD 31 0.00000 -0.02031 0.03023 0.01053 0.07419 0.05538 0.66791
KDEOS 60 0.20430 0.18814 0.10314 0.08492 0.20541 0.18926 0.85584
KDEOS 70 0.18280 0.16619 0.10249 0.08426 0.21429 0.19832 0.85675
KDEOS 98 0.19355 0.17717 0.11812 0.10020 0.20213 0.18592 0.86083
LDF 4 0.11828 0.10037 0.07805 0.05932 0.16077 0.14372 0.81949
LDF 9 0.08602 0.06745 0.10117 0.08291 0.20316 0.18697 0.89208
INFLO 13 0.09677 0.07843 0.07445 0.05565 0.15649 0.13935 0.82058
INFLO 20 0.07527 0.05648 0.08095 0.06228 0.17418 0.15741 0.83714
INFLO 27 0.06452 0.04551 0.08336 0.06474 0.16736 0.15045 0.85100
INFLO 28 0.06452 0.04551 0.08269 0.06405 0.17213 0.15531 0.85341
COF 19 0.11828 0.10037 0.11681 0.09886 0.22338 0.20760 0.89817
COF 27 0.18280 0.16619 0.12987 0.11220 0.26471 0.24977 0.89370
COF 28 0.18280 0.16619 0.13004 0.11237 0.25547 0.24035 0.89163
COF 40 0.20430 0.18814 0.11175 0.09371 0.21622 0.20029 0.86748

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO