Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Wilt (2% of outliers version#10)

The data set differentiates diseased trees from other land covers. The former are considered outliers here.

Download all data set variants used (7.2 MB). You can also access the original data. (merge train and test [wilt.zip])

Normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (39.6 MB) Download raw algorithm evaluation table (73.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.02328 0.00337 0.06038 0.04123 0.60448
KNNW 1 0.00000 -0.02039 0.02547 0.00560 0.06230 0.04319 0.63361
KNNW 2 0.00000 -0.02039 0.02451 0.00462 0.06250 0.04339 0.62310
LOF 1 0.06452 0.04545 0.03186 0.01213 0.08734 0.06873 0.56418
LOF 3 0.04301 0.02350 0.04114 0.02159 0.09549 0.07705 0.70131
LOF 4 0.05376 0.03447 0.03938 0.01979 0.10746 0.08927 0.65641
SimplifiedLOF 3 0.06452 0.04545 0.04457 0.02509 0.10164 0.08333 0.72672
SimplifiedLOF 7 0.03226 0.01253 0.04792 0.02851 0.11606 0.09804 0.74301
SimplifiedLOF 8 0.03226 0.01253 0.04736 0.02794 0.12129 0.10338 0.74418
LoOP 1 0.05376 0.03447 0.04089 0.02134 0.07681 0.05799 0.64417
LoOP 7 0.04301 0.02350 0.04900 0.02962 0.12000 0.10206 0.74113
LoOP 8 0.02151 0.00156 0.04831 0.02890 0.12800 0.11022 0.73830
LDOF 8 0.03226 0.01253 0.05480 0.03554 0.12993 0.11219 0.76706
LDOF 13 0.05376 0.03447 0.05311 0.03381 0.11326 0.09518 0.77716
LDOF 14 0.05376 0.03447 0.05278 0.03347 0.11364 0.09557 0.77727
ODIN 3 0.07983 0.06107 0.04774 0.02833 0.11480 0.09676 0.72967
ODIN 4 0.06250 0.04339 0.04608 0.02664 0.10158 0.08326 0.73862
ODIN 9 0.06164 0.04251 0.04952 0.03015 0.12121 0.10330 0.71941
FastABOD 3 0.00000 -0.02039 0.02314 0.00323 0.05578 0.03653 0.59395
FastABOD 6 0.00000 -0.02039 0.02455 0.00466 0.06196 0.04284 0.61948
FastABOD 8 0.00000 -0.02039 0.02432 0.00443 0.06304 0.04394 0.61789
KDEOS 23 0.10753 0.08933 0.08212 0.06341 0.16722 0.15025 0.79097
KDEOS 25 0.10753 0.08933 0.08018 0.06142 0.16626 0.14926 0.79382
KDEOS 28 0.13978 0.12225 0.08332 0.06463 0.15752 0.14034 0.79171
KDEOS 31 0.15054 0.13322 0.07791 0.05912 0.15718 0.14000 0.79318
LDF 1 0.04301 0.02350 0.02763 0.00781 0.07663 0.05780 0.54510
LDF 3 0.03226 0.01253 0.03609 0.01644 0.07745 0.05865 0.68613
LDF 4 0.03226 0.01253 0.03420 0.01451 0.08752 0.06891 0.64371
INFLO 4 0.06452 0.04545 0.03869 0.01909 0.09045 0.07191 0.67109
INFLO 7 0.02151 0.00156 0.03527 0.01560 0.09924 0.08087 0.63407
COF 1 0.05376 0.03447 0.04141 0.02187 0.07681 0.05799 0.64831
COF 3 0.04301 0.02350 0.04429 0.02481 0.10490 0.08665 0.72491
COF 4 0.03226 0.01253 0.04300 0.02349 0.09335 0.07487 0.72607

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (39.7 MB) Download raw algorithm evaluation table (71.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.02048 0.00058 0.05183 0.03257 0.55445
KNNW 1 0.00000 -0.02031 0.02359 0.00375 0.05988 0.04078 0.60903
LOF 1 0.04301 0.02357 0.02466 0.00484 0.06154 0.04247 0.54590
LOF 2 0.02151 0.00163 0.02931 0.00959 0.07425 0.05545 0.61542
LOF 3 0.02151 0.00163 0.02938 0.00966 0.06935 0.05044 0.61752
LOF 9 0.02151 0.00163 0.02716 0.00740 0.05760 0.03846 0.62575
SimplifiedLOF 3 0.03226 0.01260 0.03303 0.01339 0.07174 0.05288 0.68917
SimplifiedLOF 6 0.02151 0.00163 0.03708 0.01752 0.09149 0.07303 0.70404
SimplifiedLOF 7 0.03226 0.01260 0.03735 0.01779 0.09412 0.07572 0.69999
LoOP 6 0.02151 0.00163 0.03809 0.01855 0.09392 0.07552 0.70906
LoOP 7 0.03226 0.01260 0.03853 0.01900 0.10089 0.08263 0.69725
LDOF 6 0.01075 -0.00934 0.04444 0.02502 0.10393 0.08572 0.74752
LDOF 9 0.03226 0.01260 0.04366 0.02423 0.09286 0.07443 0.75820
LDOF 11 0.05376 0.03454 0.04476 0.02535 0.09756 0.07923 0.75466
ODIN 2 0.05421 0.03499 0.04028 0.02078 0.09236 0.07392 0.71587
ODIN 5 0.06093 0.04186 0.04167 0.02220 0.10074 0.08247 0.69168
ODIN 7 0.05457 0.03537 0.04248 0.02303 0.09884 0.08053 0.68455
FastABOD 3 0.00000 -0.02031 0.02041 0.00051 0.05004 0.03074 0.55131
FastABOD 5 0.00000 -0.02031 0.02225 0.00238 0.05528 0.03608 0.58581
FastABOD 6 0.00000 -0.02031 0.02220 0.00233 0.05451 0.03531 0.58643
FastABOD 7 0.00000 -0.02031 0.02195 0.00208 0.05639 0.03722 0.58133
KDEOS 15 0.08602 0.06745 0.07258 0.05374 0.14286 0.12544 0.75009
KDEOS 21 0.09677 0.07843 0.06833 0.04940 0.16064 0.14359 0.75574
KDEOS 22 0.11828 0.10037 0.06673 0.04777 0.15000 0.13273 0.75518
LDF 2 0.02151 0.00163 0.02846 0.00872 0.06522 0.04623 0.61403
LDF 6 0.03226 0.01260 0.02483 0.00502 0.05551 0.03632 0.57597
INFLO 2 0.01075 -0.00934 0.02720 0.00744 0.06399 0.04498 0.63134
INFLO 4 0.03226 0.01260 0.03035 0.01065 0.07221 0.05336 0.62551
INFLO 6 0.01075 -0.00934 0.03236 0.01270 0.08647 0.06791 0.61048
INFLO 7 0.01075 -0.00934 0.03143 0.01176 0.08791 0.06938 0.59104
COF 3 0.01075 -0.00934 0.03453 0.01491 0.07508 0.05629 0.70073
COF 5 0.00000 -0.02031 0.03491 0.01530 0.07709 0.05834 0.68470
COF 7 0.03226 0.01260 0.03418 0.01455 0.07427 0.05546 0.66603
COF 8 0.01075 -0.00934 0.03359 0.01396 0.08088 0.06221 0.66836

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (40.1 MB) Download raw algorithm evaluation table (72.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.04518 0.02571 0.10475 0.08650 0.79695
KNN 2 0.00000 -0.02039 0.04524 0.02577 0.11095 0.09282 0.79697
KNN 3 0.00000 -0.02039 0.04488 0.02541 0.11174 0.09363 0.79267
KNNW 1 0.00000 -0.02039 0.04300 0.02349 0.10772 0.08953 0.77127
KNNW 4 0.00000 -0.02039 0.04562 0.02616 0.10981 0.09166 0.79820
KNNW 7 0.00000 -0.02039 0.04460 0.02513 0.11133 0.09322 0.79212
LOF 3 0.08602 0.06739 0.05850 0.03931 0.13821 0.12064 0.72690
LOF 10 0.08602 0.06739 0.08969 0.07113 0.17457 0.15775 0.87422
LOF 19 0.05376 0.03447 0.09617 0.07775 0.20247 0.18621 0.86730
SimplifiedLOF 5 0.09677 0.07836 0.06374 0.04465 0.14969 0.13235 0.74928
SimplifiedLOF 19 0.05376 0.03447 0.09146 0.07294 0.19598 0.17959 0.87637
SimplifiedLOF 27 0.03226 0.01253 0.09162 0.07310 0.18305 0.16640 0.87871
SimplifiedLOF 29 0.03226 0.01253 0.09228 0.07378 0.19196 0.17549 0.87766
LoOP 13 0.09677 0.07836 0.07844 0.05965 0.16779 0.15082 0.83798
LoOP 34 0.05376 0.03447 0.09055 0.07202 0.19077 0.17427 0.87229
LoOP 43 0.05376 0.03447 0.08805 0.06946 0.19820 0.18185 0.86609
LDOF 13 0.09677 0.07836 0.05467 0.03540 0.12362 0.10575 0.75826
LDOF 44 0.06452 0.04545 0.08245 0.06375 0.17722 0.16044 0.87010
LDOF 53 0.05376 0.03447 0.08204 0.06333 0.18512 0.16851 0.86784
ODIN 27 0.11214 0.09404 0.10560 0.08737 0.20635 0.19017 0.87424
ODIN 39 0.12610 0.10828 0.11000 0.09185 0.21023 0.19413 0.87142
ODIN 41 0.12784 0.11006 0.10987 0.09172 0.21728 0.20133 0.86803
ODIN 81 0.15054 0.13322 0.08546 0.06682 0.16425 0.14721 0.82139
FastABOD 4 0.01075 -0.00941 0.02987 0.01009 0.07047 0.05152 0.67112
FastABOD 13 0.00000 -0.02039 0.03110 0.01135 0.07692 0.05811 0.68220
FastABOD 15 0.00000 -0.02039 0.03111 0.01136 0.07550 0.05666 0.68300
KDEOS 62 0.21505 0.19905 0.11836 0.10038 0.21714 0.20118 0.85613
KDEOS 69 0.21505 0.19905 0.12614 0.10832 0.22460 0.20879 0.85486
KDEOS 90 0.19355 0.17711 0.13556 0.11794 0.20930 0.19318 0.85952
KDEOS 100 0.19355 0.17711 0.14975 0.13242 0.20755 0.19139 0.85762
LDF 2 0.09677 0.07836 0.04730 0.02788 0.11141 0.09329 0.66988
LDF 8 0.04301 0.02350 0.09278 0.07428 0.19277 0.17632 0.88242
LDF 14 0.06452 0.04545 0.09701 0.07861 0.21032 0.19422 0.86773
LDF 15 0.08602 0.06739 0.09733 0.07893 0.20594 0.18975 0.86290
INFLO 6 0.11828 0.10030 0.05970 0.04054 0.14173 0.12424 0.74409
INFLO 19 0.07527 0.05642 0.08089 0.06215 0.18582 0.16922 0.83645
INFLO 27 0.06452 0.04545 0.07980 0.06104 0.19883 0.18250 0.83984
INFLO 28 0.05376 0.03447 0.07896 0.06018 0.20000 0.18369 0.83451
COF 20 0.11828 0.10030 0.10873 0.09056 0.21569 0.19970 0.88746
COF 29 0.22581 0.21002 0.12760 0.10981 0.26733 0.25239 0.87727

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (40.2 MB) Download raw algorithm evaluation table (72.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.03940 0.01989 0.09504 0.07665 0.76595
KNNW 1 0.00000 -0.02031 0.03685 0.01729 0.08791 0.06938 0.73892
KNNW 2 0.00000 -0.02031 0.03890 0.01937 0.09836 0.08004 0.76282
KNNW 3 0.00000 -0.02031 0.03911 0.01959 0.09365 0.07524 0.76702
KNNW 4 0.00000 -0.02031 0.03894 0.01942 0.09614 0.07778 0.76736
LOF 8 0.09677 0.07843 0.06839 0.04947 0.13972 0.12224 0.82370
LOF 16 0.07527 0.05648 0.07820 0.05947 0.15611 0.13897 0.86621
LOF 19 0.09677 0.07843 0.07778 0.05904 0.16044 0.14339 0.86232
SimplifiedLOF 8 0.07527 0.05648 0.05546 0.03628 0.12476 0.10698 0.76389
SimplifiedLOF 27 0.07527 0.05648 0.07470 0.05591 0.15212 0.13490 0.86497
SimplifiedLOF 28 0.06452 0.04551 0.07489 0.05610 0.15314 0.13594 0.86495
SimplifiedLOF 31 0.05376 0.03454 0.07454 0.05574 0.15489 0.13772 0.86368
LoOP 13 0.08602 0.06745 0.06085 0.04178 0.12805 0.11034 0.80396
LoOP 27 0.08602 0.06745 0.07380 0.05499 0.15493 0.13776 0.85984
LoOP 39 0.05376 0.03454 0.07392 0.05511 0.15705 0.13993 0.85695
LoOP 41 0.05376 0.03454 0.07442 0.05562 0.15385 0.13666 0.85602
LDOF 21 0.06452 0.04551 0.05411 0.03490 0.11068 0.09262 0.79760
LDOF 42 0.05376 0.03454 0.07086 0.05199 0.16000 0.14294 0.85554
LDOF 52 0.03226 0.01260 0.07272 0.05388 0.15892 0.14183 0.86083
ODIN 27 0.08864 0.07013 0.09114 0.07267 0.17162 0.15479 0.87421
ODIN 37 0.08423 0.06563 0.09237 0.07393 0.17791 0.16121 0.86594
ODIN 42 0.10000 0.08172 0.08994 0.07146 0.18561 0.16907 0.85778
ODIN 50 0.12043 0.10256 0.08736 0.06882 0.18182 0.16520 0.84613
FastABOD 3 0.00000 -0.02031 0.02560 0.00581 0.05859 0.03947 0.61700
FastABOD 4 0.00000 -0.02031 0.02631 0.00653 0.05803 0.03890 0.62869
FastABOD 5 0.00000 -0.02031 0.02613 0.00635 0.05836 0.03923 0.63122
FastABOD 44 0.00000 -0.02031 0.02536 0.00556 0.06360 0.04457 0.62320
KDEOS 76 0.17204 0.15522 0.09836 0.08004 0.19266 0.17626 0.84998
KDEOS 83 0.16129 0.14425 0.09513 0.07675 0.19802 0.18173 0.85121
KDEOS 93 0.19355 0.17717 0.09568 0.07731 0.20988 0.19383 0.84945
LDF 11 0.07527 0.05648 0.07874 0.06003 0.15625 0.13911 0.86391
LDF 12 0.08602 0.06745 0.07878 0.06007 0.15632 0.13918 0.86083
LDF 26 0.04301 0.02357 0.07095 0.05207 0.16875 0.15186 0.82366
INFLO 12 0.09677 0.07843 0.05485 0.03565 0.12214 0.10430 0.75119
INFLO 27 0.06452 0.04551 0.06517 0.04618 0.13998 0.12251 0.82353
INFLO 51 0.02151 0.00163 0.05341 0.03418 0.14315 0.12574 0.73938
COF 19 0.16129 0.14425 0.08994 0.07145 0.17308 0.15628 0.87066
COF 23 0.15054 0.13328 0.09912 0.08082 0.18815 0.17166 0.87884
COF 25 0.15054 0.13328 0.10256 0.08433 0.19811 0.18182 0.87500
COF 27 0.12903 0.11134 0.09921 0.08091 0.20576 0.18963 0.86300

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO