Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Wilt (2% of outliers version#04)

The data set differentiates diseased trees from other land covers. The former are considered outliers here.

Download all data set variants used (7.2 MB). You can also access the original data. (merge train and test [wilt.zip])

Normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (39.6 MB) Download raw algorithm evaluation table (73.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.02252 0.00259 0.05617 0.03693 0.59181
KNNW 1 0.00000 -0.02039 0.02430 0.00441 0.05903 0.03984 0.60699
LOF 4 0.04301 0.02350 0.03713 0.01750 0.09747 0.07907 0.65102
LOF 6 0.01075 -0.00941 0.03417 0.01448 0.07508 0.05622 0.66266
SimplifiedLOF 1 0.04301 0.02350 0.03303 0.01332 0.06977 0.05080 0.60337
SimplifiedLOF 6 0.01075 -0.00941 0.04653 0.02710 0.11780 0.09982 0.74039
SimplifiedLOF 7 0.03226 0.01253 0.04611 0.02667 0.10995 0.09180 0.74051
LoOP 1 0.04301 0.02350 0.03225 0.01252 0.06977 0.05080 0.60929
LoOP 6 0.02151 0.00156 0.04779 0.02838 0.11622 0.09820 0.74976
LoOP 8 0.03226 0.01253 0.04737 0.02795 0.11198 0.09387 0.75035
LDOF 10 0.03226 0.01253 0.05386 0.03457 0.12322 0.10535 0.77902
LDOF 14 0.05376 0.03447 0.04903 0.02965 0.10643 0.08821 0.77133
ODIN 3 0.07563 0.05679 0.04649 0.02705 0.10876 0.09059 0.72881
ODIN 4 0.06897 0.04999 0.04721 0.02779 0.11257 0.09448 0.73080
ODIN 7 0.07527 0.05642 0.05011 0.03074 0.10851 0.09034 0.74993
FastABOD 3 0.00000 -0.02039 0.02106 0.00111 0.05081 0.03146 0.55631
FastABOD 5 0.00000 -0.02039 0.02395 0.00405 0.05640 0.03716 0.60647
FastABOD 8 0.00000 -0.02039 0.02319 0.00327 0.05835 0.03915 0.59860
KDEOS 21 0.11828 0.10030 0.07093 0.05199 0.12281 0.10492 0.78448
KDEOS 22 0.11828 0.10030 0.07396 0.05508 0.13158 0.11388 0.78210
KDEOS 29 0.07527 0.05642 0.07309 0.05419 0.15075 0.13344 0.77913
KDEOS 52 0.12903 0.11128 0.06508 0.04602 0.14468 0.12724 0.76067
LDF 2 0.05376 0.03447 0.03326 0.01355 0.08132 0.06259 0.63107
LDF 4 0.02151 0.00156 0.03357 0.01387 0.08282 0.06412 0.64249
INFLO 4 0.04301 0.02350 0.03505 0.01538 0.07901 0.06023 0.65689
INFLO 6 0.00000 -0.02039 0.03709 0.01746 0.09343 0.07495 0.65530
INFLO 8 0.01075 -0.00941 0.03541 0.01575 0.10035 0.08201 0.64127
COF 1 0.04301 0.02350 0.03313 0.01342 0.06977 0.05080 0.60605
COF 2 0.03226 0.01253 0.03787 0.01826 0.09935 0.08099 0.66216
COF 4 0.04301 0.02350 0.04196 0.02243 0.09629 0.07786 0.69753
COF 5 0.03226 0.01253 0.04081 0.02126 0.09819 0.07981 0.70955

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (39.7 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.02230 0.00243 0.05634 0.03717 0.58730
KNNW 1 0.00000 -0.02031 0.02463 0.00481 0.06034 0.04125 0.61758
LOF 2 0.05376 0.03454 0.03215 0.01249 0.07432 0.05551 0.60681
LOF 5 0.03226 0.01260 0.03640 0.01683 0.08468 0.06608 0.69419
LOF 6 0.01075 -0.00934 0.03642 0.01684 0.07981 0.06112 0.69689
SimplifiedLOF 3 0.05376 0.03454 0.04018 0.02068 0.08662 0.06807 0.70836
SimplifiedLOF 6 0.05376 0.03454 0.04776 0.02842 0.10746 0.08932 0.73816
SimplifiedLOF 7 0.04301 0.02357 0.04772 0.02837 0.10952 0.09143 0.74878
SimplifiedLOF 8 0.05376 0.03454 0.04701 0.02765 0.10594 0.08778 0.75352
LoOP 6 0.05376 0.03454 0.05047 0.03118 0.11590 0.09794 0.74444
LoOP 7 0.04301 0.02357 0.05086 0.03158 0.12202 0.10419 0.75807
LoOP 9 0.04301 0.02357 0.04927 0.02995 0.11271 0.09468 0.76547
LDOF 5 0.07527 0.05648 0.04824 0.02891 0.10000 0.08172 0.74238
LDOF 8 0.04301 0.02357 0.05658 0.03742 0.12097 0.10311 0.78126
LDOF 10 0.04301 0.02357 0.05627 0.03710 0.13089 0.11323 0.77758
ODIN 4 0.08772 0.06919 0.05443 0.03522 0.12360 0.10579 0.75058
ODIN 7 0.08439 0.06579 0.06284 0.04380 0.14508 0.12771 0.77627
FastABOD 3 0.00000 -0.02031 0.02297 0.00312 0.05109 0.03181 0.57709
FastABOD 6 0.00000 -0.02031 0.02355 0.00371 0.06026 0.04117 0.60311
FastABOD 7 0.00000 -0.02031 0.02353 0.00370 0.06073 0.04165 0.60327
KDEOS 23 0.10753 0.08940 0.08280 0.06416 0.18293 0.16633 0.80281
KDEOS 30 0.09677 0.07843 0.07809 0.05936 0.17521 0.15846 0.80508
KDEOS 57 0.12903 0.11134 0.06710 0.04815 0.14634 0.12900 0.77980
LDF 2 0.04301 0.02357 0.03175 0.01208 0.07898 0.06027 0.60937
LDF 4 0.00000 -0.02031 0.03260 0.01294 0.07196 0.05311 0.67631
INFLO 4 0.06452 0.04551 0.03848 0.01894 0.08851 0.07000 0.67691
INFLO 6 0.04301 0.02357 0.03949 0.01998 0.09371 0.07530 0.67872
INFLO 9 0.02151 0.00163 0.03806 0.01851 0.11002 0.09194 0.64473
COF 2 0.05376 0.03454 0.03506 0.01546 0.07317 0.05434 0.67249
COF 4 0.03226 0.01260 0.03910 0.01958 0.09283 0.07440 0.69491
COF 9 0.01075 -0.00934 0.03872 0.01920 0.09787 0.07955 0.70005

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (40.1 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.04245 0.02293 0.09922 0.08086 0.77463
KNN 4 0.00000 -0.02039 0.04277 0.02325 0.10037 0.08203 0.78312
KNNW 1 0.00000 -0.02039 0.03982 0.02024 0.10736 0.08916 0.73123
KNNW 7 0.00000 -0.02039 0.04261 0.02309 0.10241 0.08411 0.78194
KNNW 9 0.00000 -0.02039 0.04238 0.02285 0.10164 0.08333 0.78209
LOF 8 0.08602 0.06739 0.08681 0.06819 0.18810 0.17155 0.86837
LOF 15 0.07527 0.05642 0.09886 0.08049 0.20209 0.18582 0.88645
LOF 16 0.06452 0.04545 0.10010 0.08175 0.20361 0.18738 0.88565
LOF 22 0.06452 0.04545 0.09831 0.07993 0.21607 0.20009 0.87684
SimplifiedLOF 2 0.07527 0.05642 0.03376 0.01406 0.08040 0.06166 0.57595
SimplifiedLOF 27 0.05376 0.03447 0.09306 0.07457 0.19474 0.17832 0.88245
SimplifiedLOF 28 0.05376 0.03447 0.09265 0.07415 0.20053 0.18423 0.88167
LoOP 16 0.08602 0.06739 0.08178 0.06306 0.16104 0.14394 0.86554
LoOP 34 0.03226 0.01253 0.09155 0.07304 0.19271 0.17625 0.87797
LoOP 37 0.03226 0.01253 0.09099 0.07246 0.20000 0.18369 0.87633
LDOF 5 0.07527 0.05642 0.03500 0.01533 0.08377 0.06509 0.61404
LDOF 45 0.06452 0.04545 0.08165 0.06293 0.18033 0.16362 0.86795
LDOF 51 0.04301 0.02350 0.08346 0.06478 0.17538 0.15857 0.86982
ODIN 26 0.13891 0.12136 0.11337 0.09529 0.21021 0.19411 0.88927
ODIN 27 0.12560 0.10778 0.11349 0.09542 0.21636 0.20038 0.88769
ODIN 49 0.15591 0.13871 0.11011 0.09197 0.22300 0.20716 0.86951
ODIN 53 0.16176 0.14467 0.10561 0.08738 0.21754 0.20159 0.86330
FastABOD 3 0.01075 -0.00941 0.02935 0.00956 0.06714 0.04812 0.65995
FastABOD 15 0.00000 -0.02039 0.03115 0.01140 0.07395 0.05507 0.67715
FastABOD 20 0.00000 -0.02039 0.03124 0.01150 0.07297 0.05407 0.67806
KDEOS 99 0.24731 0.23197 0.14994 0.13261 0.25843 0.24331 0.86408
KDEOS 100 0.25806 0.24294 0.14659 0.12920 0.26087 0.24580 0.86407
LDF 7 0.11828 0.10030 0.09222 0.07372 0.18885 0.17232 0.87683
LDF 10 0.07527 0.05642 0.10267 0.08438 0.19692 0.18055 0.88403
LDF 15 0.06452 0.04545 0.10049 0.08215 0.21192 0.19585 0.87618
INFLO 2 0.08602 0.06739 0.03789 0.01827 0.09950 0.08115 0.58922
INFLO 16 0.07527 0.05642 0.07808 0.05929 0.16049 0.14338 0.84756
INFLO 27 0.07527 0.05642 0.08288 0.06419 0.18321 0.16656 0.84437
COF 22 0.17204 0.15516 0.11765 0.09966 0.25000 0.23471 0.87234
COF 24 0.17204 0.15516 0.12121 0.10330 0.24189 0.22643 0.87145
COF 27 0.18280 0.16614 0.11763 0.09964 0.22305 0.20721 0.87687
COF 28 0.20430 0.18808 0.11834 0.10037 0.22785 0.21211 0.87531

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (40.2 MB) Download raw algorithm evaluation table (71.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.04316 0.02372 0.09831 0.07999 0.78054
KNN 3 0.00000 -0.02031 0.04173 0.02226 0.10526 0.08709 0.77646
KNNW 1 0.00000 -0.02031 0.04178 0.02231 0.09924 0.08095 0.75941
KNNW 3 0.00000 -0.02031 0.04342 0.02399 0.10061 0.08234 0.78395
KNNW 8 0.00000 -0.02031 0.04199 0.02253 0.10433 0.08614 0.78072
LOF 8 0.09677 0.07843 0.08317 0.06454 0.17337 0.15658 0.85719
LOF 15 0.08602 0.06745 0.09258 0.07415 0.18433 0.16776 0.87330
LOF 19 0.07527 0.05648 0.09433 0.07593 0.18978 0.17332 0.87303
LOF 27 0.05376 0.03454 0.08858 0.07007 0.19763 0.18133 0.85946
SimplifiedLOF 5 0.09677 0.07843 0.06009 0.04100 0.13142 0.11377 0.75371
SimplifiedLOF 26 0.08602 0.06745 0.09254 0.07410 0.18256 0.16595 0.87931
LoOP 16 0.09677 0.07843 0.08020 0.06152 0.15571 0.13856 0.85720
LoOP 26 0.08602 0.06745 0.08979 0.07130 0.18113 0.16450 0.87435
LoOP 45 0.04301 0.02357 0.08788 0.06935 0.18911 0.17264 0.87012
LDOF 36 0.08602 0.06745 0.07592 0.05714 0.14605 0.12871 0.86221
LDOF 51 0.07527 0.05648 0.08509 0.06650 0.17224 0.15542 0.87321
ODIN 27 0.12155 0.10371 0.10534 0.08717 0.18978 0.17332 0.87648
ODIN 40 0.14370 0.12630 0.10424 0.08605 0.19858 0.18230 0.86581
ODIN 48 0.12810 0.11038 0.10034 0.08206 0.20648 0.19036 0.85338
FastABOD 3 0.02151 0.00163 0.02633 0.00655 0.05927 0.04016 0.62947
FastABOD 15 0.00000 -0.02031 0.02893 0.00920 0.06923 0.05032 0.66281
FastABOD 52 0.00000 -0.02031 0.02816 0.00842 0.07198 0.05313 0.65644
KDEOS 95 0.23656 0.22105 0.11847 0.10056 0.24044 0.22501 0.86092
KDEOS 99 0.20430 0.18814 0.11601 0.09806 0.22115 0.20533 0.86266
LDF 5 0.09677 0.07843 0.07800 0.05927 0.15702 0.13990 0.83687
LDF 10 0.07527 0.05648 0.09414 0.07574 0.19020 0.17375 0.87248
LDF 15 0.08602 0.06745 0.09650 0.07814 0.20408 0.18791 0.86704
LDF 22 0.07527 0.05648 0.09048 0.07201 0.21483 0.19888 0.85291
INFLO 7 0.09677 0.07843 0.05730 0.03815 0.15116 0.13392 0.72412
INFLO 21 0.08602 0.06745 0.07859 0.05987 0.16148 0.14444 0.83699
INFLO 26 0.08602 0.06745 0.07879 0.06008 0.15777 0.14066 0.83062
INFLO 35 0.05376 0.03454 0.07254 0.05370 0.16812 0.15122 0.80784
COF 25 0.16129 0.14425 0.11578 0.09781 0.22973 0.21408 0.87250
COF 26 0.15054 0.13328 0.11403 0.09603 0.23913 0.22367 0.86952
COF 28 0.19355 0.17717 0.11186 0.09381 0.20588 0.18975 0.87663

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO