Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Wilt (2% of outliers version#05)

The data set differentiates diseased trees from other land covers. The former are considered outliers here.

Download all data set variants used (7.2 MB). You can also access the original data. (merge train and test [wilt.zip])

Normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (39.6 MB) Download raw algorithm evaluation table (73.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.02300 0.00308 0.05673 0.03750 0.59346
KNNW 1 0.00000 -0.02039 0.02376 0.00385 0.05751 0.03829 0.59190
KNNW 2 0.00000 -0.02039 0.02338 0.00347 0.05904 0.03985 0.59506
LOF 2 0.04301 0.02350 0.03673 0.01709 0.08950 0.07094 0.64056
LOF 4 0.03226 0.01253 0.03725 0.01762 0.09886 0.08049 0.67089
LOF 6 0.03226 0.01253 0.03896 0.01937 0.09349 0.07501 0.69837
SimplifiedLOF 2 0.04301 0.02350 0.03518 0.01551 0.08541 0.06676 0.66653
SimplifiedLOF 7 0.03226 0.01253 0.04709 0.02766 0.11311 0.09503 0.74553
SimplifiedLOF 8 0.02151 0.00156 0.04752 0.02810 0.10969 0.09154 0.74824
SimplifiedLOF 9 0.02151 0.00156 0.04715 0.02773 0.10882 0.09065 0.74844
LoOP 2 0.05376 0.03447 0.03440 0.01472 0.07692 0.05811 0.66148
LoOP 7 0.02151 0.00156 0.04828 0.02888 0.12594 0.10812 0.75324
LoOP 8 0.02151 0.00156 0.04897 0.02958 0.11854 0.10057 0.75666
LDOF 6 0.06452 0.04545 0.04935 0.02997 0.11429 0.09623 0.74575
LDOF 9 0.06452 0.04545 0.05358 0.03429 0.11315 0.09507 0.78396
LDOF 11 0.05376 0.03447 0.05289 0.03358 0.11696 0.09896 0.78776
LDOF 13 0.05376 0.03447 0.05212 0.03279 0.11884 0.10087 0.78539
ODIN 5 0.03708 0.01745 0.04586 0.02641 0.10526 0.08702 0.74351
ODIN 7 0.06515 0.04609 0.05019 0.03083 0.12174 0.10384 0.74021
ODIN 8 0.05914 0.03996 0.05020 0.03084 0.11533 0.09729 0.74123
FastABOD 3 0.00000 -0.02039 0.02276 0.00284 0.05164 0.03231 0.57323
FastABOD 5 0.00000 -0.02039 0.02439 0.00451 0.06044 0.04129 0.60455
FastABOD 10 0.00000 -0.02039 0.02366 0.00376 0.06305 0.04395 0.59730
KDEOS 25 0.06452 0.04545 0.07236 0.05345 0.15736 0.14018 0.78227
KDEOS 27 0.08602 0.06739 0.06779 0.04878 0.16062 0.14351 0.78096
KDEOS 29 0.08602 0.06739 0.06787 0.04887 0.14815 0.13078 0.78421
KDEOS 31 0.09677 0.07836 0.06635 0.04732 0.15324 0.13598 0.78171
LDF 2 0.04301 0.02350 0.03433 0.01464 0.08202 0.06331 0.64438
LDF 4 0.03226 0.01253 0.03509 0.01542 0.08904 0.07047 0.67042
INFLO 2 0.05376 0.03447 0.03356 0.01386 0.07851 0.05973 0.63285
INFLO 6 0.01075 -0.00941 0.03775 0.01814 0.09612 0.07769 0.67195
COF 3 0.06452 0.04545 0.03865 0.01905 0.08242 0.06371 0.67687
COF 4 0.05376 0.03447 0.03867 0.01908 0.08065 0.06190 0.67756
COF 5 0.04301 0.02350 0.03801 0.01840 0.08086 0.06213 0.68598
COF 7 0.05376 0.03447 0.03827 0.01866 0.08955 0.07099 0.67288

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (39.7 MB) Download raw algorithm evaluation table (72.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.02109 0.00121 0.05157 0.03230 0.56378
KNNW 1 0.00000 -0.02031 0.02175 0.00188 0.05502 0.03583 0.56851
KNNW 2 0.00000 -0.02031 0.02156 0.00168 0.05478 0.03558 0.57121
LOF 2 0.05376 0.03454 0.03310 0.01346 0.08031 0.06162 0.61322
LOF 4 0.02151 0.00163 0.03144 0.01177 0.08406 0.06545 0.60967
LOF 8 0.01075 -0.00934 0.02957 0.00986 0.06652 0.04756 0.63602
SimplifiedLOF 1 0.03226 0.01260 0.02914 0.00942 0.07349 0.05467 0.61002
SimplifiedLOF 4 0.02151 0.00163 0.03875 0.01923 0.09602 0.07766 0.68740
SimplifiedLOF 8 0.02151 0.00163 0.04002 0.02052 0.09440 0.07600 0.70999
LoOP 1 0.03226 0.01260 0.03022 0.01052 0.07557 0.05679 0.60876
LoOP 5 0.01075 -0.00934 0.04091 0.02142 0.10270 0.08447 0.70682
LoOP 8 0.02151 0.00163 0.04280 0.02336 0.09756 0.07923 0.72497
LDOF 5 0.03226 0.01260 0.03947 0.01996 0.08539 0.06681 0.72167
LDOF 10 0.01075 -0.00934 0.04153 0.02206 0.08396 0.06535 0.74512
LDOF 11 0.02151 0.00163 0.04090 0.02141 0.08624 0.06767 0.73713
ODIN 6 0.07013 0.05124 0.05193 0.03267 0.12121 0.10336 0.71513
ODIN 7 0.09005 0.07157 0.04953 0.03022 0.10327 0.08505 0.71031
ODIN 11 0.04728 0.02793 0.04402 0.02460 0.09795 0.07962 0.72675
FastABOD 3 0.00000 -0.02031 0.02278 0.00293 0.05476 0.03556 0.58509
FastABOD 5 0.00000 -0.02031 0.02302 0.00317 0.05484 0.03564 0.59707
FastABOD 8 0.00000 -0.02031 0.02256 0.00270 0.05726 0.03811 0.59270
KDEOS 23 0.09677 0.07843 0.06773 0.04879 0.15917 0.14209 0.76453
KDEOS 33 0.10753 0.08940 0.06779 0.04885 0.15652 0.13939 0.76285
KDEOS 42 0.11828 0.10037 0.06490 0.04591 0.14847 0.13117 0.75150
KDEOS 46 0.11828 0.10037 0.06671 0.04776 0.16327 0.14627 0.75394
LDF 2 0.05376 0.03454 0.03069 0.01099 0.07442 0.05562 0.60456
LDF 3 0.02151 0.00163 0.02806 0.00832 0.05924 0.04013 0.61059
INFLO 1 0.03226 0.01260 0.02783 0.00808 0.06115 0.04208 0.58763
INFLO 4 0.03226 0.01260 0.03516 0.01556 0.09309 0.07467 0.65859
INFLO 6 0.02151 0.00163 0.03367 0.01404 0.09375 0.07534 0.63720
COF 3 0.04301 0.02357 0.03398 0.01436 0.07333 0.05451 0.66705
COF 5 0.02151 0.00163 0.03474 0.01513 0.07850 0.05978 0.66075
COF 8 0.02151 0.00163 0.03474 0.01513 0.08103 0.06236 0.66374

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (40.1 MB) Download raw algorithm evaluation table (72.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.04484 0.02536 0.10469 0.08644 0.78911
KNN 2 0.00000 -0.02039 0.04582 0.02637 0.10828 0.09010 0.79460
KNN 3 0.00000 -0.02039 0.04513 0.02567 0.11348 0.09540 0.79094
KNNW 1 0.00000 -0.02039 0.04144 0.02190 0.09931 0.08095 0.75394
KNNW 5 0.00000 -0.02039 0.04533 0.02587 0.10695 0.08875 0.79270
KNNW 11 0.00000 -0.02039 0.04317 0.02366 0.10864 0.09047 0.78188
LOF 3 0.10753 0.08933 0.06090 0.04176 0.12060 0.10268 0.76924
LOF 8 0.09677 0.07836 0.09133 0.07281 0.18021 0.16350 0.87184
LOF 10 0.08602 0.06739 0.09174 0.07323 0.20079 0.18449 0.87015
LOF 16 0.07527 0.05642 0.09403 0.07556 0.19811 0.18177 0.86096
SimplifiedLOF 2 0.10753 0.08933 0.04490 0.02543 0.11429 0.09623 0.62755
SimplifiedLOF 23 0.06452 0.04545 0.09171 0.07320 0.19130 0.17482 0.87565
SimplifiedLOF 31 0.02151 0.00156 0.08957 0.07101 0.19204 0.17557 0.87184
LoOP 2 0.10753 0.08933 0.04463 0.02515 0.11561 0.09758 0.63169
LoOP 20 0.06452 0.04545 0.08800 0.06940 0.18298 0.16632 0.86748
LoOP 27 0.05376 0.03447 0.08766 0.06906 0.18569 0.16909 0.86871
LoOP 31 0.04301 0.02350 0.08726 0.06865 0.19172 0.17524 0.86663
LDOF 7 0.09677 0.07836 0.04758 0.02816 0.12010 0.10217 0.70018
LDOF 42 0.04301 0.02350 0.08094 0.06220 0.18101 0.16431 0.87118
LDOF 45 0.04301 0.02350 0.08263 0.06393 0.17518 0.15837 0.87290
LDOF 48 0.04301 0.02350 0.08297 0.06428 0.17945 0.16272 0.87197
ODIN 20 0.12308 0.10521 0.10269 0.08440 0.19950 0.18318 0.86457
ODIN 24 0.13763 0.12005 0.10366 0.08538 0.18777 0.17122 0.85914
ODIN 34 0.10323 0.08494 0.10087 0.08254 0.21118 0.19510 0.84740
ODIN 56 0.15323 0.13596 0.09231 0.07381 0.18142 0.16473 0.82741
FastABOD 3 0.00000 -0.02039 0.02768 0.00786 0.06202 0.04289 0.64257
FastABOD 15 0.00000 -0.02039 0.03038 0.01061 0.07267 0.05377 0.66269
FastABOD 16 0.00000 -0.02039 0.03050 0.01074 0.07201 0.05309 0.66374
KDEOS 58 0.16129 0.14419 0.09883 0.08046 0.17857 0.16183 0.85201
KDEOS 79 0.19355 0.17711 0.10395 0.08568 0.19355 0.17711 0.84967
KDEOS 98 0.17204 0.15516 0.11721 0.09921 0.19910 0.18277 0.84625
KDEOS 99 0.17204 0.15516 0.11673 0.09872 0.20091 0.18462 0.84620
LDF 4 0.10753 0.08933 0.07447 0.05561 0.16500 0.14798 0.83642
LDF 8 0.10753 0.08933 0.09641 0.07799 0.19881 0.18247 0.87620
LDF 11 0.05376 0.03447 0.09814 0.07975 0.20853 0.19240 0.86529
LDF 15 0.05376 0.03447 0.09299 0.07450 0.20976 0.19365 0.84994
INFLO 2 0.12903 0.11128 0.05345 0.03416 0.13889 0.12133 0.64172
INFLO 26 0.05376 0.03447 0.07802 0.05923 0.17937 0.16264 0.83053
INFLO 27 0.05376 0.03447 0.07828 0.05949 0.18615 0.16956 0.82679
COF 21 0.11828 0.10030 0.10340 0.08512 0.20755 0.19139 0.87687
COF 24 0.11828 0.10030 0.10826 0.09008 0.20130 0.18502 0.87233
COF 29 0.18280 0.16614 0.10675 0.08854 0.19658 0.18020 0.86432
COF 40 0.10753 0.08933 0.09578 0.07735 0.21399 0.19797 0.84790

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (40.2 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.03959 0.02008 0.08874 0.07023 0.76585
KNN 2 0.00000 -0.02031 0.04176 0.02230 0.09689 0.07854 0.78161
KNN 5 0.00000 -0.02031 0.04063 0.02114 0.10023 0.08195 0.77852
KNNW 1 0.00000 -0.02031 0.03372 0.01409 0.08385 0.06524 0.69737
KNNW 7 0.00000 -0.02031 0.04026 0.02076 0.09494 0.07655 0.77553
KNNW 13 0.00000 -0.02031 0.03904 0.01952 0.09509 0.07671 0.76891
LOF 10 0.09677 0.07843 0.08978 0.07129 0.18227 0.16565 0.87924
LOF 13 0.04301 0.02357 0.09360 0.07519 0.18795 0.17146 0.89020
SimplifiedLOF 3 0.08602 0.06745 0.03836 0.01882 0.08861 0.07009 0.63901
SimplifiedLOF 19 0.06452 0.04551 0.08552 0.06694 0.17997 0.16331 0.87350
SimplifiedLOF 23 0.06452 0.04551 0.08714 0.06859 0.17875 0.16207 0.88237
SimplifiedLOF 32 0.03226 0.01260 0.08681 0.06826 0.17117 0.15433 0.88916
LoOP 9 0.07527 0.05648 0.06703 0.04808 0.15009 0.13282 0.80795
LoOP 26 0.07527 0.05648 0.08529 0.06671 0.18371 0.16713 0.88047
LoOP 32 0.04301 0.02357 0.08592 0.06735 0.17750 0.16079 0.88696
LDOF 7 0.07527 0.05648 0.03749 0.01793 0.08138 0.06272 0.66555
LDOF 53 0.03226 0.01260 0.07671 0.05795 0.16463 0.14766 0.87988
LDOF 61 0.03226 0.01260 0.07628 0.05752 0.16937 0.15250 0.87887
ODIN 27 0.11060 0.09253 0.10910 0.09100 0.21739 0.20149 0.89824
ODIN 28 0.11543 0.09746 0.10877 0.09067 0.21346 0.19748 0.89857
ODIN 77 0.14168 0.12425 0.08177 0.06311 0.17778 0.16107 0.83021
FastABOD 3 0.01075 -0.00934 0.02535 0.00555 0.05964 0.04054 0.59810
FastABOD 15 0.00000 -0.02031 0.02859 0.00886 0.06902 0.05010 0.65844
FastABOD 17 0.00000 -0.02031 0.02873 0.00900 0.06782 0.04888 0.65996
FastABOD 18 0.00000 -0.02031 0.02873 0.00900 0.06772 0.04878 0.65990
KDEOS 78 0.19355 0.17717 0.11608 0.09813 0.19459 0.17823 0.86851
KDEOS 93 0.18280 0.16619 0.12741 0.10968 0.21477 0.19881 0.87296
KDEOS 100 0.18280 0.16619 0.14532 0.12796 0.19895 0.18268 0.87440
LDF 7 0.08602 0.06745 0.08884 0.07033 0.19099 0.17456 0.87738
LDF 11 0.05376 0.03454 0.09496 0.07657 0.18788 0.17138 0.89288
LDF 12 0.04301 0.02357 0.09545 0.07707 0.19195 0.17554 0.89019
LDF 13 0.08602 0.06745 0.09537 0.07699 0.20482 0.18867 0.88712
INFLO 7 0.07527 0.05648 0.05496 0.03576 0.13060 0.11294 0.72747
INFLO 14 0.05376 0.03454 0.07543 0.05664 0.17910 0.16243 0.82723
INFLO 18 0.06452 0.04551 0.07812 0.05939 0.17822 0.16152 0.85065
INFLO 33 0.01075 -0.00934 0.07434 0.05553 0.15559 0.13843 0.85233
COF 24 0.15054 0.13328 0.10606 0.08790 0.20884 0.19276 0.86793
COF 31 0.13978 0.12231 0.10887 0.09077 0.22840 0.21272 0.87111
COF 47 0.10753 0.08940 0.10329 0.08507 0.25545 0.24033 0.85875

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO