Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Wilt (2% of outliers version#07)

The data set differentiates diseased trees from other land covers. The former are considered outliers here.

Download all data set variants used (7.2 MB). You can also access the original data. (merge train and test [wilt.zip])

Normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (39.6 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.02058 0.00061 0.05270 0.03339 0.55721
KNNW 1 0.00000 -0.02039 0.02269 0.00277 0.05835 0.03915 0.58569
LOF 1 0.05376 0.03447 0.03191 0.01217 0.09845 0.08007 0.57109
LOF 2 0.06452 0.04545 0.03776 0.01814 0.09068 0.07214 0.62559
LOF 3 0.03226 0.01253 0.03758 0.01796 0.08598 0.06735 0.66350
SimplifiedLOF 2 0.07527 0.05642 0.04240 0.02288 0.09848 0.08011 0.69621
SimplifiedLOF 4 0.04301 0.02350 0.04850 0.02910 0.11327 0.09520 0.73320
SimplifiedLOF 6 0.04301 0.02350 0.04694 0.02751 0.10422 0.08596 0.73589
LoOP 2 0.08602 0.06739 0.04420 0.02472 0.10631 0.08809 0.69484
LoOP 4 0.04301 0.02350 0.04945 0.03007 0.11644 0.09843 0.73352
LoOP 5 0.05376 0.03447 0.05000 0.03063 0.10680 0.08859 0.73144
LoOP 7 0.03226 0.01253 0.04705 0.02763 0.10296 0.08467 0.73778
LDOF 5 0.08602 0.06739 0.05783 0.03862 0.12950 0.11175 0.75845
LDOF 6 0.02151 0.00156 0.05695 0.03772 0.13043 0.11271 0.77339
LDOF 12 0.04301 0.02350 0.05340 0.03410 0.11750 0.09951 0.79320
ODIN 4 0.09402 0.07555 0.05299 0.03369 0.11985 0.10191 0.74641
ODIN 5 0.08289 0.06419 0.05360 0.03430 0.12048 0.10255 0.74779
ODIN 7 0.06371 0.04463 0.05128 0.03194 0.12069 0.10276 0.74011
FastABOD 3 0.01075 -0.00941 0.02409 0.00420 0.05653 0.03730 0.60666
FastABOD 4 0.00000 -0.02039 0.02375 0.00384 0.06061 0.04146 0.60297
KDEOS 21 0.09677 0.07836 0.07836 0.05957 0.16749 0.15052 0.78363
KDEOS 29 0.13978 0.12225 0.07896 0.06019 0.15564 0.13843 0.77857
KDEOS 40 0.16129 0.14419 0.07585 0.05702 0.16575 0.14874 0.77032
LDF 2 0.05376 0.03447 0.03337 0.01367 0.07958 0.06082 0.62089
LDF 3 0.02151 0.00156 0.03044 0.01068 0.07242 0.05351 0.62758
INFLO 2 0.07527 0.05642 0.03939 0.01981 0.08857 0.06999 0.65914
INFLO 4 0.04301 0.02350 0.04162 0.02208 0.09887 0.08050 0.68054
INFLO 5 0.05376 0.03447 0.03956 0.01998 0.10058 0.08225 0.65266
COF 2 0.07527 0.05642 0.04416 0.02467 0.10219 0.08389 0.68945
COF 4 0.09677 0.07836 0.04495 0.02548 0.09890 0.08053 0.69939

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (39.7 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.02138 0.00150 0.05393 0.03471 0.56882
KNNW 1 0.00000 -0.02031 0.02386 0.00404 0.05972 0.04061 0.60899
LOF 2 0.02151 0.00163 0.03473 0.01512 0.09302 0.07460 0.62544
LOF 3 0.01075 -0.00934 0.03765 0.01810 0.08788 0.06935 0.67844
SimplifiedLOF 2 0.03226 0.01260 0.03600 0.01642 0.07692 0.05817 0.68184
SimplifiedLOF 5 0.01075 -0.00934 0.04414 0.02472 0.10390 0.08569 0.71506
SimplifiedLOF 6 0.02151 0.00163 0.04304 0.02360 0.10484 0.08665 0.71746
LoOP 2 0.04301 0.02357 0.03661 0.01704 0.07890 0.06018 0.68266
LoOP 5 0.01075 -0.00934 0.04483 0.02543 0.10784 0.08972 0.71420
LoOP 6 0.01075 -0.00934 0.04483 0.02542 0.10941 0.09132 0.72053
LDOF 3 0.05376 0.03454 0.03570 0.01611 0.07379 0.05497 0.71463
LDOF 6 0.02151 0.00163 0.05013 0.03083 0.11312 0.09511 0.77435
LDOF 8 0.02151 0.00163 0.05043 0.03114 0.10849 0.09038 0.79124
LDOF 10 0.01075 -0.00934 0.05063 0.03135 0.11189 0.09385 0.78662
ODIN 4 0.10256 0.08433 0.05182 0.03256 0.11429 0.09629 0.72706
ODIN 8 0.04624 0.02686 0.04788 0.02854 0.12381 0.10601 0.71764
FastABOD 3 0.00000 -0.02031 0.02247 0.00261 0.05344 0.03422 0.57310
FastABOD 5 0.00000 -0.02031 0.02339 0.00355 0.05782 0.03868 0.59946
FastABOD 8 0.00000 -0.02031 0.02263 0.00277 0.05990 0.04080 0.59095
KDEOS 22 0.10753 0.08940 0.06821 0.04929 0.17049 0.15364 0.74626
KDEOS 24 0.11828 0.10037 0.07469 0.05589 0.16901 0.15213 0.74919
KDEOS 29 0.13978 0.12231 0.07016 0.05127 0.14130 0.12386 0.75416
KDEOS 34 0.11828 0.10037 0.06595 0.04697 0.13734 0.11981 0.75587
LDF 2 0.01075 -0.00934 0.03052 0.01082 0.07606 0.05729 0.61420
LDF 3 0.01075 -0.00934 0.03281 0.01316 0.07014 0.05125 0.66830
LDF 4 0.02151 0.00163 0.02967 0.00995 0.06853 0.04960 0.63794
INFLO 3 0.04301 0.02357 0.03244 0.01279 0.08949 0.07100 0.60773
INFLO 4 0.03226 0.01260 0.03488 0.01528 0.08939 0.07089 0.66492
INFLO 6 0.00000 -0.02031 0.03505 0.01544 0.10217 0.08393 0.63326
COF 2 0.04301 0.02357 0.03671 0.01714 0.07552 0.05674 0.67564
COF 4 0.02151 0.00163 0.03890 0.01937 0.09213 0.07369 0.69993
COF 5 0.00000 -0.02031 0.03804 0.01849 0.09538 0.07700 0.69587

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (40.1 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.03973 0.02015 0.09465 0.07619 0.76917
KNN 3 0.00000 -0.02039 0.03848 0.01888 0.09614 0.07771 0.76286
KNNW 1 0.00000 -0.02039 0.03652 0.01688 0.08755 0.06895 0.73844
KNNW 3 0.00000 -0.02039 0.03927 0.01969 0.09635 0.07793 0.76905
KNNW 4 0.00000 -0.02039 0.03925 0.01966 0.09865 0.08027 0.76888
LOF 2 0.05376 0.03447 0.03492 0.01524 0.07600 0.05717 0.64163
LOF 15 0.05376 0.03447 0.07805 0.05926 0.17230 0.15542 0.85553
LOF 20 0.04301 0.02350 0.07711 0.05829 0.17759 0.16082 0.84883
SimplifiedLOF 5 0.06452 0.04545 0.04198 0.02245 0.09524 0.07679 0.69956
SimplifiedLOF 25 0.03226 0.01253 0.07334 0.05445 0.16393 0.14689 0.84951
SimplifiedLOF 30 0.03226 0.01253 0.07278 0.05388 0.16518 0.14816 0.84859
LoOP 2 0.05376 0.03447 0.02957 0.00979 0.06639 0.04736 0.60035
LoOP 29 0.04301 0.02350 0.07130 0.05237 0.16205 0.14496 0.84675
LoOP 33 0.04301 0.02350 0.07132 0.05239 0.16535 0.14834 0.84633
LoOP 34 0.04301 0.02350 0.07142 0.05249 0.15984 0.14272 0.84621
LDOF 6 0.04301 0.02350 0.02944 0.00966 0.07394 0.05507 0.59876
LDOF 44 0.01075 -0.00941 0.06655 0.04752 0.15773 0.14056 0.84288
LDOF 54 0.01075 -0.00941 0.06814 0.04914 0.15328 0.13601 0.84578
ODIN 24 0.07918 0.06041 0.09133 0.07280 0.19459 0.17818 0.86198
ODIN 25 0.08523 0.06659 0.09084 0.07231 0.19263 0.17618 0.86229
ODIN 50 0.11968 0.10174 0.08036 0.06161 0.16822 0.15127 0.82157
FastABOD 3 0.01075 -0.00941 0.02694 0.00710 0.06226 0.04314 0.61579
FastABOD 26 0.00000 -0.02039 0.02560 0.00573 0.06203 0.04290 0.62715
FastABOD 100 0.00000 -0.02039 0.02442 0.00454 0.06305 0.04395 0.61110
KDEOS 77 0.16129 0.14419 0.08638 0.06775 0.17000 0.15308 0.83552
KDEOS 81 0.17204 0.15516 0.08585 0.06721 0.17297 0.15611 0.83422
KDEOS 94 0.13978 0.12225 0.08517 0.06652 0.17526 0.15844 0.83218
KDEOS 100 0.12903 0.11128 0.08884 0.07026 0.15385 0.13660 0.83237
LDF 2 0.05376 0.03447 0.03871 0.01911 0.08290 0.06421 0.68231
LDF 9 0.02151 0.00156 0.07894 0.06016 0.17164 0.15476 0.85243
LDF 12 0.04301 0.02350 0.08125 0.06252 0.17593 0.15913 0.84863
INFLO 2 0.06452 0.04545 0.03086 0.01110 0.06667 0.04764 0.60628
INFLO 25 0.03226 0.01253 0.06302 0.04392 0.15679 0.13960 0.81222
INFLO 27 0.03226 0.01253 0.06325 0.04416 0.15254 0.13527 0.81516
COF 26 0.10753 0.08933 0.09893 0.08056 0.21574 0.19976 0.86845
COF 30 0.16129 0.14419 0.10426 0.08600 0.22857 0.21285 0.86230
COF 32 0.12903 0.11128 0.10494 0.08669 0.23768 0.22214 0.86119
COF 39 0.11828 0.10030 0.09586 0.07742 0.25090 0.23562 0.84849

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (40.2 MB) Download raw algorithm evaluation table (71.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.03727 0.01771 0.08678 0.06823 0.74473
KNN 4 0.00000 -0.02031 0.03892 0.01939 0.09333 0.07491 0.76387
KNN 6 0.00000 -0.02031 0.03829 0.01875 0.09375 0.07534 0.76088
KNNW 1 0.00000 -0.02031 0.03505 0.01545 0.08095 0.06228 0.70265
KNNW 7 0.00000 -0.02031 0.03828 0.01874 0.09286 0.07443 0.75987
KNNW 8 0.00000 -0.02031 0.03825 0.01872 0.09377 0.07536 0.76016
KNNW 9 0.00000 -0.02031 0.03818 0.01865 0.09388 0.07547 0.75983
LOF 5 0.10753 0.08940 0.06773 0.04879 0.14328 0.12588 0.77113
LOF 15 0.09677 0.07843 0.09161 0.07315 0.17827 0.16158 0.87524
LOF 28 0.05376 0.03454 0.08456 0.06596 0.19525 0.17890 0.85356
SimplifiedLOF 6 0.11828 0.10037 0.06144 0.04238 0.13605 0.11850 0.73124
SimplifiedLOF 25 0.08602 0.06745 0.08527 0.06669 0.17600 0.15926 0.86874
SimplifiedLOF 27 0.07527 0.05648 0.08650 0.06794 0.17591 0.15917 0.87361
SimplifiedLOF 30 0.07527 0.05648 0.08625 0.06769 0.17277 0.15596 0.87537
LoOP 19 0.11828 0.10037 0.08253 0.06389 0.16760 0.15069 0.85520
LoOP 26 0.06452 0.04551 0.08429 0.06569 0.17939 0.16272 0.86847
LoOP 29 0.06452 0.04551 0.08511 0.06652 0.17014 0.15329 0.87298
LoOP 35 0.05376 0.03454 0.08455 0.06595 0.16901 0.15213 0.87409
LDOF 12 0.08602 0.06745 0.05316 0.03393 0.10795 0.08983 0.76186
LDOF 47 0.07527 0.05648 0.08073 0.06206 0.16069 0.14364 0.87576
LDOF 54 0.07527 0.05648 0.07918 0.06047 0.16535 0.14839 0.87291
ODIN 24 0.16227 0.14525 0.10485 0.08667 0.18919 0.17272 0.87063
ODIN 26 0.15432 0.13714 0.10788 0.08976 0.19335 0.17697 0.87323
ODIN 31 0.15188 0.13465 0.10933 0.09123 0.20994 0.19390 0.86795
ODIN 34 0.14247 0.12505 0.11127 0.09322 0.19876 0.18248 0.86938
FastABOD 3 0.00000 -0.02031 0.02553 0.00574 0.06061 0.04152 0.59588
FastABOD 4 0.00000 -0.02031 0.02606 0.00628 0.05900 0.03988 0.60931
FastABOD 17 0.00000 -0.02031 0.02584 0.00605 0.05676 0.03760 0.61320
KDEOS 96 0.23656 0.22105 0.14132 0.12388 0.24419 0.22883 0.85214
KDEOS 99 0.22581 0.21008 0.14894 0.13165 0.24468 0.22934 0.85130
KDEOS 100 0.23656 0.22105 0.15192 0.13469 0.23913 0.22367 0.85167
LDF 6 0.12903 0.11134 0.07842 0.05969 0.15451 0.13733 0.84073
LDF 13 0.09677 0.07843 0.09190 0.07345 0.17683 0.16011 0.87070
LDF 15 0.09677 0.07843 0.09222 0.07378 0.19217 0.17576 0.86853
LDF 21 0.05376 0.03454 0.08776 0.06923 0.19619 0.17986 0.85594
INFLO 9 0.11828 0.10037 0.06098 0.04190 0.13003 0.11236 0.74724
INFLO 16 0.09677 0.07843 0.07429 0.05549 0.16770 0.15079 0.80133
INFLO 21 0.09677 0.07843 0.07561 0.05683 0.15272 0.13550 0.82591
COF 4 0.15054 0.13328 0.05679 0.03763 0.15470 0.13752 0.66302
COF 29 0.08602 0.06745 0.10399 0.08578 0.21698 0.20107 0.88110
COF 34 0.12903 0.11134 0.10946 0.09137 0.22630 0.21058 0.87666
COF 35 0.11828 0.10037 0.10645 0.08830 0.23030 0.21467 0.87179

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO