Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Wilt (2% of outliers version#08)

The data set differentiates diseased trees from other land covers. The former are considered outliers here.

Download all data set variants used (7.2 MB). You can also access the original data. (merge train and test [wilt.zip])

Normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (39.6 MB) Download raw algorithm evaluation table (73.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.02365 0.00375 0.06104 0.04190 0.60971
KNNW 1 0.00000 -0.02039 0.02586 0.00600 0.06278 0.04367 0.63949
KNNW 2 0.00000 -0.02039 0.02483 0.00495 0.06333 0.04423 0.63008
LOF 2 0.08602 0.06739 0.04705 0.02762 0.11438 0.09632 0.67947
LOF 3 0.03226 0.01253 0.04250 0.02298 0.11111 0.09299 0.68876
SimplifiedLOF 2 0.08602 0.06739 0.04601 0.02656 0.09659 0.07817 0.72587
SimplifiedLOF 4 0.04301 0.02350 0.05680 0.03757 0.13828 0.12071 0.75633
LoOP 2 0.05376 0.03447 0.04463 0.02515 0.09272 0.07422 0.72284
LoOP 4 0.03226 0.01253 0.05452 0.03525 0.13580 0.11819 0.74955
LoOP 8 0.03226 0.01253 0.05037 0.03101 0.12112 0.10320 0.75041
LDOF 3 0.06452 0.04545 0.03554 0.01588 0.08054 0.06179 0.65169
LDOF 6 0.04301 0.02350 0.05372 0.03443 0.14675 0.12936 0.75500
LDOF 10 0.03226 0.01253 0.05559 0.03634 0.13158 0.11388 0.78701
ODIN 3 0.08861 0.07003 0.05156 0.03222 0.12727 0.10948 0.73765
ODIN 4 0.06195 0.04282 0.05343 0.03413 0.13109 0.11337 0.74554
ODIN 5 0.06839 0.04940 0.05418 0.03490 0.12754 0.10975 0.73756
ODIN 6 0.07040 0.05145 0.05376 0.03447 0.13974 0.12220 0.73085
FastABOD 3 0.00000 -0.02039 0.02464 0.00476 0.05594 0.03669 0.60816
FastABOD 4 0.00000 -0.02039 0.02567 0.00581 0.06215 0.04303 0.63250
FastABOD 5 0.00000 -0.02039 0.02544 0.00557 0.06348 0.04439 0.63328
FastABOD 9 0.00000 -0.02039 0.02447 0.00458 0.06403 0.04495 0.62282
KDEOS 21 0.10753 0.08933 0.08261 0.06391 0.18576 0.16916 0.79016
KDEOS 22 0.11828 0.10030 0.08130 0.06257 0.18621 0.16962 0.79034
KDEOS 25 0.13978 0.12225 0.07821 0.05942 0.15942 0.14228 0.78685
KDEOS 30 0.12903 0.11128 0.07960 0.06084 0.15493 0.13770 0.79122
LDF 2 0.09677 0.07836 0.04421 0.02473 0.10196 0.08365 0.66301
LDF 3 0.02151 0.00156 0.03668 0.01704 0.08491 0.06625 0.67398
INFLO 1 0.06452 0.04545 0.03695 0.01732 0.08194 0.06322 0.66578
INFLO 4 0.05376 0.03447 0.04235 0.02282 0.10335 0.08507 0.70068
INFLO 6 0.02151 0.00156 0.03851 0.01891 0.11250 0.09441 0.65642
COF 4 0.07527 0.05642 0.05176 0.03243 0.12325 0.10538 0.74576
COF 5 0.07527 0.05642 0.05300 0.03370 0.11550 0.09747 0.73989

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (39.7 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.02346 0.00362 0.05892 0.03980 0.61267
KNNW 1 0.00000 -0.02031 0.02344 0.00360 0.05941 0.04030 0.60312
KNNW 2 0.00000 -0.02031 0.02389 0.00406 0.06058 0.04149 0.61971
LOF 4 0.03226 0.01260 0.03961 0.02010 0.09964 0.08135 0.68559
LOF 5 0.03226 0.01260 0.04008 0.02058 0.09293 0.07450 0.70679
LOF 6 0.02151 0.00163 0.03934 0.01983 0.08880 0.07029 0.71178
LOF 8 0.04301 0.02357 0.03458 0.01497 0.07414 0.05533 0.68414
SimplifiedLOF 5 0.02151 0.00163 0.05011 0.03081 0.12007 0.10219 0.75598
SimplifiedLOF 7 0.04301 0.02357 0.04931 0.03000 0.11138 0.09333 0.76140
SimplifiedLOF 9 0.03226 0.01260 0.04792 0.02858 0.10588 0.08772 0.76514
LoOP 5 0.02151 0.00163 0.05160 0.03233 0.12524 0.10747 0.75872
LoOP 9 0.02151 0.00163 0.05013 0.03084 0.11816 0.10025 0.76944
LoOP 11 0.04301 0.02357 0.04732 0.02797 0.09929 0.08099 0.75563
LDOF 5 0.10753 0.08940 0.05779 0.03865 0.13270 0.11508 0.77618
LDOF 6 0.06452 0.04551 0.05746 0.03831 0.13567 0.11811 0.77835
LDOF 7 0.06452 0.04551 0.05936 0.04025 0.13426 0.11667 0.78034
LDOF 12 0.04301 0.02357 0.05617 0.03700 0.11733 0.09940 0.80232
ODIN 4 0.05455 0.03534 0.05231 0.03306 0.11963 0.10174 0.76818
ODIN 6 0.07509 0.05630 0.05490 0.03570 0.12200 0.10417 0.75052
ODIN 7 0.06452 0.04551 0.05585 0.03667 0.12090 0.10304 0.76512
ODIN 9 0.07968 0.06098 0.04951 0.03020 0.10386 0.08566 0.74396
FastABOD 3 0.00000 -0.02031 0.02442 0.00461 0.05698 0.03783 0.61378
FastABOD 5 0.00000 -0.02031 0.02532 0.00552 0.06165 0.04259 0.63406
FastABOD 7 0.00000 -0.02031 0.02447 0.00465 0.06259 0.04355 0.62597
KDEOS 19 0.07527 0.05648 0.07459 0.05579 0.14685 0.12952 0.79906
KDEOS 22 0.10753 0.08940 0.07404 0.05523 0.16162 0.14458 0.80811
KDEOS 25 0.09677 0.07843 0.07264 0.05380 0.16505 0.14809 0.80504
KDEOS 28 0.11828 0.10037 0.06823 0.04930 0.13368 0.11608 0.79963
LDF 2 0.03226 0.01260 0.03316 0.01351 0.07692 0.05817 0.64826
LDF 4 0.03226 0.01260 0.03431 0.01469 0.07536 0.05657 0.66219
LDF 5 0.02151 0.00163 0.03302 0.01338 0.07740 0.05866 0.66605
LDF 6 0.01075 -0.00934 0.03210 0.01244 0.07197 0.05312 0.66825
INFLO 3 0.03226 0.01260 0.03497 0.01537 0.09091 0.07244 0.63724
INFLO 4 0.02151 0.00163 0.04016 0.02066 0.09793 0.07961 0.70420
INFLO 6 0.01075 -0.00934 0.03968 0.02017 0.11439 0.09640 0.67661
COF 2 0.04301 0.02357 0.03504 0.01544 0.07724 0.05849 0.65423
COF 4 0.03226 0.01260 0.03850 0.01897 0.08351 0.06489 0.70526
COF 6 0.02151 0.00163 0.03502 0.01542 0.08031 0.06162 0.71087

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (40.1 MB) Download raw algorithm evaluation table (72.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.04230 0.02278 0.10542 0.08719 0.77990
KNN 3 0.00000 -0.02039 0.04282 0.02330 0.10424 0.08598 0.78983
KNNW 1 0.00000 -0.02039 0.03991 0.02033 0.09677 0.07836 0.74035
KNNW 6 0.00000 -0.02039 0.04249 0.02297 0.10139 0.08308 0.78753
KNNW 7 0.00000 -0.02039 0.04247 0.02295 0.10239 0.08410 0.78802
KNNW 10 0.00000 -0.02039 0.04202 0.02249 0.10428 0.08602 0.78625
LOF 1 0.06452 0.04545 0.03130 0.01155 0.09524 0.07679 0.55720
LOF 19 0.02151 0.00156 0.09203 0.07352 0.19298 0.17653 0.88013
LOF 20 0.03226 0.01253 0.09220 0.07370 0.20118 0.18490 0.87965
LOF 23 0.04301 0.02350 0.09082 0.07229 0.20609 0.18990 0.87585
SimplifiedLOF 1 0.05376 0.03447 0.03139 0.01165 0.06803 0.04903 0.57471
SimplifiedLOF 27 0.03226 0.01253 0.08553 0.06689 0.19008 0.17357 0.87434
SimplifiedLOF 35 0.00000 -0.02039 0.08356 0.06488 0.18124 0.16455 0.87589
LoOP 6 0.06452 0.04545 0.05239 0.03307 0.15082 0.13351 0.70442
LoOP 27 0.03226 0.01253 0.08218 0.06347 0.18421 0.16758 0.86963
LoOP 35 0.01075 -0.00941 0.08145 0.06273 0.17778 0.16102 0.87357
LDOF 7 0.06452 0.04545 0.03868 0.01908 0.09567 0.07724 0.65510
LDOF 57 0.02151 0.00156 0.07552 0.05667 0.17379 0.15695 0.86656
LDOF 58 0.02151 0.00156 0.07576 0.05692 0.17630 0.15951 0.86627
LDOF 59 0.02151 0.00156 0.07577 0.05693 0.17589 0.15909 0.86587
ODIN 25 0.09429 0.07583 0.09346 0.07498 0.17664 0.15985 0.87594
ODIN 32 0.08602 0.06739 0.09776 0.07937 0.18667 0.17009 0.87044
ODIN 36 0.09720 0.07880 0.09341 0.07493 0.19005 0.17353 0.86682
ODIN 54 0.12698 0.10919 0.08815 0.06956 0.17004 0.15312 0.84289
FastABOD 3 0.00000 -0.02039 0.02624 0.00639 0.05675 0.03752 0.63226
FastABOD 17 0.00000 -0.02039 0.02881 0.00901 0.07369 0.05481 0.65934
FastABOD 23 0.00000 -0.02039 0.02874 0.00894 0.07467 0.05580 0.65787
KDEOS 67 0.15054 0.13322 0.08384 0.06516 0.15054 0.13322 0.84004
KDEOS 96 0.15054 0.13322 0.10266 0.08437 0.17178 0.15490 0.84885
KDEOS 100 0.15054 0.13322 0.10194 0.08363 0.17391 0.15707 0.84924
LDF 1 0.08602 0.06739 0.03123 0.01148 0.08738 0.06877 0.56629
LDF 12 0.03226 0.01253 0.09748 0.07908 0.20669 0.19051 0.88275
LDF 13 0.04301 0.02350 0.09745 0.07905 0.21472 0.19872 0.88164
INFLO 13 0.07527 0.05642 0.06028 0.04112 0.15011 0.13278 0.76698
INFLO 26 0.03226 0.01253 0.07367 0.05479 0.18994 0.17343 0.83783
COF 27 0.18280 0.16614 0.12582 0.10800 0.23308 0.21745 0.88044
COF 31 0.21505 0.19905 0.13055 0.11283 0.24080 0.22533 0.87699
COF 34 0.22581 0.21002 0.12929 0.11154 0.25758 0.24244 0.87081
COF 36 0.19355 0.17711 0.12503 0.10719 0.26148 0.24643 0.86991

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (40.2 MB) Download raw algorithm evaluation table (71.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.04450 0.02509 0.10627 0.08811 0.78567
KNN 3 0.00000 -0.02031 0.04592 0.02654 0.11197 0.09393 0.79728
KNNW 1 0.00000 -0.02031 0.03889 0.01936 0.09420 0.07580 0.72853
KNNW 5 0.00000 -0.02031 0.04505 0.02565 0.10759 0.08946 0.79462
KNNW 6 0.00000 -0.02031 0.04498 0.02558 0.10928 0.09119 0.79435
LOF 5 0.09677 0.07843 0.07453 0.05573 0.17349 0.15670 0.80399
LOF 13 0.07527 0.05648 0.10220 0.08396 0.20441 0.18825 0.88102
LOF 18 0.07527 0.05648 0.10100 0.08274 0.21222 0.19622 0.88030
SimplifiedLOF 5 0.08602 0.06745 0.05483 0.03562 0.13592 0.11837 0.72076
SimplifiedLOF 22 0.07527 0.05648 0.09777 0.07944 0.21142 0.19540 0.87749
SimplifiedLOF 28 0.05376 0.03454 0.09639 0.07803 0.19789 0.18160 0.88099
LoOP 15 0.08602 0.06745 0.08601 0.06744 0.18231 0.16569 0.84091
LoOP 22 0.07527 0.05648 0.09346 0.07504 0.21145 0.19543 0.87105
LoOP 31 0.06452 0.04551 0.09412 0.07572 0.20147 0.18525 0.87827
LoOP 34 0.04301 0.02357 0.09338 0.07496 0.19957 0.18331 0.87862
LDOF 19 0.11828 0.10037 0.07028 0.05139 0.14583 0.12848 0.80913
LDOF 41 0.04301 0.02357 0.08697 0.06842 0.20441 0.18825 0.87192
LDOF 50 0.05376 0.03454 0.08830 0.06978 0.18930 0.17283 0.87443
LDOF 55 0.05376 0.03454 0.08818 0.06966 0.18299 0.16639 0.87560
ODIN 27 0.11203 0.09399 0.11189 0.09385 0.21809 0.20220 0.88201
ODIN 32 0.10249 0.08425 0.11493 0.09695 0.23729 0.22179 0.88099
ODIN 42 0.15152 0.13428 0.11039 0.09232 0.20690 0.19079 0.87011
FastABOD 4 0.01075 -0.00934 0.03108 0.01140 0.07447 0.05567 0.66552
FastABOD 6 0.00000 -0.02031 0.03126 0.01158 0.07505 0.05626 0.67353
FastABOD 14 0.00000 -0.02031 0.03191 0.01224 0.07306 0.05423 0.67713
KDEOS 96 0.20430 0.18814 0.12476 0.10698 0.22936 0.21370 0.85544
KDEOS 97 0.19355 0.17717 0.12616 0.10841 0.21905 0.20318 0.85543
KDEOS 99 0.18280 0.16619 0.12589 0.10813 0.21739 0.20149 0.85549
LDF 5 0.10753 0.08940 0.08704 0.06849 0.18400 0.16742 0.84899
LDF 8 0.09677 0.07843 0.10348 0.08527 0.21159 0.19557 0.88531
LDF 10 0.09677 0.07843 0.10582 0.08766 0.21765 0.20175 0.88462
LDF 14 0.08602 0.06745 0.10224 0.08400 0.21839 0.20251 0.88027
INFLO 3 0.08602 0.06745 0.03881 0.01929 0.10453 0.08634 0.63631
INFLO 22 0.08602 0.06745 0.08265 0.06401 0.18337 0.16678 0.83905
INFLO 28 0.06452 0.04551 0.08159 0.06293 0.18526 0.16871 0.84249
INFLO 40 0.03226 0.01260 0.07628 0.05752 0.18688 0.17036 0.82112
COF 21 0.17204 0.15522 0.11584 0.09788 0.22010 0.20425 0.88633
COF 28 0.18280 0.16619 0.12028 0.10240 0.23776 0.22228 0.88502
COF 32 0.16129 0.14425 0.11428 0.09628 0.24603 0.23072 0.87857

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO