Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Annthyroid (2% of outliers version#03)

This data set contains medical data on hypothyroidism. Three classes relate to the conditions normal, hyperfunction, and subnormal functioning. Classes other than normal condition were defined as outliers here.

Download all data set variants used (9.9 MB). You can also access the original data. (merge train and test [ann-test.data and ann-train.data])

Normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (58.4 MB) Download raw algorithm evaluation table (72.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.06716 0.04821 0.03837 0.01883 0.09366 0.07524 0.67943
KNNW 1 0.07463 0.05582 0.04373 0.02430 0.10289 0.08467 0.69448
LOF 1 0.08209 0.06344 0.03845 0.01891 0.10081 0.08254 0.58420
LOF 4 0.06716 0.04821 0.05332 0.03409 0.13235 0.11472 0.72403
LOF 7 0.03731 0.01775 0.05007 0.03076 0.11742 0.09949 0.73541
LOF 11 0.02239 0.00252 0.04957 0.03026 0.13249 0.11487 0.71934
SimplifiedLOF 2 0.07463 0.05582 0.04002 0.02051 0.10390 0.08569 0.65050
SimplifiedLOF 7 0.04478 0.02537 0.04823 0.02889 0.11945 0.10156 0.72435
SimplifiedLOF 11 0.03731 0.01775 0.04913 0.02981 0.11404 0.09603 0.73999
LoOP 3 0.09701 0.07867 0.04675 0.02739 0.10753 0.08939 0.68583
LoOP 11 0.06716 0.04821 0.05337 0.03414 0.12053 0.10266 0.73958
LoOP 24 0.04478 0.02537 0.04970 0.03039 0.12635 0.10860 0.71882
LDOF 15 0.11194 0.09390 0.06044 0.04134 0.11742 0.09948 0.75083
LDOF 17 0.09701 0.07867 0.06296 0.04392 0.12719 0.10946 0.75573
LDOF 26 0.08955 0.07105 0.06612 0.04715 0.13839 0.12089 0.74843
LDOF 33 0.09701 0.07867 0.06506 0.04607 0.14042 0.12295 0.74481
ODIN 19 0.07688 0.05812 0.04988 0.03057 0.10327 0.08505 0.72092
ODIN 39 0.09638 0.07802 0.05743 0.03828 0.13354 0.11594 0.69977
ODIN 43 0.09734 0.07900 0.05801 0.03887 0.12935 0.11166 0.69769
ODIN 59 0.10821 0.09009 0.05620 0.03703 0.11490 0.09692 0.67646
FastABOD 3 0.05224 0.03298 0.03535 0.01575 0.06936 0.05046 0.65541
FastABOD 4 0.05224 0.03298 0.03409 0.01447 0.06925 0.05034 0.65666
FastABOD 5 0.05970 0.04060 0.03280 0.01315 0.06734 0.04839 0.65481
FastABOD 7 0.05970 0.04060 0.03263 0.01298 0.06969 0.05078 0.64856
KDEOS 17 0.08955 0.07105 0.05020 0.03090 0.09859 0.08028 0.70812
KDEOS 22 0.06716 0.04821 0.05504 0.03584 0.10359 0.08537 0.71631
KDEOS 23 0.07463 0.05582 0.05395 0.03473 0.10492 0.08673 0.72025
KDEOS 36 0.06716 0.04821 0.05332 0.03409 0.09654 0.07819 0.72587
LDF 3 0.09701 0.07867 0.04868 0.02935 0.14006 0.12258 0.66058
LDF 4 0.03731 0.01775 0.05232 0.03306 0.13559 0.11803 0.70025
LDF 5 0.03731 0.01775 0.05011 0.03081 0.13849 0.12099 0.70930
INFLO 1 0.08955 0.07105 0.03825 0.01871 0.09392 0.07551 0.60159
INFLO 4 0.07463 0.05582 0.04759 0.02824 0.11321 0.09519 0.69333
INFLO 8 0.05224 0.03298 0.04667 0.02730 0.11784 0.09992 0.69958
INFLO 15 0.03731 0.01775 0.04489 0.02548 0.12312 0.10530 0.66977
COF 2 0.08209 0.06344 0.04000 0.02049 0.09821 0.07989 0.63975
COF 10 0.05224 0.03298 0.05191 0.03264 0.12319 0.10537 0.72807
COF 15 0.04478 0.02537 0.04943 0.03012 0.13651 0.11896 0.71372

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.9 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05882 0.03962 0.05769 0.03847 0.15054 0.13321 0.75830
KNNW 1 0.08088 0.06213 0.06327 0.04416 0.16102 0.14390 0.78075
LOF 1 0.05882 0.03962 0.03841 0.01879 0.10762 0.08942 0.62256
LOF 8 0.01471 -0.00540 0.07161 0.05267 0.17940 0.16266 0.79613
LOF 11 0.00735 -0.01290 0.07450 0.05562 0.19744 0.18107 0.79279
SimplifiedLOF 1 0.02941 0.00961 0.03215 0.01241 0.07795 0.05914 0.63730
SimplifiedLOF 11 0.01471 -0.00540 0.06773 0.04871 0.15869 0.14152 0.81789
SimplifiedLOF 12 0.00735 -0.01290 0.06906 0.05007 0.16638 0.14938 0.81780
SimplifiedLOF 13 0.00735 -0.01290 0.06894 0.04994 0.17223 0.15534 0.81574
LoOP 12 0.06618 0.04712 0.07518 0.05631 0.15758 0.14039 0.82993
LoOP 14 0.06618 0.04712 0.07763 0.05881 0.17606 0.15925 0.82749
LoOP 21 0.05882 0.03962 0.07763 0.05881 0.19723 0.18085 0.81780
LoOP 90 0.10294 0.08464 0.06583 0.04677 0.13251 0.11481 0.74649
LDOF 24 0.11029 0.09214 0.09283 0.07432 0.19963 0.18330 0.84423
LDOF 29 0.13971 0.12215 0.09942 0.08105 0.21355 0.19751 0.83839
LDOF 50 0.19853 0.18218 0.11292 0.09482 0.20820 0.19205 0.82741
LDOF 59 0.19853 0.18218 0.11380 0.09572 0.20884 0.19269 0.82323
ODIN 14 0.06506 0.04598 0.06199 0.04285 0.12177 0.10385 0.81085
ODIN 49 0.09191 0.07338 0.07578 0.05692 0.18662 0.17003 0.78089
ODIN 56 0.09559 0.07714 0.07460 0.05572 0.18764 0.17107 0.76934
ODIN 99 0.11460 0.09654 0.06753 0.04851 0.17073 0.15381 0.72650
FastABOD 4 0.04412 0.02462 0.04464 0.02515 0.10137 0.08304 0.74416
FastABOD 84 0.04412 0.02462 0.04042 0.02084 0.10191 0.08359 0.69716
KDEOS 12 0.12500 0.10715 0.06084 0.04168 0.12958 0.11182 0.75335
KDEOS 26 0.08824 0.06963 0.06755 0.04853 0.14646 0.12905 0.76815
KDEOS 64 0.12500 0.10715 0.08268 0.06396 0.14319 0.12571 0.79651
KDEOS 66 0.11029 0.09214 0.08138 0.06263 0.13517 0.11752 0.79720
LDF 2 0.12500 0.10715 0.05039 0.03102 0.13964 0.12209 0.66459
LDF 6 0.00735 -0.01290 0.07440 0.05551 0.19590 0.17949 0.77325
LDF 8 0.00000 -0.02040 0.06694 0.04791 0.19770 0.18133 0.75335
INFLO 1 0.06618 0.04712 0.04000 0.02041 0.10266 0.08435 0.65238
INFLO 12 0.03676 0.01711 0.07565 0.05680 0.19016 0.17364 0.80030
COF 13 0.03676 0.01711 0.06817 0.04916 0.17021 0.15328 0.78996
COF 20 0.04412 0.02462 0.07167 0.05273 0.20044 0.18412 0.77448
COF 21 0.04412 0.02462 0.07025 0.05128 0.20131 0.18502 0.76798
COF 61 0.13235 0.11465 0.06023 0.04106 0.13433 0.11667 0.71259

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (57.8 MB) Download raw algorithm evaluation table (72.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05970 0.04060 0.03659 0.01701 0.08281 0.06417 0.67165
KNNW 1 0.07463 0.05582 0.04212 0.02265 0.08696 0.06840 0.69757
LOF 2 0.07463 0.05582 0.04748 0.02813 0.13075 0.11309 0.65139
LOF 6 0.03731 0.01775 0.05671 0.03755 0.15917 0.14209 0.72581
LOF 10 0.01493 -0.00509 0.05145 0.03218 0.14757 0.13025 0.73173
SimplifiedLOF 2 0.05970 0.04060 0.04270 0.02325 0.10900 0.09090 0.69194
SimplifiedLOF 6 0.04478 0.02537 0.05356 0.03433 0.13643 0.11889 0.74296
SimplifiedLOF 14 0.01493 -0.00509 0.05022 0.03093 0.13168 0.11403 0.75403
LoOP 4 0.11194 0.09390 0.05497 0.03577 0.12834 0.11063 0.72878
LoOP 9 0.07463 0.05582 0.05976 0.04065 0.14655 0.12921 0.75217
LoOP 14 0.05970 0.04060 0.05968 0.04058 0.14451 0.12713 0.75443
LDOF 9 0.13433 0.11674 0.06793 0.04900 0.14400 0.12661 0.76619
LDOF 22 0.11194 0.09390 0.07745 0.05870 0.16102 0.14397 0.77956
LDOF 23 0.10448 0.08628 0.07779 0.05905 0.15880 0.14171 0.78163
LDOF 33 0.11194 0.09390 0.07908 0.06037 0.14876 0.13146 0.75722
ODIN 10 0.12096 0.10310 0.06147 0.04240 0.14170 0.12426 0.73749
ODIN 21 0.11392 0.09592 0.06572 0.04674 0.15962 0.14255 0.72518
ODIN 27 0.10829 0.09017 0.06750 0.04855 0.15196 0.13473 0.71138
ODIN 49 0.13513 0.11756 0.06313 0.04410 0.13585 0.11829 0.67813
FastABOD 4 0.05224 0.03298 0.03206 0.01239 0.06498 0.04598 0.64246
FastABOD 6 0.05224 0.03298 0.03241 0.01275 0.06934 0.05043 0.64505
FastABOD 9 0.05224 0.03298 0.03193 0.01226 0.07302 0.05419 0.63451
KDEOS 20 0.09701 0.07867 0.05663 0.03746 0.10408 0.08588 0.74221
KDEOS 22 0.07463 0.05582 0.05800 0.03886 0.10797 0.08984 0.74826
KDEOS 47 0.05970 0.04060 0.05133 0.03205 0.11280 0.09477 0.72096
LDF 1 0.07463 0.05582 0.03298 0.01334 0.08434 0.06573 0.58026
LDF 4 0.04478 0.02537 0.05179 0.03252 0.14286 0.12544 0.71583
INFLO 1 0.08955 0.07105 0.04321 0.02377 0.11243 0.09439 0.66185
INFLO 8 0.04478 0.02537 0.05086 0.03158 0.13430 0.11671 0.70519
INFLO 13 0.02239 0.00252 0.04708 0.02772 0.14391 0.12651 0.66727
COF 4 0.08955 0.07105 0.05299 0.03375 0.13793 0.12042 0.70765
COF 7 0.06716 0.04821 0.05867 0.03954 0.14701 0.12968 0.74320

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.3 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.02206 0.00211 0.04132 0.02176 0.10719 0.08897 0.68676
KNNW 1 0.01471 -0.00540 0.04062 0.02105 0.09524 0.07678 0.69926
KNNW 2 0.01471 -0.00540 0.04119 0.02163 0.10667 0.08844 0.69419
LOF 1 0.04412 0.02462 0.02824 0.00842 0.07426 0.05538 0.55731
LOF 6 0.01471 -0.00540 0.06373 0.04462 0.17667 0.15987 0.73790
LOF 8 0.00735 -0.01290 0.06189 0.04275 0.18070 0.16398 0.73644
SimplifiedLOF 1 0.02941 0.00961 0.03163 0.01187 0.08019 0.06142 0.63226
SimplifiedLOF 9 0.00735 -0.01290 0.05920 0.04001 0.15537 0.13814 0.77011
SimplifiedLOF 10 0.00735 -0.01290 0.05941 0.04022 0.15100 0.13368 0.77100
LoOP 6 0.07353 0.05463 0.06096 0.04180 0.14828 0.13091 0.76697
LoOP 9 0.06618 0.04712 0.06642 0.04737 0.16088 0.14376 0.77801
LoOP 13 0.05147 0.03212 0.06793 0.04891 0.17647 0.15967 0.76980
LoOP 15 0.05147 0.03212 0.06850 0.04950 0.16880 0.15184 0.77148
LDOF 16 0.13971 0.12215 0.08910 0.07051 0.18638 0.16978 0.81086
LDOF 30 0.15441 0.13716 0.09400 0.07552 0.20144 0.18515 0.78383
LDOF 32 0.16176 0.14466 0.09271 0.07419 0.20101 0.18470 0.77428
LDOF 41 0.14706 0.12966 0.08745 0.06883 0.20588 0.18968 0.76359
ODIN 11 0.11279 0.09469 0.07462 0.05574 0.16667 0.14966 0.76857
ODIN 36 0.15517 0.13793 0.08338 0.06468 0.22571 0.20991 0.73462
ODIN 38 0.16415 0.14710 0.08418 0.06550 0.22508 0.20927 0.73276
ODIN 52 0.20637 0.19018 0.08142 0.06268 0.21365 0.19761 0.70858
FastABOD 3 0.01471 -0.00540 0.03173 0.01197 0.07446 0.05558 0.64932
FastABOD 4 0.01471 -0.00540 0.03317 0.01344 0.07492 0.05605 0.65606
FastABOD 6 0.01471 -0.00540 0.03319 0.01346 0.07668 0.05784 0.65346
FastABOD 24 0.01471 -0.00540 0.03051 0.01073 0.07702 0.05819 0.63407
KDEOS 20 0.13235 0.11465 0.06564 0.04658 0.14035 0.12281 0.75490
KDEOS 30 0.08824 0.06963 0.06111 0.04196 0.12176 0.10384 0.76164
KDEOS 66 0.12500 0.10715 0.07470 0.05583 0.14966 0.13231 0.73999
LDF 2 0.06618 0.04712 0.03896 0.01935 0.11173 0.09361 0.61155
LDF 4 0.02206 0.00211 0.05688 0.03764 0.16667 0.14966 0.69513
LDF 5 0.00735 -0.01290 0.05311 0.03379 0.16701 0.15002 0.70087
LDF 8 0.00735 -0.01290 0.05324 0.03393 0.16064 0.14352 0.70141
INFLO 1 0.04412 0.02462 0.03352 0.01380 0.08195 0.06322 0.63057
INFLO 7 0.02206 0.00211 0.05628 0.03702 0.16207 0.14498 0.71122
INFLO 8 0.02206 0.00211 0.05789 0.03867 0.15779 0.14061 0.73160
INFLO 10 0.01471 -0.00540 0.05807 0.03885 0.15504 0.13780 0.71942
COF 2 0.02941 0.00961 0.03499 0.01531 0.08455 0.06587 0.65752
COF 8 0.01471 -0.00540 0.05749 0.03827 0.16636 0.14935 0.73119
COF 11 0.02941 0.00961 0.05840 0.03919 0.17045 0.15353 0.73101

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO