Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Annthyroid (2% of outliers version#06)

This data set contains medical data on hypothyroidism. Three classes relate to the conditions normal, hyperfunction, and subnormal functioning. Classes other than normal condition were defined as outliers here.

Download all data set variants used (9.9 MB). You can also access the original data. (merge train and test [ann-test.data and ann-train.data])

Normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (58.4 MB) Download raw algorithm evaluation table (72.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.04478 0.02537 0.03852 0.01899 0.08780 0.06927 0.67020
KNNW 1 0.06716 0.04821 0.04345 0.02401 0.10201 0.08376 0.69041
LOF 1 0.06716 0.04821 0.03630 0.01672 0.09745 0.07911 0.61022
LOF 6 0.02239 0.00252 0.05074 0.03145 0.13793 0.12042 0.73813
SimplifiedLOF 2 0.05970 0.04060 0.04240 0.02294 0.09979 0.08150 0.68905
SimplifiedLOF 7 0.02985 0.01014 0.05104 0.03176 0.12028 0.10241 0.76599
SimplifiedLOF 8 0.02239 0.00252 0.05054 0.03125 0.12385 0.10605 0.76515
LoOP 2 0.07463 0.05582 0.04575 0.02636 0.11609 0.09814 0.68762
LoOP 7 0.05970 0.04060 0.05433 0.03512 0.12519 0.10741 0.76994
LoOP 16 0.02985 0.01014 0.05024 0.03095 0.12959 0.11190 0.73734
LDOF 11 0.10448 0.08628 0.05954 0.04043 0.12658 0.10884 0.77159
LDOF 12 0.09701 0.07867 0.06212 0.04306 0.13309 0.11547 0.77508
LDOF 13 0.09701 0.07867 0.06293 0.04389 0.13283 0.11521 0.77377
LDOF 16 0.08209 0.06344 0.05975 0.04064 0.14103 0.12357 0.76178
ODIN 4 0.09470 0.07630 0.05120 0.03192 0.12563 0.10786 0.73946
ODIN 16 0.09701 0.07867 0.06064 0.04155 0.12429 0.10650 0.72808
ODIN 45 0.11567 0.09770 0.05347 0.03424 0.15625 0.13911 0.67416
FastABOD 4 0.04478 0.02537 0.03388 0.01425 0.07237 0.05352 0.65667
FastABOD 5 0.05224 0.03298 0.03313 0.01349 0.07229 0.05344 0.65078
FastABOD 48 0.02985 0.01014 0.03216 0.01249 0.07614 0.05737 0.62127
KDEOS 8 0.11194 0.09390 0.05407 0.03485 0.11952 0.10163 0.73928
KDEOS 16 0.08955 0.07105 0.05374 0.03452 0.10076 0.08249 0.75569
KDEOS 23 0.08955 0.07105 0.05743 0.03827 0.10160 0.08334 0.75058
LDF 2 0.05970 0.04060 0.03636 0.01679 0.09140 0.07294 0.63874
LDF 5 0.01493 -0.00509 0.04460 0.02519 0.12393 0.10613 0.68879
INFLO 2 0.06716 0.04821 0.04401 0.02458 0.11070 0.09263 0.69328
INFLO 3 0.05970 0.04060 0.04617 0.02679 0.13002 0.11234 0.68629
INFLO 8 0.02985 0.01014 0.04765 0.02830 0.11818 0.10026 0.72521
COF 2 0.08955 0.07105 0.04313 0.02368 0.11312 0.09510 0.67966
COF 5 0.05224 0.03298 0.04719 0.02783 0.10773 0.08960 0.73435
COF 9 0.04478 0.02537 0.04750 0.02814 0.11741 0.09948 0.71624
COF 14 0.03731 0.01775 0.04526 0.02586 0.12214 0.10430 0.69422

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.9 MB) Download raw algorithm evaluation table (71.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.07353 0.05463 0.04227 0.02273 0.10995 0.09179 0.66501
KNNW 1 0.07353 0.05463 0.04560 0.02612 0.11152 0.09340 0.68765
KNNW 2 0.08088 0.06213 0.04381 0.02430 0.11258 0.09448 0.67761
LOF 1 0.06618 0.04712 0.03604 0.01638 0.09042 0.07186 0.61156
LOF 8 0.04412 0.02462 0.05026 0.03089 0.14236 0.12486 0.70657
SimplifiedLOF 8 0.04412 0.02462 0.04972 0.03033 0.12358 0.10570 0.73257
SimplifiedLOF 9 0.04412 0.02462 0.05021 0.03084 0.13380 0.11613 0.73069
SimplifiedLOF 10 0.04412 0.02462 0.04987 0.03049 0.12766 0.10986 0.73295
LoOP 8 0.06618 0.04712 0.05261 0.03329 0.12632 0.10849 0.74328
LoOP 16 0.08088 0.06213 0.05650 0.03726 0.14553 0.12810 0.73498
LoOP 18 0.07353 0.05463 0.05545 0.03618 0.14683 0.12942 0.73046
LoOP 26 0.08824 0.06963 0.05263 0.03330 0.12838 0.11060 0.71997
LDOF 12 0.10294 0.08464 0.06707 0.04804 0.13208 0.11437 0.75734
LDOF 20 0.11029 0.09214 0.07234 0.05341 0.14257 0.12507 0.74379
LDOF 24 0.12500 0.10715 0.06729 0.04826 0.14798 0.13060 0.73611
LDOF 31 0.11765 0.09965 0.06568 0.04662 0.16071 0.14359 0.72646
ODIN 5 0.08163 0.06290 0.04429 0.02479 0.09639 0.07795 0.72347
ODIN 75 0.11581 0.09777 0.06307 0.04395 0.14334 0.12587 0.67008
ODIN 92 0.14037 0.12284 0.05936 0.04016 0.14796 0.13058 0.65584
ODIN 93 0.13427 0.11661 0.06023 0.04105 0.14987 0.13253 0.65541
FastABOD 4 0.05147 0.03212 0.03405 0.01434 0.07912 0.06033 0.63466
FastABOD 11 0.05882 0.03962 0.03720 0.01755 0.08251 0.06379 0.62736
FastABOD 17 0.04412 0.02462 0.03832 0.01870 0.08118 0.06244 0.62171
FastABOD 25 0.05147 0.03212 0.03510 0.01541 0.08302 0.06431 0.61769
KDEOS 28 0.08088 0.06213 0.05708 0.03784 0.09953 0.08115 0.73085
KDEOS 34 0.09559 0.07714 0.05363 0.03433 0.09942 0.08104 0.73273
KDEOS 37 0.08824 0.06963 0.05180 0.03245 0.11258 0.09448 0.73714
LDF 2 0.08088 0.06213 0.03845 0.01883 0.10507 0.08681 0.61139
LDF 4 0.03676 0.01711 0.04695 0.02751 0.12805 0.11026 0.66715
LDF 5 0.03676 0.01711 0.04548 0.02601 0.11928 0.10132 0.66997
INFLO 1 0.05882 0.03962 0.03489 0.01520 0.08511 0.06644 0.62633
INFLO 4 0.04412 0.02462 0.04150 0.02194 0.10420 0.08592 0.69128
INFLO 8 0.05147 0.03212 0.05048 0.03111 0.12787 0.11008 0.68362
INFLO 15 0.03676 0.01711 0.04677 0.02732 0.13636 0.11874 0.65335
COF 10 0.06618 0.04712 0.04875 0.02934 0.12006 0.10211 0.71305
COF 13 0.05882 0.03962 0.04799 0.02857 0.12389 0.10602 0.69017
COF 17 0.08088 0.06213 0.04806 0.02864 0.12009 0.10214 0.68128
COF 46 0.07353 0.05463 0.04979 0.03041 0.09167 0.07313 0.67297

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (57.8 MB) Download raw algorithm evaluation table (73.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.02985 0.01014 0.03705 0.01749 0.08544 0.06685 0.66412
KNN 3 0.01493 -0.00509 0.03448 0.01486 0.08757 0.06903 0.64293
KNNW 1 0.05224 0.03298 0.04286 0.02341 0.10031 0.08203 0.69705
LOF 1 0.08955 0.07105 0.04539 0.02599 0.12759 0.10987 0.62288
LOF 6 0.01493 -0.00509 0.05292 0.03367 0.15086 0.13361 0.70645
LOF 8 0.01493 -0.00509 0.05032 0.03102 0.14614 0.12879 0.71143
SimplifiedLOF 1 0.05970 0.04060 0.04995 0.03065 0.09970 0.08141 0.68417
SimplifiedLOF 7 0.02239 0.00252 0.05672 0.03755 0.14313 0.12572 0.76361
LoOP 4 0.09701 0.07867 0.06031 0.04122 0.15205 0.13482 0.75361
LoOP 7 0.07463 0.05582 0.06329 0.04426 0.15111 0.13386 0.77279
LDOF 7 0.14925 0.13197 0.07726 0.05851 0.16349 0.14649 0.78978
LDOF 9 0.09701 0.07867 0.07353 0.05470 0.15254 0.13532 0.79370
ODIN 9 0.13875 0.12125 0.06681 0.04785 0.14555 0.12819 0.74370
ODIN 11 0.14689 0.12956 0.07185 0.05299 0.15337 0.13617 0.74010
ODIN 12 0.14730 0.12998 0.06956 0.05066 0.15686 0.13973 0.73056
ODIN 26 0.13324 0.11563 0.06434 0.04533 0.17112 0.15428 0.69713
FastABOD 4 0.05224 0.03298 0.03358 0.01394 0.07461 0.05581 0.64934
FastABOD 5 0.05970 0.04060 0.03355 0.01392 0.07128 0.05241 0.64744
FastABOD 7 0.04478 0.02537 0.03381 0.01418 0.07221 0.05336 0.64599
KDEOS 10 0.11940 0.10151 0.08509 0.06650 0.12903 0.11134 0.74518
KDEOS 11 0.14179 0.12435 0.08415 0.06554 0.14876 0.13146 0.74545
KDEOS 15 0.11940 0.10151 0.07428 0.05547 0.13014 0.11246 0.75350
KDEOS 24 0.14179 0.12435 0.07369 0.05486 0.15328 0.13608 0.74594
LDF 2 0.08209 0.06344 0.03608 0.01650 0.09978 0.08149 0.60943
LDF 6 0.01493 -0.00509 0.04502 0.02562 0.13360 0.11600 0.67742
INFLO 1 0.08955 0.07105 0.04869 0.02936 0.12549 0.10772 0.67962
INFLO 4 0.03731 0.01775 0.05206 0.03280 0.13793 0.12042 0.71201
INFLO 6 0.03731 0.01775 0.05294 0.03370 0.13808 0.12057 0.70888
INFLO 9 0.02239 0.00252 0.04906 0.02974 0.14801 0.13070 0.67363
COF 2 0.08955 0.07105 0.05017 0.03087 0.15159 0.13435 0.70074
COF 7 0.06716 0.04821 0.05372 0.03449 0.13357 0.11596 0.73034
COF 8 0.05970 0.04060 0.05343 0.03420 0.13592 0.11837 0.73114

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.3 MB) Download raw algorithm evaluation table (71.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.06618 0.04712 0.03962 0.02003 0.09442 0.07594 0.65749
KNNW 1 0.06618 0.04712 0.04547 0.02599 0.12364 0.10576 0.68755
LOF 1 0.06618 0.04712 0.03692 0.01727 0.08901 0.07043 0.62730
LOF 6 0.03676 0.01711 0.05461 0.03532 0.14222 0.12472 0.71762
LOF 8 0.03676 0.01711 0.05246 0.03313 0.14552 0.12809 0.71093
SimplifiedLOF 1 0.04412 0.02462 0.03745 0.01781 0.08830 0.06970 0.66608
SimplifiedLOF 7 0.03676 0.01711 0.05286 0.03354 0.12056 0.10261 0.74676
SimplifiedLOF 9 0.03676 0.01711 0.05360 0.03429 0.12931 0.11155 0.74616
SimplifiedLOF 10 0.03676 0.01711 0.05343 0.03412 0.13151 0.11379 0.74604
LoOP 3 0.10294 0.08464 0.04716 0.02772 0.11724 0.09923 0.70923
LoOP 7 0.08088 0.06213 0.05904 0.03985 0.13804 0.12046 0.75669
LoOP 10 0.07353 0.05463 0.06239 0.04327 0.15693 0.13973 0.75445
LDOF 9 0.11029 0.09214 0.07036 0.05139 0.15481 0.13757 0.77884
LDOF 18 0.14706 0.12966 0.07775 0.05893 0.17094 0.15403 0.76373
LDOF 21 0.12500 0.10715 0.08016 0.06139 0.17552 0.15870 0.75679
LDOF 26 0.13971 0.12215 0.08315 0.06445 0.16268 0.14560 0.74889
ODIN 13 0.11408 0.09601 0.06609 0.04704 0.15385 0.13658 0.73469
ODIN 19 0.12567 0.10783 0.06793 0.04891 0.15846 0.14129 0.72471
ODIN 23 0.11838 0.10040 0.06520 0.04613 0.17043 0.15350 0.71298
ODIN 49 0.15567 0.13845 0.06287 0.04375 0.15613 0.13892 0.68006
FastABOD 3 0.05147 0.03212 0.03325 0.01352 0.07513 0.05627 0.62704
FastABOD 4 0.05147 0.03212 0.03572 0.01605 0.08000 0.06123 0.63196
FastABOD 8 0.05147 0.03212 0.03455 0.01485 0.08240 0.06368 0.62548
KDEOS 24 0.10294 0.08464 0.05974 0.04056 0.12987 0.11212 0.74015
KDEOS 25 0.08824 0.06963 0.05983 0.04065 0.12698 0.10917 0.73730
LDF 2 0.09559 0.07714 0.04270 0.02317 0.12500 0.10715 0.61173
LDF 3 0.05882 0.03962 0.04894 0.02954 0.14376 0.12629 0.65222
LDF 5 0.02941 0.00961 0.04539 0.02592 0.12314 0.10525 0.67150
INFLO 3 0.05882 0.03962 0.04205 0.02251 0.11461 0.09655 0.66898
INFLO 8 0.05882 0.03962 0.05505 0.03577 0.14724 0.12984 0.71350
INFLO 11 0.05147 0.03212 0.05142 0.03207 0.14815 0.13077 0.66971
COF 4 0.05147 0.03212 0.04791 0.02848 0.10959 0.09142 0.72774
COF 6 0.05882 0.03962 0.05163 0.03228 0.13269 0.11499 0.72146
COF 10 0.07353 0.05463 0.05310 0.03378 0.13245 0.11475 0.71237
COF 12 0.08824 0.06963 0.05154 0.03219 0.12860 0.11082 0.70007

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO