Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Annthyroid (2% of outliers version#05)

This data set contains medical data on hypothyroidism. Three classes relate to the conditions normal, hyperfunction, and subnormal functioning. Classes other than normal condition were defined as outliers here.

Download all data set variants used (9.9 MB). You can also access the original data. (merge train and test [ann-test.data and ann-train.data])

Normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (58.4 MB) Download raw algorithm evaluation table (72.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.07463 0.05582 0.04392 0.02449 0.09623 0.07787 0.71190
KNNW 1 0.06716 0.04821 0.05008 0.03077 0.11111 0.09305 0.74033
LOF 1 0.07463 0.05582 0.03993 0.02042 0.10101 0.08274 0.62944
LOF 6 0.02985 0.01014 0.05777 0.03863 0.14679 0.12945 0.73905
LOF 8 0.02985 0.01014 0.05458 0.03537 0.15180 0.13457 0.72735
SimplifiedLOF 3 0.05224 0.03298 0.04972 0.03041 0.11382 0.09582 0.73275
SimplifiedLOF 6 0.02985 0.01014 0.05643 0.03725 0.13915 0.12166 0.76118
SimplifiedLOF 7 0.02985 0.01014 0.05791 0.03877 0.14615 0.12881 0.75642
LoOP 8 0.09701 0.07867 0.06166 0.04259 0.13821 0.12070 0.75730
LoOP 10 0.08209 0.06344 0.06252 0.04347 0.14717 0.12984 0.76202
LoOP 14 0.05970 0.04060 0.06107 0.04199 0.15606 0.13891 0.75044
LDOF 10 0.11940 0.10151 0.07451 0.05570 0.14360 0.12620 0.78744
LDOF 15 0.14179 0.12435 0.07776 0.05902 0.16254 0.14553 0.78263
LDOF 16 0.14925 0.13197 0.07614 0.05737 0.15225 0.13502 0.77958
LDOF 18 0.14179 0.12435 0.07900 0.06029 0.15714 0.14002 0.77458
ODIN 11 0.09664 0.07829 0.05728 0.03813 0.12059 0.10272 0.74492
ODIN 44 0.11387 0.09586 0.06127 0.04219 0.15802 0.14092 0.70414
ODIN 82 0.12758 0.10985 0.06225 0.04320 0.15103 0.13378 0.67659
ODIN 94 0.11791 0.09999 0.06291 0.04387 0.14987 0.13260 0.66913
FastABOD 3 0.04478 0.02537 0.04350 0.02407 0.08651 0.06795 0.70279
FastABOD 4 0.05970 0.04060 0.04703 0.02767 0.08625 0.06768 0.69869
FastABOD 5 0.06716 0.04821 0.04039 0.02089 0.08362 0.06500 0.69579
KDEOS 11 0.11940 0.10151 0.05656 0.03739 0.13263 0.11500 0.73819
KDEOS 14 0.09701 0.07867 0.05614 0.03697 0.13441 0.11682 0.74522
KDEOS 15 0.08955 0.07105 0.06018 0.04109 0.13127 0.11362 0.74980
KDEOS 18 0.10448 0.08628 0.06918 0.05027 0.11982 0.10193 0.74519
LDF 3 0.08955 0.07105 0.05043 0.03113 0.14118 0.12373 0.68157
LDF 4 0.02985 0.01014 0.05146 0.03219 0.16667 0.14973 0.67737
LDF 5 0.02985 0.01014 0.05245 0.03320 0.15584 0.13869 0.69983
INFLO 1 0.07463 0.05582 0.04053 0.02103 0.10119 0.08293 0.63885
INFLO 4 0.04478 0.02537 0.05331 0.03407 0.13477 0.11719 0.70511
INFLO 8 0.04478 0.02537 0.05531 0.03611 0.13933 0.12184 0.70391
INFLO 14 0.03731 0.01775 0.04967 0.03036 0.14754 0.13022 0.67857
COF 3 0.08209 0.06344 0.05073 0.03144 0.11633 0.09838 0.71512
COF 5 0.07463 0.05582 0.05489 0.03569 0.13212 0.11448 0.72889
COF 8 0.06716 0.04821 0.05820 0.03906 0.15625 0.13911 0.72133
COF 9 0.05224 0.03298 0.05727 0.03812 0.15638 0.13924 0.72055

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.9 MB) Download raw algorithm evaluation table (71.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05147 0.03212 0.04024 0.02066 0.09778 0.07937 0.68316
KNNW 1 0.05882 0.03962 0.04387 0.02436 0.09959 0.08121 0.69972
KNNW 2 0.05147 0.03212 0.04174 0.02219 0.10646 0.08823 0.69228
LOF 1 0.05882 0.03962 0.03102 0.01125 0.07792 0.05911 0.59443
LOF 5 0.02206 0.00211 0.04967 0.03029 0.12932 0.11156 0.73026
LOF 6 0.03676 0.01711 0.05054 0.03117 0.12613 0.10830 0.73219
LOF 7 0.02941 0.00961 0.04972 0.03033 0.11985 0.10189 0.73250
SimplifiedLOF 3 0.02941 0.00961 0.03719 0.01755 0.09786 0.07945 0.66110
SimplifiedLOF 8 0.02941 0.00961 0.04579 0.02632 0.11429 0.09622 0.72925
SimplifiedLOF 11 0.02941 0.00961 0.04659 0.02714 0.10403 0.08575 0.73733
LoOP 5 0.08088 0.06213 0.04774 0.02831 0.11966 0.10170 0.69537
LoOP 11 0.08088 0.06213 0.05165 0.03230 0.11538 0.09734 0.74104
LoOP 17 0.07353 0.05463 0.05255 0.03322 0.11765 0.09965 0.73779
LoOP 31 0.07353 0.05463 0.05092 0.03156 0.12273 0.10484 0.72414
LDOF 15 0.11029 0.09214 0.06299 0.04387 0.11444 0.09637 0.75086
LDOF 39 0.12500 0.10715 0.07008 0.05111 0.13622 0.11860 0.74930
LDOF 40 0.12500 0.10715 0.07044 0.05148 0.13924 0.12168 0.74844
ODIN 11 0.09559 0.07714 0.05340 0.03408 0.10817 0.08998 0.74003
ODIN 37 0.11159 0.09347 0.06446 0.04537 0.14213 0.12463 0.72960
ODIN 51 0.10948 0.09131 0.06110 0.04195 0.15060 0.13327 0.71499
ODIN 99 0.12166 0.10374 0.06179 0.04265 0.12844 0.11066 0.69031
FastABOD 4 0.02941 0.00961 0.03251 0.01277 0.07045 0.05148 0.65535
FastABOD 6 0.04412 0.02462 0.03388 0.01417 0.07532 0.05646 0.65177
FastABOD 17 0.04412 0.02462 0.03500 0.01531 0.07321 0.05430 0.63944
KDEOS 8 0.08088 0.06213 0.04639 0.02693 0.09816 0.07976 0.69672
KDEOS 25 0.08088 0.06213 0.05412 0.03482 0.10345 0.08516 0.72717
KDEOS 38 0.05147 0.03212 0.04705 0.02761 0.09756 0.07915 0.73392
KDEOS 58 0.04412 0.02462 0.04416 0.02466 0.10803 0.08983 0.72123
LDF 2 0.06618 0.04712 0.03483 0.01514 0.08230 0.06358 0.61755
LDF 4 0.05147 0.03212 0.04399 0.02448 0.10780 0.08960 0.69928
LDF 6 0.02941 0.00961 0.04796 0.02853 0.12565 0.10782 0.69328
INFLO 3 0.05882 0.03962 0.04175 0.02220 0.12251 0.10460 0.63542
INFLO 10 0.05882 0.03962 0.04519 0.02571 0.10896 0.09078 0.68681
INFLO 11 0.05882 0.03962 0.04547 0.02599 0.11111 0.09298 0.68336
INFLO 20 0.02941 0.00961 0.04191 0.02236 0.12405 0.10618 0.64250
COF 4 0.07353 0.05463 0.04232 0.02278 0.10223 0.08391 0.67335
COF 10 0.06618 0.04712 0.04798 0.02855 0.11417 0.09610 0.71233
COF 19 0.06618 0.04712 0.04829 0.02887 0.12147 0.10355 0.70758

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (57.8 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.04478 0.02537 0.03932 0.01980 0.08365 0.06503 0.69705
KNNW 1 0.05970 0.04060 0.04707 0.02770 0.09857 0.08025 0.73517
LOF 1 0.08209 0.06344 0.04441 0.02499 0.10120 0.08294 0.65266
LOF 5 0.02985 0.01014 0.05490 0.03570 0.14583 0.12848 0.71682
LOF 6 0.02985 0.01014 0.05483 0.03562 0.14604 0.12869 0.71965
SimplifiedLOF 1 0.05224 0.03298 0.04113 0.02165 0.09472 0.07633 0.69344
SimplifiedLOF 5 0.03731 0.01775 0.05630 0.03713 0.13293 0.11531 0.75588
SimplifiedLOF 6 0.02239 0.00252 0.05607 0.03689 0.13465 0.11707 0.75820
LoOP 9 0.06716 0.04821 0.06234 0.04329 0.14468 0.12730 0.76313
LoOP 11 0.06716 0.04821 0.06158 0.04251 0.15082 0.13357 0.76527
LoOP 12 0.08209 0.06344 0.06162 0.04256 0.14863 0.13133 0.76207
LDOF 12 0.11194 0.09390 0.07766 0.05892 0.15241 0.13519 0.79600
LDOF 20 0.14925 0.13197 0.08025 0.06156 0.16446 0.14748 0.78039
LDOF 21 0.13433 0.11674 0.07951 0.06081 0.16622 0.14928 0.77624
ODIN 9 0.11570 0.09773 0.06249 0.04344 0.13043 0.11277 0.74151
ODIN 33 0.12565 0.10789 0.06452 0.04551 0.17263 0.15582 0.69722
ODIN 50 0.15672 0.13958 0.06572 0.04673 0.16228 0.14526 0.68695
ODIN 61 0.15387 0.13668 0.06796 0.04902 0.15721 0.14008 0.67919
FastABOD 3 0.03731 0.01775 0.03581 0.01622 0.07874 0.06002 0.67078
FastABOD 4 0.03731 0.01775 0.03654 0.01697 0.07395 0.05513 0.67466
FastABOD 5 0.03731 0.01775 0.03708 0.01752 0.07309 0.05425 0.67396
FastABOD 16 0.05224 0.03298 0.03440 0.01478 0.07275 0.05391 0.66314
KDEOS 15 0.08955 0.07105 0.06378 0.04475 0.12202 0.10418 0.75807
KDEOS 21 0.11940 0.10151 0.06554 0.04656 0.13289 0.11527 0.75544
KDEOS 23 0.09701 0.07867 0.06480 0.04579 0.13439 0.11680 0.75730
LDF 1 0.06716 0.04821 0.03492 0.01531 0.09336 0.07494 0.58965
LDF 6 0.02239 0.00252 0.05002 0.03072 0.14026 0.12279 0.68464
INFLO 1 0.08209 0.06344 0.04895 0.02963 0.11368 0.09568 0.69756
INFLO 2 0.05970 0.04060 0.04855 0.02922 0.10368 0.08547 0.71704
INFLO 8 0.03731 0.01775 0.05241 0.03316 0.14141 0.12397 0.69916
INFLO 9 0.02985 0.01014 0.05007 0.03077 0.14672 0.12938 0.67401
COF 5 0.06716 0.04821 0.05423 0.03502 0.12804 0.11032 0.73552
COF 7 0.08209 0.06344 0.05348 0.03425 0.14203 0.12460 0.72570
COF 8 0.08955 0.07105 0.05349 0.03426 0.13983 0.12235 0.72231

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.3 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05147 0.03212 0.03807 0.01845 0.09325 0.07475 0.66786
KNNW 1 0.05882 0.03962 0.04317 0.02365 0.10741 0.08920 0.69317
LOF 1 0.04412 0.02462 0.03682 0.01717 0.10162 0.08329 0.61577
LOF 6 0.02941 0.00961 0.05747 0.03824 0.14671 0.12930 0.72816
LOF 9 0.02206 0.00211 0.05326 0.03394 0.14242 0.12493 0.72913
SimplifiedLOF 7 0.02206 0.00211 0.05435 0.03505 0.13212 0.11442 0.75618
SimplifiedLOF 8 0.02941 0.00961 0.05327 0.03395 0.13089 0.11316 0.75364
SimplifiedLOF 10 0.02941 0.00961 0.05409 0.03479 0.13879 0.12122 0.75670
LoOP 5 0.10294 0.08464 0.05773 0.03851 0.13922 0.12166 0.74429
LoOP 8 0.07353 0.05463 0.06090 0.04174 0.13559 0.11796 0.76627
LoOP 10 0.06618 0.04712 0.06222 0.04308 0.14262 0.12513 0.76460
LoOP 14 0.07353 0.05463 0.06053 0.04136 0.14590 0.12847 0.74965
LDOF 12 0.11765 0.09965 0.07498 0.05611 0.14625 0.12883 0.78652
LDOF 20 0.13971 0.12215 0.08181 0.06308 0.15521 0.13798 0.77414
LDOF 23 0.12500 0.10715 0.08233 0.06361 0.16629 0.14928 0.76570
LDOF 31 0.13971 0.12215 0.08604 0.06739 0.15309 0.13581 0.75912
ODIN 11 0.14495 0.12750 0.07930 0.06051 0.15359 0.13633 0.75679
ODIN 16 0.13330 0.11561 0.08553 0.06688 0.15314 0.13586 0.75072
ODIN 28 0.12102 0.10308 0.07580 0.05694 0.16790 0.15092 0.72511
ODIN 39 0.15556 0.13833 0.07241 0.05349 0.15730 0.14011 0.72139
FastABOD 3 0.03676 0.01711 0.03082 0.01105 0.07807 0.05926 0.62476
FastABOD 5 0.05147 0.03212 0.03382 0.01410 0.07792 0.05911 0.63791
FastABOD 6 0.05882 0.03962 0.03396 0.01425 0.07661 0.05777 0.63735
KDEOS 23 0.08824 0.06963 0.06170 0.04255 0.11554 0.09749 0.74741
KDEOS 32 0.06618 0.04712 0.05357 0.03426 0.10515 0.08690 0.75058
KDEOS 40 0.08088 0.06213 0.05398 0.03468 0.12435 0.10649 0.73768
KDEOS 66 0.10294 0.08464 0.05219 0.03286 0.12357 0.10569 0.71548
LDF 2 0.08824 0.06963 0.03972 0.02013 0.09412 0.07564 0.63659
LDF 6 0.02206 0.00211 0.04924 0.02984 0.13675 0.11914 0.69192
LDF 8 0.00735 -0.01290 0.04855 0.02914 0.14149 0.12398 0.69781
LDF 9 0.00735 -0.01290 0.04716 0.02772 0.13487 0.11722 0.69856
INFLO 4 0.05882 0.03962 0.05277 0.03344 0.14090 0.12337 0.72189
INFLO 6 0.04412 0.02462 0.05299 0.03367 0.13312 0.11543 0.72307
INFLO 10 0.03676 0.01711 0.05257 0.03324 0.14560 0.12817 0.69807
COF 5 0.07353 0.05463 0.04981 0.03043 0.11552 0.09748 0.71793
COF 10 0.06618 0.04712 0.05461 0.03533 0.13907 0.12151 0.72925
COF 13 0.06618 0.04712 0.05368 0.03437 0.14286 0.12537 0.71542

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO