Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Annthyroid (2% of outliers version#04)

This data set contains medical data on hypothyroidism. Three classes relate to the conditions normal, hyperfunction, and subnormal functioning. Classes other than normal condition were defined as outliers here.

Download all data set variants used (9.9 MB). You can also access the original data. (merge train and test [ann-test.data and ann-train.data])

Normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (58.4 MB) Download raw algorithm evaluation table (73.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.04478 0.02537 0.03538 0.01578 0.07854 0.05982 0.68000
KNN 4 0.00000 -0.02032 0.03266 0.01301 0.07946 0.06076 0.66398
KNNW 1 0.05224 0.03298 0.03963 0.02012 0.08427 0.06566 0.70073
LOF 1 0.04478 0.02537 0.03124 0.01156 0.07821 0.05948 0.60564
LOF 6 0.00746 -0.01270 0.04434 0.02492 0.11275 0.09472 0.70570
LOF 7 0.00746 -0.01270 0.04349 0.02405 0.11078 0.09271 0.70663
LOF 20 0.00000 -0.02032 0.04061 0.02112 0.12229 0.10446 0.68762
SimplifiedLOF 1 0.02985 0.01014 0.03008 0.01038 0.06768 0.04874 0.61817
SimplifiedLOF 8 0.00746 -0.01270 0.04347 0.02404 0.10707 0.08893 0.72412
SimplifiedLOF 10 0.00746 -0.01270 0.04310 0.02366 0.10704 0.08889 0.73139
SimplifiedLOF 22 0.00746 -0.01270 0.04105 0.02157 0.11217 0.09413 0.71969
LoOP 5 0.06716 0.04821 0.04204 0.02257 0.09841 0.08009 0.69806
LoOP 12 0.03731 0.01775 0.04474 0.02533 0.09974 0.08145 0.73136
LoOP 16 0.02985 0.01014 0.04525 0.02585 0.10536 0.08718 0.73103
LoOP 31 0.02985 0.01014 0.04336 0.02392 0.11373 0.09572 0.71277
LDOF 15 0.07463 0.05582 0.05215 0.03289 0.11568 0.09771 0.74679
LDOF 31 0.08209 0.06344 0.05672 0.03756 0.11905 0.10115 0.73906
LDOF 32 0.08955 0.07105 0.05634 0.03716 0.12375 0.10595 0.73522
LDOF 34 0.08955 0.07105 0.05599 0.03680 0.12632 0.10856 0.72995
ODIN 19 0.06242 0.04336 0.04540 0.02600 0.10292 0.08469 0.71119
ODIN 41 0.09222 0.07377 0.05206 0.03280 0.13536 0.11780 0.68653
ODIN 42 0.09370 0.07528 0.05256 0.03331 0.13518 0.11761 0.68737
ODIN 50 0.10000 0.08171 0.05113 0.03185 0.13147 0.11383 0.68059
FastABOD 4 0.03731 0.01775 0.03214 0.01248 0.06766 0.04872 0.66718
FastABOD 6 0.02985 0.01014 0.03223 0.01256 0.06564 0.04665 0.66534
FastABOD 10 0.04478 0.02537 0.03112 0.01144 0.06365 0.04462 0.65944
KDEOS 36 0.06716 0.04821 0.04608 0.02670 0.09091 0.07244 0.72836
KDEOS 42 0.08209 0.06344 0.04845 0.02911 0.09836 0.08004 0.72746
KDEOS 64 0.08955 0.07105 0.04448 0.02507 0.09492 0.07653 0.71562
KDEOS 86 0.06716 0.04821 0.04502 0.02562 0.10200 0.08376 0.70793
LDF 1 0.07463 0.05582 0.03163 0.01195 0.09615 0.07779 0.58768
LDF 3 0.05970 0.04060 0.04228 0.02282 0.12528 0.10751 0.64487
LDF 6 0.00746 -0.01270 0.04561 0.02622 0.12500 0.10722 0.70430
INFLO 3 0.04478 0.02537 0.03827 0.01873 0.10030 0.08202 0.64177
INFLO 4 0.03731 0.01775 0.04008 0.02058 0.10219 0.08395 0.67348
INFLO 8 0.02239 0.00252 0.04022 0.02072 0.10292 0.08470 0.67226
INFLO 28 0.00746 -0.01270 0.03526 0.01566 0.11309 0.09507 0.62093
COF 4 0.06716 0.04821 0.04032 0.02083 0.09438 0.07598 0.67063
COF 8 0.03731 0.01775 0.04199 0.02253 0.10214 0.08390 0.69888
COF 10 0.02985 0.01014 0.04086 0.02137 0.10282 0.08459 0.70230
COF 26 0.01493 -0.00509 0.03869 0.01916 0.11400 0.09600 0.68008

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.9 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.06618 0.04712 0.04473 0.02524 0.11694 0.09893 0.69108
KNNW 1 0.07353 0.05463 0.04901 0.02961 0.13196 0.11425 0.71248
LOF 1 0.09559 0.07714 0.04366 0.02415 0.11892 0.10094 0.64155
LOF 8 0.03676 0.01711 0.05600 0.03674 0.14576 0.12833 0.75910
LOF 11 0.03676 0.01711 0.05520 0.03592 0.15094 0.13362 0.74472
SimplifiedLOF 3 0.05882 0.03962 0.04371 0.02420 0.10598 0.08774 0.69832
SimplifiedLOF 9 0.03676 0.01711 0.05467 0.03538 0.12671 0.10890 0.76475
SimplifiedLOF 12 0.03676 0.01711 0.05458 0.03529 0.12975 0.11200 0.76645
SimplifiedLOF 16 0.03676 0.01711 0.05241 0.03308 0.13536 0.11772 0.75753
LoOP 5 0.11029 0.09214 0.05566 0.03639 0.12121 0.10328 0.73723
LoOP 12 0.06618 0.04712 0.06348 0.04437 0.14176 0.12425 0.77944
LoOP 14 0.07353 0.05463 0.06433 0.04524 0.14504 0.12760 0.77768
LoOP 23 0.08088 0.06213 0.06101 0.04185 0.14682 0.12941 0.76391
LDOF 23 0.11029 0.09214 0.07657 0.05773 0.14545 0.12802 0.78205
LDOF 35 0.13235 0.11465 0.07875 0.05995 0.15267 0.13538 0.76877
LDOF 40 0.14706 0.12966 0.08103 0.06228 0.14815 0.13077 0.76638
ODIN 17 0.10255 0.08424 0.06551 0.04644 0.14198 0.12448 0.78049
ODIN 50 0.10485 0.08658 0.06792 0.04891 0.16667 0.14966 0.71781
ODIN 97 0.13655 0.11894 0.06610 0.04705 0.15743 0.14024 0.68510
FastABOD 3 0.04412 0.02462 0.03731 0.01767 0.08885 0.07026 0.67678
FastABOD 6 0.04412 0.02462 0.03939 0.01980 0.09034 0.07178 0.66270
FastABOD 20 0.05882 0.03962 0.03797 0.01834 0.09619 0.07775 0.65016
FastABOD 24 0.06618 0.04712 0.03802 0.01839 0.09524 0.07678 0.64854
KDEOS 15 0.08824 0.06963 0.05648 0.03723 0.11074 0.09260 0.74969
KDEOS 19 0.06618 0.04712 0.06336 0.04425 0.11850 0.10052 0.75773
KDEOS 32 0.08824 0.06963 0.06155 0.04240 0.12521 0.10736 0.76652
KDEOS 39 0.08824 0.06963 0.05861 0.03940 0.11380 0.09572 0.77018
LDF 1 0.11765 0.09965 0.04541 0.02593 0.13636 0.11874 0.63584
LDF 4 0.05147 0.03212 0.05468 0.03539 0.15196 0.13466 0.71918
LDF 6 0.03676 0.01711 0.05190 0.03256 0.15578 0.13856 0.70569
INFLO 3 0.08824 0.06963 0.04790 0.02847 0.12077 0.10283 0.67713
INFLO 8 0.07353 0.05463 0.05569 0.03643 0.13712 0.11952 0.73502
INFLO 12 0.04412 0.02462 0.05625 0.03699 0.14162 0.12411 0.72494
INFLO 16 0.04412 0.02462 0.05242 0.03309 0.14587 0.12844 0.69113
COF 4 0.08088 0.06213 0.04554 0.02607 0.10687 0.08865 0.70012
COF 10 0.05882 0.03962 0.05366 0.03435 0.12903 0.11126 0.73951
COF 15 0.05882 0.03962 0.05470 0.03541 0.14159 0.12408 0.72642
COF 19 0.05147 0.03212 0.05386 0.03455 0.14447 0.12701 0.70303

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (57.8 MB) Download raw algorithm evaluation table (73.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.03731 0.01775 0.03410 0.01447 0.07597 0.05719 0.67969
KNN 3 0.00000 -0.02032 0.03107 0.01138 0.07602 0.05724 0.65110
KNNW 1 0.04478 0.02537 0.03946 0.01994 0.08661 0.06806 0.70743
LOF 3 0.03731 0.01775 0.03911 0.01959 0.11111 0.09305 0.65274
LOF 6 0.00746 -0.01270 0.04757 0.02822 0.12995 0.11227 0.70744
LOF 10 0.00000 -0.02032 0.04634 0.02696 0.12428 0.10648 0.71658
SimplifiedLOF 1 0.02239 0.00252 0.03136 0.01167 0.07273 0.05389 0.64999
SimplifiedLOF 7 0.00746 -0.01270 0.04509 0.02569 0.10363 0.08541 0.73977
SimplifiedLOF 11 0.00000 -0.02032 0.04539 0.02599 0.11494 0.09696 0.73960
SimplifiedLOF 12 0.00000 -0.02032 0.04493 0.02552 0.11609 0.09814 0.73713
LoOP 4 0.05970 0.04060 0.04375 0.02432 0.10405 0.08584 0.69486
LoOP 7 0.05224 0.03298 0.04754 0.02819 0.11002 0.09194 0.73850
LoOP 11 0.02985 0.01014 0.04793 0.02859 0.11257 0.09453 0.73560
LoOP 22 0.02985 0.01014 0.04674 0.02737 0.11844 0.10053 0.72786
LDOF 10 0.10448 0.08628 0.05555 0.03636 0.11828 0.10036 0.75075
LDOF 14 0.09701 0.07867 0.06039 0.04130 0.13191 0.11428 0.76986
LDOF 23 0.08955 0.07105 0.06460 0.04559 0.13817 0.12066 0.75977
LDOF 31 0.09701 0.07867 0.06623 0.04726 0.13283 0.11521 0.75401
ODIN 14 0.10746 0.08933 0.05535 0.03616 0.11881 0.10091 0.72335
ODIN 19 0.09453 0.07613 0.05629 0.03712 0.13406 0.11646 0.72920
ODIN 43 0.10255 0.08432 0.06129 0.04222 0.15261 0.13539 0.69948
ODIN 56 0.09587 0.07750 0.05663 0.03747 0.15542 0.13826 0.68443
FastABOD 4 0.03731 0.01775 0.02951 0.00979 0.06093 0.04185 0.64342
FastABOD 5 0.02985 0.01014 0.03020 0.01050 0.06288 0.04384 0.65134
KDEOS 14 0.07463 0.05582 0.05616 0.03698 0.08589 0.06732 0.71240
KDEOS 35 0.06716 0.04821 0.05117 0.03189 0.10445 0.08625 0.74495
KDEOS 38 0.06716 0.04821 0.05091 0.03162 0.11570 0.09773 0.73940
KDEOS 92 0.06716 0.04821 0.05758 0.03843 0.10366 0.08545 0.70757
LDF 2 0.05970 0.04060 0.03375 0.01412 0.08417 0.06556 0.61510
LDF 5 0.00000 -0.02032 0.04576 0.02637 0.12500 0.10722 0.69098
LDF 7 0.00000 -0.02032 0.04485 0.02544 0.11892 0.10102 0.69968
INFLO 3 0.04478 0.02537 0.03753 0.01798 0.10156 0.08331 0.65064
INFLO 6 0.02985 0.01014 0.04113 0.02165 0.11287 0.09485 0.66967
INFLO 12 0.00000 -0.02032 0.03906 0.01954 0.11945 0.10156 0.65770
COF 3 0.05224 0.03298 0.03890 0.01938 0.08736 0.06882 0.67688
COF 10 0.04478 0.02537 0.04490 0.02549 0.11015 0.09207 0.72225
COF 16 0.02985 0.01014 0.04269 0.02324 0.11667 0.09872 0.69943

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.3 MB) Download raw algorithm evaluation table (71.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05147 0.03212 0.04228 0.02274 0.10795 0.08975 0.68229
KNNW 1 0.07353 0.05463 0.04598 0.02652 0.12081 0.10287 0.69512
LOF 1 0.08088 0.06213 0.03824 0.01862 0.10680 0.08857 0.60834
LOF 6 0.02206 0.00211 0.05649 0.03725 0.15822 0.14104 0.72905
LOF 8 0.02206 0.00211 0.05537 0.03609 0.16296 0.14589 0.72695
LOF 9 0.02206 0.00211 0.05287 0.03355 0.15282 0.13554 0.72984
SimplifiedLOF 1 0.02941 0.00961 0.03605 0.01638 0.08802 0.06941 0.65089
SimplifiedLOF 8 0.02206 0.00211 0.05567 0.03641 0.13854 0.12097 0.75511
SimplifiedLOF 10 0.02206 0.00211 0.05493 0.03565 0.14103 0.12350 0.75570
SimplifiedLOF 11 0.02206 0.00211 0.05475 0.03547 0.13871 0.12114 0.75602
LoOP 5 0.11029 0.09214 0.05955 0.04036 0.13830 0.12072 0.75540
LoOP 8 0.08824 0.06963 0.06445 0.04536 0.14737 0.12997 0.76656
LoOP 11 0.05882 0.03962 0.06331 0.04420 0.15726 0.14006 0.76817
LoOP 12 0.08088 0.06213 0.06424 0.04515 0.15663 0.13942 0.76949
LDOF 14 0.12500 0.10715 0.07503 0.05616 0.15470 0.13745 0.78236
LDOF 31 0.13971 0.12215 0.08865 0.07005 0.16152 0.14441 0.75847
LDOF 39 0.16176 0.14466 0.08430 0.06562 0.16312 0.14605 0.74950
LDOF 44 0.15441 0.13716 0.08082 0.06207 0.17143 0.15452 0.74097
ODIN 11 0.12793 0.11014 0.07243 0.05350 0.14004 0.12250 0.75953
ODIN 12 0.13228 0.11458 0.07447 0.05558 0.15023 0.13290 0.75952
ODIN 22 0.12373 0.10585 0.07185 0.05292 0.18511 0.16849 0.73170
ODIN 93 0.16820 0.15123 0.06998 0.05101 0.17109 0.15418 0.67398
FastABOD 5 0.04412 0.02462 0.03537 0.01569 0.08194 0.06321 0.64243
FastABOD 6 0.05147 0.03212 0.03598 0.01631 0.08257 0.06385 0.64072
FastABOD 7 0.05882 0.03962 0.03524 0.01556 0.08757 0.06896 0.63873
FastABOD 8 0.05882 0.03962 0.03512 0.01543 0.08978 0.07121 0.63286
KDEOS 20 0.09559 0.07714 0.06664 0.04760 0.11454 0.09647 0.75090
KDEOS 21 0.08824 0.06963 0.06278 0.04366 0.11700 0.09899 0.75452
KDEOS 35 0.09559 0.07714 0.05647 0.03722 0.10795 0.08975 0.75876
KDEOS 72 0.11029 0.09214 0.06011 0.04094 0.11111 0.09298 0.72780
LDF 2 0.11029 0.09214 0.04727 0.02783 0.12162 0.10370 0.66268
LDF 3 0.09559 0.07714 0.05297 0.03365 0.15461 0.13737 0.67403
LDF 5 0.01471 -0.00540 0.05109 0.03173 0.14403 0.12657 0.70940
INFLO 1 0.06618 0.04712 0.03809 0.01847 0.09811 0.07971 0.64341
INFLO 4 0.03676 0.01711 0.05093 0.03157 0.12950 0.11174 0.71879
INFLO 8 0.04412 0.02462 0.05552 0.03625 0.15175 0.13444 0.70744
INFLO 11 0.02941 0.00961 0.05305 0.03373 0.15608 0.13886 0.68100
COF 8 0.09559 0.07714 0.05478 0.03549 0.13248 0.11478 0.72615
COF 20 0.04412 0.02462 0.04644 0.02698 0.14196 0.12446 0.67698

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO