Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Annthyroid (2% of outliers version#01)

This data set contains medical data on hypothyroidism. Three classes relate to the conditions normal, hyperfunction, and subnormal functioning. Classes other than normal condition were defined as outliers here.

Download all data set variants used (9.9 MB). You can also access the original data. (merge train and test [ann-test.data and ann-train.data])

Normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (58.4 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.07463 0.05582 0.04251 0.02305 0.11720 0.09927 0.65554
KNNW 1 0.09701 0.07867 0.04527 0.02587 0.11080 0.09274 0.67769
KNNW 2 0.07463 0.05582 0.04340 0.02397 0.11662 0.09867 0.66690
LOF 2 0.07463 0.05582 0.04105 0.02157 0.10902 0.09092 0.61921
LOF 6 0.02985 0.01014 0.05356 0.03433 0.14318 0.12577 0.71664
LOF 8 0.04478 0.02537 0.05219 0.03293 0.14403 0.12664 0.72218
LOF 11 0.03731 0.01775 0.05020 0.03090 0.14830 0.13099 0.70935
SimplifiedLOF 3 0.05970 0.04060 0.04282 0.02337 0.10633 0.08817 0.67607
SimplifiedLOF 8 0.03731 0.01775 0.05265 0.03340 0.12500 0.10722 0.74166
SimplifiedLOF 9 0.03731 0.01775 0.05311 0.03387 0.12500 0.10722 0.74106
SimplifiedLOF 14 0.04478 0.02537 0.04984 0.03053 0.13504 0.11746 0.72870
LoOP 8 0.05970 0.04060 0.05469 0.03548 0.12371 0.10591 0.74909
LoOP 18 0.08209 0.06344 0.05694 0.03778 0.15315 0.13595 0.73471
LoOP 21 0.08955 0.07105 0.05629 0.03711 0.14624 0.12889 0.73729
LoOP 67 0.06716 0.04821 0.06149 0.04242 0.10773 0.08960 0.68706
LDOF 14 0.10448 0.08628 0.07020 0.05131 0.15000 0.13273 0.76194
LDOF 17 0.13433 0.11674 0.07496 0.05617 0.17035 0.15349 0.75869
LDOF 19 0.15672 0.13958 0.07505 0.05626 0.15970 0.14262 0.75788
LDOF 86 0.11940 0.10151 0.07935 0.06064 0.13521 0.11764 0.70112
ODIN 19 0.08668 0.06812 0.05341 0.03418 0.12295 0.10513 0.73078
ODIN 56 0.14272 0.12531 0.06051 0.04142 0.15457 0.13739 0.68732
ODIN 66 0.12687 0.10912 0.06133 0.04226 0.16484 0.14787 0.67613
ODIN 67 0.12687 0.10912 0.06100 0.04192 0.16667 0.14973 0.67499
FastABOD 4 0.06716 0.04821 0.03875 0.01922 0.09766 0.07932 0.65254
FastABOD 17 0.05970 0.04060 0.04037 0.02087 0.10283 0.08460 0.63614
FastABOD 31 0.05970 0.04060 0.03633 0.01675 0.10336 0.08514 0.63123
KDEOS 24 0.11194 0.09390 0.05647 0.03730 0.12885 0.11115 0.71949
KDEOS 26 0.11194 0.09390 0.06231 0.04326 0.13468 0.11710 0.72276
KDEOS 27 0.11194 0.09390 0.06335 0.04432 0.11475 0.09677 0.72456
KDEOS 37 0.07463 0.05582 0.05579 0.03660 0.10622 0.08806 0.74161
LDF 3 0.08209 0.06344 0.04336 0.02392 0.13408 0.11648 0.63459
LDF 5 0.04478 0.02537 0.05280 0.03356 0.14141 0.12397 0.69951
INFLO 2 0.07463 0.05582 0.03846 0.01893 0.09524 0.07685 0.63831
INFLO 8 0.04478 0.02537 0.05156 0.03229 0.12971 0.11202 0.70038
INFLO 14 0.04478 0.02537 0.04734 0.02799 0.14499 0.12762 0.66229
COF 11 0.06716 0.04821 0.05500 0.03580 0.14403 0.12664 0.71087
COF 13 0.07463 0.05582 0.05461 0.03541 0.14498 0.12761 0.70329
COF 26 0.08955 0.07105 0.04856 0.02923 0.11675 0.09881 0.68803

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.9 MB) Download raw algorithm evaluation table (70.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.04412 0.02462 0.03922 0.01962 0.10543 0.08718 0.66118
KNNW 1 0.05882 0.03962 0.04108 0.02151 0.10629 0.08806 0.66550
KNNW 2 0.04412 0.02462 0.03974 0.02014 0.10714 0.08893 0.66397
LOF 1 0.05147 0.03212 0.02881 0.00900 0.08656 0.06792 0.53362
LOF 7 0.00735 -0.01290 0.04283 0.02330 0.10509 0.08683 0.71353
LOF 11 0.00735 -0.01290 0.04594 0.02647 0.13758 0.11999 0.70738
SimplifiedLOF 12 0.00735 -0.01290 0.04480 0.02531 0.11618 0.09815 0.72965
SimplifiedLOF 13 0.00735 -0.01290 0.04494 0.02546 0.11631 0.09828 0.72690
SimplifiedLOF 14 0.00735 -0.01290 0.04493 0.02544 0.12121 0.10328 0.72518
SimplifiedLOF 94 0.05147 0.03212 0.03086 0.01109 0.07414 0.05525 0.64419
LoOP 14 0.03676 0.01711 0.04931 0.02991 0.12500 0.10715 0.73339
LoOP 22 0.04412 0.02462 0.04904 0.02964 0.14286 0.12537 0.72448
LoOP 49 0.07353 0.05463 0.04294 0.02341 0.10765 0.08944 0.69028
LoOP 99 0.05882 0.03962 0.05195 0.03261 0.09449 0.07601 0.65489
LDOF 14 0.07353 0.05463 0.05685 0.03761 0.11940 0.10144 0.76607
LDOF 31 0.09559 0.07714 0.06307 0.04396 0.16895 0.15199 0.73688
LDOF 50 0.13971 0.12215 0.06042 0.04125 0.14545 0.12802 0.71421
ODIN 18 0.04412 0.02462 0.04339 0.02388 0.09585 0.07740 0.72095
ODIN 94 0.10504 0.08678 0.04655 0.02710 0.12105 0.10312 0.65172
ODIN 100 0.10729 0.08907 0.04717 0.02773 0.11825 0.10026 0.65141
FastABOD 4 0.03676 0.01711 0.03290 0.01317 0.08089 0.06214 0.63589
FastABOD 5 0.02941 0.00961 0.03305 0.01332 0.08356 0.06486 0.63577
FastABOD 6 0.02941 0.00961 0.03282 0.01308 0.08443 0.06575 0.63345
KDEOS 16 0.10294 0.08464 0.05436 0.03507 0.10526 0.08701 0.70606
KDEOS 18 0.08824 0.06963 0.05992 0.04074 0.10468 0.08642 0.70946
KDEOS 19 0.10294 0.08464 0.05738 0.03815 0.11470 0.09663 0.70194
KDEOS 31 0.05882 0.03962 0.04414 0.02464 0.09390 0.07541 0.71310
LDF 3 0.07353 0.05463 0.03512 0.01544 0.09740 0.07899 0.61337
LDF 6 0.02206 0.00211 0.04133 0.02177 0.11924 0.10127 0.66472
LDF 7 0.00000 -0.02040 0.03887 0.01926 0.10864 0.09046 0.66590
INFLO 1 0.05147 0.03212 0.02907 0.00926 0.08497 0.06630 0.54852
INFLO 8 0.01471 -0.00540 0.03987 0.02028 0.10475 0.08649 0.67076
INFLO 13 0.02206 0.00211 0.04410 0.02460 0.13243 0.11473 0.66341
INFLO 15 0.02206 0.00211 0.04257 0.02304 0.14054 0.12301 0.64392
COF 13 0.03676 0.01711 0.03982 0.02023 0.10212 0.08380 0.68249
COF 16 0.03676 0.01711 0.04021 0.02062 0.11826 0.10027 0.66420
COF 28 0.08088 0.06213 0.03939 0.01979 0.10417 0.08589 0.64752
COF 69 0.07353 0.05463 0.04480 0.02531 0.07895 0.06016 0.64794

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (57.8 MB) Download raw algorithm evaluation table (72.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05224 0.03298 0.04123 0.02175 0.12227 0.10444 0.66008
KNNW 1 0.10448 0.08628 0.04476 0.02535 0.11155 0.09350 0.68662
LOF 2 0.08209 0.06344 0.04795 0.02861 0.11603 0.09807 0.66937
LOF 4 0.04478 0.02537 0.05387 0.03465 0.15510 0.13794 0.70301
LOF 8 0.02985 0.01014 0.04988 0.03058 0.16107 0.14403 0.69482
SimplifiedLOF 1 0.05224 0.03298 0.03461 0.01499 0.08160 0.06294 0.63829
SimplifiedLOF 6 0.02985 0.01014 0.05732 0.03816 0.13265 0.11503 0.76047
SimplifiedLOF 9 0.02985 0.01014 0.05361 0.03438 0.14062 0.12316 0.74701
LoOP 4 0.10448 0.08628 0.05686 0.03770 0.13065 0.11299 0.74419
LoOP 6 0.09701 0.07867 0.06192 0.04286 0.13292 0.11530 0.76613
LoOP 15 0.09701 0.07867 0.05809 0.03896 0.15385 0.13665 0.73471
LDOF 12 0.14179 0.12435 0.07301 0.05418 0.15441 0.13723 0.79234
LDOF 19 0.15672 0.13958 0.08079 0.06211 0.17632 0.15959 0.77770
LDOF 20 0.15672 0.13958 0.08068 0.06200 0.18085 0.16421 0.77725
LDOF 32 0.14925 0.13197 0.08473 0.06613 0.17241 0.15560 0.74433
ODIN 13 0.10775 0.08962 0.05944 0.04033 0.14035 0.12288 0.72529
ODIN 36 0.16339 0.14640 0.07021 0.05132 0.17955 0.16288 0.69052
ODIN 37 0.17164 0.15481 0.06995 0.05105 0.17664 0.15991 0.69006
ODIN 43 0.16554 0.14858 0.06919 0.05028 0.18349 0.16690 0.68482
FastABOD 4 0.07463 0.05582 0.03636 0.01678 0.08485 0.06625 0.64524
FastABOD 6 0.05970 0.04060 0.03750 0.01794 0.08783 0.06930 0.64108
FastABOD 40 0.04478 0.02537 0.03401 0.01438 0.09337 0.07494 0.61896
KDEOS 13 0.11194 0.09390 0.05620 0.03703 0.11321 0.09519 0.74614
KDEOS 21 0.08955 0.07105 0.05771 0.03856 0.11211 0.09407 0.74966
KDEOS 33 0.05970 0.04060 0.05553 0.03634 0.12598 0.10823 0.74180
KDEOS 40 0.07463 0.05582 0.06047 0.04138 0.11321 0.09519 0.73363
LDF 2 0.11194 0.09390 0.04688 0.02751 0.14375 0.12635 0.62635
LDF 3 0.07463 0.05582 0.04720 0.02784 0.14067 0.12321 0.65453
LDF 4 0.05224 0.03298 0.04619 0.02681 0.14684 0.12950 0.65998
LDF 7 0.02239 0.00252 0.04179 0.02232 0.12357 0.10576 0.66050
INFLO 3 0.08209 0.06344 0.04776 0.02841 0.11111 0.09305 0.68426
INFLO 4 0.05970 0.04060 0.05261 0.03336 0.13043 0.11277 0.70289
INFLO 6 0.05224 0.03298 0.05212 0.03287 0.13913 0.12164 0.70592
INFLO 8 0.04478 0.02537 0.05220 0.03294 0.14770 0.13039 0.69478
COF 4 0.09701 0.07867 0.05271 0.03346 0.12269 0.10486 0.70549
COF 6 0.08955 0.07105 0.05601 0.03683 0.13761 0.12009 0.71661
COF 9 0.07463 0.05582 0.05326 0.03402 0.14545 0.12809 0.70743

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.3 MB) Download raw algorithm evaluation table (71.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.03676 0.01711 0.03801 0.01839 0.09982 0.08146 0.65312
KNNW 1 0.05147 0.03212 0.04093 0.02136 0.10297 0.08467 0.66735
LOF 1 0.02941 0.00961 0.03042 0.01064 0.08736 0.06874 0.56985
LOF 10 0.00735 -0.01290 0.04866 0.02925 0.14618 0.12876 0.71712
SimplifiedLOF 1 0.01471 -0.00540 0.03520 0.01552 0.09451 0.07603 0.62842
SimplifiedLOF 10 0.00735 -0.01290 0.04996 0.03058 0.12679 0.10897 0.74114
SimplifiedLOF 12 0.00735 -0.01290 0.04842 0.02900 0.13028 0.11254 0.73319
LoOP 10 0.04412 0.02462 0.05351 0.03420 0.12903 0.11126 0.74319
LoOP 13 0.04412 0.02462 0.05390 0.03460 0.14000 0.12245 0.73768
LoOP 19 0.05882 0.03962 0.05291 0.03358 0.14909 0.13173 0.73286
LoOP 32 0.08088 0.06213 0.04812 0.02870 0.12717 0.10936 0.70490
LDOF 17 0.11765 0.09965 0.06883 0.04983 0.14493 0.12748 0.78159
LDOF 31 0.14706 0.12966 0.07833 0.05953 0.17222 0.15533 0.75109
LDOF 40 0.16912 0.15217 0.07581 0.05696 0.17094 0.15403 0.73416
LDOF 44 0.16912 0.15217 0.07364 0.05474 0.18254 0.16586 0.72590
ODIN 13 0.08566 0.06701 0.05270 0.03337 0.11836 0.10037 0.73490
ODIN 45 0.10649 0.08826 0.05451 0.03522 0.15063 0.13330 0.67575
ODIN 73 0.12868 0.11090 0.05413 0.03483 0.14388 0.12642 0.65255
ODIN 100 0.11765 0.09965 0.05649 0.03724 0.14097 0.12344 0.65409
FastABOD 5 0.02941 0.00961 0.03183 0.01208 0.07231 0.05339 0.62971
FastABOD 7 0.02206 0.00211 0.03221 0.01247 0.07282 0.05390 0.63136
KDEOS 19 0.08088 0.06213 0.05376 0.03445 0.12143 0.10350 0.73097
KDEOS 21 0.08824 0.06963 0.05203 0.03269 0.12022 0.10227 0.73033
KDEOS 23 0.08824 0.06963 0.05235 0.03302 0.12429 0.10643 0.73176
KDEOS 24 0.08824 0.06963 0.05251 0.03318 0.11282 0.09472 0.73561
LDF 3 0.06618 0.04712 0.03791 0.01828 0.11392 0.09585 0.61313
LDF 5 0.01471 -0.00540 0.04149 0.02193 0.11915 0.10118 0.66314
LDF 6 0.01471 -0.00540 0.04244 0.02291 0.13187 0.11416 0.65501
INFLO 1 0.03676 0.01711 0.03419 0.01449 0.09185 0.07332 0.60658
INFLO 10 0.00735 -0.01290 0.04700 0.02756 0.13765 0.12006 0.68239
INFLO 12 0.00735 -0.01290 0.04686 0.02742 0.14085 0.12332 0.67788
COF 11 0.04412 0.02462 0.04634 0.02688 0.12646 0.10863 0.70786
COF 13 0.03676 0.01711 0.04521 0.02573 0.13008 0.11233 0.69879
COF 65 0.05882 0.03962 0.03667 0.01702 0.07528 0.05642 0.63055

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO