Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Annthyroid (2% of outliers version#02)

This data set contains medical data on hypothyroidism. Three classes relate to the conditions normal, hyperfunction, and subnormal functioning. Classes other than normal condition were defined as outliers here.

Download all data set variants used (9.9 MB). You can also access the original data. (merge train and test [ann-test.data and ann-train.data])

Normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (58.4 MB) Download raw algorithm evaluation table (73.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05224 0.03298 0.03848 0.01895 0.09383 0.07542 0.68380
KNNW 1 0.02985 0.01014 0.03969 0.02018 0.09677 0.07842 0.70063
KNNW 2 0.05224 0.03298 0.03945 0.01993 0.09689 0.07854 0.69279
LOF 2 0.07463 0.05582 0.05095 0.03166 0.12708 0.10934 0.70084
LOF 5 0.02985 0.01014 0.05878 0.03966 0.16610 0.14916 0.72966
SimplifiedLOF 4 0.05970 0.04060 0.05163 0.03237 0.12963 0.11195 0.74118
SimplifiedLOF 5 0.04478 0.02537 0.05556 0.03637 0.13859 0.12108 0.76204
SimplifiedLOF 6 0.02985 0.01014 0.05645 0.03728 0.13929 0.12180 0.76065
SimplifiedLOF 9 0.03731 0.01775 0.05466 0.03546 0.14694 0.12961 0.75208
LoOP 5 0.09701 0.07867 0.05965 0.04055 0.13824 0.12073 0.76193
LoOP 9 0.08209 0.06344 0.05970 0.04060 0.15210 0.13488 0.74997
LoOP 60 0.04478 0.02537 0.06156 0.04249 0.08902 0.07051 0.68035
LDOF 6 0.10448 0.08628 0.06471 0.04570 0.13730 0.11977 0.77035
LDOF 10 0.09701 0.07867 0.06899 0.05008 0.14579 0.12844 0.77586
LDOF 13 0.09701 0.07867 0.07103 0.05215 0.15896 0.14188 0.76613
ODIN 8 0.07471 0.05591 0.05043 0.03113 0.14052 0.12305 0.69973
ODIN 16 0.10733 0.08919 0.05620 0.03702 0.13103 0.11338 0.72437
ODIN 21 0.08366 0.06504 0.05929 0.04018 0.13231 0.11468 0.73469
ODIN 23 0.07794 0.05921 0.05979 0.04069 0.13429 0.11670 0.72892
FastABOD 4 0.02985 0.01014 0.03369 0.01405 0.07375 0.05493 0.66758
FastABOD 5 0.02985 0.01014 0.03336 0.01372 0.07517 0.05638 0.66395
FastABOD 7 0.03731 0.01775 0.03282 0.01317 0.07492 0.05613 0.66100
KDEOS 10 0.11940 0.10151 0.05474 0.03553 0.13091 0.11325 0.72964
KDEOS 16 0.09701 0.07867 0.05288 0.03364 0.12022 0.10234 0.73731
KDEOS 23 0.08955 0.07105 0.05588 0.03670 0.10230 0.08406 0.72797
LDF 1 0.05970 0.04060 0.02848 0.00874 0.07198 0.05312 0.53483
LDF 4 0.04478 0.02537 0.04841 0.02907 0.13443 0.11684 0.69851
LDF 5 0.02985 0.01014 0.04861 0.02928 0.12789 0.11017 0.69197
INFLO 1 0.08209 0.06344 0.03790 0.01835 0.11299 0.09497 0.59714
INFLO 6 0.05224 0.03298 0.05474 0.03554 0.15168 0.13444 0.71571
INFLO 9 0.04478 0.02537 0.05307 0.03383 0.15675 0.13962 0.67104
COF 3 0.06716 0.04821 0.04677 0.02740 0.11222 0.09419 0.70429
COF 5 0.05970 0.04060 0.05002 0.03072 0.11521 0.09724 0.74329
COF 6 0.05970 0.04060 0.05122 0.03195 0.13519 0.11762 0.72994
COF 7 0.05224 0.03298 0.05105 0.03177 0.13611 0.11855 0.72004

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.9 MB) Download raw algorithm evaluation table (71.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05147 0.03212 0.04259 0.02305 0.10095 0.08261 0.69437
KNNW 1 0.07353 0.05463 0.04733 0.02789 0.11041 0.09226 0.71505
LOF 1 0.06618 0.04712 0.03681 0.01716 0.09456 0.07609 0.63033
LOF 6 0.02206 0.00211 0.05445 0.03516 0.13876 0.12119 0.73261
LOF 7 0.02206 0.00211 0.05372 0.03441 0.13371 0.11604 0.73785
LOF 12 0.02206 0.00211 0.05186 0.03252 0.15431 0.13706 0.71195
SimplifiedLOF 11 0.02206 0.00211 0.05182 0.03247 0.12218 0.10427 0.74474
SimplifiedLOF 23 0.01471 -0.00540 0.04629 0.02683 0.13947 0.12192 0.71221
SimplifiedLOF 96 0.03676 0.01711 0.03135 0.01159 0.07891 0.06012 0.65990
LoOP 5 0.08088 0.06213 0.04996 0.03058 0.11751 0.09950 0.72701
LoOP 12 0.05882 0.03962 0.05868 0.03947 0.13964 0.12209 0.74931
LoOP 13 0.05147 0.03212 0.05869 0.03949 0.14177 0.12426 0.74644
LoOP 26 0.05147 0.03212 0.05407 0.03477 0.14953 0.13218 0.71860
LDOF 13 0.13235 0.11465 0.06677 0.04773 0.13776 0.12016 0.75308
LDOF 16 0.13235 0.11465 0.06839 0.04939 0.15000 0.13266 0.75774
LDOF 25 0.12500 0.10715 0.07321 0.05430 0.15730 0.14011 0.75367
LDOF 38 0.12500 0.10715 0.07010 0.05113 0.16438 0.14734 0.74645
ODIN 15 0.10822 0.09002 0.05655 0.03730 0.12466 0.10680 0.73646
ODIN 51 0.10131 0.08297 0.06493 0.04586 0.15805 0.14088 0.68272
ODIN 61 0.10735 0.08914 0.06587 0.04681 0.14617 0.12875 0.68159
ODIN 95 0.12724 0.10943 0.05929 0.04010 0.12871 0.11094 0.67950
FastABOD 5 0.04412 0.02462 0.03538 0.01570 0.07837 0.05956 0.67567
FastABOD 8 0.04412 0.02462 0.03468 0.01498 0.07940 0.06062 0.66904
KDEOS 14 0.09559 0.07714 0.05003 0.03065 0.12033 0.10238 0.72385
KDEOS 15 0.08824 0.06963 0.05287 0.03354 0.12576 0.10792 0.72303
KDEOS 23 0.08088 0.06213 0.06051 0.04134 0.10143 0.08309 0.72350
KDEOS 35 0.07353 0.05463 0.05600 0.03674 0.10619 0.08796 0.73713
LDF 1 0.07353 0.05463 0.03313 0.01341 0.08443 0.06575 0.60040
LDF 4 0.03676 0.01711 0.05051 0.03114 0.14097 0.12344 0.69487
LDF 5 0.01471 -0.00540 0.04907 0.02967 0.13457 0.11691 0.69977
INFLO 2 0.05147 0.03212 0.04044 0.02086 0.11111 0.09298 0.66510
INFLO 6 0.03676 0.01711 0.04979 0.03041 0.12722 0.10941 0.71273
INFLO 11 0.02941 0.00961 0.05176 0.03241 0.13743 0.11983 0.70370
INFLO 18 0.02941 0.00961 0.04692 0.02747 0.14529 0.12785 0.66575
COF 8 0.05147 0.03212 0.04950 0.03010 0.11508 0.09703 0.73088
COF 10 0.06618 0.04712 0.04989 0.03050 0.12482 0.10696 0.72939
COF 12 0.06618 0.04712 0.05147 0.03212 0.13592 0.11829 0.71930
COF 13 0.05882 0.03962 0.05150 0.03215 0.13270 0.11501 0.72372

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (57.8 MB) Download raw algorithm evaluation table (73.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.04478 0.02537 0.03704 0.01747 0.08631 0.06775 0.68000
KNNW 1 0.03731 0.01775 0.03916 0.01964 0.08824 0.06971 0.70118
KNNW 2 0.04478 0.02537 0.03803 0.01848 0.08696 0.06840 0.69052
LOF 1 0.08209 0.06344 0.03781 0.01826 0.10082 0.08255 0.61766
LOF 3 0.04478 0.02537 0.06054 0.04145 0.16154 0.14450 0.71207
LOF 6 0.02985 0.01014 0.05864 0.03951 0.15973 0.14266 0.72964
SimplifiedLOF 2 0.05970 0.04060 0.04221 0.02275 0.10084 0.08257 0.69789
SimplifiedLOF 5 0.02985 0.01014 0.05897 0.03985 0.16012 0.14306 0.75560
SimplifiedLOF 6 0.02985 0.01014 0.05945 0.04034 0.15576 0.13861 0.75735
SimplifiedLOF 7 0.03731 0.01775 0.05832 0.03919 0.14956 0.13228 0.76150
LoOP 3 0.08955 0.07105 0.05713 0.03797 0.12414 0.10634 0.73734
LoOP 5 0.07463 0.05582 0.06440 0.04539 0.17143 0.15459 0.75367
LoOP 7 0.08209 0.06344 0.06379 0.04477 0.16348 0.14648 0.76356
LDOF 9 0.11940 0.10151 0.07166 0.05280 0.16320 0.14620 0.78255
LDOF 14 0.09701 0.07867 0.07620 0.05743 0.16015 0.14308 0.78894
LDOF 19 0.11940 0.10151 0.07779 0.05905 0.17423 0.15745 0.77713
LDOF 24 0.11194 0.09390 0.08120 0.06253 0.16112 0.14408 0.76934
ODIN 17 0.11547 0.09750 0.06771 0.04876 0.16142 0.14438 0.73622
ODIN 20 0.11712 0.09918 0.06818 0.04924 0.15281 0.13560 0.73547
ODIN 79 0.12801 0.11030 0.05930 0.04019 0.14205 0.12461 0.65615
FastABOD 4 0.02985 0.01014 0.03128 0.01160 0.07094 0.05206 0.64449
FastABOD 5 0.03731 0.01775 0.03159 0.01191 0.07064 0.05175 0.64885
KDEOS 11 0.11194 0.09390 0.06539 0.04640 0.12552 0.10776 0.74229
KDEOS 15 0.10448 0.08628 0.05837 0.03924 0.11986 0.10198 0.74373
KDEOS 23 0.10448 0.08628 0.06244 0.04339 0.12968 0.11199 0.73870
LDF 1 0.06716 0.04821 0.03308 0.01343 0.08571 0.06714 0.58912
LDF 4 0.03731 0.01775 0.04710 0.02773 0.13827 0.12076 0.68192
LDF 6 0.02239 0.00252 0.04689 0.02752 0.12362 0.10581 0.69681
INFLO 1 0.08209 0.06344 0.04062 0.02113 0.10185 0.08360 0.64483
INFLO 4 0.03731 0.01775 0.05975 0.04064 0.16772 0.15081 0.73665
COF 2 0.06716 0.04821 0.04095 0.02146 0.09281 0.07437 0.68495
COF 5 0.04478 0.02537 0.05259 0.03334 0.13527 0.11770 0.72728
COF 6 0.02985 0.01014 0.05282 0.03357 0.13869 0.12119 0.72293
COF 7 0.04478 0.02537 0.05179 0.03252 0.13893 0.12144 0.72355

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.3 MB) Download raw algorithm evaluation table (72.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.03676 0.01711 0.04103 0.02146 0.09633 0.07789 0.69063
KNNW 1 0.05882 0.03962 0.04570 0.02623 0.10460 0.08633 0.71732
LOF 1 0.05882 0.03962 0.03838 0.01876 0.10309 0.08479 0.62806
LOF 6 0.01471 -0.00540 0.06235 0.04322 0.17174 0.15484 0.72892
LOF 10 0.02206 0.00211 0.05754 0.03831 0.17771 0.16093 0.72185
SimplifiedLOF 2 0.02206 0.00211 0.04299 0.02346 0.11399 0.09591 0.70003
SimplifiedLOF 8 0.02206 0.00211 0.05806 0.03884 0.14888 0.13151 0.75026
SimplifiedLOF 9 0.02206 0.00211 0.05789 0.03867 0.15361 0.13635 0.75273
SimplifiedLOF 10 0.02206 0.00211 0.05756 0.03834 0.15573 0.13850 0.75174
LoOP 5 0.11765 0.09965 0.06186 0.04272 0.13966 0.12211 0.74415
LoOP 12 0.08088 0.06213 0.06798 0.04896 0.16159 0.14448 0.76319
LoOP 20 0.06618 0.04712 0.06337 0.04426 0.17895 0.16220 0.73892
LDOF 12 0.15441 0.13716 0.08622 0.06758 0.17722 0.16043 0.78784
LDOF 15 0.16912 0.15217 0.08719 0.06856 0.17903 0.16228 0.78025
LDOF 21 0.16176 0.14466 0.09367 0.07518 0.20213 0.18585 0.77742
LDOF 24 0.16176 0.14466 0.09159 0.07305 0.21219 0.19612 0.76585
ODIN 11 0.13211 0.11441 0.06877 0.04977 0.16049 0.14337 0.73986
ODIN 27 0.17195 0.15505 0.08393 0.06524 0.18828 0.17172 0.71833
ODIN 30 0.17115 0.15424 0.08398 0.06530 0.19289 0.17643 0.71416
ODIN 40 0.16195 0.14486 0.08069 0.06194 0.20763 0.19146 0.69551
FastABOD 3 0.02206 0.00211 0.03461 0.01492 0.08392 0.06523 0.65777
FastABOD 5 0.04412 0.02462 0.03507 0.01538 0.07867 0.05987 0.66333
KDEOS 23 0.09559 0.07714 0.05801 0.03879 0.10753 0.08932 0.73729
KDEOS 61 0.08824 0.06963 0.06037 0.04120 0.13442 0.11676 0.71510
KDEOS 67 0.11029 0.09214 0.06207 0.04294 0.12685 0.10904 0.71434
KDEOS 100 0.10294 0.08464 0.07433 0.05545 0.12308 0.10519 0.70240
LDF 1 0.09559 0.07714 0.04045 0.02087 0.12102 0.10309 0.61247
LDF 4 0.02941 0.00961 0.05749 0.03826 0.17069 0.15377 0.70397
INFLO 1 0.05147 0.03212 0.03846 0.01885 0.10602 0.08779 0.64245
INFLO 4 0.03676 0.01711 0.05812 0.03891 0.13889 0.12132 0.73273
INFLO 8 0.03676 0.01711 0.05962 0.04044 0.16000 0.14286 0.71687
INFLO 12 0.02941 0.00961 0.05694 0.03770 0.17532 0.15850 0.70094
COF 5 0.07353 0.05463 0.05368 0.03438 0.13559 0.11796 0.72354
COF 6 0.06618 0.04712 0.05509 0.03581 0.14330 0.12582 0.72414
COF 10 0.06618 0.04712 0.05863 0.03943 0.16458 0.14754 0.71704
COF 13 0.05882 0.03962 0.05627 0.03702 0.16846 0.15149 0.71165

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO