Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Annthyroid (2% of outliers version#09)

This data set contains medical data on hypothyroidism. Three classes relate to the conditions normal, hyperfunction, and subnormal functioning. Classes other than normal condition were defined as outliers here.

Download all data set variants used (9.9 MB). You can also access the original data. (merge train and test [ann-test.data and ann-train.data])

Normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (58.4 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05970 0.04060 0.04064 0.02115 0.09541 0.07703 0.67410
KNNW 1 0.06716 0.04821 0.04357 0.02414 0.11156 0.09351 0.69640
LOF 2 0.06716 0.04821 0.04642 0.02704 0.11798 0.10006 0.68151
LOF 5 0.02985 0.01014 0.05658 0.03742 0.13984 0.12236 0.73158
SimplifiedLOF 2 0.05224 0.03298 0.04116 0.02167 0.09524 0.07685 0.69378
SimplifiedLOF 7 0.03731 0.01775 0.05509 0.03590 0.13069 0.11303 0.75616
SimplifiedLOF 9 0.03731 0.01775 0.05435 0.03514 0.12628 0.10853 0.75732
LoOP 5 0.11940 0.10151 0.05949 0.04038 0.14933 0.13205 0.75013
LoOP 7 0.11194 0.09390 0.06045 0.04136 0.13555 0.11799 0.75698
LDOF 9 0.13433 0.11674 0.06822 0.04929 0.13534 0.11777 0.77232
LDOF 12 0.14179 0.12435 0.07003 0.05114 0.14615 0.12881 0.77135
LDOF 15 0.13433 0.11674 0.06791 0.04897 0.14847 0.13117 0.76307
ODIN 6 0.07879 0.06007 0.04862 0.02929 0.10601 0.08784 0.73041
ODIN 39 0.08396 0.06534 0.05667 0.03751 0.13506 0.11749 0.70048
ODIN 42 0.09142 0.07296 0.05575 0.03656 0.14092 0.12347 0.69908
ODIN 57 0.10821 0.09009 0.05301 0.03377 0.13029 0.11262 0.68100
FastABOD 4 0.03731 0.01775 0.03409 0.01446 0.07692 0.05817 0.65368
FastABOD 7 0.05224 0.03298 0.03292 0.01327 0.07442 0.05561 0.64176
FastABOD 96 0.01493 -0.00509 0.03005 0.01034 0.07971 0.06101 0.61103
KDEOS 16 0.06716 0.04821 0.05232 0.03306 0.12297 0.10515 0.74851
KDEOS 21 0.09701 0.07867 0.05690 0.03773 0.11351 0.09550 0.73110
KDEOS 28 0.05224 0.03298 0.06321 0.04418 0.10056 0.08228 0.73026
LDF 2 0.08955 0.07105 0.04064 0.02115 0.10596 0.08779 0.65764
LDF 5 0.02985 0.01014 0.05073 0.03145 0.13542 0.11785 0.70774
INFLO 2 0.08209 0.06344 0.04447 0.02506 0.11458 0.09659 0.66337
INFLO 4 0.07463 0.05582 0.05354 0.03431 0.13827 0.12076 0.70986
INFLO 6 0.06716 0.04821 0.05171 0.03245 0.12329 0.10547 0.71944
COF 5 0.08955 0.07105 0.05341 0.03418 0.11613 0.09817 0.73240
COF 9 0.07463 0.05582 0.05200 0.03274 0.14141 0.12397 0.70759

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.9 MB) Download raw algorithm evaluation table (71.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05882 0.03962 0.04068 0.02111 0.09756 0.07915 0.68292
KNNW 1 0.04412 0.02462 0.04434 0.02484 0.10317 0.08488 0.71163
KNNW 2 0.06618 0.04712 0.04247 0.02294 0.10511 0.08685 0.69690
LOF 1 0.03676 0.01711 0.03083 0.01106 0.08095 0.06220 0.59704
LOF 6 0.01471 -0.00540 0.05024 0.03086 0.13518 0.11753 0.71413
LOF 8 0.01471 -0.00540 0.04969 0.03030 0.13758 0.11999 0.71792
SimplifiedLOF 9 0.00735 -0.01290 0.04834 0.02893 0.12780 0.11000 0.73520
SimplifiedLOF 11 0.00735 -0.01290 0.04905 0.02965 0.12121 0.10328 0.74227
SimplifiedLOF 89 0.03676 0.01711 0.03092 0.01115 0.07616 0.05732 0.65178
LoOP 11 0.05882 0.03962 0.05618 0.03692 0.14493 0.12748 0.75288
LoOP 12 0.05882 0.03962 0.05670 0.03746 0.13937 0.12181 0.75354
LoOP 14 0.06618 0.04712 0.05730 0.03806 0.13879 0.12122 0.74977
LoOP 15 0.06618 0.04712 0.05766 0.03844 0.13910 0.12153 0.74924
LDOF 13 0.09559 0.07714 0.06440 0.04531 0.14118 0.12365 0.77184
LDOF 19 0.14706 0.12966 0.06973 0.05075 0.15029 0.13295 0.76788
LDOF 28 0.13235 0.11465 0.07588 0.05703 0.16078 0.14366 0.76424
LDOF 32 0.14706 0.12966 0.07627 0.05742 0.16000 0.14286 0.75455
ODIN 11 0.07892 0.06013 0.05470 0.03542 0.11644 0.09841 0.75843
ODIN 26 0.09467 0.07620 0.05909 0.03989 0.13462 0.11696 0.73857
ODIN 31 0.08956 0.07098 0.05847 0.03926 0.14508 0.12764 0.72436
ODIN 99 0.11826 0.10027 0.05516 0.03588 0.12014 0.10219 0.66663
FastABOD 5 0.04412 0.02462 0.03320 0.01348 0.07432 0.05543 0.66469
FastABOD 6 0.04412 0.02462 0.03335 0.01362 0.07699 0.05816 0.66392
KDEOS 16 0.10294 0.08464 0.06544 0.04638 0.12165 0.10373 0.73758
KDEOS 17 0.10294 0.08464 0.07114 0.05219 0.11503 0.09697 0.73921
KDEOS 33 0.08088 0.06213 0.06387 0.04477 0.13575 0.11811 0.76009
KDEOS 40 0.08088 0.06213 0.05982 0.04064 0.14595 0.12852 0.74332
LDF 2 0.09559 0.07714 0.04139 0.02183 0.11461 0.09655 0.64263
LDF 5 0.00735 -0.01290 0.04594 0.02647 0.12366 0.10578 0.69769
LDF 6 0.02206 0.00211 0.04644 0.02698 0.11852 0.10053 0.70424
INFLO 1 0.03676 0.01711 0.03272 0.01299 0.08753 0.06891 0.61343
INFLO 10 0.02941 0.00961 0.05089 0.03152 0.12615 0.10832 0.70520
INFLO 11 0.02941 0.00961 0.05134 0.03198 0.13240 0.11470 0.70355
INFLO 14 0.02206 0.00211 0.05033 0.03096 0.13468 0.11703 0.70131
COF 9 0.02941 0.00961 0.04622 0.02676 0.11438 0.09632 0.71617
COF 10 0.03676 0.01711 0.04578 0.02632 0.11200 0.09388 0.71663
COF 16 0.05882 0.03962 0.04611 0.02665 0.12403 0.10616 0.71070
COF 17 0.03676 0.01711 0.04585 0.02638 0.12542 0.10758 0.70213

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 6729 objects, 134 outliers (1.99%)

Download raw algorithm results (57.8 MB) Download raw algorithm evaluation table (72.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.05224 0.03298 0.03989 0.02038 0.09122 0.07275 0.67239
KNN 3 0.02239 0.00252 0.03564 0.01605 0.09155 0.07309 0.64877
KNNW 1 0.05970 0.04060 0.04315 0.02370 0.10123 0.08297 0.70142
LOF 1 0.08955 0.07105 0.04288 0.02343 0.11831 0.10040 0.62576
LOF 6 0.04478 0.02537 0.05514 0.03594 0.15812 0.14101 0.71856
SimplifiedLOF 4 0.04478 0.02537 0.05311 0.03387 0.11354 0.09553 0.75825
SimplifiedLOF 6 0.02985 0.01014 0.05682 0.03765 0.13333 0.11572 0.77043
SimplifiedLOF 9 0.03731 0.01775 0.05488 0.03568 0.14685 0.12952 0.76455
LoOP 6 0.11940 0.10151 0.06280 0.04376 0.13402 0.11643 0.76979
LoOP 7 0.11194 0.09390 0.06399 0.04498 0.13965 0.12217 0.77431
LoOP 12 0.09701 0.07867 0.06161 0.04255 0.15717 0.14005 0.75702
LDOF 9 0.11194 0.09390 0.07117 0.05229 0.13636 0.11882 0.79585
LDOF 15 0.11940 0.10151 0.07740 0.05866 0.16055 0.14349 0.77633
LDOF 17 0.12687 0.10912 0.07735 0.05860 0.16532 0.14836 0.76156
LDOF 20 0.14925 0.13197 0.07623 0.05747 0.15550 0.13834 0.75350
ODIN 4 0.06746 0.04851 0.04786 0.02851 0.10710 0.08896 0.74752
ODIN 31 0.11330 0.09528 0.06052 0.04143 0.17090 0.15405 0.68652
ODIN 42 0.13072 0.11305 0.06268 0.04364 0.16290 0.14589 0.68325
ODIN 90 0.14290 0.12548 0.06213 0.04308 0.15108 0.13383 0.66073
FastABOD 4 0.05224 0.03298 0.03369 0.01406 0.07870 0.05998 0.63929
FastABOD 5 0.05224 0.03298 0.03410 0.01448 0.08314 0.06451 0.63675
FastABOD 6 0.05970 0.04060 0.03400 0.01437 0.07963 0.06092 0.63449
KDEOS 11 0.08209 0.06344 0.06612 0.04714 0.10753 0.08939 0.74297
KDEOS 13 0.11194 0.09390 0.05827 0.03914 0.11321 0.09519 0.74588
KDEOS 14 0.11194 0.09390 0.05806 0.03893 0.11659 0.09864 0.74514
LDF 2 0.08955 0.07105 0.04456 0.02514 0.12393 0.10613 0.64848
LDF 5 0.03731 0.01775 0.04918 0.02986 0.13542 0.11785 0.69972
LDF 6 0.02985 0.01014 0.04866 0.02933 0.14128 0.12383 0.68901
INFLO 4 0.08955 0.07105 0.05453 0.03532 0.12647 0.10872 0.73533
INFLO 9 0.04478 0.02537 0.05037 0.03108 0.14914 0.13185 0.67818
COF 6 0.08955 0.07105 0.05496 0.03576 0.12684 0.10910 0.73871
COF 13 0.05970 0.04060 0.04900 0.02968 0.13729 0.11976 0.69474

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 6802 objects, 136 outliers (2.00%)

Download raw algorithm results (58.3 MB) Download raw algorithm evaluation table (72.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.04412 0.02462 0.03932 0.01972 0.09113 0.07258 0.68938
KNNW 1 0.05147 0.03212 0.04532 0.02584 0.10402 0.08574 0.71815
KNNW 2 0.05882 0.03962 0.04162 0.02206 0.09828 0.07988 0.70235
LOF 2 0.05147 0.03212 0.04607 0.02661 0.13362 0.11594 0.66968
LOF 6 0.00735 -0.01290 0.05723 0.03800 0.14653 0.12912 0.72938
LOF 11 0.00000 -0.02040 0.05207 0.03273 0.14911 0.13175 0.69638
SimplifiedLOF 1 0.02941 0.00961 0.03615 0.01649 0.08358 0.06489 0.67268
SimplifiedLOF 9 0.00000 -0.02040 0.05516 0.03588 0.13660 0.11899 0.76413
SimplifiedLOF 11 0.00000 -0.02040 0.05433 0.03503 0.13814 0.12055 0.76098
LoOP 9 0.05882 0.03962 0.06414 0.04505 0.15419 0.13693 0.77830
LoOP 10 0.05882 0.03962 0.06444 0.04536 0.15470 0.13745 0.77361
LoOP 16 0.08824 0.06963 0.06210 0.04296 0.14615 0.12873 0.75390
LDOF 13 0.13235 0.11465 0.08238 0.06366 0.16635 0.14934 0.80614
LDOF 20 0.16912 0.15217 0.09002 0.07145 0.18029 0.16357 0.79854
LDOF 25 0.15441 0.13716 0.09245 0.07393 0.17699 0.16020 0.78390
LDOF 33 0.15441 0.13716 0.08952 0.07095 0.18997 0.17345 0.75583
ODIN 8 0.11490 0.09684 0.06483 0.04575 0.13636 0.11874 0.75992
ODIN 18 0.12453 0.10667 0.07236 0.05344 0.17690 0.16011 0.73884
ODIN 24 0.12754 0.10974 0.06960 0.05062 0.18142 0.16472 0.72909
ODIN 27 0.14346 0.12599 0.06916 0.05017 0.16465 0.14761 0.72296
FastABOD 4 0.02941 0.00961 0.03379 0.01407 0.07677 0.05794 0.66200
FastABOD 6 0.03676 0.01711 0.03323 0.01350 0.07977 0.06099 0.65751
FastABOD 16 0.04412 0.02462 0.03264 0.01291 0.07420 0.05531 0.64717
KDEOS 21 0.10294 0.08464 0.06944 0.05046 0.12058 0.10264 0.76635
KDEOS 31 0.07353 0.05463 0.06319 0.04407 0.13550 0.11786 0.75814
LDF 1 0.07353 0.05463 0.03558 0.01590 0.09722 0.07880 0.57573
LDF 6 0.00735 -0.01290 0.04987 0.03048 0.13725 0.11965 0.68941
LDF 8 0.00000 -0.02040 0.04687 0.02742 0.13746 0.11986 0.66851
INFLO 1 0.05147 0.03212 0.03623 0.01657 0.08392 0.06523 0.65625
INFLO 2 0.04412 0.02462 0.04701 0.02757 0.11417 0.09610 0.73803
INFLO 6 0.02206 0.00211 0.05387 0.03456 0.14840 0.13102 0.72077
COF 8 0.05882 0.03962 0.05395 0.03465 0.12967 0.11191 0.73418
COF 9 0.06618 0.04712 0.05365 0.03435 0.13239 0.11469 0.72768
COF 14 0.05882 0.03962 0.05049 0.03112 0.13855 0.12098 0.71102

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO