Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#05)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (72.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 92 0.42233 0.27791 0.43110 0.28887 0.43845 0.29807 0.67850
KNN 98 0.43447 0.29308 0.43402 0.29252 0.43765 0.29707 0.68144
KNN 100 0.43447 0.29308 0.43524 0.29405 0.43552 0.29440 0.68287
KNNW 59 0.42233 0.27791 0.37980 0.22475 0.42553 0.28191 0.61622
KNNW 66 0.41990 0.27488 0.38345 0.22932 0.43147 0.28934 0.62095
KNNW 100 0.41990 0.27488 0.39702 0.24628 0.42857 0.28571 0.64025
LOF 100 0.34466 0.18083 0.30879 0.13599 0.40296 0.25370 0.66375
SimplifiedLOF 26 0.34466 0.18083 0.29663 0.12079 0.35294 0.19118 0.59625
SimplifiedLOF 31 0.33495 0.16869 0.30024 0.12530 0.35015 0.18769 0.59769
SimplifiedLOF 80 0.32039 0.15049 0.28672 0.10840 0.36930 0.21163 0.60390
SimplifiedLOF 100 0.31796 0.14745 0.28917 0.11146 0.36671 0.20839 0.61150
LoOP 20 0.33010 0.16262 0.27979 0.09974 0.34238 0.17797 0.57777
LoOP 34 0.32039 0.15049 0.28564 0.10705 0.34130 0.17663 0.58127
LoOP 84 0.31796 0.14745 0.27981 0.09976 0.36543 0.20679 0.60076
LoOP 100 0.31553 0.14442 0.28198 0.10247 0.35849 0.19811 0.60695
LDOF 81 0.32039 0.15049 0.27697 0.09621 0.35806 0.19758 0.58945
LDOF 96 0.31796 0.14745 0.27781 0.09726 0.36609 0.20761 0.58810
LDOF 98 0.31796 0.14745 0.27845 0.09807 0.36502 0.20627 0.58810
ODIN 85 0.31351 0.14189 0.28158 0.10197 0.37411 0.21764 0.62093
ODIN 96 0.30392 0.12990 0.28590 0.10737 0.38355 0.22943 0.63064
ODIN 100 0.30299 0.12874 0.28855 0.11068 0.38348 0.22935 0.63398
FastABOD 86 0.29369 0.11711 0.27197 0.08996 0.34537 0.18171 0.58320
FastABOD 89 0.28883 0.11104 0.27198 0.08998 0.34573 0.18216 0.58349
FastABOD 100 0.29369 0.11711 0.27323 0.09153 0.34504 0.18131 0.58495
KDEOS 11 0.24029 0.05036 0.24636 0.05795 0.33835 0.17293 0.54076
KDEOS 19 0.25728 0.07160 0.23525 0.04406 0.34171 0.17714 0.56007
KDEOS 27 0.23544 0.04430 0.22967 0.03709 0.34658 0.18322 0.55831
LDF 26 0.33738 0.17172 0.30533 0.13167 0.36670 0.20837 0.61175
LDF 100 0.33495 0.16869 0.37921 0.22401 0.41583 0.26978 0.69415
INFLO 99 0.32767 0.15959 0.28572 0.10715 0.40580 0.25725 0.63005
INFLO 100 0.32767 0.15959 0.28650 0.10812 0.40606 0.25757 0.63299
COF 28 0.30825 0.13532 0.29135 0.11419 0.35879 0.19849 0.59198
COF 41 0.32039 0.15049 0.30767 0.13458 0.34240 0.17800 0.58113
COF 57 0.29612 0.12015 0.31991 0.14989 0.33380 0.16725 0.56157

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 70 0.42373 0.27992 0.39856 0.24848 0.42612 0.28291 0.65169
KNN 96 0.42373 0.27992 0.41747 0.27210 0.43006 0.28783 0.66944
KNN 100 0.42373 0.27992 0.41889 0.27387 0.42811 0.28539 0.67126
KNNW 88 0.40678 0.25874 0.37082 0.21381 0.41333 0.26693 0.61773
KNNW 100 0.40678 0.25874 0.37556 0.21973 0.41644 0.27081 0.62457
LOF 97 0.33898 0.17403 0.29768 0.12242 0.39821 0.24803 0.64952
LOF 98 0.34140 0.17705 0.29857 0.12353 0.39564 0.24483 0.65041
LOF 100 0.34140 0.17705 0.30020 0.12556 0.39687 0.24636 0.65277
SimplifiedLOF 16 0.31477 0.14377 0.28843 0.11086 0.33583 0.17008 0.57591
SimplifiedLOF 97 0.33172 0.16495 0.27898 0.09905 0.35659 0.19603 0.60028
SimplifiedLOF 100 0.32203 0.15285 0.27953 0.09974 0.36068 0.20114 0.60155
LoOP 15 0.30508 0.13167 0.27857 0.09855 0.33293 0.16647 0.56274
LoOP 95 0.32203 0.15285 0.27214 0.09051 0.34950 0.18717 0.59469
LoOP 99 0.32203 0.15285 0.27352 0.09223 0.34894 0.18647 0.59684
LoOP 100 0.31961 0.14982 0.27345 0.09215 0.35028 0.18815 0.59636
LDOF 20 0.29540 0.11957 0.27717 0.09679 0.33347 0.16714 0.55272
LDOF 78 0.30751 0.13470 0.26599 0.08282 0.35184 0.19009 0.57783
LDOF 85 0.31477 0.14377 0.26599 0.08282 0.34935 0.18699 0.57742
LDOF 100 0.30993 0.13772 0.26719 0.08432 0.35651 0.19593 0.57678
ODIN 92 0.31053 0.13848 0.27825 0.09814 0.37522 0.21930 0.61726
ODIN 96 0.30310 0.12920 0.27973 0.09999 0.37809 0.22290 0.62044
ODIN 100 0.30464 0.13112 0.28237 0.10329 0.37619 0.22052 0.62433
FastABOD 6 0.23487 0.04393 0.23829 0.04820 0.33989 0.17517 0.54020
FastABOD 99 0.28329 0.10444 0.26394 0.08026 0.33941 0.17456 0.56500
FastABOD 100 0.28329 0.10444 0.26412 0.08048 0.33973 0.17496 0.56513
KDEOS 16 0.26150 0.07721 0.24027 0.05069 0.33490 0.16893 0.55068
KDEOS 20 0.24697 0.05906 0.23708 0.04670 0.34469 0.18117 0.55617
KDEOS 21 0.24697 0.05906 0.23843 0.04838 0.33950 0.17467 0.55646
LDF 96 0.32688 0.15890 0.35649 0.19590 0.41091 0.26391 0.67646
LDF 97 0.32446 0.15588 0.35795 0.19773 0.41252 0.26592 0.67810
LDF 100 0.32446 0.15588 0.36418 0.20551 0.41129 0.26438 0.68241
INFLO 99 0.32930 0.16193 0.27895 0.09902 0.39725 0.24684 0.62469
COF 28 0.26634 0.08326 0.27676 0.09628 0.35288 0.19140 0.56632
COF 41 0.30508 0.13167 0.28712 0.10923 0.33974 0.17497 0.55346
COF 56 0.27603 0.09536 0.29865 0.12363 0.33374 0.16747 0.53890

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 91 0.40777 0.25971 0.36556 0.20695 0.41855 0.27319 0.68469
KNN 95 0.40777 0.25971 0.36601 0.20752 0.42174 0.27717 0.68551
KNN 100 0.40534 0.25667 0.36652 0.20815 0.42037 0.27546 0.68636
KNNW 95 0.40291 0.25364 0.35135 0.18918 0.40878 0.26098 0.67172
KNNW 100 0.40291 0.25364 0.35221 0.19027 0.41026 0.26282 0.67278
LOF 95 0.44175 0.30218 0.35129 0.18912 0.45625 0.32031 0.69557
LOF 98 0.43932 0.29915 0.35276 0.19094 0.45763 0.32203 0.69777
LOF 100 0.43932 0.29915 0.35345 0.19182 0.45570 0.31962 0.69854
SimplifiedLOF 92 0.38592 0.23240 0.32377 0.15471 0.42606 0.28257 0.65969
SimplifiedLOF 98 0.38835 0.23544 0.32561 0.15701 0.42504 0.28130 0.66226
SimplifiedLOF 100 0.39563 0.24454 0.32608 0.15760 0.42487 0.28109 0.66214
LoOP 96 0.37864 0.22330 0.31979 0.14974 0.41724 0.27155 0.64910
LoOP 97 0.38107 0.22633 0.32027 0.15034 0.41608 0.27010 0.64875
LoOP 99 0.37621 0.22027 0.32072 0.15091 0.41955 0.27444 0.64826
LoOP 100 0.38107 0.22633 0.32070 0.15087 0.42116 0.27645 0.64719
LDOF 98 0.36650 0.20813 0.31337 0.14172 0.41249 0.26561 0.64225
LDOF 99 0.36165 0.20206 0.31393 0.14241 0.41294 0.26618 0.64283
LDOF 100 0.36650 0.20813 0.31395 0.14244 0.41425 0.26781 0.64264
ODIN 100 0.39520 0.24400 0.34571 0.18214 0.42308 0.27885 0.67225
FastABOD 4 0.28155 0.10194 0.28530 0.10663 0.35786 0.19733 0.59572
FastABOD 50 0.29369 0.11711 0.28017 0.10021 0.36407 0.20508 0.59654
FastABOD 81 0.29126 0.11408 0.28246 0.10307 0.36827 0.21033 0.59908
FastABOD 100 0.29126 0.11408 0.28396 0.10495 0.36792 0.20991 0.60051
KDEOS 71 0.27427 0.09284 0.25184 0.06480 0.36879 0.21099 0.59746
KDEOS 99 0.26942 0.08677 0.25676 0.07094 0.38301 0.22876 0.60750
KDEOS 100 0.27184 0.08981 0.25671 0.07088 0.38284 0.22855 0.60769
LDF 100 0.41990 0.27488 0.36913 0.21141 0.42986 0.28733 0.70627
INFLO 90 0.37136 0.21420 0.32533 0.15666 0.44592 0.30740 0.67635
INFLO 92 0.37864 0.22330 0.32600 0.15751 0.44592 0.30740 0.67760
INFLO 96 0.38350 0.22937 0.32679 0.15848 0.44560 0.30699 0.67646
INFLO 99 0.38592 0.23240 0.32673 0.15841 0.44527 0.30659 0.67519
COF 76 0.34223 0.17779 0.30568 0.13210 0.35796 0.19745 0.60031
COF 80 0.35437 0.19296 0.30364 0.12955 0.36836 0.21046 0.60196
COF 82 0.34709 0.18386 0.30330 0.12913 0.37093 0.21367 0.60146
COF 86 0.35194 0.18993 0.30397 0.12997 0.36242 0.20302 0.60404

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.9 MB) Download raw algorithm evaluation table (75.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 95 0.39467 0.24362 0.35143 0.18958 0.40870 0.26114 0.66923
KNN 99 0.39709 0.24664 0.35192 0.19020 0.40489 0.25639 0.66992
KNN 100 0.39952 0.24967 0.35186 0.19012 0.40400 0.25526 0.67001
KNNW 99 0.38499 0.23151 0.33533 0.16946 0.40172 0.25243 0.65468
KNNW 100 0.38499 0.23151 0.33542 0.16958 0.40138 0.25199 0.65485
LOF 85 0.41404 0.26782 0.33555 0.16973 0.44553 0.30716 0.67526
LOF 97 0.42373 0.27992 0.34215 0.17798 0.44397 0.30522 0.68372
LOF 100 0.42131 0.27690 0.34337 0.17950 0.44421 0.30551 0.68546
SimplifiedLOF 99 0.38257 0.22849 0.31614 0.14548 0.41779 0.27251 0.65186
SimplifiedLOF 100 0.38257 0.22849 0.31692 0.14646 0.42054 0.27594 0.65215
LoOP 93 0.37530 0.21941 0.31002 0.13784 0.40675 0.25870 0.63810
LoOP 98 0.38015 0.22546 0.31145 0.13963 0.41125 0.26433 0.63668
LoOP 100 0.38015 0.22546 0.31230 0.14069 0.41279 0.26625 0.63689
LDOF 96 0.36077 0.20126 0.30586 0.13264 0.40797 0.26023 0.63438
LDOF 99 0.36077 0.20126 0.30602 0.13284 0.40844 0.26081 0.63536
LDOF 100 0.36077 0.20126 0.30673 0.13373 0.40755 0.25970 0.63559
ODIN 99 0.38660 0.23353 0.33319 0.16679 0.41888 0.27386 0.66037
ODIN 100 0.38579 0.23252 0.33400 0.16780 0.42229 0.27812 0.66108
FastABOD 12 0.26392 0.08024 0.27075 0.08876 0.35866 0.19861 0.58037
FastABOD 25 0.28329 0.10444 0.27222 0.09060 0.35348 0.19214 0.58238
FastABOD 99 0.27603 0.09536 0.27685 0.09638 0.35678 0.19626 0.58724
FastABOD 100 0.27603 0.09536 0.27676 0.09628 0.35678 0.19626 0.58732
KDEOS 97 0.26634 0.08326 0.25153 0.06475 0.37082 0.21381 0.60357
KDEOS 98 0.27119 0.08931 0.25146 0.06466 0.37132 0.21443 0.60371
KDEOS 100 0.27361 0.09234 0.25048 0.06344 0.37351 0.21717 0.60330
LDF 74 0.39952 0.24967 0.33981 0.17506 0.41841 0.27328 0.67024
LDF 99 0.41646 0.27085 0.35605 0.19535 0.41822 0.27304 0.68930
LDF 100 0.41404 0.26782 0.35604 0.19534 0.41715 0.27170 0.68968
INFLO 87 0.38257 0.22849 0.31260 0.14107 0.42004 0.27532 0.64909
INFLO 99 0.37772 0.22244 0.31693 0.14647 0.42751 0.28465 0.65509
INFLO 100 0.37772 0.22244 0.31717 0.14677 0.42910 0.28664 0.65508
COF 75 0.30024 0.12562 0.28938 0.11204 0.34459 0.18104 0.58258
COF 87 0.31961 0.14982 0.28501 0.10659 0.34890 0.18642 0.58274
COF 92 0.31235 0.14075 0.28111 0.10172 0.35112 0.18920 0.58409
COF 93 0.31719 0.14680 0.27968 0.09992 0.34986 0.18763 0.58419

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO