Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#01)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (71.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 98 0.42476 0.28095 0.44989 0.31236 0.43220 0.29025 0.68083
KNN 100 0.42233 0.27791 0.45090 0.31363 0.43404 0.29255 0.68202
KNNW 76 0.40777 0.25971 0.39685 0.24606 0.41250 0.26562 0.62965
KNNW 91 0.40777 0.25971 0.40427 0.25533 0.41484 0.26854 0.63786
KNNW 100 0.40777 0.25971 0.40764 0.25955 0.41484 0.26854 0.64200
LOF 100 0.37621 0.22027 0.31849 0.14811 0.41195 0.26494 0.66849
SimplifiedLOF 22 0.34223 0.17779 0.29374 0.11717 0.35074 0.18842 0.58690
SimplifiedLOF 24 0.32767 0.15959 0.29597 0.11996 0.34985 0.18732 0.58971
SimplifiedLOF 99 0.31796 0.14745 0.28776 0.10970 0.37706 0.22133 0.61636
SimplifiedLOF 100 0.31553 0.14442 0.28817 0.11021 0.37580 0.21975 0.61705
LoOP 16 0.32282 0.15352 0.27785 0.09731 0.34100 0.17625 0.57167
LoOP 30 0.30583 0.13228 0.28315 0.10394 0.34095 0.17618 0.58172
LoOP 98 0.30583 0.13228 0.27912 0.09890 0.37049 0.21311 0.60968
LoOP 100 0.30825 0.13532 0.27972 0.09966 0.36923 0.21154 0.61158
LDOF 20 0.32039 0.15049 0.27838 0.09797 0.33633 0.17041 0.56282
LDOF 24 0.30097 0.12621 0.28011 0.10013 0.33989 0.17487 0.56860
LDOF 97 0.30340 0.12925 0.26740 0.08425 0.35592 0.19490 0.58133
LDOF 100 0.30825 0.13532 0.26800 0.08500 0.35588 0.19485 0.58236
ODIN 86 0.30340 0.12925 0.27914 0.09892 0.37795 0.22244 0.62676
ODIN 100 0.30049 0.12561 0.28842 0.11053 0.38689 0.23361 0.64002
FastABOD 86 0.31068 0.13835 0.27586 0.09482 0.34144 0.17680 0.57609
FastABOD 87 0.31311 0.14138 0.27598 0.09498 0.34144 0.17680 0.57624
FastABOD 100 0.31311 0.14138 0.27802 0.09753 0.34054 0.17567 0.57803
KDEOS 16 0.25243 0.06553 0.23269 0.04086 0.33801 0.17251 0.54450
KDEOS 26 0.23544 0.04430 0.22860 0.03575 0.34816 0.18520 0.55231
KDEOS 32 0.24757 0.05947 0.22752 0.03440 0.34249 0.17812 0.55412
LDF 59 0.34951 0.18689 0.33057 0.16322 0.36868 0.21085 0.63504
LDF 98 0.33981 0.17476 0.42157 0.27696 0.41205 0.26507 0.69257
LDF 100 0.34223 0.17779 0.42485 0.28106 0.41047 0.26309 0.69511
INFLO 96 0.33495 0.16869 0.28730 0.10913 0.40758 0.25948 0.63636
INFLO 98 0.33252 0.16566 0.28864 0.11080 0.41033 0.26291 0.63739
INFLO 100 0.33252 0.16566 0.28940 0.11175 0.41096 0.26370 0.63666
COF 24 0.30583 0.13228 0.28531 0.10664 0.33824 0.17279 0.58046
COF 27 0.29612 0.12015 0.28475 0.10594 0.34422 0.18028 0.57888
COF 41 0.32282 0.15352 0.29030 0.11287 0.33538 0.16923 0.56562
COF 85 0.29612 0.12015 0.30545 0.13181 0.33333 0.16667 0.55160

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 68 0.43584 0.29505 0.42004 0.27532 0.43636 0.29571 0.66222
KNN 72 0.43341 0.29202 0.42299 0.27899 0.43765 0.29732 0.66612
KNN 100 0.43341 0.29202 0.44062 0.30103 0.43735 0.29695 0.68484
KNNW 85 0.41889 0.27387 0.39468 0.24362 0.43073 0.28867 0.62975
KNNW 99 0.42615 0.28295 0.40069 0.25114 0.43019 0.28799 0.63754
KNNW 100 0.42615 0.28295 0.40113 0.25169 0.43035 0.28819 0.63808
LOF 98 0.34867 0.18613 0.30896 0.13652 0.40135 0.25196 0.66078
LOF 99 0.35109 0.18916 0.30990 0.13769 0.40034 0.25069 0.66215
LOF 100 0.34867 0.18613 0.31075 0.13875 0.40067 0.25110 0.66359
SimplifiedLOF 24 0.34625 0.18310 0.29371 0.11746 0.34755 0.18473 0.58778
SimplifiedLOF 34 0.32930 0.16193 0.29791 0.12270 0.34446 0.18087 0.59223
SimplifiedLOF 81 0.31961 0.14982 0.28573 0.10749 0.36247 0.20338 0.60105
SimplifiedLOF 100 0.32446 0.15588 0.28936 0.11202 0.36227 0.20313 0.60942
LoOP 33 0.31961 0.14982 0.28410 0.10545 0.33455 0.16849 0.58090
LoOP 92 0.32203 0.15285 0.28039 0.10081 0.35811 0.19793 0.60094
LoOP 97 0.32930 0.16193 0.28090 0.10145 0.35719 0.19678 0.60316
LoOP 100 0.32446 0.15588 0.28171 0.10246 0.35777 0.19751 0.60473
LDOF 67 0.29782 0.12259 0.27719 0.09682 0.34783 0.18508 0.58653
LDOF 82 0.31719 0.14680 0.27677 0.09629 0.35647 0.19588 0.58794
LDOF 84 0.32203 0.15285 0.27560 0.09483 0.35484 0.19384 0.58685
LDOF 98 0.31961 0.14982 0.27630 0.09571 0.35988 0.20014 0.58492
ODIN 89 0.31477 0.14377 0.28326 0.10440 0.37492 0.21894 0.62511
ODIN 98 0.30239 0.12831 0.28642 0.10835 0.38322 0.22930 0.63326
ODIN 100 0.30428 0.13066 0.28840 0.11082 0.38265 0.22859 0.63497
FastABOD 67 0.30751 0.13470 0.28247 0.10341 0.35455 0.19347 0.59432
FastABOD 84 0.31477 0.14377 0.28531 0.10696 0.35301 0.19155 0.59681
FastABOD 100 0.31235 0.14075 0.28719 0.10931 0.35327 0.19188 0.59873
KDEOS 12 0.23002 0.03788 0.24885 0.06140 0.33579 0.17004 0.53434
KDEOS 16 0.25666 0.07116 0.23959 0.04983 0.33655 0.17099 0.54744
KDEOS 26 0.24697 0.05906 0.23241 0.04086 0.33708 0.17165 0.55397
KDEOS 100 0.24213 0.05301 0.23250 0.04098 0.34181 0.17756 0.55088
LDF 94 0.33656 0.17100 0.37459 0.21852 0.41151 0.26466 0.68672
LDF 100 0.33656 0.17100 0.38626 0.23310 0.41808 0.27286 0.69643
INFLO 99 0.34383 0.18008 0.28900 0.11157 0.40903 0.26155 0.63531
INFLO 100 0.34383 0.18008 0.28987 0.11266 0.40984 0.26256 0.63617
COF 28 0.30751 0.13470 0.29697 0.12153 0.34637 0.18326 0.58547
COF 29 0.30266 0.12865 0.29673 0.12123 0.34922 0.18682 0.58406
COF 40 0.31719 0.14680 0.30316 0.12927 0.34020 0.17555 0.57331
COF 82 0.28814 0.11049 0.31729 0.14693 0.33306 0.16663 0.54427

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 82 0.39806 0.24757 0.36307 0.20384 0.41924 0.27405 0.68269
KNN 93 0.40777 0.25971 0.36434 0.20542 0.41810 0.27263 0.68459
KNN 100 0.40049 0.25061 0.36535 0.20669 0.41840 0.27300 0.68567
KNNW 71 0.38107 0.22633 0.34430 0.18038 0.41503 0.26878 0.66703
KNNW 100 0.39078 0.23847 0.35053 0.18816 0.41297 0.26621 0.67346
LOF 86 0.44903 0.31129 0.35236 0.19045 0.46394 0.32992 0.69650
LOF 100 0.44903 0.31129 0.35909 0.19886 0.46566 0.33207 0.70513
SimplifiedLOF 98 0.40291 0.25364 0.33129 0.16412 0.44467 0.30583 0.67218
SimplifiedLOF 99 0.40534 0.25667 0.33183 0.16479 0.44356 0.30446 0.67262
SimplifiedLOF 100 0.40534 0.25667 0.33214 0.16518 0.44356 0.30445 0.67299
LoOP 100 0.40291 0.25364 0.32633 0.15792 0.43658 0.29572 0.66106
LDOF 89 0.37379 0.21723 0.30929 0.13661 0.42424 0.28030 0.64613
LDOF 100 0.36650 0.20813 0.31629 0.14536 0.43141 0.28926 0.65315
ODIN 99 0.41578 0.26972 0.34349 0.17937 0.43907 0.29884 0.67684
ODIN 100 0.41475 0.26843 0.34421 0.18026 0.43849 0.29812 0.67744
FastABOD 18 0.27670 0.09587 0.27595 0.09494 0.35342 0.19177 0.58148
FastABOD 81 0.29854 0.12318 0.28098 0.10123 0.34893 0.18616 0.58455
FastABOD 100 0.29612 0.12015 0.28226 0.10283 0.35168 0.18960 0.58570
KDEOS 71 0.26456 0.08070 0.24965 0.06206 0.37262 0.21578 0.60168
KDEOS 100 0.25485 0.06857 0.25580 0.06975 0.38495 0.23119 0.61361
LDF 56 0.41505 0.26881 0.34664 0.18330 0.44004 0.30006 0.67799
LDF 99 0.42476 0.28095 0.37237 0.21546 0.43262 0.29077 0.71022
LDF 100 0.42233 0.27791 0.37267 0.21583 0.43379 0.29224 0.71099
INFLO 84 0.39320 0.24150 0.32957 0.16197 0.44767 0.30959 0.68054
INFLO 95 0.40291 0.25364 0.33146 0.16432 0.44706 0.30882 0.67912
INFLO 100 0.40049 0.25061 0.33166 0.16458 0.44871 0.31089 0.67703
COF 69 0.32282 0.15352 0.29877 0.12346 0.37023 0.21279 0.59389
COF 84 0.33252 0.16566 0.30641 0.13301 0.36051 0.20063 0.59594
COF 86 0.33010 0.16262 0.30329 0.12912 0.35600 0.19500 0.59611
COF 99 0.34223 0.17779 0.30017 0.12521 0.35657 0.19572 0.59405

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.9 MB) Download raw algorithm evaluation table (75.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 82 0.40920 0.26177 0.36539 0.20703 0.42057 0.27598 0.68180
KNN 96 0.41404 0.26782 0.36747 0.20962 0.41537 0.26947 0.68492
KNN 100 0.40920 0.26177 0.36781 0.21005 0.41686 0.27134 0.68555
KNNW 97 0.40436 0.25572 0.35099 0.18903 0.41030 0.26314 0.67058
KNNW 100 0.40436 0.25572 0.35155 0.18973 0.41204 0.26531 0.67122
LOF 91 0.44068 0.30110 0.34887 0.18638 0.46296 0.32895 0.69450
LOF 97 0.44794 0.31018 0.35220 0.19054 0.46154 0.32717 0.69849
LOF 100 0.44794 0.31018 0.35396 0.19274 0.46082 0.32627 0.70037
SimplifiedLOF 97 0.39467 0.24362 0.32349 0.15467 0.42485 0.28132 0.66459
SimplifiedLOF 100 0.39467 0.24362 0.32436 0.15576 0.42785 0.28507 0.66543
LoOP 91 0.38741 0.23454 0.31464 0.14361 0.41939 0.27450 0.65113
LoOP 99 0.39467 0.24362 0.31910 0.14919 0.41717 0.27172 0.65464
LDOF 92 0.37288 0.21639 0.30717 0.13428 0.40470 0.25614 0.64190
LDOF 100 0.36077 0.20126 0.31217 0.14052 0.41431 0.26816 0.64724
ODIN 99 0.39692 0.24643 0.33952 0.17470 0.42956 0.28720 0.67292
ODIN 100 0.39766 0.24735 0.34004 0.17535 0.42914 0.28669 0.67370
FastABOD 19 0.28087 0.10142 0.28586 0.10765 0.36393 0.20520 0.59961
FastABOD 92 0.29782 0.12259 0.29229 0.11569 0.36364 0.20483 0.60678
FastABOD 100 0.29782 0.12259 0.29261 0.11608 0.36340 0.20454 0.60743
KDEOS 18 0.25424 0.06813 0.23051 0.03849 0.35654 0.19597 0.55945
KDEOS 100 0.25182 0.06511 0.25582 0.07011 0.37992 0.22519 0.60509
LDF 56 0.40678 0.25874 0.34189 0.17766 0.43090 0.28889 0.67091
LDF 98 0.42373 0.27992 0.36989 0.21265 0.42410 0.28038 0.70543
LDF 100 0.42373 0.27992 0.37095 0.21397 0.42476 0.28121 0.70700
INFLO 85 0.38499 0.23151 0.31803 0.14785 0.43682 0.29627 0.66700
INFLO 100 0.38499 0.23151 0.32355 0.15475 0.44214 0.30292 0.67156
COF 74 0.30024 0.12562 0.29601 0.12033 0.37581 0.22005 0.59102
COF 81 0.32930 0.16193 0.30047 0.12590 0.36975 0.21247 0.59325
COF 92 0.33656 0.17100 0.29270 0.11620 0.36963 0.21232 0.59765
COF 98 0.34625 0.18310 0.29535 0.11950 0.35552 0.19469 0.59518

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO