Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#04)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (71.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 96 0.41262 0.26578 0.41751 0.27189 0.42371 0.27964 0.67472
KNN 100 0.41019 0.26274 0.41996 0.27494 0.42675 0.28344 0.67714
KNNW 62 0.39563 0.24454 0.35937 0.19921 0.40617 0.25771 0.60087
KNNW 99 0.40291 0.25364 0.37652 0.22065 0.40568 0.25711 0.62639
KNNW 100 0.40291 0.25364 0.37689 0.22111 0.40568 0.25711 0.62697
LOF 100 0.33738 0.17172 0.29666 0.12083 0.40100 0.25125 0.66148
SimplifiedLOF 16 0.33495 0.16869 0.28312 0.10390 0.34127 0.17659 0.57814
SimplifiedLOF 21 0.33252 0.16566 0.28577 0.10722 0.34358 0.17948 0.58472
SimplifiedLOF 81 0.32039 0.15049 0.27432 0.09290 0.36446 0.20558 0.59447
SimplifiedLOF 100 0.31796 0.14745 0.27583 0.09478 0.36041 0.20051 0.60285
LoOP 17 0.32767 0.15959 0.27225 0.09031 0.33630 0.17037 0.56730
LoOP 21 0.31311 0.14138 0.27310 0.09138 0.33728 0.17160 0.57051
LoOP 81 0.31068 0.13835 0.26818 0.08523 0.35747 0.19684 0.59158
LoOP 100 0.32039 0.15049 0.27070 0.08838 0.35501 0.19376 0.59804
LDOF 62 0.30340 0.12925 0.27075 0.08844 0.35494 0.19368 0.58299
LDOF 73 0.31796 0.14745 0.27171 0.08963 0.36157 0.20197 0.58252
LDOF 95 0.33252 0.16566 0.27116 0.08895 0.35860 0.19824 0.57941
LDOF 100 0.33010 0.16262 0.27207 0.09009 0.36142 0.20178 0.58054
ODIN 85 0.30874 0.13592 0.27122 0.08902 0.37146 0.21433 0.61700
ODIN 100 0.30394 0.12992 0.27807 0.09759 0.38078 0.22597 0.63062
FastABOD 95 0.28641 0.10801 0.26092 0.07615 0.33611 0.17014 0.56128
FastABOD 99 0.28641 0.10801 0.26147 0.07684 0.33611 0.17014 0.56184
FastABOD 100 0.28641 0.10801 0.26146 0.07683 0.33642 0.17053 0.56201
KDEOS 16 0.25485 0.06857 0.23496 0.04370 0.33803 0.17254 0.54596
KDEOS 17 0.25971 0.07464 0.23284 0.04105 0.34319 0.17899 0.54846
KDEOS 22 0.25485 0.06857 0.22756 0.03445 0.33942 0.17427 0.54918
LDF 99 0.33981 0.17476 0.37578 0.21973 0.42919 0.28649 0.70071
LDF 100 0.33738 0.17172 0.37761 0.22202 0.42918 0.28647 0.70136
INFLO 92 0.33495 0.16869 0.27276 0.09095 0.39065 0.23831 0.61531
INFLO 100 0.33010 0.16262 0.27589 0.09486 0.39586 0.24483 0.62082
COF 17 0.29126 0.11408 0.27643 0.09554 0.34871 0.18588 0.57874
COF 35 0.31553 0.14442 0.29102 0.11378 0.33656 0.17070 0.56640
COF 57 0.29369 0.11711 0.29409 0.11761 0.33347 0.16684 0.54211

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 68 0.42615 0.28295 0.41068 0.26362 0.42874 0.28618 0.66524
KNN 99 0.42373 0.27992 0.43031 0.28814 0.43770 0.29738 0.68281
KNN 100 0.42373 0.27992 0.43052 0.28841 0.43716 0.29670 0.68313
KNNW 87 0.40678 0.25874 0.38276 0.22873 0.41302 0.26654 0.63515
KNNW 97 0.40678 0.25874 0.38740 0.23453 0.41562 0.26979 0.64047
KNNW 100 0.40678 0.25874 0.38847 0.23587 0.41562 0.26979 0.64200
LOF 98 0.34383 0.18008 0.31040 0.13831 0.40765 0.25984 0.66520
LOF 100 0.34383 0.18008 0.31171 0.13996 0.40896 0.26146 0.66684
SimplifiedLOF 22 0.33172 0.16495 0.29311 0.11670 0.34941 0.18706 0.58937
SimplifiedLOF 25 0.32688 0.15890 0.29451 0.11846 0.34677 0.18376 0.58885
SimplifiedLOF 95 0.31961 0.14982 0.28642 0.10835 0.36913 0.21170 0.61270
SimplifiedLOF 100 0.31477 0.14377 0.28753 0.10973 0.36835 0.21073 0.61488
LoOP 29 0.32203 0.15285 0.27955 0.09977 0.33886 0.17387 0.57841
LoOP 99 0.31719 0.14680 0.28071 0.10121 0.36463 0.20607 0.60885
LoOP 100 0.31235 0.14075 0.28086 0.10140 0.36290 0.20391 0.60868
LDOF 24 0.30508 0.13167 0.27666 0.09615 0.33621 0.17057 0.56326
LDOF 98 0.31719 0.14680 0.27368 0.09243 0.35959 0.19978 0.58261
LDOF 100 0.31719 0.14680 0.27426 0.09316 0.36179 0.20253 0.58343
ODIN 86 0.31356 0.14226 0.27834 0.09826 0.37750 0.22215 0.62395
ODIN 100 0.29960 0.12481 0.28690 0.10895 0.38605 0.23284 0.63591
FastABOD 89 0.28329 0.10444 0.26851 0.08596 0.34418 0.18052 0.58064
FastABOD 94 0.28814 0.11049 0.26887 0.08642 0.34399 0.18029 0.58136
FastABOD 100 0.28814 0.11049 0.26973 0.08749 0.34370 0.17993 0.58224
KDEOS 12 0.23002 0.03788 0.23769 0.04745 0.33482 0.16883 0.53447
KDEOS 24 0.24213 0.05301 0.23314 0.04177 0.34187 0.17763 0.55793
KDEOS 26 0.23245 0.04090 0.22752 0.03474 0.34613 0.18296 0.55586
KDEOS 93 0.25666 0.07116 0.23275 0.04129 0.34343 0.17958 0.55441
LDF 6 0.35109 0.18916 0.31791 0.14770 0.36218 0.20302 0.59877
LDF 100 0.34140 0.17705 0.38059 0.22602 0.41723 0.27181 0.69306
INFLO 94 0.33414 0.16798 0.28369 0.10494 0.40000 0.25027 0.62647
INFLO 97 0.32930 0.16193 0.28637 0.10829 0.40383 0.25506 0.63103
INFLO 100 0.33172 0.16495 0.28700 0.10908 0.40164 0.25233 0.62877
COF 13 0.30024 0.12562 0.27739 0.09706 0.33776 0.17249 0.56260
COF 21 0.28814 0.11049 0.27847 0.09841 0.34697 0.18401 0.56803
COF 24 0.28814 0.11049 0.28672 0.10872 0.34551 0.18218 0.57463
COF 53 0.27361 0.09234 0.30203 0.12785 0.33293 0.16647 0.54698

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 84 0.40291 0.25364 0.35677 0.19597 0.41686 0.27107 0.67548
KNN 90 0.40777 0.25971 0.35813 0.19766 0.41611 0.27013 0.67705
KNN 100 0.40291 0.25364 0.35979 0.19973 0.41501 0.26876 0.67949
KNNW 95 0.39078 0.23847 0.34006 0.17507 0.40413 0.25516 0.65951
KNNW 97 0.39320 0.24150 0.34051 0.17563 0.40413 0.25516 0.66009
KNNW 100 0.39320 0.24150 0.34124 0.17655 0.40409 0.25512 0.66088
LOF 93 0.43447 0.29308 0.34326 0.17908 0.45455 0.31818 0.68609
LOF 98 0.43447 0.29308 0.34619 0.18274 0.45975 0.32468 0.69028
LOF 100 0.43447 0.29308 0.34675 0.18344 0.45860 0.32325 0.69132
SimplifiedLOF 99 0.39078 0.23847 0.31689 0.14612 0.43313 0.29142 0.65350
SimplifiedLOF 100 0.39078 0.23847 0.31695 0.14619 0.43170 0.28962 0.65367
LoOP 98 0.37621 0.22027 0.31185 0.13981 0.42296 0.27870 0.64354
LoOP 99 0.38107 0.22633 0.31199 0.13999 0.42339 0.27923 0.64211
LoOP 100 0.38107 0.22633 0.31208 0.14010 0.42308 0.27885 0.64195
LDOF 97 0.35922 0.19903 0.30608 0.13260 0.41963 0.27454 0.63628
LDOF 98 0.36165 0.20206 0.30643 0.13303 0.41914 0.27392 0.63721
LDOF 99 0.36165 0.20206 0.30667 0.13333 0.41843 0.27303 0.63707
LDOF 100 0.35922 0.19903 0.30653 0.13316 0.41963 0.27454 0.63746
ODIN 99 0.39583 0.24479 0.33852 0.17316 0.42813 0.28517 0.66676
ODIN 100 0.39684 0.24606 0.33886 0.17358 0.42699 0.28374 0.66718
FastABOD 4 0.25243 0.06553 0.27033 0.08791 0.34810 0.18513 0.57470
FastABOD 5 0.25728 0.07160 0.27157 0.08946 0.34842 0.18553 0.57416
FastABOD 10 0.24272 0.05340 0.26685 0.08356 0.35425 0.19282 0.56953
FastABOD 81 0.28155 0.10194 0.26769 0.08461 0.34678 0.18348 0.57235
KDEOS 68 0.24757 0.05947 0.23787 0.04733 0.36325 0.20406 0.57972
KDEOS 99 0.23301 0.04126 0.24278 0.05347 0.37508 0.21886 0.59480
KDEOS 100 0.23058 0.03823 0.24246 0.05308 0.37653 0.22067 0.59497
LDF 69 0.40777 0.25971 0.34292 0.17865 0.43038 0.28797 0.67577
LDF 86 0.41990 0.27488 0.35437 0.19296 0.42521 0.28152 0.69036
LDF 99 0.41748 0.27184 0.36285 0.20357 0.42707 0.28384 0.70021
LDF 100 0.41990 0.27488 0.36283 0.20354 0.42766 0.28457 0.70072
INFLO 92 0.39078 0.23847 0.31709 0.14636 0.42975 0.28719 0.66400
INFLO 100 0.39078 0.23847 0.31981 0.14977 0.43491 0.29364 0.66883
COF 75 0.32039 0.15049 0.29055 0.11319 0.34495 0.18119 0.57348
COF 77 0.32524 0.15655 0.28903 0.11128 0.35079 0.18848 0.57276
COF 80 0.32039 0.15049 0.28836 0.11045 0.35661 0.19576 0.57477
COF 100 0.31311 0.14138 0.28604 0.10755 0.34797 0.18497 0.58054

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.9 MB) Download raw algorithm evaluation table (75.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 90 0.39952 0.24967 0.35904 0.19909 0.41567 0.26985 0.68088
KNN 95 0.40436 0.25572 0.36023 0.20058 0.41485 0.26882 0.68213
KNN 100 0.40194 0.25269 0.36042 0.20081 0.41443 0.26831 0.68281
KNNW 94 0.38983 0.23756 0.34331 0.17943 0.40890 0.26139 0.66750
KNNW 100 0.38983 0.23756 0.34447 0.18089 0.40956 0.26221 0.66880
LOF 93 0.43341 0.29202 0.34767 0.18489 0.46122 0.32677 0.69491
LOF 94 0.43826 0.29808 0.34811 0.18544 0.46058 0.32597 0.69544
LOF 100 0.43584 0.29505 0.35109 0.18916 0.45992 0.32514 0.69908
SimplifiedLOF 99 0.38983 0.23756 0.32191 0.15270 0.43002 0.28778 0.66284
SimplifiedLOF 100 0.38983 0.23756 0.32197 0.15277 0.43145 0.28957 0.66291
LoOP 91 0.38015 0.22546 0.31339 0.14204 0.42022 0.27554 0.64745
LoOP 93 0.38499 0.23151 0.31440 0.14331 0.41888 0.27386 0.64831
LoOP 99 0.38499 0.23151 0.31736 0.14701 0.41879 0.27376 0.64941
LDOF 91 0.36077 0.20126 0.30585 0.13263 0.41367 0.26735 0.64080
LDOF 96 0.35835 0.19823 0.30845 0.13588 0.41744 0.27206 0.64383
LDOF 100 0.36077 0.20126 0.30975 0.13749 0.41300 0.26652 0.64489
ODIN 100 0.39952 0.24967 0.34169 0.17741 0.42800 0.28526 0.67200
FastABOD 24 0.27845 0.09839 0.27637 0.09579 0.36384 0.20508 0.59274
FastABOD 91 0.28814 0.11049 0.28153 0.10224 0.36175 0.20248 0.59941
FastABOD 100 0.28814 0.11049 0.28214 0.10300 0.36195 0.20272 0.60024
KDEOS 46 0.26150 0.07721 0.23631 0.04573 0.35697 0.19651 0.58527
KDEOS 95 0.25424 0.06813 0.25415 0.06802 0.38218 0.22801 0.60894
KDEOS 100 0.25666 0.07116 0.25557 0.06980 0.38165 0.22734 0.61060
LDF 95 0.41404 0.26782 0.36532 0.20694 0.42706 0.28409 0.70554
LDF 98 0.41162 0.26479 0.36777 0.21000 0.43033 0.28817 0.70822
LDF 100 0.41404 0.26782 0.36902 0.21156 0.42874 0.28618 0.70944
INFLO 100 0.39225 0.24059 0.32457 0.15602 0.44412 0.30540 0.67565
COF 77 0.30751 0.13470 0.29509 0.11918 0.34763 0.18484 0.58184
COF 85 0.32203 0.15285 0.29266 0.11615 0.35163 0.18983 0.58722
COF 96 0.30508 0.13167 0.28532 0.10698 0.35721 0.19680 0.58682
COF 98 0.30024 0.12562 0.28833 0.11074 0.35682 0.19631 0.58919

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO