Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#08)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (72.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 96 0.42476 0.28095 0.42607 0.28259 0.43488 0.29360 0.66986
KNN 99 0.42961 0.28701 0.42706 0.28383 0.43315 0.29144 0.67094
KNN 100 0.42718 0.28398 0.42768 0.28459 0.43225 0.29031 0.67143
KNNW 57 0.41019 0.26274 0.37139 0.21424 0.41169 0.26462 0.61120
KNNW 80 0.40534 0.25667 0.38262 0.22827 0.41905 0.27381 0.62481
KNNW 100 0.41019 0.26274 0.39087 0.23859 0.41734 0.27168 0.63455
LOF 97 0.33738 0.17172 0.30706 0.13383 0.40762 0.25953 0.65881
LOF 99 0.33495 0.16869 0.30726 0.13407 0.41202 0.26502 0.66048
LOF 100 0.33495 0.16869 0.30779 0.13474 0.41096 0.26370 0.66118
SimplifiedLOF 21 0.32767 0.15959 0.29190 0.11487 0.34935 0.18669 0.59494
SimplifiedLOF 34 0.31796 0.14745 0.29419 0.11774 0.35370 0.19212 0.60174
SimplifiedLOF 94 0.30825 0.13532 0.28858 0.11072 0.37552 0.21940 0.61382
SimplifiedLOF 100 0.31311 0.14138 0.28962 0.11203 0.37300 0.21625 0.61604
LoOP 72 0.32282 0.15352 0.27997 0.09997 0.36418 0.20522 0.60078
LoOP 96 0.31311 0.14138 0.28211 0.10264 0.36658 0.20822 0.61126
LoOP 99 0.31553 0.14442 0.28207 0.10259 0.36817 0.21021 0.61324
LDOF 21 0.30097 0.12621 0.28125 0.10156 0.33374 0.16717 0.56263
LDOF 87 0.30825 0.13532 0.27793 0.09741 0.36697 0.20872 0.59332
LDOF 98 0.31311 0.14138 0.27786 0.09732 0.37321 0.21652 0.59200
LDOF 99 0.32039 0.15049 0.27846 0.09808 0.37030 0.21288 0.59233
ODIN 87 0.31380 0.14225 0.28190 0.10237 0.37745 0.22181 0.62232
ODIN 99 0.30057 0.12571 0.28863 0.11078 0.38490 0.23112 0.63088
ODIN 100 0.30112 0.12640 0.28888 0.11110 0.38471 0.23089 0.63203
FastABOD 86 0.27427 0.09284 0.26429 0.08037 0.34429 0.18036 0.56956
FastABOD 94 0.28398 0.10498 0.26449 0.08062 0.34394 0.17993 0.57091
FastABOD 100 0.28398 0.10498 0.26552 0.08190 0.34391 0.17988 0.57205
KDEOS 11 0.22330 0.02913 0.24025 0.05031 0.33620 0.17025 0.53088
KDEOS 32 0.23058 0.03823 0.22923 0.03654 0.34720 0.18400 0.55830
KDEOS 92 0.25971 0.07464 0.23180 0.03975 0.34502 0.18128 0.56192
KDEOS 100 0.24757 0.05947 0.23566 0.04457 0.34566 0.18207 0.56376
LDF 94 0.33010 0.16262 0.35318 0.19148 0.40140 0.25175 0.67476
LDF 100 0.33010 0.16262 0.36823 0.21029 0.40935 0.26169 0.68315
INFLO 82 0.33738 0.17172 0.28025 0.10031 0.40028 0.25035 0.62421
INFLO 97 0.33010 0.16262 0.28991 0.11239 0.41058 0.26322 0.63748
INFLO 100 0.33010 0.16262 0.28991 0.11239 0.40987 0.26234 0.63648
COF 29 0.28883 0.11104 0.27952 0.09940 0.35130 0.18912 0.58151
COF 30 0.29369 0.11711 0.28429 0.10537 0.34646 0.18308 0.58228
COF 40 0.32039 0.15049 0.28785 0.10982 0.34063 0.17579 0.56877
COF 55 0.29612 0.12015 0.30038 0.12548 0.33376 0.16720 0.55099

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (72.9 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 90 0.39952 0.24967 0.41432 0.26816 0.41695 0.27145 0.66033
KNN 99 0.40920 0.26177 0.41800 0.27276 0.41550 0.26964 0.66399
KNN 100 0.40920 0.26177 0.41885 0.27383 0.41469 0.26862 0.66431
KNNW 23 0.40194 0.25269 0.33937 0.17452 0.40194 0.25269 0.57135
KNNW 86 0.39709 0.24664 0.37772 0.22243 0.40811 0.26041 0.61689
KNNW 100 0.39952 0.24967 0.38312 0.22918 0.40755 0.25970 0.62382
LOF 96 0.34140 0.17705 0.30617 0.13302 0.39747 0.24711 0.65740
LOF 98 0.33898 0.17403 0.30728 0.13442 0.39964 0.24982 0.65866
LOF 100 0.34140 0.17705 0.30905 0.13663 0.39918 0.24925 0.66052
SimplifiedLOF 24 0.34140 0.17705 0.29989 0.12518 0.35925 0.19935 0.59844
SimplifiedLOF 33 0.32446 0.15588 0.30118 0.12679 0.35688 0.19640 0.60096
SimplifiedLOF 92 0.31961 0.14982 0.29095 0.11401 0.37052 0.21344 0.61133
SimplifiedLOF 100 0.31961 0.14982 0.29175 0.11501 0.36829 0.21064 0.61392
LoOP 22 0.32203 0.15285 0.28878 0.11130 0.35660 0.19605 0.58586
LoOP 24 0.33172 0.16495 0.28871 0.11121 0.35338 0.19201 0.58695
LoOP 96 0.31961 0.14982 0.28341 0.10459 0.36505 0.20660 0.60930
LoOP 100 0.31961 0.14982 0.28453 0.10598 0.36317 0.20425 0.61149
LDOF 22 0.30024 0.12562 0.28419 0.10556 0.34173 0.17746 0.56916
LDOF 80 0.31719 0.14680 0.28208 0.10293 0.36397 0.20526 0.58976
LDOF 82 0.33172 0.16495 0.28248 0.10343 0.36278 0.20377 0.58937
LDOF 95 0.32203 0.15285 0.28168 0.10242 0.36645 0.20835 0.58843
ODIN 85 0.31356 0.14226 0.28281 0.10384 0.37465 0.21859 0.62006
ODIN 100 0.30180 0.12756 0.29094 0.11400 0.38031 0.22567 0.63237
FastABOD 90 0.29056 0.11352 0.26493 0.08150 0.34139 0.17703 0.57491
FastABOD 96 0.29056 0.11352 0.26532 0.08198 0.34231 0.17819 0.57578
FastABOD 100 0.29056 0.11352 0.26581 0.08259 0.34195 0.17774 0.57639
KDEOS 12 0.24939 0.06208 0.24424 0.05564 0.33537 0.16951 0.53815
KDEOS 15 0.26392 0.08024 0.23855 0.04853 0.33851 0.17344 0.54978
KDEOS 95 0.24939 0.06208 0.23476 0.04380 0.34473 0.18121 0.56129
KDEOS 100 0.24455 0.05603 0.23546 0.04467 0.34455 0.18098 0.56242
LDF 26 0.32930 0.16193 0.30870 0.13619 0.35552 0.19469 0.60439
LDF 100 0.31961 0.14982 0.36764 0.20984 0.41185 0.26507 0.68285
INFLO 80 0.33898 0.17403 0.28414 0.10550 0.39592 0.24517 0.62442
INFLO 94 0.33656 0.17100 0.28987 0.11266 0.40683 0.25880 0.63524
INFLO 98 0.33656 0.17100 0.28960 0.11232 0.40718 0.25924 0.63419
INFLO 99 0.33656 0.17100 0.29067 0.11366 0.40687 0.25886 0.63509
COF 27 0.28571 0.10747 0.27957 0.09979 0.34409 0.18040 0.57774
COF 30 0.27361 0.09234 0.28540 0.10707 0.34894 0.18647 0.57747
COF 50 0.30993 0.13772 0.30433 0.13073 0.33293 0.16647 0.55085

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.9 MB) Download raw algorithm evaluation table (73.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 74 0.40049 0.25061 0.35921 0.19902 0.42045 0.27557 0.67732
KNN 88 0.41262 0.26578 0.36089 0.20112 0.41722 0.27152 0.67954
KNN 100 0.40534 0.25667 0.36291 0.20363 0.41850 0.27313 0.68177
KNNW 72 0.38350 0.22937 0.34056 0.17570 0.41096 0.26370 0.66409
KNNW 96 0.40049 0.25061 0.34629 0.18287 0.40944 0.26179 0.66942
KNNW 100 0.40049 0.25061 0.34699 0.18374 0.40917 0.26146 0.67009
LOF 94 0.43689 0.29612 0.35089 0.18862 0.46488 0.33110 0.69786
LOF 97 0.44417 0.30522 0.35209 0.19012 0.46154 0.32692 0.69923
LOF 100 0.44175 0.30218 0.35327 0.19159 0.46225 0.32782 0.70046
SimplifiedLOF 97 0.39806 0.24757 0.32469 0.15586 0.43164 0.28955 0.66932
SimplifiedLOF 100 0.39806 0.24757 0.32616 0.15771 0.43505 0.29381 0.66994
LoOP 100 0.39563 0.24454 0.32019 0.15024 0.42960 0.28700 0.65536
LDOF 93 0.37379 0.21723 0.31058 0.13823 0.42251 0.27814 0.64580
LDOF 100 0.37136 0.21420 0.31383 0.14229 0.42626 0.28283 0.64870
ODIN 100 0.39988 0.24985 0.33687 0.17109 0.42745 0.28431 0.66918
FastABOD 91 0.28641 0.10801 0.27589 0.09486 0.36068 0.20085 0.58479
FastABOD 100 0.28641 0.10801 0.27682 0.09603 0.36184 0.20230 0.58562
KDEOS 71 0.26214 0.07767 0.25317 0.06646 0.37169 0.21461 0.60070
KDEOS 99 0.25485 0.06857 0.25567 0.06959 0.38391 0.22988 0.61141
KDEOS 100 0.25243 0.06553 0.25367 0.06709 0.38304 0.22879 0.61142
LDF 47 0.39563 0.24454 0.33265 0.16581 0.43625 0.29531 0.66098
LDF 54 0.41505 0.26881 0.33995 0.17493 0.43043 0.28804 0.66980
LDF 100 0.41505 0.26881 0.36646 0.20808 0.42857 0.28571 0.70344
INFLO 82 0.38592 0.23240 0.31876 0.14845 0.44103 0.30128 0.66990
INFLO 95 0.39563 0.24454 0.32289 0.15362 0.43575 0.29468 0.66849
INFLO 96 0.39320 0.24150 0.32410 0.15513 0.43985 0.29982 0.67231
INFLO 100 0.38592 0.23240 0.32452 0.15566 0.43982 0.29978 0.66920
COF 79 0.29126 0.11408 0.29102 0.11378 0.36953 0.21191 0.59480
COF 81 0.29854 0.12318 0.29224 0.11530 0.36473 0.20591 0.59578
COF 84 0.29612 0.12015 0.29003 0.11254 0.36127 0.20159 0.59619
COF 89 0.32524 0.15655 0.29011 0.11264 0.34853 0.18566 0.59270

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.9 MB) Download raw algorithm evaluation table (75.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 87 0.39467 0.24362 0.35290 0.19142 0.41090 0.26389 0.67055
KNN 96 0.40194 0.25269 0.35428 0.19315 0.41003 0.26281 0.67281
KNN 100 0.39952 0.24967 0.35493 0.19396 0.40879 0.26126 0.67359
KNNW 95 0.38983 0.23756 0.33860 0.17354 0.39828 0.24813 0.65719
KNNW 100 0.38741 0.23454 0.33942 0.17457 0.40069 0.25113 0.65824
LOF 96 0.43584 0.29505 0.34672 0.18370 0.45708 0.32160 0.68818
LOF 99 0.44068 0.30110 0.34773 0.18496 0.45464 0.31855 0.68965
LOF 100 0.44068 0.30110 0.34819 0.18553 0.45455 0.31843 0.69029
SimplifiedLOF 98 0.39467 0.24362 0.32287 0.15389 0.43227 0.29060 0.65984
SimplifiedLOF 99 0.39225 0.24059 0.32359 0.15479 0.43200 0.29026 0.66008
SimplifiedLOF 100 0.39225 0.24059 0.32351 0.15470 0.43098 0.28899 0.66022
LoOP 100 0.38983 0.23756 0.32009 0.15042 0.42520 0.28176 0.64804
LDOF 92 0.36320 0.20428 0.30862 0.13608 0.41903 0.27405 0.63839
LDOF 99 0.36320 0.20428 0.31160 0.13981 0.42667 0.28359 0.64234
LDOF 100 0.36077 0.20126 0.31178 0.14004 0.42654 0.28344 0.64273
ODIN 98 0.39661 0.24604 0.33981 0.17507 0.42557 0.28223 0.66467
ODIN 100 0.39467 0.24362 0.34106 0.17663 0.42771 0.28490 0.66616
FastABOD 24 0.26392 0.08024 0.26496 0.08153 0.35294 0.19147 0.57452
FastABOD 81 0.28329 0.10444 0.26999 0.08781 0.35103 0.18908 0.57905
FastABOD 100 0.28087 0.10142 0.27127 0.08942 0.35268 0.19115 0.58048
KDEOS 87 0.26392 0.08024 0.25240 0.06583 0.37705 0.22159 0.60313
KDEOS 99 0.25424 0.06813 0.25383 0.06763 0.38153 0.22719 0.60708
KDEOS 100 0.25666 0.07116 0.25476 0.06879 0.38051 0.22591 0.60698
LDF 57 0.39467 0.24362 0.33511 0.16919 0.42149 0.27712 0.65807
LDF 92 0.41162 0.26479 0.35394 0.19271 0.41527 0.26936 0.68667
LDF 100 0.40920 0.26177 0.35770 0.19742 0.41497 0.26897 0.69207
INFLO 97 0.39225 0.24059 0.32300 0.15406 0.43755 0.29719 0.67052
INFLO 98 0.39225 0.24059 0.32335 0.15449 0.44015 0.30044 0.67072
INFLO 99 0.38741 0.23454 0.32373 0.15497 0.43950 0.29962 0.66943
INFLO 100 0.38983 0.23756 0.32369 0.15492 0.43870 0.29863 0.67098
COF 77 0.30508 0.13167 0.28802 0.11035 0.35563 0.19483 0.57909
COF 84 0.31719 0.14680 0.28749 0.10968 0.35585 0.19510 0.58242
COF 90 0.32203 0.15285 0.28386 0.10514 0.34983 0.18758 0.58319
COF 98 0.30993 0.13772 0.28275 0.10376 0.35096 0.18899 0.58418

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO