Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#09)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (72.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 99 0.43447 0.29308 0.42806 0.28508 0.43966 0.29957 0.68189
KNN 100 0.43204 0.29005 0.42908 0.28635 0.44156 0.30195 0.68285
KNNW 43 0.41019 0.26274 0.35942 0.19927 0.41297 0.26621 0.59820
KNNW 99 0.40777 0.25971 0.38777 0.23471 0.41919 0.27399 0.63789
KNNW 100 0.40777 0.25971 0.38815 0.23519 0.41919 0.27399 0.63839
LOF 99 0.34951 0.18689 0.31139 0.13924 0.40881 0.26102 0.66586
LOF 100 0.34709 0.18386 0.31066 0.13832 0.40813 0.26016 0.66704
SimplifiedLOF 22 0.34223 0.17779 0.29696 0.12120 0.35526 0.19408 0.58484
SimplifiedLOF 32 0.32282 0.15352 0.29903 0.12379 0.35370 0.19213 0.59064
SimplifiedLOF 82 0.32282 0.15352 0.28622 0.10777 0.37140 0.21425 0.60634
SimplifiedLOF 100 0.32282 0.15352 0.28892 0.11115 0.36920 0.21150 0.61448
LoOP 26 0.33252 0.16566 0.28292 0.10365 0.34051 0.17564 0.57640
LoOP 32 0.31796 0.14745 0.28556 0.10695 0.34375 0.17969 0.58022
LoOP 83 0.31553 0.14442 0.27924 0.09905 0.36456 0.20570 0.60167
LoOP 99 0.32524 0.15655 0.28321 0.10401 0.36168 0.20210 0.60949
LDOF 21 0.32039 0.15049 0.27978 0.09973 0.33387 0.16734 0.55531
LDOF 24 0.31068 0.13835 0.28087 0.10109 0.33360 0.16700 0.55639
LDOF 91 0.31311 0.14138 0.27592 0.09490 0.36218 0.20272 0.58555
ODIN 87 0.31796 0.14745 0.28452 0.10565 0.37500 0.21875 0.62525
ODIN 99 0.30886 0.13607 0.29054 0.11317 0.38216 0.22770 0.63498
ODIN 100 0.31103 0.13878 0.29115 0.11394 0.38210 0.22762 0.63587
FastABOD 73 0.29126 0.11408 0.26478 0.08098 0.34370 0.17962 0.56775
FastABOD 94 0.28641 0.10801 0.26768 0.08460 0.34521 0.18151 0.57101
FastABOD 100 0.29126 0.11408 0.26821 0.08527 0.34483 0.18103 0.57191
KDEOS 16 0.25485 0.06857 0.24430 0.05537 0.33751 0.17188 0.54713
KDEOS 93 0.25971 0.07464 0.23634 0.04543 0.34704 0.18380 0.55527
KDEOS 100 0.25485 0.06857 0.23804 0.04755 0.34595 0.18243 0.55775
LDF 100 0.34223 0.17779 0.37993 0.22492 0.42249 0.27811 0.69553
INFLO 95 0.33495 0.16869 0.28698 0.10872 0.40683 0.25853 0.63662
INFLO 96 0.33738 0.17172 0.28711 0.10889 0.40685 0.25856 0.63643
INFLO 100 0.33495 0.16869 0.28801 0.11001 0.40825 0.26031 0.63481
COF 19 0.30825 0.13532 0.28984 0.11230 0.34550 0.18187 0.57849
COF 27 0.31553 0.14442 0.29018 0.11272 0.34628 0.18285 0.57052
COF 39 0.33252 0.16566 0.30297 0.12872 0.33945 0.17431 0.56608
COF 54 0.31068 0.13835 0.31636 0.14545 0.33347 0.16684 0.54702

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 99 0.40920 0.26177 0.41250 0.26589 0.41778 0.27249 0.67411
KNN 100 0.40920 0.26177 0.41274 0.26619 0.41881 0.27378 0.67411
KNNW 82 0.38741 0.23454 0.36152 0.20219 0.39426 0.24309 0.61508
KNNW 95 0.38741 0.23454 0.36709 0.20915 0.39683 0.24631 0.62307
KNNW 100 0.38499 0.23151 0.36934 0.21196 0.39474 0.24370 0.62584
LOF 99 0.34383 0.18008 0.29179 0.11506 0.39661 0.24604 0.65641
LOF 100 0.33898 0.17403 0.29236 0.11577 0.39563 0.24481 0.65720
SimplifiedLOF 16 0.31235 0.14075 0.27798 0.09780 0.34677 0.18376 0.57912
SimplifiedLOF 18 0.31961 0.14982 0.27624 0.09563 0.34359 0.17979 0.57885
SimplifiedLOF 73 0.29540 0.11957 0.26598 0.08281 0.35934 0.19946 0.59177
SimplifiedLOF 100 0.30993 0.13772 0.26988 0.08768 0.35800 0.19779 0.60135
LoOP 13 0.30751 0.13470 0.26930 0.08696 0.33293 0.16647 0.56308
LoOP 16 0.31477 0.14377 0.26887 0.08642 0.33917 0.17426 0.56913
LoOP 79 0.30266 0.12865 0.26113 0.07674 0.35523 0.19433 0.58969
LoOP 100 0.30508 0.13167 0.26392 0.08023 0.35267 0.19113 0.59697
LDOF 19 0.29056 0.11352 0.26973 0.08749 0.33347 0.16714 0.55555
LDOF 60 0.27119 0.08931 0.26304 0.07914 0.35192 0.19019 0.57878
LDOF 96 0.30266 0.12865 0.26276 0.07878 0.35667 0.19612 0.57613
LDOF 100 0.31235 0.14075 0.26271 0.07872 0.35314 0.19172 0.57609
ODIN 88 0.29540 0.11957 0.26487 0.08142 0.37522 0.21930 0.61591
ODIN 98 0.29263 0.11611 0.26997 0.08779 0.38286 0.22885 0.62562
ODIN 100 0.29464 0.11862 0.27115 0.08927 0.38264 0.22858 0.62738
FastABOD 92 0.29056 0.11352 0.26229 0.07820 0.34308 0.17914 0.56939
FastABOD 97 0.29298 0.11654 0.26274 0.07876 0.34289 0.17891 0.56999
FastABOD 100 0.29298 0.11654 0.26323 0.07937 0.34232 0.17820 0.57038
KDEOS 12 0.24213 0.05301 0.23307 0.04168 0.33470 0.16867 0.54196
KDEOS 19 0.25908 0.07419 0.23060 0.03859 0.34060 0.17605 0.55584
KDEOS 83 0.22034 0.02578 0.22146 0.02718 0.34344 0.17960 0.54303
LDF 97 0.32930 0.16193 0.36790 0.21016 0.42291 0.27890 0.69008
LDF 100 0.32930 0.16193 0.37548 0.21963 0.42420 0.28051 0.69300
INFLO 94 0.32930 0.16193 0.27094 0.08900 0.39918 0.24925 0.62447
INFLO 100 0.32688 0.15890 0.27305 0.09164 0.40163 0.25231 0.62645
COF 15 0.30508 0.13167 0.26349 0.07970 0.33867 0.17364 0.55876
COF 18 0.29298 0.11654 0.25867 0.07368 0.34347 0.17963 0.55679
COF 65 0.27603 0.09536 0.27963 0.09986 0.33293 0.16647 0.53066

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 86 0.40534 0.25667 0.36223 0.20279 0.41723 0.27154 0.68172
KNN 95 0.41262 0.26578 0.36364 0.20455 0.41564 0.26955 0.68326
KNN 100 0.40777 0.25971 0.36430 0.20538 0.41477 0.26846 0.68435
KNNW 98 0.39320 0.24150 0.34698 0.18372 0.40948 0.26184 0.66983
KNNW 99 0.39320 0.24150 0.34722 0.18403 0.40986 0.26233 0.67005
KNNW 100 0.39320 0.24150 0.34740 0.18426 0.40986 0.26233 0.67024
LOF 85 0.42718 0.28398 0.34568 0.18210 0.46263 0.32828 0.69051
LOF 94 0.43689 0.29612 0.35185 0.18982 0.46185 0.32731 0.69765
LOF 100 0.43689 0.29612 0.35490 0.19362 0.45771 0.32214 0.70083
SimplifiedLOF 98 0.38592 0.23240 0.32451 0.15564 0.42678 0.28347 0.66226
SimplifiedLOF 100 0.39078 0.23847 0.32497 0.15621 0.42518 0.28148 0.66255
LoOP 97 0.38592 0.23240 0.31828 0.14785 0.41543 0.26929 0.64799
LoOP 99 0.39078 0.23847 0.31905 0.14881 0.41879 0.27348 0.64777
LoOP 100 0.39320 0.24150 0.31949 0.14937 0.41714 0.27143 0.64783
LDOF 99 0.37379 0.21723 0.30967 0.13708 0.41651 0.27064 0.64005
LDOF 100 0.36893 0.21117 0.30955 0.13693 0.41856 0.27320 0.63998
ODIN 100 0.40049 0.25061 0.34260 0.17825 0.42998 0.28748 0.67042
FastABOD 35 0.29612 0.12015 0.28036 0.10045 0.35978 0.19972 0.59073
FastABOD 97 0.29369 0.11711 0.28478 0.10598 0.36203 0.20254 0.59601
FastABOD 99 0.29369 0.11711 0.28505 0.10631 0.36203 0.20254 0.59616
FastABOD 100 0.29369 0.11711 0.28499 0.10624 0.36160 0.20200 0.59621
KDEOS 43 0.25971 0.07464 0.23277 0.04096 0.35580 0.19475 0.57669
KDEOS 98 0.25243 0.06553 0.25417 0.06771 0.37395 0.21744 0.60477
KDEOS 100 0.25485 0.06857 0.25559 0.06949 0.37372 0.21715 0.60594
LDF 55 0.41262 0.26578 0.34309 0.17886 0.43784 0.29730 0.67307
LDF 85 0.42476 0.28095 0.36112 0.20140 0.42921 0.28652 0.69815
LDF 100 0.41990 0.27488 0.37092 0.21366 0.43509 0.29386 0.70926
INFLO 94 0.38835 0.23544 0.32498 0.15622 0.44428 0.30535 0.66975
INFLO 99 0.38350 0.22937 0.32668 0.15835 0.44727 0.30909 0.67231
INFLO 100 0.38592 0.23240 0.32785 0.15981 0.44725 0.30906 0.67446
COF 74 0.29854 0.12318 0.29755 0.12194 0.36394 0.20493 0.59007
COF 87 0.32039 0.15049 0.30160 0.12700 0.35726 0.19658 0.59602
COF 91 0.32524 0.15655 0.29885 0.12357 0.35525 0.19406 0.59527

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (75.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 96 0.39952 0.24967 0.34772 0.18494 0.41081 0.26378 0.67070
KNN 100 0.39952 0.24967 0.34830 0.18567 0.41150 0.26465 0.67144
KNNW 83 0.38983 0.23756 0.33022 0.16308 0.39888 0.24887 0.65492
KNNW 96 0.38257 0.22849 0.33286 0.16638 0.40103 0.25156 0.65767
KNNW 100 0.38499 0.23151 0.33363 0.16734 0.40068 0.25113 0.65844
LOF 97 0.43584 0.29505 0.33890 0.17392 0.44776 0.30995 0.68698
LOF 100 0.43341 0.29202 0.34025 0.17562 0.45191 0.31514 0.68868
SimplifiedLOF 98 0.38741 0.23454 0.31252 0.14096 0.42932 0.28691 0.65672
SimplifiedLOF 99 0.38983 0.23756 0.31282 0.14134 0.42842 0.28579 0.65679
SimplifiedLOF 100 0.38983 0.23756 0.31305 0.14162 0.42677 0.28372 0.65687
LoOP 98 0.38257 0.22849 0.30724 0.13436 0.42115 0.27670 0.64260
LoOP 99 0.38499 0.23151 0.30757 0.13478 0.41954 0.27469 0.64289
LoOP 100 0.38741 0.23454 0.30761 0.13483 0.41992 0.27517 0.64198
LDOF 90 0.36320 0.20428 0.29798 0.12279 0.40926 0.26184 0.63475
LDOF 99 0.35835 0.19823 0.30292 0.12897 0.41093 0.26393 0.64031
LDOF 100 0.36077 0.20126 0.30305 0.12913 0.41360 0.26726 0.64018
ODIN 98 0.38923 0.23681 0.33174 0.16498 0.42672 0.28366 0.66079
ODIN 100 0.38776 0.23497 0.33257 0.16601 0.42436 0.28071 0.66164
FastABOD 28 0.27845 0.09839 0.27208 0.09043 0.35718 0.19677 0.58444
FastABOD 47 0.28571 0.10747 0.27408 0.09293 0.35466 0.19362 0.58631
FastABOD 99 0.28329 0.10444 0.27738 0.09705 0.35637 0.19576 0.58998
FastABOD 100 0.28329 0.10444 0.27736 0.09703 0.35660 0.19604 0.59003
KDEOS 99 0.25666 0.07116 0.24679 0.05883 0.37895 0.22397 0.60610
KDEOS 100 0.25908 0.07419 0.24613 0.05800 0.38013 0.22545 0.60616
LDF 71 0.38741 0.23454 0.33497 0.16902 0.41860 0.27352 0.67238
LDF 91 0.40678 0.25874 0.34717 0.18426 0.41322 0.26679 0.68671
LDF 100 0.40194 0.25269 0.35198 0.19026 0.41648 0.27086 0.69313
INFLO 90 0.37530 0.21941 0.31288 0.14141 0.43620 0.29551 0.66342
INFLO 96 0.38741 0.23454 0.31442 0.14334 0.43478 0.29373 0.66059
INFLO 100 0.38499 0.23151 0.31531 0.14444 0.43357 0.29221 0.66478
COF 13 0.27361 0.09234 0.25124 0.06439 0.35227 0.19063 0.57181
COF 82 0.31719 0.14680 0.28195 0.10276 0.34175 0.17749 0.57795
COF 86 0.32446 0.15588 0.27975 0.10002 0.34336 0.17949 0.57798
COF 91 0.31235 0.14075 0.27927 0.09941 0.34477 0.18126 0.58348

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO