Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (22% of outliers)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2114 objects, 466 outliers (22.04%)

Download raw algorithm results (18.3 MB) Download raw algorithm evaluation table (70.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 97 0.43133 0.27053 0.44136 0.28340 0.44148 0.28355 0.66513
KNN 100 0.42704 0.26502 0.44215 0.28441 0.44012 0.28181 0.66669
KNNW 41 0.40987 0.24300 0.37129 0.19352 0.41207 0.24582 0.58176
KNNW 79 0.40773 0.24025 0.39381 0.22240 0.42402 0.26115 0.61008
KNNW 100 0.40987 0.24300 0.40216 0.23311 0.42317 0.26007 0.62187
LOF 98 0.36481 0.18520 0.31790 0.12502 0.41418 0.24853 0.64441
LOF 100 0.36266 0.18244 0.31951 0.12709 0.41626 0.25120 0.64696
SimplifiedLOF 19 0.34764 0.16317 0.30720 0.11129 0.36533 0.18587 0.57845
SimplifiedLOF 35 0.32189 0.13014 0.31084 0.11596 0.36691 0.18789 0.58248
SimplifiedLOF 98 0.32403 0.13289 0.29935 0.10122 0.38232 0.20766 0.59663
SimplifiedLOF 100 0.32403 0.13289 0.30035 0.10251 0.38124 0.20627 0.59792
LoOP 26 0.33476 0.14666 0.29388 0.09422 0.36124 0.18062 0.56969
LoOP 36 0.32189 0.13014 0.29707 0.09831 0.36124 0.18062 0.57022
LoOP 99 0.32189 0.13014 0.29374 0.09403 0.37765 0.20166 0.59458
LoOP 100 0.32403 0.13289 0.29461 0.09514 0.37756 0.20156 0.59500
LDOF 24 0.30687 0.11087 0.29213 0.09197 0.36166 0.18116 0.55211
LDOF 87 0.32189 0.13014 0.29044 0.08980 0.37386 0.19681 0.57821
LDOF 100 0.32618 0.13565 0.29173 0.09146 0.37529 0.19864 0.57689
ODIN 98 0.31813 0.12532 0.30061 0.10285 0.40130 0.23201 0.61926
ODIN 99 0.31695 0.12381 0.30095 0.10328 0.40343 0.23474 0.62034
ODIN 100 0.31623 0.12288 0.30248 0.10525 0.40340 0.23470 0.62115
FastABOD 88 0.28755 0.08610 0.27678 0.07228 0.36227 0.18194 0.55564
FastABOD 96 0.29185 0.09160 0.27750 0.07320 0.36180 0.18134 0.55668
FastABOD 100 0.29185 0.09160 0.27796 0.07379 0.36180 0.18134 0.55736
KDEOS 15 0.24893 0.03655 0.25470 0.04395 0.36278 0.18259 0.53743
KDEOS 21 0.26609 0.05857 0.25007 0.03802 0.36443 0.18471 0.54545
KDEOS 22 0.26395 0.05582 0.25064 0.03875 0.36407 0.18425 0.54736
KDEOS 98 0.25966 0.05031 0.24933 0.03707 0.36858 0.19004 0.54572
LDF 98 0.34979 0.16593 0.38580 0.21212 0.42685 0.26479 0.67427
LDF 100 0.34979 0.16593 0.38979 0.21724 0.43099 0.27009 0.67707
INFLO 94 0.33691 0.14941 0.28726 0.08572 0.37679 0.20057 0.59115
INFLO 99 0.33047 0.14115 0.29104 0.09056 0.38497 0.21106 0.59811
INFLO 100 0.32833 0.13840 0.29118 0.09074 0.38264 0.20807 0.59837
COF 20 0.30687 0.11087 0.29784 0.09930 0.37165 0.19397 0.56827
COF 23 0.31116 0.11638 0.29524 0.09596 0.37245 0.19500 0.56608
COF 42 0.33047 0.14115 0.30967 0.11447 0.36848 0.18991 0.55425
COF 58 0.30472 0.10812 0.32031 0.12812 0.36124 0.18062 0.53536

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2126 objects, 471 outliers (22.15%)

Download raw algorithm results (18.3 MB) Download raw algorithm evaluation table (71.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 87 0.43100 0.26906 0.43523 0.27451 0.43680 0.27652 0.65927
KNN 96 0.42887 0.26634 0.44103 0.28195 0.44219 0.28344 0.66472
KNN 100 0.43100 0.26906 0.44239 0.28370 0.44085 0.28172 0.66652
KNNW 44 0.41189 0.24452 0.37310 0.19469 0.41277 0.24564 0.58356
KNNW 79 0.40764 0.23906 0.39335 0.22070 0.42527 0.26171 0.60876
KNNW 100 0.41189 0.24452 0.40162 0.23133 0.42447 0.26068 0.62049
LOF 98 0.36730 0.18724 0.31734 0.12306 0.41479 0.24825 0.64211
LOF 99 0.36943 0.18997 0.31814 0.12409 0.41455 0.24793 0.64338
LOF 100 0.36730 0.18724 0.31897 0.12516 0.41344 0.24651 0.64463
SimplifiedLOF 19 0.34183 0.15451 0.30574 0.10816 0.36745 0.18743 0.57629
SimplifiedLOF 32 0.32059 0.12724 0.31031 0.11403 0.36819 0.18839 0.58180
SimplifiedLOF 98 0.32909 0.13815 0.29910 0.09963 0.38127 0.20518 0.59471
SimplifiedLOF 100 0.32909 0.13815 0.29994 0.10071 0.38110 0.20497 0.59594
LoOP 26 0.33121 0.14088 0.29424 0.09339 0.36273 0.18136 0.56929
LoOP 34 0.31847 0.12451 0.29722 0.09721 0.36273 0.18136 0.57160
LoOP 99 0.32696 0.13542 0.29324 0.09210 0.37683 0.19948 0.59208
LoOP 100 0.32909 0.13815 0.29417 0.09329 0.37791 0.20087 0.59171
LDOF 24 0.30573 0.10815 0.29331 0.09219 0.36315 0.18190 0.55257
LDOF 84 0.31635 0.12179 0.29112 0.08938 0.37310 0.19469 0.57678
LDOF 97 0.32272 0.12997 0.29062 0.08874 0.37592 0.19831 0.57459
LDOF 100 0.32909 0.13815 0.29121 0.08949 0.37388 0.19569 0.57461
ODIN 90 0.32059 0.12724 0.29742 0.09747 0.39520 0.22308 0.60980
ODIN 100 0.31741 0.12315 0.30248 0.10398 0.40600 0.23695 0.61982
FastABOD 78 0.28662 0.08360 0.27608 0.07006 0.36493 0.18420 0.55456
FastABOD 98 0.29512 0.09451 0.27810 0.07265 0.36469 0.18389 0.55762
FastABOD 100 0.29512 0.09451 0.27850 0.07317 0.36469 0.18389 0.55802
KDEOS 12 0.23779 0.02087 0.25832 0.04724 0.36392 0.18289 0.52249
KDEOS 22 0.26752 0.05906 0.25102 0.03787 0.36644 0.18614 0.54721
KDEOS 99 0.25902 0.04815 0.24877 0.03498 0.37034 0.19114 0.54480
LDF 100 0.35032 0.16542 0.38702 0.21257 0.42941 0.26702 0.67449
INFLO 88 0.33758 0.14906 0.28573 0.08246 0.37153 0.19267 0.58397
INFLO 99 0.33333 0.14361 0.29094 0.08915 0.38507 0.21006 0.59605
INFLO 100 0.33546 0.14633 0.29127 0.08958 0.38411 0.20884 0.59748
COF 23 0.29936 0.09997 0.29371 0.09270 0.37705 0.19976 0.55844
COF 29 0.29724 0.09724 0.29740 0.09744 0.37234 0.19371 0.55969
COF 43 0.32059 0.12724 0.30690 0.10965 0.37046 0.19130 0.54490
COF 58 0.29512 0.09451 0.31412 0.11893 0.36293 0.18163 0.52907

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2114 objects, 466 outliers (22.04%)

Download raw algorithm results (18.3 MB) Download raw algorithm evaluation table (75.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 67 0.41845 0.25401 0.37251 0.19508 0.42324 0.26016 0.66162
KNN 94 0.40987 0.24300 0.38001 0.20470 0.42799 0.26625 0.67040
KNN 100 0.41416 0.24851 0.38066 0.20553 0.42640 0.26420 0.67147
KNNW 97 0.40773 0.24025 0.36199 0.18158 0.42089 0.25713 0.65492
KNNW 100 0.40773 0.24025 0.36269 0.18248 0.42262 0.25936 0.65557
LOF 99 0.45279 0.29806 0.36716 0.18821 0.46823 0.31787 0.68648
LOF 100 0.45064 0.29530 0.36775 0.18897 0.47104 0.32147 0.68704
SimplifiedLOF 98 0.39700 0.22649 0.33629 0.14861 0.43699 0.27779 0.65003
SimplifiedLOF 99 0.39914 0.22924 0.33676 0.14922 0.43659 0.27728 0.65015
SimplifiedLOF 100 0.39914 0.22924 0.33694 0.14945 0.43592 0.27642 0.65019
LoOP 97 0.39485 0.22373 0.33081 0.14159 0.42997 0.26879 0.63754
LoOP 98 0.39270 0.22098 0.33124 0.14214 0.43209 0.27150 0.63697
LoOP 99 0.39700 0.22649 0.33173 0.14277 0.43169 0.27100 0.63725
LoOP 100 0.39700 0.22649 0.33199 0.14310 0.43169 0.27100 0.63722
LDOF 91 0.37983 0.20446 0.31925 0.12676 0.41567 0.25044 0.62401
LDOF 99 0.37554 0.19896 0.32457 0.13358 0.42324 0.26015 0.63004
LDOF 100 0.37768 0.20171 0.32496 0.13408 0.42226 0.25890 0.63039
ODIN 99 0.41309 0.24713 0.35338 0.17054 0.43438 0.27444 0.66014
ODIN 100 0.41144 0.24502 0.35451 0.17199 0.43407 0.27404 0.66088
FastABOD 17 0.27897 0.07509 0.28365 0.08108 0.37200 0.19442 0.56200
FastABOD 93 0.29614 0.09711 0.28720 0.08565 0.36812 0.18944 0.56674
FastABOD 100 0.29614 0.09711 0.28810 0.08680 0.36830 0.18967 0.56733
KDEOS 91 0.27897 0.07509 0.27005 0.06364 0.38880 0.21597 0.59212
KDEOS 99 0.27897 0.07509 0.27046 0.06418 0.39270 0.22098 0.59557
KDEOS 100 0.27253 0.06683 0.27186 0.06597 0.39229 0.22045 0.59576
LDF 73 0.42060 0.25677 0.36865 0.19013 0.44045 0.28223 0.67503
LDF 99 0.43348 0.27328 0.38570 0.21200 0.43726 0.27814 0.69309
LDF 100 0.43348 0.27328 0.38584 0.21217 0.43726 0.27814 0.69358
INFLO 87 0.39914 0.22924 0.32578 0.13514 0.43009 0.26894 0.64136
INFLO 92 0.39700 0.22649 0.32915 0.13946 0.43924 0.28067 0.64544
INFLO 99 0.39485 0.22373 0.33064 0.14136 0.43297 0.27264 0.64834
INFLO 100 0.39485 0.22373 0.33137 0.14231 0.43247 0.27200 0.64794
COF 83 0.31545 0.12188 0.30445 0.10777 0.37034 0.19229 0.57198
COF 93 0.32833 0.13840 0.29934 0.10121 0.37308 0.19581 0.57512
COF 99 0.31116 0.11638 0.30122 0.10363 0.37516 0.19848 0.57731
COF 100 0.31116 0.11638 0.30220 0.10488 0.37447 0.19759 0.57848

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2126 objects, 471 outliers (22.15%)

Download raw algorithm results (18.3 MB) Download raw algorithm evaluation table (75.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 73 0.42251 0.25815 0.37577 0.19812 0.42487 0.26119 0.66262
KNN 98 0.41189 0.24452 0.38172 0.20576 0.42944 0.26706 0.66998
KNN 100 0.41401 0.24725 0.38172 0.20576 0.42857 0.26595 0.67016
KNNW 96 0.40977 0.24179 0.36313 0.18188 0.42323 0.25909 0.65383
KNNW 100 0.40977 0.24179 0.36401 0.18301 0.42476 0.26106 0.65472
LOF 99 0.45435 0.29907 0.36847 0.18874 0.46980 0.31891 0.68417
LOF 100 0.45435 0.29907 0.36901 0.18944 0.47454 0.32500 0.68494
SimplifiedLOF 100 0.40340 0.23361 0.33849 0.15023 0.43848 0.27868 0.65037
LoOP 97 0.39703 0.22543 0.33190 0.14176 0.43088 0.26891 0.63726
LoOP 99 0.39703 0.22543 0.33323 0.14347 0.43238 0.27085 0.63806
LoOP 100 0.39703 0.22543 0.33335 0.14363 0.43148 0.26968 0.63736
LDOF 91 0.38004 0.20361 0.31941 0.12572 0.41558 0.24926 0.62323
LDOF 99 0.37580 0.19815 0.32498 0.13287 0.42570 0.26226 0.62952
LDOF 100 0.37580 0.19815 0.32535 0.13335 0.42392 0.25998 0.62985
ODIN 99 0.41491 0.24839 0.35448 0.17077 0.43395 0.27285 0.65877
ODIN 100 0.41462 0.24802 0.35525 0.17176 0.43395 0.27285 0.65943
FastABOD 17 0.27813 0.07269 0.28504 0.08156 0.37309 0.19467 0.56290
FastABOD 91 0.29724 0.09724 0.28881 0.08641 0.37091 0.19187 0.56802
FastABOD 100 0.29724 0.09724 0.28971 0.08757 0.37072 0.19163 0.56873
KDEOS 92 0.28450 0.08088 0.27014 0.06243 0.39030 0.21678 0.59195
KDEOS 100 0.27601 0.06997 0.27150 0.06418 0.39457 0.22227 0.59509
LDF 81 0.42675 0.26361 0.37307 0.19465 0.43935 0.27979 0.67627
LDF 100 0.43524 0.27452 0.38590 0.21113 0.43524 0.27452 0.68984
INFLO 87 0.40127 0.23088 0.32654 0.13487 0.43137 0.26955 0.63976
INFLO 94 0.39278 0.21997 0.33180 0.14164 0.44190 0.28307 0.64566
INFLO 99 0.39703 0.22543 0.33302 0.14320 0.43750 0.27742 0.64983
INFLO 100 0.39278 0.21997 0.33326 0.14351 0.43701 0.27679 0.64920
COF 84 0.31635 0.12179 0.30164 0.10290 0.37659 0.19917 0.56788
COF 91 0.32059 0.12724 0.29795 0.09815 0.38001 0.20356 0.56729
COF 94 0.32272 0.12997 0.29733 0.09735 0.37753 0.20038 0.56956
COF 98 0.30361 0.10542 0.29773 0.09787 0.37779 0.20071 0.57252

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO