Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Cardiotocography (20% of outliers version#10)

Data set related to heart diseases. It describes 3 classes: normal, suspect, or pathological. Normal patients are treated as inliers and the remaining as outliers.

Download all data set variants used (8.8 MB). You can also access the original data. (CTG.xls)

Normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (71.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 97 0.42476 0.28095 0.42132 0.27665 0.42657 0.28322 0.67197
KNN 99 0.42233 0.27791 0.42242 0.27802 0.42943 0.28679 0.67292
KNN 100 0.42233 0.27791 0.42261 0.27827 0.42647 0.28309 0.67337
KNNW 84 0.40291 0.25364 0.37356 0.21695 0.41667 0.27083 0.61667
KNNW 96 0.40291 0.25364 0.37908 0.22385 0.42149 0.27686 0.62391
KNNW 100 0.40291 0.25364 0.38092 0.22615 0.42091 0.27613 0.62612
LOF 99 0.33495 0.16869 0.29662 0.12078 0.39437 0.24296 0.65413
LOF 100 0.33495 0.16869 0.29749 0.12186 0.39502 0.24377 0.65497
SimplifiedLOF 19 0.33981 0.17476 0.29445 0.11806 0.35101 0.18876 0.58711
SimplifiedLOF 84 0.30583 0.13228 0.27586 0.09483 0.35959 0.19949 0.59154
SimplifiedLOF 100 0.30340 0.12925 0.27886 0.09857 0.35613 0.19517 0.59936
LoOP 18 0.32524 0.15655 0.28265 0.10331 0.34122 0.17652 0.57666
LoOP 99 0.30340 0.12925 0.27397 0.09246 0.35137 0.18921 0.59506
LoOP 100 0.30097 0.12621 0.27440 0.09300 0.35127 0.18909 0.59635
LDOF 22 0.31553 0.14442 0.27927 0.09909 0.34093 0.17616 0.56274
LDOF 79 0.30583 0.13228 0.27332 0.09165 0.35677 0.19596 0.58168
LDOF 94 0.32524 0.15655 0.27309 0.09137 0.35707 0.19634 0.57940
LDOF 97 0.32282 0.15352 0.27313 0.09141 0.36149 0.20187 0.57878
ODIN 80 0.30022 0.12528 0.27401 0.09251 0.37220 0.21525 0.60790
ODIN 100 0.29350 0.11688 0.28267 0.10334 0.38288 0.22860 0.62691
FastABOD 76 0.28398 0.10498 0.26370 0.07963 0.33917 0.17396 0.56379
FastABOD 89 0.27913 0.09891 0.26544 0.08181 0.34012 0.17515 0.56587
FastABOD 99 0.28398 0.10498 0.26682 0.08352 0.33954 0.17442 0.56725
FastABOD 100 0.28398 0.10498 0.26654 0.08318 0.33954 0.17442 0.56727
KDEOS 15 0.23786 0.04733 0.23500 0.04375 0.34064 0.17580 0.54675
KDEOS 25 0.24029 0.05036 0.23204 0.04005 0.34483 0.18103 0.55547
KDEOS 97 0.25728 0.07160 0.23394 0.04243 0.34170 0.17712 0.54743
LDF 99 0.32524 0.15655 0.36993 0.21241 0.41237 0.26546 0.68748
LDF 100 0.33010 0.16262 0.37270 0.21587 0.41209 0.26511 0.68920
INFLO 95 0.33010 0.16262 0.27616 0.09520 0.38781 0.23476 0.61667
INFLO 97 0.32524 0.15655 0.27700 0.09624 0.39164 0.23955 0.61795
INFLO 100 0.32524 0.15655 0.27812 0.09766 0.38916 0.23645 0.61642
COF 18 0.29612 0.12015 0.29203 0.11503 0.34687 0.18359 0.58200
COF 42 0.32282 0.15352 0.29648 0.12060 0.33706 0.17132 0.54956
COF 53 0.29612 0.12015 0.30981 0.13726 0.33333 0.16667 0.53618

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 96 0.42373 0.27992 0.42294 0.27894 0.42999 0.28774 0.68077
KNN 100 0.42131 0.27690 0.42446 0.28083 0.43172 0.28991 0.68296
KNNW 91 0.39952 0.24967 0.37610 0.22041 0.40979 0.26251 0.63255
KNNW 98 0.39952 0.24967 0.37910 0.22416 0.41085 0.26383 0.63640
KNNW 100 0.39952 0.24967 0.38006 0.22535 0.41032 0.26317 0.63745
LOF 98 0.32930 0.16193 0.29834 0.12324 0.39938 0.24949 0.65851
LOF 99 0.32930 0.16193 0.29776 0.12251 0.40031 0.25066 0.65953
LOF 100 0.32930 0.16193 0.29968 0.12492 0.40000 0.25027 0.66062
SimplifiedLOF 21 0.32446 0.15588 0.28468 0.10617 0.34122 0.17682 0.57862
SimplifiedLOF 30 0.31961 0.14982 0.28827 0.11066 0.34362 0.17982 0.58411
SimplifiedLOF 99 0.30508 0.13167 0.27687 0.09641 0.36035 0.20073 0.60333
SimplifiedLOF 100 0.30266 0.12865 0.27698 0.09655 0.35929 0.19940 0.60374
LoOP 19 0.31235 0.14075 0.27361 0.09234 0.33397 0.16777 0.56743
LoOP 28 0.31235 0.14075 0.27571 0.09496 0.33492 0.16895 0.57573
LoOP 97 0.29540 0.11957 0.27140 0.08958 0.35548 0.19464 0.59863
LoOP 99 0.30024 0.12562 0.27209 0.09044 0.35459 0.19353 0.59971
LDOF 23 0.30024 0.12562 0.26962 0.08736 0.33355 0.16724 0.55005
LDOF 82 0.30751 0.13470 0.26616 0.08303 0.35902 0.19907 0.57632
LDOF 83 0.29782 0.12259 0.26577 0.08254 0.36096 0.20149 0.57586
LDOF 98 0.30993 0.13772 0.26734 0.08451 0.35075 0.18873 0.57606
ODIN 86 0.30024 0.12562 0.27322 0.09186 0.37500 0.21903 0.61907
ODIN 95 0.28692 0.10898 0.27798 0.09780 0.38293 0.22894 0.62741
ODIN 100 0.28899 0.11156 0.28104 0.10162 0.38253 0.22844 0.63120
FastABOD 77 0.28814 0.11049 0.26194 0.07776 0.34376 0.18000 0.57242
FastABOD 99 0.28571 0.10747 0.26502 0.08161 0.34470 0.18117 0.57604
FastABOD 100 0.28571 0.10747 0.26516 0.08178 0.34470 0.18117 0.57625
KDEOS 16 0.25908 0.07419 0.23526 0.04442 0.33991 0.17518 0.54628
KDEOS 24 0.24213 0.05301 0.23382 0.04262 0.33846 0.17338 0.55406
KDEOS 99 0.24455 0.05603 0.23116 0.03930 0.34775 0.18498 0.55340
LDF 100 0.33656 0.17100 0.37709 0.22164 0.42061 0.27602 0.69783
INFLO 83 0.32203 0.15285 0.27128 0.08943 0.39130 0.23941 0.61553
INFLO 96 0.31719 0.14680 0.27685 0.09639 0.40137 0.25198 0.62307
INFLO 98 0.31961 0.14982 0.27656 0.09602 0.40220 0.25303 0.62213
COF 16 0.28329 0.10444 0.27755 0.09726 0.35272 0.19119 0.56575
COF 23 0.27361 0.09234 0.27562 0.09485 0.34483 0.18133 0.56939
COF 38 0.30993 0.13772 0.29814 0.12299 0.33572 0.16996 0.55209
COF 51 0.28814 0.11049 0.30633 0.13322 0.33293 0.16647 0.53694

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 21 attributes, 2060 objects, 412 outliers (20.00%)

Download raw algorithm results (17.8 MB) Download raw algorithm evaluation table (73.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 96 0.40534 0.25667 0.35625 0.19532 0.41246 0.26557 0.68157
KNN 97 0.40534 0.25667 0.35612 0.19515 0.41424 0.26780 0.68164
KNN 100 0.40291 0.25364 0.35679 0.19599 0.41228 0.26535 0.68234
KNNW 97 0.39320 0.24150 0.33928 0.17410 0.40862 0.26078 0.66471
KNNW 100 0.39320 0.24150 0.33965 0.17456 0.40897 0.26122 0.66536
LOF 100 0.43689 0.29612 0.34499 0.18123 0.45696 0.32120 0.69463
SimplifiedLOF 98 0.38835 0.23544 0.31453 0.14316 0.42360 0.27950 0.65498
SimplifiedLOF 99 0.38835 0.23544 0.31488 0.14360 0.42412 0.28016 0.65532
SimplifiedLOF 100 0.38835 0.23544 0.31501 0.14376 0.42412 0.28016 0.65564
LoOP 93 0.37379 0.21723 0.30728 0.13411 0.41365 0.26707 0.64266
LoOP 97 0.38592 0.23240 0.30900 0.13625 0.41358 0.26698 0.64081
LoOP 100 0.38592 0.23240 0.31004 0.13755 0.41506 0.26882 0.64079
LDOF 98 0.36165 0.20206 0.30398 0.12997 0.41300 0.26625 0.63504
LDOF 99 0.36650 0.20813 0.30482 0.13103 0.41198 0.26497 0.63613
LDOF 100 0.36165 0.20206 0.30466 0.13082 0.41273 0.26591 0.63639
ODIN 100 0.39293 0.24117 0.33858 0.17323 0.42733 0.28416 0.67100
FastABOD 5 0.26942 0.08677 0.27379 0.09224 0.34575 0.18219 0.56992
FastABOD 28 0.28155 0.10194 0.26735 0.08419 0.35027 0.18784 0.57153
FastABOD 51 0.29126 0.11408 0.27041 0.08801 0.34642 0.18303 0.57410
FastABOD 100 0.28641 0.10801 0.27322 0.09153 0.34681 0.18351 0.57828
KDEOS 72 0.26942 0.08677 0.24989 0.06236 0.36354 0.20442 0.59078
KDEOS 100 0.26699 0.08374 0.25459 0.06824 0.36638 0.20798 0.60057
LDF 69 0.41019 0.26274 0.34147 0.17683 0.42842 0.28552 0.67687
LDF 92 0.41990 0.27488 0.35385 0.19231 0.42644 0.28305 0.69352
LDF 100 0.41748 0.27184 0.35724 0.19655 0.42623 0.28279 0.69813
INFLO 94 0.38592 0.23240 0.31705 0.14631 0.43420 0.29275 0.66723
INFLO 99 0.38107 0.22633 0.31943 0.14929 0.43931 0.29914 0.67180
INFLO 100 0.38350 0.22937 0.31974 0.14967 0.43852 0.29815 0.67311
COF 66 0.28641 0.10801 0.28450 0.10562 0.36047 0.20059 0.58320
COF 75 0.30825 0.13532 0.28827 0.11033 0.35525 0.19406 0.58698
COF 77 0.31068 0.13835 0.28820 0.11025 0.35484 0.19355 0.58782
COF 89 0.32282 0.15352 0.28103 0.10129 0.34576 0.18220 0.58655

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 21 attributes, 2068 objects, 413 outliers (19.97%)

Download raw algorithm results (17.9 MB) Download raw algorithm evaluation table (76.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 99 0.40920 0.26177 0.35826 0.19812 0.41612 0.27042 0.68566
KNN 100 0.40678 0.25874 0.35860 0.19854 0.41703 0.27155 0.68592
KNNW 96 0.38741 0.23454 0.33834 0.17323 0.40855 0.26095 0.66763
KNNW 99 0.38741 0.23454 0.33890 0.17392 0.41161 0.26478 0.66834
KNNW 100 0.38741 0.23454 0.33913 0.17421 0.41161 0.26478 0.66859
LOF 97 0.43826 0.29808 0.34231 0.17819 0.46089 0.32635 0.69371
LOF 100 0.43826 0.29808 0.34374 0.17997 0.45946 0.32457 0.69510
SimplifiedLOF 98 0.37772 0.22244 0.31083 0.13886 0.42378 0.27998 0.65205
SimplifiedLOF 100 0.38499 0.23151 0.31199 0.14030 0.42240 0.27826 0.65271
LoOP 93 0.36562 0.20731 0.30530 0.13194 0.41700 0.27151 0.64013
LoOP 97 0.37288 0.21639 0.30707 0.13415 0.41454 0.26844 0.64047
LoOP 98 0.37288 0.21639 0.30748 0.13466 0.41659 0.27100 0.64072
LoOP 100 0.37288 0.21639 0.30808 0.13542 0.41675 0.27120 0.64055
LDOF 95 0.35109 0.18916 0.29657 0.12103 0.40223 0.25306 0.62799
LDOF 99 0.34383 0.18008 0.29851 0.12346 0.40650 0.25840 0.62980
LDOF 100 0.34383 0.18008 0.29903 0.12411 0.40649 0.25838 0.63037
ODIN 100 0.40291 0.25390 0.33466 0.16863 0.42844 0.28581 0.66962
FastABOD 47 0.27361 0.09234 0.27226 0.09066 0.35482 0.19382 0.58060
FastABOD 99 0.27119 0.08931 0.27604 0.09538 0.35648 0.19589 0.58590
FastABOD 100 0.27119 0.08931 0.27603 0.09536 0.35667 0.19613 0.58597
KDEOS 85 0.27361 0.09234 0.24938 0.06207 0.36145 0.20210 0.59296
KDEOS 91 0.25908 0.07419 0.25112 0.06424 0.36584 0.20759 0.59613
KDEOS 100 0.25666 0.07116 0.24970 0.06247 0.36941 0.21205 0.59756
LDF 94 0.41889 0.27387 0.35811 0.19793 0.42594 0.28269 0.70528
LDF 99 0.41646 0.27085 0.36305 0.20411 0.42828 0.28561 0.70984
LDF 100 0.41646 0.27085 0.36299 0.20402 0.42717 0.28422 0.70996
INFLO 87 0.38015 0.22546 0.31256 0.14101 0.43279 0.29125 0.66265
INFLO 90 0.38499 0.23151 0.31345 0.14212 0.43381 0.29252 0.65921
INFLO 94 0.38741 0.23454 0.31289 0.14143 0.42870 0.28613 0.65747
INFLO 100 0.37772 0.22244 0.31490 0.14393 0.42981 0.28752 0.65916
COF 75 0.30993 0.13772 0.28173 0.10249 0.36302 0.20406 0.57284
COF 83 0.32203 0.15285 0.27895 0.09902 0.35381 0.19255 0.57324
COF 96 0.30508 0.13167 0.27564 0.09488 0.34987 0.18764 0.57447

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO