Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (20% of outliers version#04)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (28.3 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 6 0.29589 0.11986 0.28363 0.10454 0.41462 0.26828 0.66416
KNN 10 0.30222 0.12777 0.27814 0.09767 0.40609 0.25761 0.65792
KNNW 9 0.29589 0.11986 0.26664 0.08330 0.39676 0.24595 0.63908
KNNW 14 0.28797 0.10997 0.27130 0.08912 0.40855 0.26069 0.64887
KNNW 23 0.28797 0.10997 0.26995 0.08744 0.40405 0.25506 0.65153
LOF 15 0.22468 0.03085 0.21637 0.02047 0.35080 0.18850 0.55650
LOF 95 0.20728 0.00910 0.22655 0.03319 0.36957 0.21196 0.56854
LOF 98 0.21044 0.01305 0.22637 0.03296 0.37001 0.21252 0.56816
SimplifiedLOF 1 0.22310 0.02888 0.22728 0.03410 0.33333 0.16667 0.49465
SimplifiedLOF 2 0.25475 0.06843 0.22381 0.02976 0.33421 0.16777 0.49686
SimplifiedLOF 34 0.19778 -0.00277 0.20460 0.00575 0.34847 0.18559 0.53545
SimplifiedLOF 94 0.15032 -0.06210 0.20944 0.01180 0.35678 0.19598 0.51598
LoOP 1 0.22310 0.02888 0.22902 0.03627 0.33333 0.16667 0.49537
LoOP 2 0.23259 0.04074 0.21863 0.02329 0.33333 0.16667 0.50282
LoOP 41 0.20886 0.01108 0.21165 0.01456 0.35454 0.19318 0.55053
LoOP 99 0.20570 0.00712 0.21942 0.02427 0.35880 0.19849 0.54618
LDOF 2 0.22152 0.02690 0.22787 0.03484 0.33493 0.16866 0.49190
LDOF 4 0.24209 0.05261 0.21625 0.02031 0.33430 0.16788 0.50591
LDOF 47 0.19462 -0.00672 0.20994 0.01242 0.35741 0.19676 0.54448
LDOF 51 0.20728 0.00910 0.21127 0.01409 0.35678 0.19597 0.54743
ODIN 47 0.23570 0.04463 0.23130 0.03912 0.35942 0.19928 0.57860
ODIN 83 0.26134 0.07667 0.23667 0.04584 0.35707 0.19633 0.58365
ODIN 100 0.25343 0.06679 0.23805 0.04756 0.35717 0.19646 0.58461
FastABOD 3 0.24525 0.05657 0.22745 0.03431 0.33726 0.17157 0.54605
FastABOD 55 0.25949 0.07437 0.23068 0.03835 0.33680 0.17100 0.54513
FastABOD 69 0.25633 0.07041 0.23132 0.03915 0.33680 0.17100 0.54569
FastABOD 99 0.25791 0.07239 0.23098 0.03872 0.33710 0.17137 0.54648
KDEOS 3 0.21677 0.02097 0.21413 0.01766 0.33738 0.17173 0.47914
KDEOS 41 0.21994 0.02492 0.20622 0.00777 0.34051 0.17563 0.51753
KDEOS 98 0.21044 0.01305 0.20537 0.00671 0.35460 0.19325 0.53659
KDEOS 100 0.21677 0.02097 0.20552 0.00690 0.35519 0.19398 0.53593
LDF 94 0.25000 0.06250 0.24798 0.05997 0.40644 0.25805 0.62693
LDF 100 0.25000 0.06250 0.25058 0.06323 0.40602 0.25752 0.63162
INFLO 69 0.21994 0.02492 0.22113 0.02641 0.36321 0.20401 0.54821
INFLO 93 0.21203 0.01503 0.22628 0.03285 0.36652 0.20815 0.56653
INFLO 94 0.21361 0.01701 0.22624 0.03280 0.36710 0.20888 0.56678
COF 1 0.22152 0.02690 0.22712 0.03390 0.33342 0.16678 0.49424
COF 2 0.25791 0.07239 0.22363 0.02954 0.33333 0.16667 0.49167
COF 33 0.18513 -0.01859 0.19994 -0.00008 0.33693 0.17116 0.51189
COF 69 0.15506 -0.05617 0.18608 -0.01740 0.34400 0.18000 0.48329

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (29.0 MB) Download raw algorithm evaluation table (75.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.33572 0.16966 0.30921 0.13651 0.43065 0.28832 0.69605
KNN 8 0.33716 0.17145 0.30483 0.13103 0.43142 0.28928 0.69169
KNNW 5 0.33429 0.16786 0.28545 0.10682 0.39574 0.24467 0.64419
KNNW 13 0.32855 0.16069 0.29741 0.12176 0.42580 0.28224 0.67900
KNNW 34 0.31707 0.14634 0.28901 0.11126 0.42441 0.28052 0.68189
LOF 2 0.20516 0.00646 0.21425 0.01781 0.34092 0.17614 0.52741
LOF 5 0.21090 0.01363 0.20833 0.01041 0.34318 0.17898 0.52736
LOF 15 0.20086 0.00108 0.20832 0.01040 0.35528 0.19411 0.54962
LOF 66 0.11908 -0.10115 0.19017 -0.01229 0.36944 0.21180 0.52635
SimplifiedLOF 2 0.24390 0.05488 0.22114 0.02643 0.33397 0.16747 0.52004
SimplifiedLOF 15 0.18221 -0.02224 0.19873 -0.00158 0.34468 0.18085 0.52472
SimplifiedLOF 81 0.12482 -0.09397 0.18717 -0.01603 0.36653 0.20816 0.51041
LoOP 2 0.22382 0.02977 0.22454 0.03068 0.33333 0.16667 0.53610
LoOP 16 0.18651 -0.01686 0.20548 0.00686 0.34461 0.18076 0.53642
LoOP 99 0.13486 -0.08142 0.19705 -0.00369 0.36861 0.21076 0.52827
LDOF 2 0.21090 0.01363 0.20746 0.00932 0.33381 0.16727 0.48081
LDOF 5 0.21521 0.01901 0.20012 0.00015 0.33373 0.16717 0.48337
LDOF 78 0.15495 -0.05631 0.19964 -0.00046 0.35994 0.19992 0.53066
LDOF 100 0.14204 -0.07245 0.19784 -0.00270 0.36343 0.20429 0.52580
ODIN 18 0.23022 0.03777 0.22816 0.03520 0.35907 0.19884 0.57248
ODIN 28 0.21995 0.02494 0.22367 0.02959 0.36439 0.20549 0.56960
ODIN 94 0.23582 0.04478 0.22629 0.03287 0.35196 0.18996 0.57348
ODIN 99 0.23338 0.04173 0.22754 0.03442 0.35294 0.19118 0.57494
FastABOD 57 0.27403 0.09254 0.23794 0.04742 0.36096 0.20120 0.58809
FastABOD 59 0.27403 0.09254 0.23773 0.04717 0.36121 0.20151 0.58782
FastABOD 70 0.27116 0.08895 0.23974 0.04968 0.35990 0.19988 0.58952
KDEOS 3 0.21951 0.02439 0.21771 0.02214 0.33656 0.17071 0.50867
KDEOS 77 0.19225 -0.00968 0.20334 0.00417 0.35712 0.19640 0.54116
KDEOS 100 0.18795 -0.01506 0.20527 0.00659 0.35631 0.19539 0.54704
LDF 5 0.26255 0.07819 0.24720 0.05900 0.36579 0.20724 0.58703
LDF 8 0.23529 0.04412 0.23725 0.04657 0.37816 0.22270 0.60676
LDF 10 0.19512 -0.00610 0.22284 0.02855 0.38204 0.22755 0.58971
INFLO 1 0.20086 0.00108 0.20606 0.00757 0.33494 0.16867 0.50732
INFLO 4 0.21090 0.01363 0.20019 0.00024 0.33382 0.16727 0.51192
INFLO 17 0.19225 -0.00968 0.20307 0.00383 0.35714 0.19643 0.54183
INFLO 94 0.13056 -0.08680 0.19359 -0.00801 0.37235 0.21543 0.52261
COF 1 0.22812 0.03515 0.22325 0.02907 0.34374 0.17968 0.53591
COF 2 0.23386 0.04232 0.22133 0.02666 0.34043 0.17553 0.53393
COF 79 0.13486 -0.08142 0.19222 -0.00973 0.35970 0.19963 0.51005

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (27.4 MB) Download raw algorithm evaluation table (72.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 8 0.45728 0.32160 0.44659 0.30824 0.46509 0.33136 0.74661
KNN 16 0.43987 0.29984 0.44168 0.30210 0.47188 0.33985 0.74573
KNN 65 0.42722 0.28402 0.41870 0.27337 0.46698 0.33373 0.74827
KNNW 11 0.45886 0.32358 0.44224 0.30281 0.46470 0.33088 0.73740
KNNW 17 0.45411 0.31764 0.44461 0.30576 0.46246 0.32807 0.74208
KNNW 25 0.44937 0.31171 0.44277 0.30346 0.47066 0.33833 0.74437
KNNW 88 0.42880 0.28600 0.42346 0.27933 0.46616 0.33269 0.74764
LOF 84 0.31804 0.14755 0.28646 0.10807 0.36630 0.20787 0.61456
LOF 95 0.31013 0.13766 0.29183 0.11479 0.37312 0.21640 0.62161
LOF 100 0.30696 0.13370 0.29464 0.11830 0.37240 0.21550 0.62526
SimplifiedLOF 44 0.25949 0.07437 0.24291 0.05364 0.35490 0.19362 0.56604
SimplifiedLOF 95 0.29272 0.11590 0.28329 0.10411 0.34796 0.18495 0.59103
SimplifiedLOF 100 0.29272 0.11590 0.28630 0.10788 0.34925 0.18656 0.59305
LoOP 1 0.25316 0.06646 0.25064 0.06330 0.33333 0.16667 0.50679
LoOP 55 0.25000 0.06250 0.22753 0.03441 0.34795 0.18494 0.55353
LoOP 98 0.26582 0.08228 0.24829 0.06037 0.33866 0.17333 0.56364
LoOP 99 0.26424 0.08030 0.24869 0.06086 0.33846 0.17308 0.56432
LDOF 2 0.25475 0.06843 0.24169 0.05211 0.33430 0.16788 0.46573
LDOF 55 0.17089 -0.03639 0.18832 -0.01460 0.33458 0.16822 0.47732
LDOF 100 0.21044 0.01305 0.20642 0.00802 0.33449 0.16811 0.50415
ODIN 23 0.14813 -0.06484 0.20110 0.00138 0.37263 0.21579 0.53833
ODIN 89 0.22785 0.03481 0.20902 0.01127 0.35784 0.19730 0.54311
ODIN 93 0.22251 0.02813 0.20972 0.01215 0.35787 0.19734 0.54402
ODIN 99 0.22569 0.03211 0.21003 0.01254 0.35832 0.19790 0.54387
FastABOD 3 0.41930 0.27413 0.40352 0.25439 0.45043 0.31303 0.72197
FastABOD 4 0.41930 0.27413 0.40455 0.25568 0.45143 0.31428 0.72323
FastABOD 5 0.41456 0.26820 0.40385 0.25481 0.45482 0.31853 0.72335
FastABOD 25 0.40981 0.26226 0.40207 0.25259 0.46060 0.32575 0.72261
KDEOS 3 0.21994 0.02492 0.23022 0.03777 0.33665 0.17082 0.48769
KDEOS 97 0.20411 0.00514 0.22000 0.02500 0.35525 0.19406 0.56355
KDEOS 100 0.20886 0.01108 0.22117 0.02646 0.35465 0.19331 0.56462
LDF 86 0.38608 0.23259 0.38882 0.23602 0.42951 0.28689 0.69309
LDF 100 0.40190 0.25237 0.40605 0.25756 0.42687 0.28358 0.70426
INFLO 84 0.26899 0.08623 0.26181 0.07727 0.43030 0.28788 0.61001
INFLO 89 0.27690 0.09612 0.26383 0.07979 0.43207 0.29009 0.60929
INFLO 95 0.28639 0.10799 0.26433 0.08042 0.42747 0.28434 0.60483
INFLO 97 0.28481 0.10601 0.26647 0.08309 0.43193 0.28992 0.60785
COF 96 0.32120 0.15150 0.34959 0.18698 0.38309 0.22887 0.63042
COF 99 0.32120 0.15150 0.35234 0.19043 0.38003 0.22504 0.63312
COF 100 0.32437 0.15546 0.35309 0.19137 0.37662 0.22077 0.63171

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (28.6 MB) Download raw algorithm evaluation table (73.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.46341 0.32927 0.45366 0.31708 0.47291 0.34114 0.76190
KNN 33 0.44476 0.30595 0.44224 0.30280 0.47584 0.34480 0.76691
KNN 100 0.43902 0.29878 0.42910 0.28638 0.48146 0.35183 0.76532
KNNW 15 0.46198 0.32747 0.45140 0.31425 0.46911 0.33638 0.75864
KNNW 19 0.45911 0.32389 0.45211 0.31513 0.47475 0.34343 0.76113
KNNW 52 0.44476 0.30595 0.44467 0.30584 0.47707 0.34634 0.76579
KNNW 63 0.44333 0.30416 0.44229 0.30286 0.48045 0.35056 0.76562
LOF 98 0.29412 0.11765 0.27029 0.08786 0.37865 0.22332 0.62783
LOF 99 0.28838 0.11047 0.27139 0.08924 0.38162 0.22703 0.62915
LOF 100 0.29125 0.11406 0.27235 0.09044 0.38108 0.22635 0.63041
SimplifiedLOF 2 0.25825 0.07281 0.21680 0.02100 0.33349 0.16687 0.50412
SimplifiedLOF 100 0.23529 0.04412 0.23402 0.04252 0.35512 0.19391 0.57778
LoOP 1 0.23099 0.03874 0.23072 0.03840 0.33333 0.16667 0.50027
LoOP 97 0.21377 0.01722 0.21807 0.02259 0.34043 0.17553 0.55100
LoOP 100 0.21377 0.01722 0.22029 0.02537 0.33966 0.17458 0.55257
LDOF 2 0.21951 0.02439 0.21069 0.01336 0.33333 0.16667 0.45014
LDOF 69 0.15782 -0.05273 0.17923 -0.02597 0.34312 0.17889 0.47516
LDOF 100 0.17504 -0.03121 0.19041 -0.01199 0.34096 0.17620 0.49431
ODIN 1 0.20100 0.00125 0.21176 0.01470 0.35447 0.19309 0.53804
ODIN 20 0.15629 -0.05463 0.20610 0.00762 0.37912 0.22391 0.54983
ODIN 22 0.15914 -0.05107 0.20651 0.00814 0.37894 0.22368 0.55221
FastABOD 83 0.41607 0.27009 0.42141 0.27676 0.47355 0.34194 0.74381
FastABOD 98 0.41607 0.27009 0.42174 0.27717 0.47433 0.34292 0.74410
FastABOD 100 0.41607 0.27009 0.42175 0.27719 0.47398 0.34248 0.74412
KDEOS 3 0.22525 0.03156 0.21068 0.01335 0.33445 0.16807 0.50030
KDEOS 96 0.19082 -0.01148 0.21612 0.02015 0.35418 0.19273 0.55453
KDEOS 100 0.19512 -0.00610 0.21892 0.02364 0.35335 0.19168 0.55810
LDF 93 0.37877 0.22346 0.38829 0.23537 0.41660 0.27075 0.69824
LDF 100 0.39598 0.24498 0.39778 0.24722 0.41564 0.26955 0.70355
INFLO 100 0.25108 0.06385 0.24749 0.05936 0.44153 0.30192 0.61337
COF 100 0.27260 0.09075 0.26492 0.08116 0.37416 0.21770 0.61656

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO