Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

SpamBase (20% of outliers version#03)

A data set representing emails classified as spam (outliers) or nonspam.

Download all data set variants used (25.4 MB). You can also access the original data. (spambase.data)

Normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (28.3 MB) Download raw algorithm evaluation table (73.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 5 0.31646 0.14557 0.28542 0.10678 0.41521 0.26902 0.66355
KNN 6 0.31013 0.13766 0.28593 0.10742 0.41319 0.26649 0.66527
KNN 7 0.31329 0.14161 0.28530 0.10662 0.41395 0.26744 0.66529
KNN 19 0.32120 0.15150 0.27117 0.08896 0.39982 0.24977 0.65146
KNNW 14 0.30222 0.12777 0.27529 0.09412 0.39403 0.24254 0.64757
KNNW 18 0.29747 0.12184 0.27421 0.09276 0.39923 0.24903 0.64953
KNNW 24 0.30696 0.13370 0.27299 0.09123 0.39799 0.24749 0.65060
KNNW 45 0.31013 0.13766 0.26827 0.08534 0.39590 0.24487 0.64637
LOF 87 0.25791 0.07239 0.24180 0.05225 0.37263 0.21579 0.60331
LOF 96 0.25791 0.07239 0.24208 0.05260 0.37114 0.21393 0.60436
LOF 97 0.26424 0.08030 0.24203 0.05253 0.37067 0.21334 0.60437
SimplifiedLOF 1 0.21994 0.02492 0.22742 0.03427 0.33377 0.16722 0.49321
SimplifiedLOF 2 0.22310 0.02888 0.20919 0.01148 0.33404 0.16755 0.48118
SimplifiedLOF 85 0.17089 -0.03639 0.22172 0.02714 0.36251 0.20314 0.55445
SimplifiedLOF 100 0.18038 -0.02453 0.22097 0.02621 0.36068 0.20085 0.55657
LoOP 92 0.23576 0.04470 0.23097 0.03871 0.36558 0.20698 0.58008
LoOP 99 0.24051 0.05063 0.23217 0.04021 0.36523 0.20654 0.58247
LoOP 100 0.24051 0.05063 0.23252 0.04065 0.36487 0.20609 0.58308
LDOF 2 0.21044 0.01305 0.22542 0.03177 0.33519 0.16899 0.48646
LDOF 3 0.21361 0.01701 0.20609 0.00762 0.33333 0.16667 0.48025
LDOF 100 0.20411 0.00514 0.22454 0.03068 0.35973 0.19967 0.56158
ODIN 75 0.23903 0.04879 0.23493 0.04366 0.35456 0.19319 0.58019
ODIN 82 0.25056 0.06320 0.23570 0.04463 0.35132 0.18915 0.58048
ODIN 100 0.24216 0.05270 0.23778 0.04723 0.35278 0.19097 0.58258
FastABOD 3 0.25633 0.07041 0.23121 0.03902 0.33680 0.17100 0.54975
FastABOD 4 0.25949 0.07437 0.23075 0.03843 0.33689 0.17112 0.54690
FastABOD 93 0.24684 0.05854 0.23070 0.03837 0.33734 0.17168 0.54675
KDEOS 3 0.21361 0.01701 0.21654 0.02068 0.33433 0.16791 0.47270
KDEOS 100 0.20253 0.00316 0.20544 0.00679 0.35499 0.19374 0.54302
LDF 97 0.26582 0.08228 0.24574 0.05717 0.39517 0.24396 0.61810
LDF 100 0.25949 0.07437 0.24702 0.05877 0.39673 0.24592 0.62019
INFLO 75 0.25633 0.07041 0.23716 0.04645 0.36476 0.20595 0.59291
INFLO 94 0.24367 0.05459 0.23932 0.04915 0.36738 0.20922 0.60016
INFLO 97 0.24367 0.05459 0.23961 0.04952 0.36699 0.20874 0.60112
INFLO 100 0.24684 0.05854 0.23972 0.04965 0.36636 0.20796 0.60090
COF 1 0.21994 0.02492 0.22712 0.03390 0.33413 0.16766 0.49281
COF 2 0.23892 0.04866 0.21402 0.01752 0.33333 0.16667 0.47718
COF 49 0.21361 0.01701 0.21150 0.01438 0.34765 0.18456 0.54302
COF 74 0.20570 0.00712 0.20579 0.00724 0.35348 0.19185 0.52749

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (29.0 MB) Download raw algorithm evaluation table (74.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.33572 0.16966 0.30827 0.13534 0.43745 0.29681 0.69240
KNN 9 0.32281 0.15352 0.30580 0.13225 0.43537 0.29422 0.69402
KNNW 10 0.31851 0.14813 0.28902 0.11127 0.41285 0.26606 0.66273
KNNW 15 0.31133 0.13917 0.29287 0.11609 0.41946 0.27433 0.67422
KNNW 30 0.31420 0.14275 0.28879 0.11098 0.42597 0.28246 0.67895
KNNW 35 0.31277 0.14096 0.28836 0.11045 0.42494 0.28117 0.67900
LOF 3 0.23529 0.04412 0.23133 0.03916 0.33470 0.16837 0.53073
LOF 4 0.25108 0.06385 0.22847 0.03559 0.33601 0.17001 0.53561
LOF 23 0.19512 -0.00610 0.20824 0.01030 0.36272 0.20340 0.55421
LOF 87 0.12769 -0.09039 0.20354 0.00442 0.36658 0.20822 0.52938
SimplifiedLOF 4 0.25825 0.07281 0.24032 0.05040 0.33357 0.16697 0.52896
SimplifiedLOF 100 0.13056 -0.08680 0.19372 -0.00785 0.35939 0.19924 0.50586
LoOP 4 0.24677 0.05846 0.23661 0.04576 0.33333 0.16667 0.53429
LoOP 6 0.25108 0.06385 0.23125 0.03906 0.33333 0.16667 0.53600
LoOP 53 0.18795 -0.01506 0.20436 0.00544 0.35724 0.19654 0.53925
LoOP 91 0.15208 -0.05990 0.20240 0.00300 0.36220 0.20274 0.53030
LDOF 6 0.26112 0.07640 0.22647 0.03308 0.33413 0.16767 0.50380
LDOF 79 0.17934 -0.02582 0.20700 0.00875 0.36482 0.20603 0.54209
LDOF 82 0.17934 -0.02582 0.20640 0.00800 0.36664 0.20830 0.54069
ODIN 15 0.22155 0.02693 0.23009 0.03762 0.36369 0.20461 0.56993
ODIN 94 0.24857 0.06071 0.23527 0.04409 0.35546 0.19433 0.58063
ODIN 99 0.24432 0.05540 0.23559 0.04449 0.35542 0.19427 0.58152
ODIN 100 0.24667 0.05834 0.23547 0.04434 0.35604 0.19505 0.58158
FastABOD 20 0.24821 0.06026 0.22142 0.02677 0.34770 0.18462 0.54904
FastABOD 70 0.25538 0.06923 0.22333 0.02916 0.34770 0.18462 0.55124
FastABOD 72 0.25681 0.07102 0.22327 0.02909 0.34770 0.18462 0.55119
FastABOD 98 0.25251 0.06564 0.22392 0.02990 0.34770 0.18462 0.55095
KDEOS 51 0.20803 0.01004 0.21635 0.02044 0.34367 0.17959 0.53827
KDEOS 69 0.22669 0.03336 0.21634 0.02043 0.34758 0.18448 0.55186
KDEOS 97 0.20803 0.01004 0.20612 0.00765 0.35506 0.19382 0.54358
LDF 5 0.25968 0.07461 0.26084 0.07605 0.36122 0.20153 0.58681
LDF 8 0.22669 0.03336 0.24045 0.05056 0.38307 0.22884 0.59799
INFLO 3 0.23242 0.04053 0.22567 0.03209 0.33445 0.16807 0.52597
INFLO 5 0.23386 0.04232 0.22365 0.02957 0.33690 0.17113 0.53630
INFLO 18 0.21377 0.01722 0.20795 0.00993 0.36131 0.20163 0.54154
INFLO 90 0.13486 -0.08142 0.19955 -0.00056 0.37219 0.21524 0.52046
COF 1 0.23099 0.03874 0.22451 0.03064 0.34680 0.18350 0.53254
COF 5 0.25538 0.06923 0.22973 0.03716 0.33349 0.16687 0.51773
COF 6 0.25825 0.07281 0.22910 0.03637 0.33397 0.16747 0.52022
COF 76 0.13199 -0.08501 0.18401 -0.01999 0.35030 0.18788 0.48859

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 57 attributes, 3160 objects, 632 outliers (20.00%)

Download raw algorithm results (27.4 MB) Download raw algorithm evaluation table (72.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 3 0.41930 0.27413 0.41705 0.27131 0.44039 0.30049 0.71277
KNN 6 0.43513 0.29391 0.41422 0.26777 0.44413 0.30517 0.72090
KNN 98 0.40981 0.26226 0.38834 0.23543 0.45076 0.31345 0.73041
KNNW 15 0.43513 0.29391 0.41458 0.26823 0.44159 0.30199 0.71774
KNNW 72 0.41456 0.26820 0.39958 0.24948 0.44742 0.30927 0.72847
KNNW 100 0.41139 0.26424 0.39633 0.24542 0.44692 0.30865 0.72955
LOF 84 0.29272 0.11590 0.26963 0.08704 0.37450 0.21812 0.61331
LOF 92 0.30380 0.12975 0.27268 0.09085 0.36951 0.21189 0.61716
LOF 100 0.29905 0.12381 0.27588 0.09486 0.36946 0.21182 0.62060
SimplifiedLOF 99 0.26899 0.08623 0.26891 0.08614 0.36314 0.20392 0.59422
SimplifiedLOF 100 0.27057 0.08821 0.26918 0.08647 0.36123 0.20153 0.59490
LoOP 96 0.24842 0.06052 0.24010 0.05012 0.34706 0.18382 0.56732
LoOP 99 0.25475 0.06843 0.24126 0.05158 0.34779 0.18474 0.56729
LoOP 100 0.25475 0.06843 0.24162 0.05202 0.34783 0.18478 0.56713
LDOF 2 0.21994 0.02492 0.20975 0.01219 0.33430 0.16788 0.43820
LDOF 68 0.18671 -0.01661 0.19168 -0.01040 0.33501 0.16876 0.48220
LDOF 100 0.19937 -0.00079 0.20412 0.00515 0.33457 0.16821 0.50218
ODIN 22 0.14722 -0.06597 0.20350 0.00437 0.36709 0.20887 0.54162
ODIN 96 0.20992 0.01239 0.20759 0.00948 0.35826 0.19783 0.54546
ODIN 100 0.20428 0.00535 0.20809 0.01012 0.35841 0.19801 0.54655
FastABOD 3 0.39082 0.23853 0.38333 0.22917 0.44051 0.30064 0.70343
FastABOD 4 0.38133 0.22666 0.38120 0.22650 0.44455 0.30569 0.70326
KDEOS 88 0.22785 0.03481 0.22296 0.02870 0.34991 0.18739 0.56572
KDEOS 100 0.22627 0.03283 0.22697 0.03372 0.35355 0.19193 0.57259
LDF 96 0.37975 0.22468 0.38133 0.22666 0.42586 0.28233 0.69001
LDF 99 0.38766 0.23457 0.38071 0.22588 0.42810 0.28512 0.69226
INFLO 91 0.26424 0.08030 0.25884 0.07355 0.45352 0.31691 0.62506
INFLO 93 0.26582 0.08228 0.25774 0.07218 0.45082 0.31352 0.62121
COF 83 0.30380 0.12975 0.31298 0.14122 0.35088 0.18860 0.59255
COF 99 0.30063 0.12579 0.32404 0.15505 0.36026 0.20032 0.60604
COF 100 0.30380 0.12975 0.32535 0.15668 0.35653 0.19566 0.60794

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 57 attributes, 3485 objects, 697 outliers (20.00%)

Download raw algorithm results (28.6 MB) Download raw algorithm evaluation table (72.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 7 0.45768 0.32209 0.46111 0.32639 0.47277 0.34096 0.75689
KNN 13 0.45337 0.31671 0.46232 0.32790 0.47326 0.34157 0.76081
KNN 32 0.44476 0.30595 0.44880 0.31099 0.47487 0.34359 0.76167
KNN 57 0.44476 0.30595 0.44077 0.30097 0.48035 0.35044 0.75911
KNNW 11 0.45911 0.32389 0.45268 0.31585 0.46395 0.32994 0.74492
KNNW 24 0.45337 0.31671 0.45851 0.32314 0.47084 0.33856 0.75550
KNNW 61 0.44620 0.30775 0.44840 0.31050 0.47963 0.34953 0.75976
KNNW 100 0.44333 0.30416 0.44446 0.30557 0.47940 0.34925 0.76031
LOF 99 0.29555 0.11944 0.28580 0.10725 0.38018 0.22522 0.62523
LOF 100 0.29699 0.12123 0.28700 0.10875 0.37944 0.22431 0.62638
SimplifiedLOF 2 0.26829 0.08537 0.23277 0.04096 0.33349 0.16687 0.51506
SimplifiedLOF 90 0.23529 0.04412 0.23278 0.04097 0.35211 0.19014 0.57102
SimplifiedLOF 100 0.24821 0.06026 0.24224 0.05280 0.34975 0.18719 0.57767
LoOP 1 0.23529 0.04412 0.24928 0.06160 0.33333 0.16667 0.51917
LoOP 2 0.23960 0.04950 0.24760 0.05950 0.33333 0.16667 0.53309
LoOP 84 0.21090 0.01363 0.21789 0.02237 0.34823 0.18528 0.55400
LoOP 100 0.23099 0.03874 0.22966 0.03707 0.34174 0.17718 0.55934
LDOF 2 0.25681 0.07102 0.24081 0.05101 0.33333 0.16667 0.47240
LDOF 69 0.17647 -0.02941 0.18313 -0.02109 0.34069 0.17586 0.47812
LDOF 100 0.18938 -0.01327 0.19597 -0.00504 0.34008 0.17510 0.49890
ODIN 1 0.21923 0.02404 0.21790 0.02237 0.35320 0.19150 0.55046
ODIN 22 0.17290 -0.03388 0.21110 0.01387 0.37377 0.21722 0.55880
ODIN 24 0.16906 -0.03868 0.21077 0.01347 0.37450 0.21813 0.55652
FastABOD 83 0.41033 0.26291 0.40375 0.25468 0.45604 0.32004 0.70499
FastABOD 85 0.41176 0.26471 0.40374 0.25467 0.45604 0.32004 0.70496
FastABOD 100 0.41176 0.26471 0.40399 0.25499 0.45567 0.31959 0.70524
KDEOS 3 0.22238 0.02798 0.20962 0.01203 0.33397 0.16747 0.49579
KDEOS 99 0.19369 -0.00789 0.21714 0.02143 0.34754 0.18442 0.55277
KDEOS 100 0.19082 -0.01148 0.21776 0.02220 0.34748 0.18435 0.55389
LDF 100 0.39168 0.23960 0.40726 0.25907 0.41552 0.26940 0.69279
INFLO 84 0.23386 0.04232 0.25246 0.06558 0.44455 0.30568 0.61876
INFLO 99 0.25681 0.07102 0.25887 0.07358 0.43961 0.29951 0.61209
INFLO 100 0.25681 0.07102 0.26010 0.07513 0.44093 0.30116 0.61362
COF 2 0.26973 0.08716 0.23834 0.04792 0.33770 0.17213 0.52530
COF 97 0.26399 0.07999 0.26199 0.07749 0.36214 0.20267 0.59748
COF 100 0.26973 0.08716 0.26832 0.08540 0.35976 0.19970 0.60299

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO