Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Pima (35% of outliers)

The data set contains medical data on diabetes. Patients suffering from diabetes were considered outliers.

Download all data set variants used (694.8 kB). You can also access the original data. (pima-indians-diabetes.data)

Normalized, without duplicates

This version contains 8 attributes, 768 objects, 268 outliers (34.90%)

Download raw algorithm results (6.7 MB) Download raw algorithm evaluation table (59.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 70 0.56343 0.32943 0.52731 0.27394 0.62353 0.42174 0.73110
KNN 85 0.54851 0.30651 0.52940 0.27715 0.63066 0.43269 0.73224
KNN 88 0.55224 0.31224 0.52896 0.27648 0.63142 0.43387 0.73193
KNN 98 0.54851 0.30651 0.52970 0.27762 0.62974 0.43128 0.73187
KNNW 5 0.56716 0.33516 0.52855 0.27586 0.59621 0.37979 0.71123
KNNW 6 0.56716 0.33516 0.52920 0.27685 0.59389 0.37621 0.71386
KNNW 91 0.54478 0.30078 0.52685 0.27324 0.62334 0.42145 0.72921
KNNW 100 0.54104 0.29504 0.52703 0.27352 0.62242 0.42004 0.72944
LOF 94 0.52239 0.26639 0.47742 0.19731 0.59012 0.37043 0.68449
LOF 100 0.51493 0.25493 0.48123 0.20317 0.59466 0.37740 0.68955
SimplifiedLOF 99 0.47388 0.19188 0.44007 0.13996 0.54230 0.29697 0.62085
SimplifiedLOF 100 0.47388 0.19188 0.44034 0.14036 0.54305 0.29812 0.62131
LoOP 36 0.41791 0.10591 0.41080 0.09498 0.53563 0.28672 0.58389
LoOP 99 0.47015 0.18615 0.43340 0.12970 0.53129 0.28006 0.60922
LDOF 7 0.43657 0.13457 0.40312 0.08319 0.51794 0.25956 0.56014
LDOF 18 0.40672 0.08872 0.40411 0.08471 0.52456 0.26972 0.56925
LDOF 95 0.42164 0.11164 0.41081 0.09500 0.51925 0.26157 0.56938
LDOF 98 0.42537 0.11737 0.40957 0.09310 0.51969 0.26224 0.57004
ODIN 100 0.47228 0.18942 0.45434 0.16187 0.55257 0.31274 0.63637
FastABOD 22 0.60075 0.38675 0.57534 0.34772 0.61561 0.40957 0.74828
FastABOD 49 0.58955 0.36955 0.58218 0.35823 0.62353 0.42174 0.75513
FastABOD 99 0.58955 0.36955 0.58866 0.36819 0.62215 0.41963 0.76081
KDEOS 2 0.39821 0.07565 0.39452 0.06999 0.51838 0.26022 0.55621
KDEOS 48 0.34328 -0.00872 0.36154 0.01933 0.53029 0.27852 0.53607
LDF 88 0.54851 0.30651 0.51094 0.24880 0.62069 0.41738 0.72306
LDF 94 0.54478 0.30078 0.51399 0.25349 0.62467 0.42350 0.72568
LDF 100 0.53731 0.28931 0.51736 0.25866 0.62396 0.42240 0.72888
INFLO 92 0.46642 0.18042 0.43171 0.12710 0.54108 0.29509 0.61728
INFLO 94 0.47015 0.18615 0.42956 0.12381 0.53770 0.28990 0.61375
INFLO 100 0.47015 0.18615 0.43275 0.12871 0.54261 0.29745 0.61622
COF 86 0.54478 0.30078 0.52777 0.27466 0.59821 0.38286 0.68786
COF 90 0.54104 0.29504 0.52852 0.27580 0.60377 0.39140 0.69168
COF 100 0.54478 0.30078 0.54148 0.29572 0.59866 0.38355 0.70121

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 8 attributes, 768 objects, 268 outliers (34.90%)

Download raw algorithm results (6.6 MB) Download raw algorithm evaluation table (56.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 52 0.50746 0.24346 0.49596 0.22580 0.55455 0.31578 0.64470
KNN 56 0.50000 0.23200 0.49666 0.22687 0.55418 0.31522 0.64561
KNN 80 0.51119 0.24919 0.49754 0.22823 0.54870 0.30681 0.64464
KNN 90 0.51866 0.26066 0.49554 0.22515 0.54688 0.30400 0.64119
KNNW 48 0.51119 0.24919 0.49077 0.21782 0.53731 0.28931 0.63904
KNNW 93 0.50373 0.23773 0.49454 0.22361 0.54862 0.30668 0.64249
KNNW 99 0.50746 0.24346 0.49478 0.22399 0.54703 0.30424 0.64257
KNNW 100 0.50746 0.24346 0.49485 0.22408 0.54651 0.30344 0.64255
LOF 100 0.52612 0.27212 0.48867 0.21460 0.54988 0.30861 0.65307
SimplifiedLOF 25 0.35821 0.01421 0.36099 0.01848 0.52485 0.27017 0.51618
SimplifiedLOF 99 0.44030 0.14030 0.44834 0.15265 0.52258 0.26668 0.60196
SimplifiedLOF 100 0.44030 0.14030 0.44901 0.15367 0.52248 0.26654 0.60230
LoOP 1 0.32463 -0.03737 0.34745 -0.00231 0.51737 0.25869 0.50143
LoOP 86 0.44403 0.14603 0.42952 0.12375 0.51737 0.25869 0.58466
LoOP 100 0.43657 0.13457 0.43765 0.13624 0.51737 0.25869 0.59285
LDOF 2 0.35821 0.01421 0.36632 0.02667 0.52256 0.26665 0.52340
LDOF 93 0.45149 0.15749 0.43447 0.13134 0.51838 0.26022 0.58146
LDOF 100 0.44403 0.14603 0.44012 0.14002 0.51934 0.26170 0.58717
ODIN 48 0.42276 0.11336 0.39358 0.06853 0.52148 0.26499 0.55471
ODIN 67 0.44690 0.15044 0.41064 0.09474 0.51787 0.25946 0.57264
ODIN 100 0.43159 0.12693 0.43126 0.12641 0.51744 0.25879 0.58544
FastABOD 3 0.52985 0.27785 0.50336 0.23716 0.55523 0.31684 0.66224
FastABOD 77 0.54104 0.29504 0.51839 0.26024 0.54701 0.30421 0.66721
FastABOD 100 0.53731 0.28931 0.51944 0.26186 0.55017 0.30907 0.66885
KDEOS 5 0.38060 0.04860 0.37003 0.03237 0.51737 0.25869 0.52001
KDEOS 88 0.36194 0.01994 0.35259 0.00557 0.53408 0.28434 0.53921
KDEOS 96 0.38433 0.05433 0.35637 0.01138 0.53015 0.27831 0.54377
KDEOS 100 0.37687 0.04287 0.35772 0.01346 0.53042 0.27873 0.54648
LDF 78 0.54104 0.29504 0.49788 0.22875 0.56069 0.32523 0.66499
LDF 81 0.53358 0.28358 0.49882 0.23019 0.56232 0.32772 0.66628
LDF 96 0.52985 0.27785 0.50276 0.23623 0.55878 0.32228 0.66927
LDF 100 0.53358 0.28358 0.50297 0.23656 0.56231 0.32771 0.66886
INFLO 88 0.47015 0.18615 0.43916 0.13856 0.52157 0.26513 0.60992
INFLO 94 0.47015 0.18615 0.44047 0.14057 0.52208 0.26592 0.61246
INFLO 95 0.47015 0.18615 0.44040 0.14045 0.52106 0.26434 0.61287
INFLO 99 0.46269 0.17469 0.44110 0.14153 0.52012 0.26290 0.61216
COF 87 0.50373 0.23773 0.48137 0.20338 0.56471 0.33139 0.65133
COF 99 0.50000 0.23200 0.48610 0.21066 0.56653 0.33420 0.65895
COF 100 0.49627 0.22627 0.48761 0.21296 0.56445 0.33100 0.65986

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO