Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Pima (2% of outliers version#10)

The data set contains medical data on diabetes. Patients suffering from diabetes were considered outliers.

Download all data set variants used (694.8 kB). You can also access the original data. (pima-indians-diabetes.data)

Normalized, without duplicates

This version contains 8 attributes, 510 objects, 10 outliers (1.96%)

Download raw algorithm results (4.5 MB) Download raw algorithm evaluation table (43.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02000 0.03863 0.01940 0.10811 0.09027 0.70600
KNN 8 0.00000 -0.02000 0.05155 0.03258 0.12245 0.10490 0.73620
KNN 24 0.00000 -0.02000 0.04685 0.02778 0.12766 0.11021 0.70440
KNNW 1 0.00000 -0.02000 0.03821 0.01897 0.09639 0.07831 0.68460
KNNW 11 0.00000 -0.02000 0.04308 0.02394 0.11111 0.09333 0.72520
KNNW 14 0.00000 -0.02000 0.04368 0.02455 0.11765 0.10000 0.72440
KNNW 66 0.00000 -0.02000 0.04555 0.02646 0.11538 0.09769 0.70080
LOF 1 0.00000 -0.02000 0.02398 0.00445 0.06178 0.04301 0.58050
LOF 18 0.00000 -0.02000 0.04914 0.03012 0.12903 0.11161 0.70860
LOF 61 0.00000 -0.02000 0.05222 0.03327 0.11429 0.09657 0.71480
LOF 67 0.00000 -0.02000 0.05113 0.03215 0.10345 0.08552 0.71620
SimplifiedLOF 1 0.00000 -0.02000 0.02813 0.00870 0.06897 0.05034 0.52300
SimplifiedLOF 31 0.00000 -0.02000 0.05090 0.03191 0.11321 0.09547 0.72440
SimplifiedLOF 32 0.00000 -0.02000 0.05130 0.03233 0.11765 0.10000 0.72400
LoOP 1 0.00000 -0.02000 0.02886 0.00944 0.06897 0.05034 0.56580
LoOP 22 0.00000 -0.02000 0.04346 0.02433 0.11200 0.09424 0.71680
LoOP 27 0.00000 -0.02000 0.04637 0.02730 0.11765 0.10000 0.71590
LoOP 32 0.00000 -0.02000 0.04972 0.03072 0.10714 0.08929 0.71180
LDOF 2 0.10000 0.08200 0.03712 0.01786 0.13333 0.11600 0.45880
LDOF 37 0.10000 0.08200 0.04973 0.03073 0.11111 0.09333 0.68400
LDOF 49 0.10000 0.08200 0.04636 0.02728 0.10526 0.08737 0.68760
ODIN 1 0.01648 -0.00319 0.01820 -0.00143 0.03929 0.02008 0.46520
ODIN 97 0.00000 -0.02000 0.04693 0.02786 0.09938 0.08137 0.68530
ODIN 98 0.00000 -0.02000 0.04707 0.02801 0.09938 0.08137 0.68490
ODIN 100 0.00000 -0.02000 0.04694 0.02788 0.10000 0.08200 0.68340
FastABOD 3 0.00000 -0.02000 0.03763 0.01839 0.09091 0.07273 0.66260
FastABOD 33 0.00000 -0.02000 0.04736 0.02831 0.11111 0.09333 0.73800
FastABOD 52 0.00000 -0.02000 0.04727 0.02822 0.11429 0.09657 0.73720
FastABOD 55 0.00000 -0.02000 0.04745 0.02840 0.11429 0.09657 0.73720
KDEOS 2 0.00000 -0.02000 0.02001 0.00041 0.04118 0.02200 0.48720
KDEOS 25 0.00000 -0.02000 0.03012 0.01072 0.08696 0.06870 0.58380
KDEOS 77 0.00000 -0.02000 0.02889 0.00947 0.07179 0.05323 0.63980
KDEOS 99 0.00000 -0.02000 0.03027 0.01087 0.07447 0.05596 0.63620
LDF 1 0.00000 -0.02000 0.02436 0.00485 0.05714 0.03829 0.58590
LDF 12 0.00000 -0.02000 0.05941 0.04060 0.15385 0.13692 0.72220
LDF 18 0.00000 -0.02000 0.05236 0.03341 0.13333 0.11600 0.72960
INFLO 21 0.00000 -0.02000 0.04724 0.02818 0.14141 0.12424 0.71260
INFLO 22 0.00000 -0.02000 0.04766 0.02862 0.12844 0.11101 0.70860
INFLO 27 0.10000 0.08200 0.04642 0.02735 0.10000 0.08200 0.68960
COF 26 0.00000 -0.02000 0.05698 0.03812 0.17778 0.16133 0.74160
COF 37 0.10000 0.08200 0.04736 0.02831 0.11765 0.10000 0.69940

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 8 attributes, 510 objects, 10 outliers (1.96%)

Download raw algorithm results (4.4 MB) Download raw algorithm evaluation table (39.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.10000 0.08200 0.03960 0.02039 0.10000 0.08200 0.63450
KNN 4 0.10000 0.08200 0.05342 0.03448 0.14286 0.12571 0.60210
KNN 9 0.10000 0.08200 0.05756 0.03871 0.14286 0.12571 0.60880
KNNW 2 0.10000 0.08200 0.04054 0.02135 0.10526 0.08737 0.62420
KNNW 6 0.10000 0.08200 0.05349 0.03456 0.14286 0.12571 0.61020
KNNW 41 0.10000 0.08200 0.05610 0.03722 0.14286 0.12571 0.61720
LOF 8 0.00000 -0.02000 0.03359 0.01427 0.07921 0.06079 0.67880
LOF 14 0.10000 0.08200 0.03425 0.01494 0.10000 0.08200 0.60860
LOF 36 0.10000 0.08200 0.05668 0.03781 0.14286 0.12571 0.64240
LOF 49 0.10000 0.08200 0.06135 0.04258 0.14286 0.12571 0.66380
SimplifiedLOF 1 0.10000 0.08200 0.04223 0.02307 0.10811 0.09027 0.56780
SimplifiedLOF 38 0.10000 0.08200 0.05475 0.03584 0.14286 0.12571 0.64340
SimplifiedLOF 60 0.10000 0.08200 0.06080 0.04202 0.14286 0.12571 0.66420
SimplifiedLOF 92 0.10000 0.08200 0.05822 0.03938 0.14286 0.12571 0.66940
LoOP 1 0.10000 0.08200 0.04256 0.02341 0.10811 0.09027 0.61470
LoOP 61 0.10000 0.08200 0.05626 0.03739 0.14286 0.12571 0.66440
LoOP 94 0.10000 0.08200 0.05885 0.04003 0.14286 0.12571 0.68060
LDOF 36 0.10000 0.08200 0.04275 0.02360 0.10526 0.08737 0.68120
LDOF 61 0.10000 0.08200 0.05829 0.03946 0.14286 0.12571 0.69560
ODIN 6 0.07692 0.05846 0.03053 0.01114 0.08696 0.06870 0.58280
ODIN 14 0.00000 -0.02000 0.03516 0.01587 0.07921 0.06079 0.68900
ODIN 94 0.00000 -0.02000 0.04320 0.02406 0.13333 0.11600 0.62620
ODIN 100 0.00000 -0.02000 0.04270 0.02356 0.13793 0.12069 0.61470
FastABOD 3 0.10000 0.08200 0.04427 0.02516 0.11111 0.09333 0.63460
FastABOD 5 0.10000 0.08200 0.05449 0.03558 0.14286 0.12571 0.61840
FastABOD 100 0.10000 0.08200 0.05666 0.03779 0.14286 0.12571 0.63060
KDEOS 2 0.00000 -0.02000 0.02538 0.00589 0.07407 0.05556 0.49880
KDEOS 14 0.00000 -0.02000 0.04479 0.02569 0.13333 0.11600 0.61020
KDEOS 33 0.00000 -0.02000 0.03238 0.01303 0.08000 0.06160 0.64900
LDF 3 0.00000 -0.02000 0.03393 0.01461 0.07643 0.05796 0.69780
LDF 11 0.10000 0.08200 0.04050 0.02131 0.11765 0.10000 0.61880
LDF 20 0.10000 0.08200 0.05680 0.03793 0.19048 0.17429 0.63600
LDF 31 0.10000 0.08200 0.06136 0.04259 0.14286 0.12571 0.64900
INFLO 1 0.10000 0.08200 0.04320 0.02406 0.12500 0.10750 0.59540
INFLO 43 0.10000 0.08200 0.05787 0.03902 0.14286 0.12571 0.67920
INFLO 69 0.10000 0.08200 0.06382 0.04509 0.14286 0.12571 0.75130
COF 1 0.10000 0.08200 0.04204 0.02288 0.10811 0.09027 0.56790
COF 25 0.00000 -0.02000 0.03908 0.01986 0.09639 0.07831 0.69600
COF 92 0.10000 0.08200 0.05251 0.03356 0.12500 0.10750 0.65460
COF 93 0.10000 0.08200 0.05283 0.03388 0.12500 0.10750 0.65600

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO