Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (20% of outliers version#04)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (50.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 20 0.37838 0.22505 0.47857 0.34995 0.57143 0.46571 0.79928
KNN 28 0.37838 0.22505 0.49052 0.36485 0.57944 0.47570 0.79604
KNN 38 0.48649 0.35982 0.53228 0.41691 0.54386 0.43135 0.79613
KNN 82 0.45946 0.32613 0.56816 0.46164 0.55046 0.43957 0.78883
KNNW 79 0.43243 0.29243 0.50559 0.38363 0.55856 0.44967 0.78505
KNNW 86 0.43243 0.29243 0.51058 0.38985 0.55856 0.44967 0.78649
KNNW 93 0.43243 0.29243 0.50559 0.38363 0.56364 0.45600 0.78631
KNNW 98 0.43243 0.29243 0.50895 0.38783 0.55856 0.44967 0.78703
LOF 78 0.48649 0.35982 0.52839 0.41206 0.55769 0.44859 0.78667
LOF 79 0.51351 0.39351 0.53238 0.41703 0.56000 0.45147 0.78505
LOF 84 0.48649 0.35982 0.54891 0.43764 0.56566 0.45852 0.78180
LOF 96 0.48649 0.35982 0.56392 0.45635 0.56311 0.45534 0.78559
SimplifiedLOF 86 0.35135 0.19135 0.38515 0.23349 0.49600 0.37168 0.71514
SimplifiedLOF 88 0.35135 0.19135 0.38920 0.23853 0.50794 0.38656 0.71676
SimplifiedLOF 99 0.35135 0.19135 0.42201 0.27944 0.50407 0.38173 0.73532
SimplifiedLOF 100 0.35135 0.19135 0.42045 0.27750 0.50794 0.38656 0.73568
LoOP 96 0.37838 0.22505 0.41977 0.27665 0.50000 0.37667 0.73505
LoOP 99 0.37838 0.22505 0.42835 0.28734 0.50820 0.38689 0.73703
LoOP 100 0.37838 0.22505 0.42841 0.28741 0.50407 0.38173 0.73730
LDOF 75 0.35135 0.19135 0.32715 0.16118 0.42647 0.28500 0.66577
LDOF 99 0.35135 0.19135 0.37140 0.21635 0.46970 0.33889 0.69784
LDOF 100 0.35135 0.19135 0.37219 0.21734 0.46970 0.33889 0.69892
ODIN 66 0.36486 0.20820 0.41880 0.27544 0.54545 0.43333 0.74784
ODIN 94 0.48649 0.35982 0.48864 0.36250 0.52727 0.41067 0.76694
ODIN 100 0.47748 0.34859 0.52086 0.40267 0.53704 0.42284 0.76973
FastABOD 84 0.45946 0.32613 0.48581 0.35898 0.56818 0.46167 0.80811
FastABOD 97 0.45946 0.32613 0.49305 0.36800 0.55556 0.44593 0.80955
FastABOD 98 0.48649 0.35982 0.49180 0.36645 0.55556 0.44593 0.80937
FastABOD 100 0.48649 0.35982 0.49222 0.36697 0.55556 0.44593 0.81027
KDEOS 2 0.39049 0.24015 0.30405 0.13238 0.39437 0.24498 0.63342
KDEOS 80 0.24324 0.05658 0.24208 0.05512 0.43871 0.30026 0.60955
KDEOS 100 0.27027 0.09027 0.26089 0.07858 0.43871 0.30026 0.64162
LDF 32 0.56757 0.46090 0.51552 0.39602 0.56757 0.46090 0.80829
LDF 66 0.54054 0.42721 0.62293 0.52992 0.62069 0.52713 0.82829
LDF 70 0.56757 0.46090 0.62185 0.52857 0.62791 0.53612 0.82180
LDF 76 0.54054 0.42721 0.62957 0.53820 0.60976 0.51350 0.82270
INFLO 82 0.43243 0.29243 0.48528 0.35832 0.59813 0.49900 0.78378
INFLO 100 0.43243 0.29243 0.52677 0.41004 0.65263 0.56695 0.78369
COF 69 0.51351 0.39351 0.54079 0.42751 0.56180 0.45371 0.80360
COF 82 0.48649 0.35982 0.56287 0.45504 0.60215 0.50401 0.81423
COF 100 0.48649 0.35982 0.61161 0.51580 0.58716 0.48532 0.82865

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (50.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.35135 0.19135 0.33618 0.17244 0.50877 0.38760 0.72838
KNN 9 0.37838 0.22505 0.33927 0.17629 0.48529 0.35833 0.73676
KNN 10 0.39189 0.24189 0.33564 0.17177 0.47368 0.34386 0.72739
KNNW 2 0.37838 0.22505 0.32325 0.15632 0.46400 0.33179 0.70054
KNNW 3 0.32432 0.15766 0.32987 0.16457 0.49587 0.37152 0.71423
KNNW 4 0.35135 0.19135 0.33500 0.17097 0.49206 0.36677 0.72054
KNNW 19 0.35135 0.19135 0.33344 0.16902 0.47482 0.34528 0.72775
LOF 20 0.37838 0.22505 0.30724 0.13636 0.45000 0.31433 0.69802
LOF 33 0.35135 0.19135 0.30913 0.13872 0.46980 0.33902 0.70811
SimplifiedLOF 26 0.35135 0.19135 0.29590 0.12222 0.42254 0.28009 0.67189
SimplifiedLOF 27 0.37838 0.22505 0.29481 0.12086 0.42759 0.28639 0.67279
SimplifiedLOF 64 0.32432 0.15766 0.29426 0.12018 0.45390 0.31920 0.68955
SimplifiedLOF 79 0.35135 0.19135 0.29276 0.11831 0.46043 0.32734 0.68685
LoOP 3 0.29730 0.12396 0.29477 0.12082 0.40708 0.26083 0.64865
LoOP 17 0.37838 0.22505 0.28265 0.10571 0.39744 0.24880 0.64135
LoOP 60 0.32432 0.15766 0.28422 0.10766 0.44444 0.30741 0.67586
LoOP 64 0.32432 0.15766 0.28046 0.10298 0.45802 0.32433 0.66532
LDOF 3 0.27027 0.09027 0.29700 0.12359 0.38994 0.23945 0.62541
LDOF 53 0.32432 0.15766 0.27553 0.09682 0.44776 0.31154 0.66577
LDOF 56 0.32432 0.15766 0.27851 0.10054 0.44118 0.30333 0.67225
LDOF 57 0.35135 0.19135 0.27889 0.10102 0.43478 0.29536 0.67099
ODIN 38 0.27027 0.09027 0.28843 0.11291 0.44755 0.31128 0.66649
ODIN 47 0.29730 0.12396 0.30539 0.13405 0.43038 0.28987 0.67432
ODIN 51 0.30631 0.13520 0.29856 0.12554 0.41860 0.27519 0.67577
ODIN 75 0.35135 0.19135 0.28100 0.10365 0.40984 0.26426 0.64982
FastABOD 4 0.40541 0.25874 0.34437 0.18265 0.51064 0.38993 0.72000
FastABOD 15 0.40541 0.25874 0.35528 0.19625 0.52632 0.40947 0.73676
FastABOD 18 0.40541 0.25874 0.35688 0.19825 0.54167 0.42861 0.73459
FastABOD 20 0.40541 0.25874 0.35754 0.19906 0.53061 0.41483 0.73514
KDEOS 4 0.35135 0.19135 0.34814 0.18734 0.43478 0.29536 0.65658
KDEOS 98 0.32432 0.15766 0.33051 0.16537 0.44444 0.30741 0.68685
KDEOS 100 0.32432 0.15766 0.32076 0.15321 0.44444 0.30741 0.68973
LDF 10 0.43243 0.29243 0.32421 0.15752 0.46018 0.32702 0.69964
LDF 15 0.40541 0.25874 0.34325 0.18125 0.46667 0.33511 0.72721
LDF 24 0.32432 0.15766 0.33887 0.17579 0.48062 0.35251 0.73676
LDF 25 0.32432 0.15766 0.33608 0.17232 0.48780 0.36146 0.73586
INFLO 35 0.37838 0.22505 0.31755 0.14922 0.54545 0.43333 0.71108
INFLO 49 0.35135 0.19135 0.33118 0.16621 0.59322 0.49288 0.75514
COF 14 0.40541 0.25874 0.35184 0.19196 0.52427 0.40693 0.73243
COF 17 0.43243 0.29243 0.33736 0.17391 0.48889 0.36281 0.72162
COF 24 0.40541 0.25874 0.36898 0.21333 0.48276 0.35517 0.75243

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO