Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (20% of outliers version#02)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (51.7 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 71 0.45946 0.32613 0.50368 0.38126 0.56364 0.45600 0.81198
KNN 82 0.45946 0.32613 0.50872 0.38753 0.54867 0.43735 0.81550
KNN 99 0.51351 0.39351 0.53109 0.41543 0.52336 0.40579 0.79991
KNNW 81 0.37838 0.22505 0.42766 0.28649 0.56000 0.45147 0.78955
KNNW 89 0.40541 0.25874 0.43389 0.29425 0.56000 0.45147 0.79369
KNNW 100 0.40541 0.25874 0.44904 0.31314 0.54054 0.42721 0.79369
LOF 85 0.48649 0.35982 0.49174 0.36637 0.56842 0.46196 0.80360
LOF 86 0.51351 0.39351 0.49262 0.36747 0.56842 0.46196 0.80378
LOF 99 0.48649 0.35982 0.51196 0.39158 0.56250 0.45458 0.80090
SimplifiedLOF 81 0.32432 0.15766 0.33997 0.17717 0.47826 0.34957 0.70505
SimplifiedLOF 100 0.32432 0.15766 0.36858 0.21283 0.50382 0.38142 0.74054
LoOP 86 0.35135 0.19135 0.35387 0.19449 0.48819 0.36194 0.72883
LoOP 98 0.32432 0.15766 0.38156 0.22901 0.51163 0.39116 0.74901
LoOP 100 0.32432 0.15766 0.38945 0.23885 0.51163 0.39116 0.75009
LDOF 87 0.32432 0.15766 0.31816 0.14998 0.45588 0.32167 0.68919
LDOF 98 0.32432 0.15766 0.33838 0.17518 0.47059 0.34000 0.70937
LDOF 100 0.32432 0.15766 0.34429 0.18255 0.46980 0.33902 0.71153
ODIN 74 0.40541 0.25874 0.39694 0.24818 0.52800 0.41157 0.75991
ODIN 94 0.45946 0.32613 0.43692 0.29803 0.52459 0.40732 0.77973
ODIN 95 0.45045 0.31489 0.44264 0.30516 0.51546 0.39595 0.78306
ODIN 100 0.45946 0.32613 0.44891 0.31297 0.51724 0.39816 0.78054
FastABOD 27 0.43243 0.29243 0.38428 0.23240 0.51613 0.39677 0.76000
FastABOD 74 0.40541 0.25874 0.43150 0.29127 0.56667 0.45978 0.79153
FastABOD 98 0.40541 0.25874 0.43611 0.29701 0.55738 0.44820 0.79423
KDEOS 46 0.29730 0.12396 0.25642 0.07300 0.35374 0.19433 0.55081
KDEOS 99 0.24324 0.05658 0.27368 0.09452 0.46452 0.33243 0.65694
KDEOS 100 0.24324 0.05658 0.27291 0.09356 0.46154 0.32872 0.65712
LDF 66 0.62162 0.52829 0.62719 0.53523 0.67532 0.59524 0.84757
LDF 68 0.67568 0.59568 0.63264 0.54203 0.69333 0.61769 0.84360
LDF 76 0.64865 0.56198 0.64842 0.56169 0.66667 0.58444 0.83802
INFLO 78 0.43243 0.29243 0.42649 0.28502 0.62264 0.52956 0.77982
INFLO 79 0.40541 0.25874 0.44830 0.31222 0.65421 0.56891 0.81378
INFLO 100 0.40541 0.25874 0.46861 0.33753 0.65263 0.56695 0.77486
COF 70 0.54054 0.42721 0.54723 0.43555 0.63043 0.53928 0.83982
COF 71 0.56757 0.46090 0.54253 0.42969 0.62069 0.52713 0.84108
COF 77 0.54054 0.42721 0.55790 0.44885 0.62651 0.53438 0.84288
COF 99 0.51351 0.39351 0.57284 0.46747 0.56180 0.45371 0.82486

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (49.2 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.32432 0.15766 0.31761 0.14929 0.47541 0.34601 0.68541
KNN 8 0.40541 0.25874 0.32897 0.16345 0.45217 0.31704 0.69595
KNN 10 0.40541 0.25874 0.33653 0.17288 0.46809 0.33688 0.70586
KNNW 4 0.32432 0.15766 0.31318 0.14376 0.46667 0.33511 0.68324
KNNW 11 0.40541 0.25874 0.32315 0.15619 0.45000 0.31433 0.69423
KNNW 17 0.37838 0.22505 0.32610 0.15988 0.44828 0.31218 0.69730
LOF 13 0.37838 0.22505 0.30199 0.12981 0.44444 0.30741 0.67351
LOF 20 0.37838 0.22505 0.31023 0.14009 0.43678 0.29785 0.69405
LOF 26 0.35135 0.19135 0.30320 0.13133 0.46154 0.32872 0.68721
LOF 28 0.37838 0.22505 0.31163 0.14183 0.45714 0.32324 0.68739
SimplifiedLOF 27 0.37838 0.22505 0.29603 0.12238 0.41481 0.27047 0.66486
SimplifiedLOF 29 0.40541 0.25874 0.29223 0.11764 0.41176 0.26667 0.65694
SimplifiedLOF 31 0.37838 0.22505 0.29490 0.12097 0.43182 0.29167 0.65928
SimplifiedLOF 42 0.37838 0.22505 0.28774 0.11205 0.41667 0.27278 0.66721
LoOP 20 0.35135 0.19135 0.28122 0.10392 0.42857 0.28762 0.63829
LoOP 27 0.40541 0.25874 0.28880 0.11336 0.41727 0.27353 0.65351
LDOF 57 0.35135 0.19135 0.27227 0.09276 0.40278 0.25546 0.64775
LDOF 84 0.35135 0.19135 0.27362 0.09445 0.42105 0.27825 0.64577
LDOF 88 0.32432 0.15766 0.27233 0.09284 0.42735 0.28610 0.64468
ODIN 15 0.32973 0.16440 0.30415 0.13250 0.39456 0.24522 0.64090
ODIN 17 0.31660 0.14803 0.29645 0.12291 0.40845 0.26254 0.64973
ODIN 30 0.35381 0.19441 0.29785 0.12465 0.39521 0.24603 0.64802
ODIN 47 0.25676 0.07342 0.28019 0.10264 0.43860 0.30012 0.63405
FastABOD 3 0.45946 0.32613 0.40057 0.25272 0.47500 0.34550 0.70486
FastABOD 4 0.48649 0.35982 0.36286 0.20570 0.48649 0.35982 0.71784
KDEOS 86 0.37838 0.22505 0.31077 0.14076 0.41584 0.27175 0.66198
KDEOS 98 0.29730 0.12396 0.31512 0.14619 0.41818 0.27467 0.66162
KDEOS 100 0.29730 0.12396 0.30272 0.13072 0.42478 0.28289 0.66450
LDF 8 0.43243 0.29243 0.30504 0.13361 0.43750 0.29875 0.65550
LDF 16 0.40541 0.25874 0.33708 0.17356 0.48837 0.36217 0.71297
LDF 18 0.37838 0.22505 0.33833 0.17512 0.48980 0.36395 0.71063
LDF 20 0.40541 0.25874 0.33887 0.17580 0.48936 0.36340 0.70955
INFLO 19 0.35135 0.19135 0.28515 0.10882 0.45161 0.31634 0.62667
INFLO 44 0.29730 0.12396 0.29826 0.12517 0.52991 0.41396 0.69306
INFLO 47 0.32432 0.15766 0.30425 0.13264 0.52991 0.41396 0.68919
INFLO 75 0.29730 0.12396 0.28088 0.10349 0.50000 0.37667 0.69459
COF 13 0.43243 0.29243 0.31674 0.14820 0.47059 0.34000 0.66396
COF 63 0.40541 0.25874 0.37167 0.21668 0.52083 0.40264 0.72450
COF 67 0.37838 0.22505 0.36030 0.20250 0.53488 0.42016 0.71820

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO