Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

HeartDisease (20% of outliers version#06)

A data set containing medical data on heart problems. Affected patients are considered outliers and healthy people are considered inliers.

Download all data set variants used (92.9 kB). You can also access the original data. (heart.dat)

Normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (51.0 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 47 0.54054 0.42721 0.47297 0.34296 0.54054 0.42721 0.79523
KNN 49 0.54054 0.42721 0.47834 0.34966 0.55000 0.43900 0.80018
KNN 52 0.51351 0.39351 0.47650 0.34737 0.56410 0.45658 0.79694
KNN 80 0.45946 0.32613 0.49380 0.36893 0.54118 0.42800 0.79045
KNNW 77 0.43243 0.29243 0.42520 0.28341 0.53061 0.41483 0.77928
KNNW 97 0.43243 0.29243 0.43826 0.29969 0.54545 0.43333 0.78252
KNNW 100 0.43243 0.29243 0.43992 0.30177 0.54545 0.43333 0.78378
LOF 80 0.51351 0.39351 0.47455 0.34494 0.54237 0.42949 0.79243
LOF 91 0.48649 0.35982 0.49251 0.36733 0.56075 0.45240 0.79658
LOF 93 0.45946 0.32613 0.49620 0.37193 0.55556 0.44593 0.80018
LOF 100 0.45946 0.32613 0.49770 0.37381 0.55556 0.44593 0.79441
SimplifiedLOF 68 0.32432 0.15766 0.32009 0.15238 0.46715 0.33572 0.69928
SimplifiedLOF 100 0.32432 0.15766 0.36374 0.20680 0.48352 0.35612 0.74288
LoOP 44 0.32432 0.15766 0.29217 0.11757 0.41892 0.27559 0.65423
LoOP 98 0.32432 0.15766 0.35880 0.20064 0.48889 0.36281 0.72793
LoOP 100 0.32432 0.15766 0.36302 0.20590 0.48889 0.36281 0.73117
LDOF 87 0.32432 0.15766 0.31736 0.14898 0.46154 0.32872 0.69550
LDOF 97 0.29730 0.12396 0.32887 0.16332 0.46512 0.33318 0.70486
LDOF 100 0.32432 0.15766 0.33600 0.17221 0.46154 0.32872 0.71171
ODIN 88 0.48649 0.35982 0.41140 0.26621 0.51546 0.39595 0.76261
ODIN 95 0.48649 0.35982 0.43442 0.29491 0.53763 0.42358 0.76946
ODIN 100 0.48649 0.35982 0.43743 0.29866 0.52174 0.40377 0.77234
FastABOD 70 0.43243 0.29243 0.44175 0.30404 0.56000 0.45147 0.80486
FastABOD 80 0.45946 0.32613 0.44769 0.31145 0.55357 0.44345 0.80613
FastABOD 96 0.45946 0.32613 0.45929 0.32591 0.55914 0.45039 0.81171
FastABOD 97 0.45946 0.32613 0.45929 0.32592 0.55914 0.45039 0.81171
KDEOS 4 0.32432 0.15766 0.28042 0.10293 0.40860 0.26272 0.57423
KDEOS 94 0.27027 0.09027 0.26870 0.08832 0.44595 0.30928 0.65297
KDEOS 100 0.27027 0.09027 0.26982 0.08970 0.44444 0.30741 0.65964
LDF 67 0.56757 0.46090 0.56698 0.46017 0.57534 0.47059 0.80847
LDF 69 0.56757 0.46090 0.56157 0.45342 0.58333 0.48056 0.81009
LDF 89 0.48649 0.35982 0.57179 0.46617 0.57692 0.47256 0.81820
INFLO 76 0.37838 0.22505 0.39545 0.24633 0.59459 0.49459 0.77901
INFLO 96 0.45946 0.32613 0.43929 0.30099 0.62366 0.53082 0.73306
INFLO 100 0.45946 0.32613 0.45398 0.31929 0.65957 0.57560 0.76414
COF 65 0.45946 0.32613 0.45142 0.31610 0.57471 0.46981 0.76486
COF 67 0.51351 0.39351 0.45820 0.32455 0.55696 0.44768 0.77189
COF 98 0.45946 0.32613 0.57478 0.46990 0.53763 0.42358 0.80721
COF 100 0.45946 0.32613 0.57660 0.47216 0.53571 0.42119 0.80306

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 13 attributes, 187 objects, 37 outliers (19.79%)

Download raw algorithm results (1.6 MB) Download raw algorithm evaluation table (48.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.32432 0.15766 0.30743 0.13660 0.45528 0.32092 0.67090
KNN 6 0.37838 0.22505 0.31275 0.14322 0.43750 0.29875 0.67396
KNN 9 0.35135 0.19135 0.33013 0.16489 0.43810 0.29949 0.68550
KNNW 4 0.35135 0.19135 0.30921 0.13881 0.45161 0.31634 0.67640
KNNW 9 0.37838 0.22505 0.31719 0.14877 0.44068 0.30271 0.67946
KNNW 12 0.37838 0.22505 0.32351 0.15664 0.44262 0.30514 0.68180
KNNW 14 0.37838 0.22505 0.32351 0.15664 0.43077 0.29036 0.67982
LOF 12 0.37838 0.22505 0.29492 0.12100 0.41667 0.27278 0.64685
LOF 26 0.27027 0.09027 0.30648 0.13541 0.42759 0.28639 0.65730
LOF 27 0.29730 0.12396 0.30704 0.13611 0.41892 0.27559 0.65279
SimplifiedLOF 17 0.35135 0.19135 0.30448 0.13292 0.40816 0.26218 0.65946
SimplifiedLOF 19 0.37838 0.22505 0.30525 0.13388 0.40404 0.25704 0.65802
SimplifiedLOF 27 0.35135 0.19135 0.30830 0.13768 0.40602 0.25950 0.65604
SimplifiedLOF 79 0.35135 0.19135 0.28602 0.10990 0.43478 0.29536 0.64162
LoOP 16 0.37838 0.22505 0.30970 0.13943 0.41270 0.26783 0.65964
LoOP 17 0.35135 0.19135 0.30602 0.13484 0.41758 0.27392 0.65261
LoOP 19 0.37838 0.22505 0.31053 0.14045 0.41379 0.26920 0.65532
LoOP 29 0.40541 0.25874 0.30515 0.13375 0.41096 0.26566 0.64811
LDOF 5 0.21622 0.02288 0.25004 0.06505 0.41791 0.27433 0.60937
LDOF 17 0.35135 0.19135 0.29241 0.11787 0.39735 0.24870 0.64541
LDOF 25 0.37838 0.22505 0.28783 0.11217 0.40860 0.26272 0.63946
LDOF 33 0.35135 0.19135 0.29447 0.12043 0.41379 0.26920 0.63856
ODIN 10 0.35135 0.19135 0.31659 0.14802 0.41584 0.27175 0.65306
ODIN 26 0.36757 0.21157 0.29501 0.12111 0.37989 0.22693 0.62505
FastABOD 4 0.37838 0.22505 0.36244 0.20518 0.50602 0.38418 0.70865
KDEOS 4 0.37838 0.22505 0.30120 0.12883 0.39716 0.24846 0.62883
KDEOS 30 0.27027 0.09027 0.33105 0.16605 0.38462 0.23282 0.62270
KDEOS 86 0.32432 0.15766 0.29364 0.11941 0.40909 0.26333 0.63838
KDEOS 92 0.32432 0.15766 0.29712 0.12374 0.40268 0.25535 0.64198
LDF 7 0.35135 0.19135 0.29937 0.12654 0.40000 0.25200 0.62180
LDF 21 0.32432 0.15766 0.31898 0.15099 0.42963 0.28894 0.67441
LDF 27 0.29730 0.12396 0.30035 0.12777 0.44231 0.30474 0.65225
INFLO 13 0.40541 0.25874 0.31970 0.15189 0.44961 0.31385 0.64225
INFLO 16 0.37838 0.22505 0.33304 0.16852 0.46970 0.33889 0.67045
INFLO 68 0.29730 0.12396 0.30192 0.12973 0.53448 0.41966 0.69459
COF 16 0.37838 0.22505 0.31485 0.14585 0.43182 0.29167 0.65712
COF 24 0.35135 0.19135 0.32351 0.15665 0.43956 0.30132 0.68847
COF 55 0.24324 0.05658 0.29575 0.12204 0.46667 0.33511 0.66919

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO