Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Arrhythmia (20% of outliers version#06)

Data set contains patient records classified as normal or as exhibiting some type of cardiac arrhythmia. In total, there are 14 types of arrhythmia and 1 type that brings together all the other different types. However, 3 types of arrhythmia have no data. Again, we treat healthy people as inliers and patients suffering from arrhythmia as outliers.

Download all data set variants used (9.2 MB). You can also access the original data. (arrhythmia.data)

Normalized, without duplicates

This version contains 259 attributes, 305 objects, 61 outliers (20.00%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (54.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 9 0.45902 0.32377 0.57659 0.47073 0.52121 0.40152 0.79374
KNN 10 0.45902 0.32377 0.57525 0.46906 0.51892 0.39865 0.79387
KNN 25 0.45902 0.32377 0.58357 0.47946 0.53333 0.41667 0.78924
KNN 94 0.44262 0.30328 0.59017 0.48771 0.50909 0.38636 0.78554
KNNW 1 0.46721 0.33402 0.55864 0.44830 0.52113 0.40141 0.78887
KNNW 19 0.45902 0.32377 0.57670 0.47087 0.52273 0.40341 0.79407
KNNW 24 0.45902 0.32377 0.57782 0.47227 0.52874 0.41092 0.79307
KNNW 99 0.44262 0.30328 0.58557 0.48196 0.51613 0.39516 0.78836
LOF 41 0.47541 0.34426 0.56858 0.46072 0.55090 0.43862 0.78877
LOF 45 0.47541 0.34426 0.57131 0.46414 0.55422 0.44277 0.78937
LOF 47 0.47541 0.34426 0.57149 0.46436 0.55758 0.44697 0.78803
LOF 98 0.44262 0.30328 0.57938 0.47422 0.50955 0.38694 0.78527
SimplifiedLOF 68 0.49180 0.36475 0.57310 0.46638 0.52941 0.41176 0.78816
SimplifiedLOF 78 0.49180 0.36475 0.57909 0.47386 0.53714 0.42143 0.79206
SimplifiedLOF 79 0.49180 0.36475 0.57924 0.47405 0.54023 0.42529 0.79179
SimplifiedLOF 96 0.47541 0.34426 0.58025 0.47531 0.53409 0.41761 0.79038
LoOP 41 0.49180 0.36475 0.55503 0.44379 0.51007 0.38758 0.78185
LoOP 82 0.49180 0.36475 0.57823 0.47279 0.53409 0.41761 0.79045
LoOP 83 0.49180 0.36475 0.57743 0.47179 0.53409 0.41761 0.79065
LoOP 89 0.49180 0.36475 0.57843 0.47304 0.52695 0.40868 0.79018
LDOF 80 0.45902 0.32377 0.56834 0.46042 0.51948 0.39935 0.78608
LDOF 81 0.45902 0.32377 0.56785 0.45981 0.52288 0.40359 0.78581
LDOF 84 0.47541 0.34426 0.56894 0.46117 0.51948 0.39935 0.78554
LDOF 99 0.49180 0.36475 0.56872 0.46090 0.51163 0.38953 0.78521
ODIN 73 0.47541 0.34426 0.44648 0.30810 0.51724 0.39655 0.77832
ODIN 85 0.45902 0.32377 0.46180 0.32725 0.52439 0.40549 0.77913
ODIN 92 0.47541 0.34426 0.49257 0.36571 0.51977 0.39972 0.78272
FastABOD 23 0.42623 0.28279 0.54584 0.43231 0.54023 0.42529 0.78581
FastABOD 43 0.42623 0.28279 0.55993 0.44992 0.51724 0.39655 0.79387
FastABOD 56 0.45902 0.32377 0.56208 0.45260 0.52941 0.41176 0.79078
FastABOD 78 0.44262 0.30328 0.56708 0.45885 0.53254 0.41568 0.78910
KDEOS 23 0.36066 0.20082 0.31135 0.13919 0.43478 0.29348 0.69907
KDEOS 98 0.36066 0.20082 0.32465 0.15581 0.47761 0.34701 0.72601
KDEOS 99 0.34426 0.18033 0.32474 0.15593 0.48241 0.35302 0.72601
KDEOS 100 0.34426 0.18033 0.32172 0.15215 0.48309 0.35386 0.72541
LDF 41 0.49180 0.36475 0.46635 0.33293 0.49180 0.36475 0.76122
LDF 43 0.47541 0.34426 0.49973 0.37467 0.52121 0.40152 0.79038
LDF 94 0.47541 0.34426 0.58860 0.48575 0.55319 0.44149 0.76015
INFLO 27 0.47541 0.34426 0.55055 0.43818 0.52980 0.41225 0.78205
INFLO 37 0.50820 0.38525 0.55529 0.44411 0.51613 0.39516 0.77741
INFLO 55 0.47541 0.34426 0.57255 0.46569 0.52174 0.40217 0.79696
INFLO 83 0.47541 0.34426 0.58262 0.47828 0.52174 0.40217 0.78924
COF 5 0.45902 0.32377 0.53592 0.41990 0.52439 0.40549 0.77855
COF 27 0.49180 0.36475 0.48603 0.35754 0.50955 0.38694 0.76297
COF 32 0.45902 0.32377 0.52361 0.40451 0.53333 0.41667 0.76371
COF 40 0.45902 0.32377 0.54967 0.43709 0.49664 0.37081 0.76639

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 259 attributes, 305 objects, 61 outliers (20.00%)

Download raw algorithm results (2.7 MB) Download raw algorithm evaluation table (53.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.45902 0.32377 0.52727 0.40909 0.53254 0.41568 0.77328
KNN 3 0.45902 0.32377 0.53141 0.41426 0.52000 0.40000 0.77842
KNN 9 0.49180 0.36475 0.52533 0.40666 0.50667 0.38333 0.76858
KNNW 1 0.47541 0.34426 0.51615 0.39519 0.50000 0.37500 0.76065
KNNW 4 0.47541 0.34426 0.52732 0.40915 0.53086 0.41358 0.77607
KNNW 10 0.47541 0.34426 0.52809 0.41011 0.51852 0.39815 0.77365
LOF 5 0.45902 0.32377 0.49926 0.37407 0.52023 0.40029 0.75027
LOF 8 0.45902 0.32377 0.47917 0.34896 0.53179 0.41474 0.74913
LOF 12 0.45902 0.32377 0.48762 0.35952 0.50888 0.38609 0.76216
LOF 16 0.49180 0.36475 0.48783 0.35979 0.49462 0.36828 0.75914
SimplifiedLOF 10 0.47541 0.34426 0.48070 0.35088 0.52514 0.40642 0.77083
SimplifiedLOF 17 0.50820 0.38525 0.49781 0.37227 0.51240 0.39050 0.76592
SimplifiedLOF 23 0.49180 0.36475 0.50778 0.38472 0.50000 0.37500 0.77036
LoOP 10 0.47541 0.34426 0.47520 0.34399 0.51892 0.39865 0.77294
LoOP 11 0.49180 0.36475 0.47633 0.34541 0.50515 0.38144 0.76841
LoOP 46 0.47541 0.34426 0.50393 0.37991 0.50340 0.37925 0.76250
LDOF 9 0.47541 0.34426 0.46540 0.33175 0.50292 0.37865 0.76525
LDOF 10 0.44262 0.30328 0.46858 0.33572 0.53763 0.42204 0.77694
LDOF 11 0.42623 0.28279 0.46643 0.33304 0.54023 0.42529 0.77083
LDOF 46 0.45902 0.32377 0.50997 0.38746 0.50633 0.38291 0.76848
ODIN 15 0.41070 0.26337 0.41427 0.26783 0.52023 0.40029 0.75719
ODIN 45 0.49727 0.37158 0.46682 0.33353 0.50000 0.37500 0.76062
ODIN 55 0.47541 0.34426 0.47100 0.33875 0.51007 0.38758 0.76290
ODIN 94 0.44262 0.30328 0.47486 0.34358 0.49682 0.37102 0.75235
FastABOD 5 0.49180 0.36475 0.51915 0.39893 0.53333 0.41667 0.78507
FastABOD 6 0.52459 0.40574 0.51747 0.39684 0.53333 0.41667 0.78326
KDEOS 36 0.39344 0.24180 0.37607 0.22008 0.47458 0.34322 0.73213
KDEOS 50 0.42623 0.28279 0.35685 0.19606 0.50575 0.38218 0.74093
KDEOS 83 0.36066 0.20082 0.34425 0.18031 0.52695 0.40868 0.74019
KDEOS 95 0.39344 0.24180 0.37145 0.21432 0.51765 0.39706 0.74993
LDF 2 0.40984 0.26230 0.33931 0.17414 0.42373 0.27966 0.64888
LDF 43 0.39344 0.24180 0.40449 0.25561 0.42222 0.27778 0.69733
LDF 62 0.32787 0.15984 0.39296 0.24120 0.46486 0.33108 0.68564
LDF 67 0.31148 0.13934 0.41543 0.26928 0.42424 0.28030 0.68140
INFLO 2 0.47541 0.34426 0.40010 0.25012 0.48120 0.35150 0.69202
INFLO 7 0.42623 0.28279 0.46445 0.33056 0.51899 0.39873 0.75611
INFLO 33 0.45902 0.32377 0.50146 0.37683 0.50602 0.38253 0.77956
INFLO 54 0.44262 0.30328 0.50372 0.37966 0.50314 0.37893 0.76095
COF 5 0.44262 0.30328 0.49718 0.37148 0.47393 0.34242 0.74080
COF 51 0.42623 0.28279 0.47172 0.33965 0.50649 0.38312 0.72346
COF 67 0.47541 0.34426 0.47582 0.34478 0.48855 0.36069 0.72924

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO