Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

Wilt (2% of outliers version#02)

The data set differentiates diseased trees from other land covers. The former are considered outliers here.

Download all data set variants used (7.2 MB). You can also access the original data. (merge train and test [wilt.zip])

Normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (39.6 MB) Download raw algorithm evaluation table (72.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.02219 0.00226 0.05691 0.03768 0.58465
KNNW 1 0.00000 -0.02039 0.02469 0.00481 0.06291 0.04380 0.61799
LOF 2 0.08602 0.06739 0.04391 0.02442 0.11671 0.09870 0.63120
LOF 3 0.06452 0.04545 0.04569 0.02624 0.11502 0.09697 0.67044
LOF 4 0.05376 0.03447 0.04456 0.02508 0.12098 0.10306 0.67080
LOF 6 0.03226 0.01253 0.03803 0.01841 0.09256 0.07406 0.68945
SimplifiedLOF 3 0.08602 0.06739 0.05136 0.03202 0.12069 0.10276 0.72964
SimplifiedLOF 4 0.06452 0.04545 0.05279 0.03348 0.13599 0.11837 0.73762
SimplifiedLOF 5 0.06452 0.04545 0.05244 0.03312 0.13763 0.12005 0.73671
SimplifiedLOF 7 0.04301 0.02350 0.04950 0.03012 0.12500 0.10716 0.74066
LoOP 3 0.08602 0.06739 0.05199 0.03266 0.12778 0.11000 0.72548
LoOP 5 0.05376 0.03447 0.05484 0.03558 0.14963 0.13229 0.73661
LoOP 8 0.04301 0.02350 0.05112 0.03177 0.12684 0.10904 0.74199
LDOF 4 0.08602 0.06739 0.05151 0.03217 0.11017 0.09203 0.73639
LDOF 7 0.07527 0.05642 0.05703 0.03780 0.14198 0.12448 0.75674
LDOF 10 0.04301 0.02350 0.05794 0.03874 0.13419 0.11653 0.77828
LDOF 11 0.05376 0.03447 0.05700 0.03778 0.13203 0.11434 0.78157
ODIN 4 0.09649 0.07807 0.05274 0.03343 0.12429 0.10644 0.72082
ODIN 5 0.09143 0.07290 0.05367 0.03438 0.13833 0.12076 0.72765
ODIN 8 0.07581 0.05697 0.04996 0.03060 0.11480 0.09676 0.74169
FastABOD 3 0.00000 -0.02039 0.02245 0.00253 0.05011 0.03075 0.57942
FastABOD 4 0.00000 -0.02039 0.02374 0.00384 0.05400 0.03471 0.60384
FastABOD 6 0.00000 -0.02039 0.02344 0.00353 0.05913 0.03995 0.60582
KDEOS 16 0.15054 0.13322 0.07543 0.05658 0.15936 0.14223 0.76958
KDEOS 29 0.13978 0.12225 0.09135 0.07282 0.15548 0.13826 0.78817
KDEOS 32 0.15054 0.13322 0.08624 0.06761 0.15470 0.13746 0.79153
KDEOS 47 0.12903 0.11128 0.08506 0.06641 0.17595 0.15915 0.77945
LDF 2 0.07527 0.05642 0.03664 0.01700 0.08874 0.07016 0.61831
LDF 5 0.02151 0.00156 0.03239 0.01266 0.07886 0.06008 0.66418
INFLO 4 0.07527 0.05642 0.04568 0.02622 0.11111 0.09299 0.69906
INFLO 6 0.03226 0.01253 0.04048 0.02092 0.12142 0.10351 0.65258
COF 2 0.06452 0.04545 0.04379 0.02430 0.10000 0.08165 0.69281
COF 5 0.04301 0.02350 0.04233 0.02280 0.09187 0.07335 0.71277

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (39.7 MB) Download raw algorithm evaluation table (72.3 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.02100 0.00111 0.05197 0.03271 0.56219
KNNW 1 0.00000 -0.02031 0.02223 0.00236 0.05432 0.03511 0.57427
LOF 2 0.04301 0.02357 0.03155 0.01187 0.08075 0.06207 0.59731
LOF 3 0.02151 0.00163 0.03387 0.01424 0.09497 0.07659 0.62496
LOF 4 0.03226 0.01260 0.03444 0.01483 0.08757 0.06903 0.62691
LOF 5 0.02151 0.00163 0.03318 0.01354 0.08519 0.06661 0.63712
SimplifiedLOF 3 0.04301 0.02357 0.03918 0.01967 0.09231 0.07387 0.69077
SimplifiedLOF 4 0.03226 0.01260 0.04256 0.02311 0.10284 0.08461 0.70156
SimplifiedLOF 5 0.03226 0.01260 0.04272 0.02327 0.09861 0.08030 0.71475
SimplifiedLOF 8 0.03226 0.01260 0.04190 0.02243 0.09482 0.07643 0.72441
LoOP 3 0.04301 0.02357 0.04013 0.02063 0.09384 0.07543 0.69146
LoOP 4 0.03226 0.01260 0.04355 0.02412 0.10616 0.08801 0.70835
LoOP 8 0.03226 0.01260 0.04440 0.02499 0.09882 0.08052 0.73442
LDOF 5 0.06452 0.04551 0.04268 0.02324 0.08885 0.07034 0.72897
LDOF 9 0.02151 0.00163 0.04962 0.03031 0.11263 0.09461 0.78216
LDOF 11 0.03226 0.01260 0.05046 0.03117 0.11185 0.09381 0.77972
ODIN 4 0.07018 0.05129 0.04232 0.02286 0.09160 0.07315 0.70824
ODIN 8 0.06183 0.04277 0.04990 0.03060 0.12202 0.10419 0.73147
FastABOD 3 0.01075 -0.00934 0.02351 0.00367 0.05179 0.03252 0.57883
FastABOD 8 0.00000 -0.02031 0.02130 0.00141 0.05660 0.03743 0.56324
KDEOS 24 0.10753 0.08940 0.06886 0.04994 0.15672 0.13959 0.76153
KDEOS 38 0.10753 0.08940 0.07381 0.05499 0.16384 0.14686 0.75120
KDEOS 41 0.10753 0.08940 0.07700 0.05824 0.14525 0.12789 0.74960
KDEOS 66 0.11828 0.10037 0.06328 0.04425 0.13937 0.12189 0.72508
LDF 2 0.03226 0.01260 0.03023 0.01053 0.07738 0.05864 0.58622
LDF 3 0.04301 0.02357 0.03171 0.01204 0.07077 0.05190 0.62720
INFLO 4 0.04301 0.02357 0.03561 0.01602 0.09147 0.07301 0.66004
COF 3 0.01075 -0.00934 0.03792 0.01838 0.10117 0.08291 0.67862
COF 4 0.03226 0.01260 0.03711 0.01755 0.08755 0.06902 0.67507

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 5 attributes, 4655 objects, 93 outliers (2.00%)

Download raw algorithm results (40.1 MB) Download raw algorithm evaluation table (72.8 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02039 0.04088 0.02133 0.09524 0.07679 0.77399
KNN 4 0.00000 -0.02039 0.04282 0.02330 0.10590 0.08767 0.78878
KNN 7 0.00000 -0.02039 0.04180 0.02227 0.10686 0.08865 0.78214
KNNW 1 0.00000 -0.02039 0.03598 0.01632 0.08594 0.06731 0.72559
KNNW 7 0.00000 -0.02039 0.04227 0.02275 0.10195 0.08364 0.78773
KNNW 8 0.00000 -0.02039 0.04228 0.02276 0.10393 0.08566 0.78765
LOF 2 0.08602 0.06739 0.04518 0.02572 0.10566 0.08743 0.64089
LOF 13 0.05376 0.03447 0.08914 0.07058 0.19244 0.17598 0.87745
LOF 14 0.04301 0.02350 0.08770 0.06910 0.19504 0.17863 0.87465
LOF 17 0.04301 0.02350 0.08987 0.07131 0.19033 0.17383 0.87307
SimplifiedLOF 8 0.08602 0.06739 0.06801 0.04901 0.15718 0.14000 0.78670
SimplifiedLOF 25 0.06452 0.04545 0.08429 0.06562 0.16617 0.14917 0.87181
SimplifiedLOF 33 0.04301 0.02350 0.08414 0.06547 0.17984 0.16313 0.87466
SimplifiedLOF 35 0.03226 0.01253 0.08343 0.06475 0.18097 0.16427 0.87358
LoOP 20 0.09677 0.07836 0.08111 0.06237 0.15880 0.14165 0.86022
LoOP 33 0.04301 0.02350 0.08398 0.06530 0.18289 0.16623 0.87316
LoOP 36 0.03226 0.01253 0.08334 0.06466 0.18944 0.17292 0.87210
LDOF 5 0.05376 0.03447 0.03286 0.01314 0.07656 0.05773 0.60911
LDOF 43 0.04301 0.02350 0.07502 0.05617 0.15973 0.14260 0.85776
LDOF 52 0.01075 -0.00941 0.07472 0.05585 0.16264 0.14557 0.85977
LDOF 54 0.01075 -0.00941 0.07378 0.05490 0.16858 0.15163 0.85858
ODIN 25 0.11962 0.10168 0.09893 0.08056 0.18937 0.17284 0.87385
ODIN 26 0.12584 0.10801 0.10069 0.08236 0.19338 0.17694 0.87159
ODIN 27 0.11146 0.09335 0.10067 0.08234 0.19733 0.18097 0.87238
FastABOD 4 0.01075 -0.00941 0.02611 0.00626 0.05936 0.04019 0.63077
FastABOD 26 0.00000 -0.02039 0.02844 0.00864 0.06638 0.04735 0.65823
FastABOD 68 0.00000 -0.02039 0.02752 0.00770 0.06892 0.04994 0.64750
KDEOS 100 0.21505 0.19905 0.14184 0.12435 0.22093 0.20505 0.85156
LDF 2 0.09677 0.07836 0.05135 0.03201 0.12500 0.10716 0.65694
LDF 10 0.04301 0.02350 0.09426 0.07580 0.20188 0.18561 0.88399
LDF 12 0.02151 0.00156 0.09287 0.07438 0.20583 0.18964 0.88062
INFLO 1 0.07527 0.05642 0.03318 0.01347 0.09174 0.07323 0.56751
INFLO 20 0.06452 0.04545 0.07383 0.05495 0.15842 0.14126 0.82413
INFLO 22 0.06452 0.04545 0.07269 0.05379 0.16858 0.15163 0.81621
INFLO 29 0.03226 0.01253 0.07220 0.05329 0.15486 0.13763 0.82890
COF 21 0.10753 0.08933 0.09014 0.07159 0.17365 0.15681 0.87593
COF 25 0.13978 0.12225 0.09383 0.07535 0.19162 0.17514 0.87194
COF 28 0.17204 0.15516 0.09398 0.07551 0.18261 0.16595 0.87433

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 5 attributes, 4671 objects, 93 outliers (1.99%)

Download raw algorithm results (40.2 MB) Download raw algorithm evaluation table (71.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.00000 -0.02031 0.03968 0.02017 0.09055 0.07208 0.76481
KNN 2 0.00000 -0.02031 0.04066 0.02117 0.09129 0.07283 0.77230
KNN 6 0.00000 -0.02031 0.03871 0.01918 0.09719 0.07885 0.75982
KNNW 1 0.00000 -0.02031 0.03470 0.01509 0.08382 0.06520 0.71220
KNNW 5 0.00000 -0.02031 0.04032 0.02082 0.09407 0.07567 0.77186
KNNW 10 0.00000 -0.02031 0.03939 0.01988 0.09745 0.07912 0.76576
LOF 5 0.07527 0.05648 0.06678 0.04782 0.13977 0.12229 0.81873
LOF 10 0.05376 0.03454 0.08443 0.06584 0.17403 0.15725 0.87332
LOF 16 0.03226 0.01260 0.08870 0.07019 0.18370 0.16712 0.86786
LOF 26 0.05376 0.03454 0.08424 0.06564 0.19239 0.17599 0.84901
SimplifiedLOF 6 0.06452 0.04551 0.05184 0.03258 0.13283 0.11521 0.74086
SimplifiedLOF 22 0.04301 0.02357 0.08388 0.06527 0.17466 0.15789 0.87406
SimplifiedLOF 24 0.04301 0.02357 0.08392 0.06531 0.17333 0.15654 0.87265
SimplifiedLOF 29 0.03226 0.01260 0.08179 0.06314 0.18403 0.16745 0.86913
LoOP 6 0.06452 0.04551 0.05064 0.03135 0.12795 0.11023 0.72322
LoOP 26 0.05376 0.03454 0.08270 0.06407 0.17407 0.15730 0.86660
LoOP 32 0.03226 0.01260 0.08079 0.06212 0.17563 0.15888 0.86363
LDOF 20 0.05376 0.03454 0.05872 0.03960 0.13036 0.11269 0.81394
LDOF 48 0.03226 0.01260 0.07230 0.05346 0.15773 0.14062 0.86049
LDOF 61 0.03226 0.01260 0.07085 0.05197 0.16492 0.14795 0.85589
ODIN 23 0.12882 0.11112 0.10315 0.08493 0.17869 0.16201 0.87328
ODIN 24 0.14122 0.12377 0.10258 0.08435 0.18081 0.16417 0.87171
ODIN 26 0.13455 0.11697 0.10584 0.08767 0.19033 0.17388 0.87233
ODIN 36 0.11828 0.10037 0.10108 0.08282 0.20270 0.18651 0.86387
FastABOD 4 0.02151 0.00163 0.02720 0.00744 0.06264 0.04360 0.63064
FastABOD 10 0.00000 -0.02031 0.02780 0.00805 0.06571 0.04673 0.64508
FastABOD 11 0.00000 -0.02031 0.02799 0.00824 0.06544 0.04646 0.64707
FastABOD 14 0.00000 -0.02031 0.02794 0.00820 0.06354 0.04452 0.64732
KDEOS 59 0.16129 0.14425 0.10220 0.08396 0.18543 0.16888 0.84566
KDEOS 93 0.22581 0.21008 0.13180 0.11416 0.23121 0.21560 0.84284
KDEOS 94 0.22581 0.21008 0.13148 0.11384 0.23596 0.22043 0.84337
KDEOS 99 0.21505 0.19911 0.13382 0.11622 0.21858 0.20271 0.84283
LDF 8 0.06452 0.04551 0.09080 0.07233 0.18548 0.16894 0.87794
LDF 15 0.06452 0.04551 0.08905 0.07055 0.19811 0.18182 0.85748
LDF 16 0.08602 0.06745 0.08873 0.07022 0.19294 0.17655 0.85398
INFLO 23 0.08602 0.06745 0.07380 0.05499 0.16000 0.14294 0.82739
INFLO 25 0.06452 0.04551 0.07386 0.05505 0.16883 0.15195 0.82823
INFLO 27 0.04301 0.02357 0.07321 0.05438 0.15833 0.14124 0.82864
COF 20 0.08602 0.06745 0.09483 0.07644 0.20896 0.19289 0.87246
COF 23 0.11828 0.10037 0.10300 0.08478 0.19463 0.17827 0.87864
COF 30 0.16129 0.14425 0.10325 0.08503 0.20202 0.18581 0.86608
COF 38 0.17204 0.15522 0.10251 0.08427 0.20192 0.18571 0.85427

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO