Supplementary Material for
On the Evaluation of Unsupervised Outlier Detection: Measures, Datasets, and an Empirical Study
by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery 30(4): 891-927, 2016, DOI: 10.1007/s10618-015-0444-8

PageBlocks (2% of outliers version#01)

The data set contains information about different types of blocks in document pages. The task of distinguishing them is an essential step in document analysis, namely to separate text from pictures or graphics. If the block content is text, it was labeled here as inlier, otherwise it was labeled as outlier.

Download all data set variants used (14.6 MB). You can also access the original data. (page-blocks.data.Z)

Normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (42.1 MB) Download raw algorithm evaluation table (67.1 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.46465 0.45379 0.39847 0.38627 0.48128 0.47077 0.90391
KNN 4 0.46465 0.45379 0.41960 0.40783 0.50000 0.48986 0.91639
KNN 5 0.45455 0.44349 0.41121 0.39927 0.48000 0.46946 0.92387
KNNW 6 0.47475 0.46410 0.41063 0.39868 0.50286 0.49278 0.91436
KNNW 7 0.47475 0.46410 0.41509 0.40323 0.50292 0.49285 0.91712
KNNW 12 0.46465 0.45379 0.42137 0.40964 0.46927 0.45851 0.92137
LOF 16 0.45455 0.44349 0.38999 0.37762 0.45933 0.44837 0.89489
LOF 18 0.43434 0.42288 0.39656 0.38433 0.46445 0.45360 0.89812
LOF 22 0.44444 0.43318 0.38807 0.37567 0.47826 0.46768 0.91893
LOF 100 0.38384 0.37135 0.37282 0.36010 0.42308 0.41138 0.94731
SimplifiedLOF 23 0.45455 0.44349 0.40084 0.38870 0.49451 0.48426 0.89194
SimplifiedLOF 26 0.47475 0.46410 0.40920 0.39722 0.48485 0.47440 0.90163
SimplifiedLOF 100 0.42424 0.41257 0.40324 0.39114 0.45344 0.44236 0.94884
LoOP 28 0.45455 0.44349 0.38824 0.37584 0.48235 0.47186 0.89646
LoOP 42 0.45455 0.44349 0.39295 0.38064 0.46073 0.44980 0.91008
LoOP 100 0.42424 0.41257 0.36731 0.35448 0.42982 0.41826 0.94165
LDOF 65 0.44444 0.43318 0.41766 0.40585 0.44976 0.43860 0.95022
LDOF 71 0.44444 0.43318 0.41513 0.40327 0.45361 0.44253 0.95232
LDOF 100 0.43434 0.42288 0.42020 0.40845 0.44082 0.42948 0.95895
ODIN 48 0.40582 0.39378 0.30109 0.28692 0.40609 0.39405 0.87393
ODIN 66 0.39486 0.38259 0.31909 0.30529 0.42623 0.41460 0.89804
ODIN 96 0.39618 0.38394 0.34295 0.32963 0.41667 0.40484 0.91801
ODIN 100 0.40000 0.38784 0.34121 0.32785 0.41463 0.40277 0.92059
FastABOD 10 0.44444 0.43318 0.37836 0.36575 0.45000 0.43885 0.86888
FastABOD 12 0.45455 0.44349 0.38089 0.36834 0.45918 0.44822 0.86815
FastABOD 96 0.42424 0.41257 0.39212 0.37980 0.44828 0.43709 0.86572
KDEOS 44 0.05051 0.03125 0.07539 0.05665 0.18090 0.16430 0.81929
KDEOS 87 0.12121 0.10340 0.07689 0.05818 0.15676 0.13966 0.83998
KDEOS 91 0.10101 0.08278 0.08065 0.06201 0.16021 0.14318 0.84211
KDEOS 100 0.11111 0.09309 0.08064 0.06200 0.15294 0.13577 0.84612
LDF 11 0.48485 0.47440 0.37471 0.36204 0.49485 0.48460 0.86604
LDF 55 0.36364 0.35073 0.37485 0.36218 0.42321 0.41151 0.93950
LDF 70 0.41414 0.40226 0.38568 0.37322 0.42991 0.41835 0.93877
INFLO 16 0.43434 0.42288 0.34732 0.33409 0.44000 0.42865 0.81366
INFLO 25 0.43434 0.42288 0.37365 0.36095 0.45860 0.44762 0.83002
INFLO 26 0.42424 0.41257 0.37693 0.36429 0.45000 0.43885 0.82438
INFLO 90 0.38384 0.37135 0.36913 0.35634 0.42857 0.41699 0.89446
COF 25 0.44444 0.43318 0.39858 0.38638 0.48447 0.47402 0.84702
COF 29 0.44444 0.43318 0.40501 0.39295 0.46857 0.45780 0.86904
COF 30 0.45455 0.44349 0.40241 0.39029 0.47399 0.46332 0.87440
COF 97 0.41414 0.40226 0.37635 0.36371 0.42268 0.41098 0.90904

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (42.3 MB) Download raw algorithm evaluation table (62.4 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 9 0.41000 0.39799 0.36838 0.35552 0.42105 0.40927 0.92025
KNN 10 0.41000 0.39799 0.37079 0.35798 0.43103 0.41945 0.91845
KNN 15 0.40000 0.38779 0.39711 0.38484 0.40909 0.39706 0.93621
KNN 27 0.37000 0.35718 0.38914 0.37671 0.38202 0.36944 0.93778
KNNW 19 0.40000 0.38779 0.37230 0.35953 0.43038 0.41879 0.92656
KNNW 22 0.41000 0.39799 0.38313 0.37058 0.42500 0.41330 0.92988
KNNW 33 0.40000 0.38779 0.39921 0.38698 0.41350 0.40156 0.93699
KNNW 57 0.39000 0.37758 0.39721 0.38495 0.39196 0.37958 0.93965
LOF 24 0.36000 0.34697 0.32380 0.31004 0.40000 0.38779 0.82568
LOF 25 0.38000 0.36738 0.33224 0.31864 0.39640 0.38411 0.82349
LOF 56 0.36000 0.34697 0.34578 0.33247 0.36842 0.35557 0.93796
LOF 99 0.34000 0.32657 0.32893 0.31527 0.37093 0.35812 0.95640
SimplifiedLOF 22 0.39000 0.37758 0.30882 0.29475 0.39594 0.38364 0.86225
SimplifiedLOF 38 0.38000 0.36738 0.37301 0.36024 0.41148 0.39950 0.83172
SimplifiedLOF 54 0.36000 0.34697 0.37221 0.35944 0.42581 0.41412 0.85901
SimplifiedLOF 100 0.37000 0.35718 0.35921 0.34617 0.39264 0.38028 0.94821
LoOP 36 0.38000 0.36738 0.29913 0.28487 0.38532 0.37281 0.82587
LoOP 86 0.36000 0.34697 0.33869 0.32523 0.40764 0.39559 0.92674
LoOP 100 0.35000 0.33677 0.33443 0.32088 0.38462 0.37209 0.93903
LDOF 57 0.39000 0.37758 0.34716 0.33387 0.40230 0.39013 0.91686
LDOF 72 0.38000 0.36738 0.35868 0.34563 0.41573 0.40384 0.93267
LDOF 100 0.38000 0.36738 0.38325 0.37070 0.40000 0.38779 0.95022
ODIN 67 0.37556 0.36285 0.28253 0.26793 0.40625 0.39416 0.87938
ODIN 73 0.38222 0.36965 0.28202 0.26741 0.39216 0.37978 0.89428
ODIN 100 0.37333 0.36058 0.30197 0.28776 0.38053 0.36792 0.92551
FastABOD 5 0.31000 0.29596 0.27823 0.26354 0.33333 0.31976 0.81584
FastABOD 9 0.34000 0.32657 0.31337 0.29939 0.34518 0.33185 0.81202
FastABOD 15 0.35000 0.33677 0.30065 0.28642 0.35071 0.33750 0.81083
FastABOD 93 0.35000 0.33677 0.29964 0.28538 0.36872 0.35587 0.80610
KDEOS 3 0.07000 0.05107 0.02661 0.00680 0.09302 0.07456 0.54231
KDEOS 77 0.04000 0.02046 0.05898 0.03983 0.13938 0.12187 0.78120
KDEOS 99 0.04000 0.02046 0.06152 0.04242 0.13814 0.12060 0.80085
KDEOS 100 0.04000 0.02046 0.06147 0.04236 0.13699 0.11942 0.80170
LDF 20 0.40000 0.38779 0.33451 0.32097 0.40609 0.39400 0.82626
LDF 56 0.35000 0.33677 0.37709 0.36441 0.35870 0.34564 0.95457
LDF 68 0.34000 0.32657 0.37462 0.36189 0.40000 0.38779 0.95645
LDF 97 0.35000 0.33677 0.36548 0.35257 0.42804 0.41640 0.95382
INFLO 23 0.38000 0.36738 0.28185 0.26723 0.38000 0.36738 0.77827
INFLO 31 0.37000 0.35718 0.31335 0.29938 0.39779 0.38553 0.78984
INFLO 36 0.38000 0.36738 0.32310 0.30932 0.38627 0.37377 0.79892
INFLO 100 0.34000 0.32657 0.30935 0.29529 0.39241 0.38004 0.87813
COF 30 0.47000 0.45921 0.35054 0.33732 0.47525 0.46457 0.81183
COF 36 0.43000 0.41840 0.35833 0.34527 0.44762 0.43638 0.82628
COF 98 0.34000 0.32657 0.33328 0.31971 0.35749 0.34441 0.85695

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, without duplicates

This version contains 10 attributes, 4982 objects, 99 outliers (1.99%)

Download raw algorithm results (43.2 MB) Download raw algorithm evaluation table (65.6 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 1 0.20202 0.18584 0.14065 0.12323 0.22857 0.21293 0.63438
KNN 4 0.20202 0.18584 0.15007 0.13284 0.22439 0.20867 0.64981
KNN 8 0.16162 0.14462 0.14291 0.12554 0.18667 0.17018 0.65542
KNNW 1 0.20202 0.18584 0.14396 0.12661 0.23750 0.22204 0.62346
KNNW 2 0.20202 0.18584 0.14300 0.12563 0.24242 0.22706 0.63131
KNNW 7 0.20202 0.18584 0.14834 0.13107 0.21348 0.19754 0.64921
KNNW 19 0.15152 0.13431 0.14043 0.12300 0.18301 0.16644 0.65438
LOF 42 0.44444 0.43318 0.43386 0.42238 0.44444 0.43318 0.92850
LOF 53 0.44444 0.43318 0.44492 0.43366 0.46927 0.45851 0.93445
LOF 54 0.43434 0.42288 0.44381 0.43253 0.47191 0.46120 0.93474
LOF 59 0.42424 0.41257 0.43320 0.42171 0.44211 0.43079 0.93540
SimplifiedLOF 50 0.46465 0.45379 0.44086 0.42953 0.46829 0.45751 0.93493
SimplifiedLOF 53 0.46465 0.45379 0.44810 0.43691 0.47525 0.46461 0.93840
SimplifiedLOF 60 0.46465 0.45379 0.44837 0.43719 0.46701 0.45620 0.94077
SimplifiedLOF 64 0.45455 0.44349 0.44034 0.42899 0.46392 0.45305 0.94137
LoOP 59 0.43434 0.42288 0.40083 0.38868 0.44221 0.43090 0.93124
LoOP 72 0.42424 0.41257 0.42553 0.41389 0.45882 0.44785 0.93460
LoOP 77 0.43434 0.42288 0.42914 0.41757 0.45198 0.44087 0.93570
LoOP 98 0.43434 0.42288 0.41999 0.40823 0.44335 0.43206 0.93760
LDOF 94 0.39394 0.38165 0.40832 0.39632 0.43925 0.42788 0.94307
LDOF 97 0.40404 0.39196 0.40957 0.39760 0.43269 0.42119 0.94446
LDOF 99 0.41414 0.40226 0.40859 0.39660 0.43478 0.42332 0.94450
ODIN 93 0.32867 0.31506 0.32875 0.31514 0.36478 0.35190 0.91933
ODIN 95 0.32323 0.30951 0.32794 0.31432 0.36129 0.34834 0.91942
ODIN 98 0.33189 0.31834 0.32781 0.31419 0.35065 0.33748 0.91874
FastABOD 5 0.17172 0.15492 0.12599 0.10827 0.18868 0.17223 0.52968
FastABOD 7 0.18182 0.16523 0.12624 0.10852 0.19608 0.17978 0.52812
FastABOD 10 0.16162 0.14462 0.12570 0.10797 0.20134 0.18515 0.52730
FastABOD 43 0.16162 0.14462 0.12694 0.10924 0.19580 0.17950 0.52389
KDEOS 95 0.14141 0.12401 0.06813 0.04924 0.14286 0.12548 0.78710
KDEOS 96 0.13131 0.11370 0.06886 0.04998 0.14365 0.12628 0.78862
KDEOS 100 0.12121 0.10340 0.07022 0.05137 0.13084 0.11322 0.79415
LDF 40 0.42424 0.41257 0.44929 0.43813 0.43787 0.42647 0.92019
LDF 43 0.41414 0.40226 0.44024 0.42889 0.45087 0.43973 0.92068
LDF 59 0.39394 0.38165 0.40295 0.39084 0.41791 0.40611 0.92247
INFLO 59 0.45455 0.44349 0.41552 0.40367 0.46486 0.45402 0.90242
INFLO 62 0.44444 0.43318 0.41632 0.40448 0.45745 0.44645 0.90843
INFLO 63 0.44444 0.43318 0.41441 0.40254 0.45055 0.43941 0.90850
COF 43 0.45455 0.44349 0.43700 0.42559 0.50000 0.48986 0.90541
COF 49 0.48485 0.47440 0.44592 0.43468 0.48731 0.47692 0.91406
COF 56 0.45455 0.44349 0.45147 0.44035 0.47742 0.46682 0.91620
COF 62 0.44444 0.43318 0.44631 0.43508 0.49711 0.48691 0.91690

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO

Not normalized, duplicates

This version contains 10 attributes, 5013 objects, 100 outliers (1.99%)

Download raw algorithm results (43.3 MB) Download raw algorithm evaluation table (61.5 kB)

Best Parameters

The following table contains the best (overall and per-method) results for each method and evaluation measure (when the same score was achieved twice, only the smallest k is given).
The Maximum F1-Measure is complimentary in addition to the measures in the original publication.

Algorithm k P@n Adj. P@n AP Adj. AP Max-F1 Adj. MF1 ROC AUC
KNN 2 0.16000 0.14290 0.11740 0.09944 0.17778 0.16104 0.65443
KNN 8 0.14000 0.12250 0.11343 0.09539 0.14778 0.13044 0.66985
KNN 9 0.14000 0.12250 0.11517 0.09716 0.17808 0.16135 0.66964
KNNW 1 0.15000 0.13270 0.11973 0.10181 0.15833 0.14120 0.61672
KNNW 3 0.16000 0.14290 0.11706 0.09908 0.16842 0.15149 0.64098
KNNW 5 0.16000 0.14290 0.11693 0.09896 0.17204 0.15519 0.65260
KNNW 16 0.14000 0.12250 0.11257 0.09451 0.14563 0.12824 0.66629
LOF 40 0.45000 0.43881 0.42289 0.41114 0.45226 0.44111 0.93762
LOF 43 0.44000 0.42860 0.42604 0.41436 0.47826 0.46764 0.93895
LOF 44 0.45000 0.43881 0.42305 0.41131 0.48128 0.47073 0.93932
LOF 67 0.42000 0.40819 0.39046 0.37805 0.43636 0.42489 0.94618
SimplifiedLOF 36 0.47000 0.45921 0.43427 0.42276 0.49327 0.48296 0.92306
SimplifiedLOF 40 0.48000 0.46942 0.43761 0.42616 0.48430 0.47381 0.92686
SimplifiedLOF 45 0.46000 0.44901 0.43892 0.42750 0.48128 0.47073 0.93402
SimplifiedLOF 67 0.44000 0.42860 0.41300 0.40105 0.47059 0.45981 0.95343
LoOP 57 0.46000 0.44901 0.41684 0.40497 0.48936 0.47897 0.93995
LoOP 70 0.47000 0.45921 0.43284 0.42130 0.48421 0.47371 0.94854
LoOP 74 0.48000 0.46942 0.42853 0.41690 0.48000 0.46942 0.94944
LoOP 99 0.46000 0.44901 0.42102 0.40924 0.46388 0.45297 0.95374
LDOF 73 0.44000 0.42860 0.41374 0.40180 0.45498 0.44388 0.93835
LDOF 75 0.45000 0.43881 0.41101 0.39902 0.45498 0.44388 0.94041
LDOF 85 0.44000 0.42860 0.41414 0.40222 0.45238 0.44123 0.94886
LDOF 100 0.44000 0.42860 0.41257 0.40062 0.44565 0.43437 0.95623
ODIN 60 0.36400 0.35105 0.27828 0.26359 0.37989 0.36727 0.90646
ODIN 95 0.35368 0.34053 0.34200 0.32861 0.40000 0.38779 0.93786
ODIN 96 0.35471 0.34157 0.33980 0.32636 0.40000 0.38779 0.93793
FastABOD 5 0.12000 0.10209 0.08665 0.06806 0.14634 0.12897 0.53375
FastABOD 6 0.13000 0.11229 0.10234 0.08407 0.14815 0.13081 0.53317
FastABOD 7 0.13000 0.11229 0.10246 0.08419 0.14907 0.13175 0.53252
FastABOD 9 0.13000 0.11229 0.09842 0.08007 0.15000 0.13270 0.53273
KDEOS 97 0.09000 0.07148 0.06037 0.04124 0.10124 0.08295 0.78969
KDEOS 100 0.09000 0.07148 0.06223 0.04314 0.10736 0.08919 0.79393
LDF 17 0.46000 0.44901 0.44041 0.42902 0.46465 0.45375 0.92259
LDF 22 0.44000 0.42860 0.44621 0.43493 0.45810 0.44707 0.93085
LDF 25 0.42000 0.40819 0.43717 0.42572 0.46512 0.45423 0.93593
LDF 27 0.45000 0.43881 0.43926 0.42785 0.45455 0.44344 0.93825
INFLO 39 0.45000 0.43881 0.40232 0.39015 0.48826 0.47785 0.85733
INFLO 44 0.47000 0.45921 0.40471 0.39260 0.47475 0.46406 0.87297
INFLO 61 0.46000 0.44901 0.39930 0.38708 0.46535 0.45446 0.92382
COF 36 0.47000 0.45921 0.43614 0.42467 0.48619 0.47573 0.89076
COF 39 0.48000 0.46942 0.43870 0.42727 0.48241 0.47188 0.90162
COF 43 0.46000 0.44901 0.44877 0.43755 0.47904 0.46844 0.90702
COF 63 0.40000 0.38779 0.43236 0.42081 0.43709 0.42563 0.92884

Plots

Precision at n
Adjusted precision at n
Average precision
Adjusted average precision
Maximum F1 score
Adjusted maximum F1 score
ROC AUC
Diversity
A: KNN, B: KNNW, C: LOF, D: SimplifiedLOF, E: LoOP, F: LDOF
G: ODIN, H: KDEOS, I: COF, J: FastABOD, K: LDF, L: INFLO