weka.classifiers.evaluation
Class ThresholdCurve

java.lang.Object
  extended byweka.classifiers.evaluation.ThresholdCurve

public class ThresholdCurve
extends java.lang.Object

Generates points illustrating prediction tradeoffs that can be obtained by varying the threshold value between classes. For example, the typical threshold value of 0.5 means the predicted probability of "positive" must be higher than 0.5 for the instance to be predicted as "positive". The resulting dataset can be used to visualize precision/recall tradeoff, or for ROC curve analysis (true positive rate vs false positive rate).

Version:
$Revision: 1.17 $
Author:
Len Trigg (len@reeltwo.com)

Field Summary
static java.lang.String FALLOUT_NAME
           
static java.lang.String FALSE_NEG_NAME
           
static java.lang.String FALSE_POS_NAME
           
static java.lang.String FMEASURE_NAME
           
static java.lang.String FP_RATE_NAME
           
static java.lang.String PRECISION_NAME
           
static java.lang.String RECALL_NAME
           
static java.lang.String RELATION_NAME
          The name of the relation used in threshold curve datasets
static java.lang.String THRESHOLD_NAME
           
static java.lang.String TP_RATE_NAME
           
static java.lang.String TRUE_NEG_NAME
           
static java.lang.String TRUE_POS_NAME
           
 
Constructor Summary
ThresholdCurve()
           
 
Method Summary
private static int binarySearch(int[] index, double[] vals, double target)
           
 Instances getCurve(FastVector predictions)
          Calculates the performance stats for the default class and return results as a set of Instances.
 Instances getCurve(FastVector predictions, int classIndex)
          Calculates the performance stats for the desired class and return results as a set of Instances.
static double getNPointPrecision(Instances tcurve, int n)
          Calculates the n point precision result, which is the precision averaged over n evenly spaced (w.r.t recall) samples of the curve.
private  double[] getProbabilities(FastVector predictions, int classIndex)
           
static double getROCArea(Instances tcurve)
          Calculates the area under the ROC curve.
static int getThresholdInstance(Instances tcurve, double threshold)
          Gets the index of the instance with the closest threshold value to the desired target
static void main(java.lang.String[] args)
          Tests the ThresholdCurve generation from the command line.
private  Instances makeHeader()
           
private  Instance makeInstance(TwoClassStats tc, double prob)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

RELATION_NAME

public static final java.lang.String RELATION_NAME
The name of the relation used in threshold curve datasets

See Also:
Constant Field Values

TRUE_POS_NAME

public static final java.lang.String TRUE_POS_NAME
See Also:
Constant Field Values

FALSE_NEG_NAME

public static final java.lang.String FALSE_NEG_NAME
See Also:
Constant Field Values

FALSE_POS_NAME

public static final java.lang.String FALSE_POS_NAME
See Also:
Constant Field Values

TRUE_NEG_NAME

public static final java.lang.String TRUE_NEG_NAME
See Also:
Constant Field Values

FP_RATE_NAME

public static final java.lang.String FP_RATE_NAME
See Also:
Constant Field Values

TP_RATE_NAME

public static final java.lang.String TP_RATE_NAME
See Also:
Constant Field Values

PRECISION_NAME

public static final java.lang.String PRECISION_NAME
See Also:
Constant Field Values

RECALL_NAME

public static final java.lang.String RECALL_NAME
See Also:
Constant Field Values

FALLOUT_NAME

public static final java.lang.String FALLOUT_NAME
See Also:
Constant Field Values

FMEASURE_NAME

public static final java.lang.String FMEASURE_NAME
See Also:
Constant Field Values

THRESHOLD_NAME

public static final java.lang.String THRESHOLD_NAME
See Also:
Constant Field Values
Constructor Detail

ThresholdCurve

public ThresholdCurve()
Method Detail

getCurve

public Instances getCurve(FastVector predictions)
Calculates the performance stats for the default class and return results as a set of Instances. The structure of these Instances is as follows:

For the definitions of these measures, see TwoClassStats

Returns:
datapoints as a set of instances, null if no predictions have been made.
See Also:
TwoClassStats

getCurve

public Instances getCurve(FastVector predictions,
                          int classIndex)
Calculates the performance stats for the desired class and return results as a set of Instances.

Parameters:
classIndex - index of the class of interest.
Returns:
datapoints as a set of instances.

getNPointPrecision

public static double getNPointPrecision(Instances tcurve,
                                        int n)
Calculates the n point precision result, which is the precision averaged over n evenly spaced (w.r.t recall) samples of the curve.

Parameters:
tcurve - a previously extracted threshold curve Instances.
n - the number of points to average over.
Returns:
the n-point precision.

getROCArea

public static double getROCArea(Instances tcurve)
Calculates the area under the ROC curve. This is normalised so that 0.5 is random, 1.0 is perfect and 0.0 is bizarre.

Parameters:
tcurve - a previously extracted threshold curve Instances.
Returns:
the ROC area, or Double.NaN if you don't pass in a ThresholdCurve generated Instances.

getThresholdInstance

public static int getThresholdInstance(Instances tcurve,
                                       double threshold)
Gets the index of the instance with the closest threshold value to the desired target

Parameters:
tcurve - a set of instances that have been generated by this class
threshold - the target threshold
Returns:
the index of the instance that has threshold closest to the target, or -1 if this could not be found (i.e. no data, or bad threshold target)

binarySearch

private static int binarySearch(int[] index,
                                double[] vals,
                                double target)

getProbabilities

private double[] getProbabilities(FastVector predictions,
                                  int classIndex)

makeHeader

private Instances makeHeader()

makeInstance

private Instance makeInstance(TwoClassStats tc,
                              double prob)

main

public static void main(java.lang.String[] args)
Tests the ThresholdCurve generation from the command line. The classifier is currently hardcoded. Pipe in an arff file.

Parameters:
args - currently ignored