weka.classifiers.functions
Class SMOreg

java.lang.Object
  extended byweka.classifiers.Classifier
      extended byweka.classifiers.functions.SMOreg
All Implemented Interfaces:
java.lang.Cloneable, OptionHandler, java.io.Serializable, WeightedInstancesHandler

public class SMOreg
extends Classifier
implements OptionHandler, WeightedInstancesHandler

Implements Alex J.Smola and Bernhard Scholkopf sequential minimal optimization algorithm for training a support vector regression using polynomial or RBF kernels. This implementation globally replaces all missing values and transforms nominal attributes into binary ones. It also normalizes all attributes by default. (Note that the coefficients in the output are based on the normalized/standardized data, not the original data.) For more information on the SMO algorithm, see

Alex J. Smola, Bernhard Scholkopf (1998). A Tutorial on Support Vector Regression. NeuroCOLT2 Technical Report Series - NC2-TR-1998-030.

S.K. Shevade, S.S. Keerthi, C. Bhattacharyya, K.R.K. Murthy, Improvements to SMO Algorithm for SVM Regression. Technical Report CD-99-16, Control Division Dept of Mechanical and Production Engineering, National University of Singapore.

Version:
$$
Author:
Sylvain Roy (sro33@student.canterbury.ac.nz)
See Also:
Serialized Form

Field Summary
static int FILTER_NONE
           
static int FILTER_NORMALIZE
          The filter to apply to the training data
static int FILTER_STANDARDIZE
           
private  double m_Alin
          The parameters of the linear transforamtion realized by the filter on the class attribute
private  double[] m_alpha
          The Lagrange multipliers
private  double[] m_alpha_
           
private  double m_b
          The thresholds.
private  double m_Blin
           
private  double m_bLow
          The thresholds.
private  double m_bUp
          The thresholds.
private  double m_C
          The complexity parameter
private  int m_cacheSize
          The size of the cache (a prime number)
private  boolean m_checksTurnedOff
          Turn off all checks and conversions?
private  int m_classIndex
          The class index from the training data
private  Instances m_data
          The training data.
private static double m_Del
          Precision constant for updating sets
private  double m_eps
          The parameter eps
private  double m_epsilon
          The parameter epsilon
private  double m_exponent
          The exponent for the polynomial kernel.
private  double[] m_fcache
          The current set of errors for all non-bound examples.
private  boolean m_featureSpaceNormalization
          Feature-space normalization?
private  Filter m_Filter
          The filter used to standardize/normalize all values.
private  int m_filterType
          Whether to normalize/standardize/neither
private  double m_gamma
          Gamma for the RBF kernel.
private  SMOset m_I0
          The five different sets used by the algorithm.
private  SMOset m_I1
           
private  SMOset m_I2
           
private  SMOset m_I3
           
private  int m_iLow
          The indices for m_bLow and m_bUp
private  int m_iUp
          The indices for m_bLow and m_bUp
private  Kernel m_kernel
          Kernel to use
private  boolean m_lowerOrder
          Use lower-order terms?
private  ReplaceMissingValues m_Missing
          The filter used to get rid of missing values.
private  NominalToBinary m_NominalToBinary
          The filter used to make attributes numeric.
private  boolean m_onlyNumeric
          Only numeric attributes in the dataset?
private  int[] m_sparseIndices
           
private  double[] m_sparseWeights
          Variables to hold weight vector in sparse form.
private  double m_tol
          The parameter tol
private  boolean m_useRBF
          Use RBF kernel?
private  double[] m_weights
          Weight vector for linear machine.
static Tag[] TAGS_FILTER
           
 
Fields inherited from class weka.classifiers.Classifier
m_Debug
 
Constructor Summary
SMOreg()
           
 
Method Summary
 void buildClassifier(Instances insts)
          Method for building the classifier.
 java.lang.String cacheSizeTipText()
          Returns the tip text for this property
private  void checkAlphas()
          Debuggage function Checks that : alpha*alpha_=0 sum(alpha[i] - alpha_[i]) = 0
private  void checkOptimality()
          Debuggage function.
private  void checkSets()
          Debuggage function.
 double classifyInstance(Instance inst)
          Classifies a given instance.
 java.lang.String cTipText()
          Returns the tip text for this property
private  void displayB()
          Debuggage function Compute and display bLow, lUp and so on...
private  void displayStat(int i1, int i2)
          Debuggage function.
 java.lang.String epsilonTipText()
          Returns the tip text for this property
 java.lang.String epsTipText()
          Returns the tip text for this property
private  int examineExample(int i2)
          Examines instance.
 java.lang.String exponentTipText()
          Returns the tip text for this property
 java.lang.String featureSpaceNormalizationTipText()
          Returns the tip text for this property
 java.lang.String filterTypeTipText()
          Returns the tip text for this property
 java.lang.String gammaTipText()
          Returns the tip text for this property
 double getC()
          Get the value of C.
 int getCacheSize()
          Get the size of the kernel cache
 double getEps()
          Get the value of eps.
 double getEpsilon()
          Get the value of epsilon.
 double getExponent()
          Get the value of exponent.
 boolean getFeatureSpaceNormalization()
          Check whether feature spaces is being normalized.
 SelectedTag getFilterType()
          Gets how the training data will be transformed.
 double getGamma()
          Get the value of gamma.
 boolean getLowerOrderTerms()
          Check whether lower-order terms are being used.
 java.lang.String[] getOptions()
          Gets the current settings of the classifier.
 double getToleranceParameter()
          Get the value of tolerance parameter.
 boolean getUseRBF()
          Check if the RBF kernel is to be used.
 java.lang.String globalInfo()
          Returns a string describing classifier
 java.util.Enumeration listOptions()
          Returns an enumeration describing the available options.
 java.lang.String lowerOrderTermsTipText()
          Returns the tip text for this property
static void main(java.lang.String[] argv)
          Main method for testing this class.
private  double objFun()
          Debuggage function.
private  double objFun(int i1, int i2, double alpha1, double alpha1_, double alpha2, double alpha2_)
          Debuggage function.
 void setC(double v)
          Set the value of C.
 void setCacheSize(int v)
          Set the value of the kernel cache.
 void setEps(double v)
          Set the value of eps.
 void setEpsilon(double v)
          Set the value of epsilon.
 void setExponent(double v)
          Set the value of exponent.
 void setFeatureSpaceNormalization(boolean v)
          Set whether feature space is normalized.
 void setFilterType(SelectedTag newType)
          Sets how the training data will be transformed.
 void setGamma(double v)
          Set the value of gamma.
 void setLowerOrderTerms(boolean v)
          Set whether lower-order terms are to be used.
 void setOptions(java.lang.String[] options)
          Parses a given list of options.
 void setToleranceParameter(double v)
          Set the value of tolerance parameter.
 void setUseRBF(boolean v)
          Set if the RBF kernel is to be used.
private  boolean takeStep(int i1, int i2)
          Method solving for the Lagrange multipliers for two instances.
 java.lang.String toleranceParameterTipText()
          Returns the tip text for this property
 java.lang.String toString()
          Prints out the classifier.
 void turnChecksOff()
          Turns off checks for missing values, etc.
 void turnChecksOn()
          Turns on checks for missing values, etc.
 java.lang.String useRBFTipText()
          Returns the tip text for this property
 
Methods inherited from class weka.classifiers.Classifier
debugTipText, distributionForInstance, forName, getDebug, makeCopies, setDebug
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

m_featureSpaceNormalization

private boolean m_featureSpaceNormalization
Feature-space normalization?


m_useRBF

private boolean m_useRBF
Use RBF kernel? (default: poly)


m_cacheSize

private int m_cacheSize
The size of the cache (a prime number)


m_kernel

private Kernel m_kernel
Kernel to use


m_lowerOrder

private boolean m_lowerOrder
Use lower-order terms?


m_exponent

private double m_exponent
The exponent for the polynomial kernel.


m_gamma

private double m_gamma
Gamma for the RBF kernel.


m_classIndex

private int m_classIndex
The class index from the training data


m_onlyNumeric

private boolean m_onlyNumeric
Only numeric attributes in the dataset?


m_NominalToBinary

private NominalToBinary m_NominalToBinary
The filter used to make attributes numeric.


FILTER_NORMALIZE

public static final int FILTER_NORMALIZE
The filter to apply to the training data

See Also:
Constant Field Values

FILTER_STANDARDIZE

public static final int FILTER_STANDARDIZE
See Also:
Constant Field Values

FILTER_NONE

public static final int FILTER_NONE
See Also:
Constant Field Values

TAGS_FILTER

public static final Tag[] TAGS_FILTER

m_Filter

private Filter m_Filter
The filter used to standardize/normalize all values.


m_filterType

private int m_filterType
Whether to normalize/standardize/neither


m_Missing

private ReplaceMissingValues m_Missing
The filter used to get rid of missing values.


m_checksTurnedOff

private boolean m_checksTurnedOff
Turn off all checks and conversions? Turning them off assumes that data is purely numeric, doesn't contain any missing values, and has a numeric class. Turning them off also means that no header information will be stored if the machine is linear. Finally, it also assumes that no instance has a weight equal to 0.


m_data

private Instances m_data
The training data.


m_C

private double m_C
The complexity parameter


m_alpha

private double[] m_alpha
The Lagrange multipliers


m_alpha_

private double[] m_alpha_

m_b

private double m_b
The thresholds.


m_bLow

private double m_bLow
The thresholds.


m_bUp

private double m_bUp
The thresholds.


m_iLow

private int m_iLow
The indices for m_bLow and m_bUp


m_iUp

private int m_iUp
The indices for m_bLow and m_bUp


m_weights

private double[] m_weights
Weight vector for linear machine.


m_fcache

private double[] m_fcache
The current set of errors for all non-bound examples.


m_I0

private SMOset m_I0
The five different sets used by the algorithm.


m_I1

private SMOset m_I1

m_I2

private SMOset m_I2

m_I3

private SMOset m_I3

m_epsilon

private double m_epsilon
The parameter epsilon


m_tol

private double m_tol
The parameter tol


m_eps

private double m_eps
The parameter eps


m_Del

private static double m_Del
Precision constant for updating sets


m_Alin

private double m_Alin
The parameters of the linear transforamtion realized by the filter on the class attribute


m_Blin

private double m_Blin

m_sparseWeights

private double[] m_sparseWeights
Variables to hold weight vector in sparse form. (To reduce storage requirements.)


m_sparseIndices

private int[] m_sparseIndices
Constructor Detail

SMOreg

public SMOreg()
Method Detail

globalInfo

public java.lang.String globalInfo()
Returns a string describing classifier

Returns:
a description suitable for displaying in the explorer/experimenter gui

buildClassifier

public void buildClassifier(Instances insts)
                     throws java.lang.Exception
Method for building the classifier.

Specified by:
buildClassifier in class Classifier
Parameters:
insts - the set of training instances
Throws:
java.lang.Exception - if the classifier can't be built successfully

examineExample

private int examineExample(int i2)
                    throws java.lang.Exception
Examines instance. (As defined in Shevade's paper.)

Parameters:
i2 - index of instance to examine
Returns:
true if examination was successfull
Throws:
java.lang.Exception - if something goes wrong

takeStep

private boolean takeStep(int i1,
                         int i2)
                  throws java.lang.Exception
Method solving for the Lagrange multipliers for two instances. (As defined in Shevade's paper.)

Parameters:
i1 - index of the first instance
i2 - index of the second instance
Returns:
true if multipliers could be found
Throws:
java.lang.Exception - if something goes wrong

classifyInstance

public double classifyInstance(Instance inst)
                        throws java.lang.Exception
Classifies a given instance.

Overrides:
classifyInstance in class Classifier
Parameters:
inst - the instance to be classified
Returns:
the classification
Throws:
java.lang.Exception - if instance could not be classified successfully

listOptions

public java.util.Enumeration listOptions()
Returns an enumeration describing the available options.

Specified by:
listOptions in interface OptionHandler
Overrides:
listOptions in class Classifier
Returns:
an enumeration of all the available options.

setOptions

public void setOptions(java.lang.String[] options)
                throws java.lang.Exception
Parses a given list of options. Valid options are:

-S num
The amount up to which deviation are tolerated (epsilon). (default 1e-3) Watch out, the value of epsilon is used with the (normalized/standardize) data

-C num
The complexity constant C. (default 1)

-E num
The exponent for the polynomial kernel. (default 1)

-G num
Gamma for the RBF kernel. (default 0.01)

-N <0|1|2>
Whether to 0=normalize/1=standardize/2=neither. (default 0=normalize)

-F
Feature-space normalization (only for non-linear polynomial kernels).

-O
Use lower-order terms (only for non-linear polynomial kernels).

-R
Use RBF kernel (default poly).

-A num
Sets the size of the kernel cache. Should be a prime number. (default 1000003)

-T num
Sets the tolerance parameter. (default 1.0e-3)

-P num
Sets the epsilon for round-off error. (default 1.0e-12)

Specified by:
setOptions in interface OptionHandler
Overrides:
setOptions in class Classifier
Parameters:
options - the list of options as an array of strings
Throws:
java.lang.Exception - if an option is not supported

getOptions

public java.lang.String[] getOptions()
Gets the current settings of the classifier.

Specified by:
getOptions in interface OptionHandler
Overrides:
getOptions in class Classifier
Returns:
an array of strings suitable for passing to setOptions

filterTypeTipText

public java.lang.String filterTypeTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getFilterType

public SelectedTag getFilterType()
Gets how the training data will be transformed. Will be one of FILTER_NORMALIZE, FILTER_STANDARDIZE, FILTER_NONE.

Returns:
the filtering mode

setFilterType

public void setFilterType(SelectedTag newType)
Sets how the training data will be transformed. Should be one of FILTER_NORMALIZE, FILTER_STANDARDIZE, FILTER_NONE.

Parameters:
newType - the new filtering mode

exponentTipText

public java.lang.String exponentTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getExponent

public double getExponent()
Get the value of exponent.

Returns:
Value of exponent.

setExponent

public void setExponent(double v)
Set the value of exponent. If linear kernel is used, rescaling and lower-order terms are turned off.

Parameters:
v - Value to assign to exponent.

gammaTipText

public java.lang.String gammaTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getGamma

public double getGamma()
Get the value of gamma.

Returns:
Value of gamma.

setGamma

public void setGamma(double v)
Set the value of gamma.

Parameters:
v - Value to assign to gamma.

cTipText

public java.lang.String cTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getC

public double getC()
Get the value of C.

Returns:
Value of C.

setC

public void setC(double v)
Set the value of C.

Parameters:
v - Value to assign to C.

toleranceParameterTipText

public java.lang.String toleranceParameterTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getToleranceParameter

public double getToleranceParameter()
Get the value of tolerance parameter.

Returns:
Value of tolerance parameter.

setToleranceParameter

public void setToleranceParameter(double v)
Set the value of tolerance parameter.

Parameters:
v - Value to assign to tolerance parameter.

epsTipText

public java.lang.String epsTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getEps

public double getEps()
Get the value of eps.

Returns:
Value of eps.

setEps

public void setEps(double v)
Set the value of eps.

Parameters:
v - Value to assign to epsilon.

epsilonTipText

public java.lang.String epsilonTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getEpsilon

public double getEpsilon()
Get the value of epsilon.

Returns:
Value of epsilon.

setEpsilon

public void setEpsilon(double v)
Set the value of epsilon.

Parameters:
v - Value to assign to epsilon.

cacheSizeTipText

public java.lang.String cacheSizeTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getCacheSize

public int getCacheSize()
Get the size of the kernel cache

Returns:
Size of kernel cache.

setCacheSize

public void setCacheSize(int v)
Set the value of the kernel cache.

Parameters:
v - Size of kernel cache.

useRBFTipText

public java.lang.String useRBFTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getUseRBF

public boolean getUseRBF()
Check if the RBF kernel is to be used.

Returns:
true if RBF

setUseRBF

public void setUseRBF(boolean v)
Set if the RBF kernel is to be used.

Parameters:
v - true if RBF

featureSpaceNormalizationTipText

public java.lang.String featureSpaceNormalizationTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getFeatureSpaceNormalization

public boolean getFeatureSpaceNormalization()
                                     throws java.lang.Exception
Check whether feature spaces is being normalized.

Returns:
true if feature space is normalized.
Throws:
java.lang.Exception

setFeatureSpaceNormalization

public void setFeatureSpaceNormalization(boolean v)
                                  throws java.lang.Exception
Set whether feature space is normalized.

Parameters:
v - true if feature space is to be normalized.
Throws:
java.lang.Exception

lowerOrderTermsTipText

public java.lang.String lowerOrderTermsTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

getLowerOrderTerms

public boolean getLowerOrderTerms()
Check whether lower-order terms are being used.

Returns:
Value of lowerOrder.

setLowerOrderTerms

public void setLowerOrderTerms(boolean v)
Set whether lower-order terms are to be used. Defaults to false if a linear machine is built.

Parameters:
v - Value to assign to lowerOrder.

turnChecksOff

public void turnChecksOff()
Turns off checks for missing values, etc. Use with caution.


turnChecksOn

public void turnChecksOn()
Turns on checks for missing values, etc.


toString

public java.lang.String toString()
Prints out the classifier.

Returns:
a description of the classifier as a string

main

public static void main(java.lang.String[] argv)
Main method for testing this class.


objFun

private double objFun()
               throws java.lang.Exception
Debuggage function. Compute the value of the objective function.

Throws:
java.lang.Exception

objFun

private double objFun(int i1,
                      int i2,
                      double alpha1,
                      double alpha1_,
                      double alpha2,
                      double alpha2_)
               throws java.lang.Exception
Debuggage function. Compute the value of the objective function.

Throws:
java.lang.Exception

checkSets

private void checkSets()
                throws java.lang.Exception
Debuggage function. Check that the set I0, I1, I2 and I3 cover the whole set of index and that no attribute appears in two different sets.

Throws:
java.lang.Exception

checkAlphas

private void checkAlphas()
                  throws java.lang.Exception
Debuggage function Checks that : alpha*alpha_=0 sum(alpha[i] - alpha_[i]) = 0

Throws:
java.lang.Exception

displayStat

private void displayStat(int i1,
                         int i2)
                  throws java.lang.Exception
Debuggage function. Display the current status of the program.

Parameters:
i1 - the first current indice
i2 - the second current indice
Throws:
java.lang.Exception

displayB

private void displayB()
               throws java.lang.Exception
Debuggage function Compute and display bLow, lUp and so on...

Throws:
java.lang.Exception

checkOptimality

private void checkOptimality()
                      throws java.lang.Exception
Debuggage function. Checks if the equations (6), (8a), (8b), (8c), (8d) hold. (Refers to "Improvements to SMO Algorithm for SVM Regression".) Prints warnings for each equation which doesn't hold.

Throws:
java.lang.Exception