Chained CostSensitiveClassifier

Question

By mistake, I was using a CostSensitiveAnalysis inside other, each with different weight matrices. One was

(matrix 1a)
0 3
1 0

The other was

(matrix 1b)
0 1
3 0

And was getting really good results. When I found out the mistake, I changed my weight matrix to:

(matrix 2)
0 1
1 0

But couldn't get the same results as before. I also tried

(matrix 3)
0 3
3 0

I thought that, by using the cost sensitive analysis over other cost sensitive analysis, with matrices 1a and 1b, I would get the same results as matrix 2 or even 3, but results are very different.

Do the cost sensitive analysis change the cost value in other way than using the specified weights?

Thanks

--

I wrote a test unit for illustration, it should work on any dataset with the class attribute as the last, and with two classes.

import static org.junit.Assert.*;
import java.io.IOException;
import org.apache.log4j.Logger;
import org.apache.log4j.lf5.util.Resource;
import org.junit.Test;

import weka.classifiers.Classifier;
import weka.classifiers.Evaluation;
import weka.classifiers.meta.CostSensitiveClassifier;
import weka.classifiers.meta.FilteredClassifier;
import weka.classifiers.trees.J48;
import weka.core.Instances;

public class WekaFacadeTest {
private Logger logger = Logger.getLogger(WekaFacadeTest.class);

private CostMatrix createCostMatrix(double weightFalsePositive, double weightFalseNegative) {
    CostMatrix costMatrix = new CostMatrix(2);
    costMatrix.setCell(0, 0, 0.0);
    costMatrix.setCell(1, 0, weightFalsePositive);
    costMatrix.setCell(0, 1, weightFalseNegative);
    costMatrix.setCell(1, 1, 0.0);
    return costMatrix;
}   

@Test
public void testDoubleCost() throws Exception {
    Instances data = WekaFacade.loadArff("test.arff");
    data.setClassIndex(data.numAttributes()-1); 

    // c1 => cost sensitive classifier applied to j48, cost = 1.0 
    CostSensitiveClassifier c1 = new CostSensitiveClassifier();
    c1.setClassifier(new J48());

    c1.setCostMatrix( createCostMatrix(1.0, 1.0));
    c1.buildClassifier(data);

    Evaluation ec1 = new Evaluation(data,c1.getCostMatrix());
    ec1.evaluateModel(c1, data);

    // c2 => no cost sensitive classifier, straight j48 
    J48 c2 = new J48();

    c2.buildClassifier(data);

    Evaluation ec2 = new Evaluation(data); 
    ec2.evaluateModel(c2, data);

    // should c1 errorRate be equal to c2?
    logger.info(String.format("Cost ec1=%f, ec2=%f",ec1.errorRate(),ec2.errorRate()));
    assertEquals(ec1.errorRate(),ec2.errorRate(),0.0001);
    // success!     

    // c3 => cost sensitive classifier applied to cost sensitive classifier  applied to j48, cost = 1.0 
    CostSensitiveClassifier c3 = new CostSensitiveClassifier();
    c3.setClassifier(new CostSensitiveClassifier());
    ((CostSensitiveClassifier)c3.getClassifier()).setClassifier(new J48());

    c3.setCostMatrix( WekaFacade.createCostMatrix(1.0, 1.0));
    ((CostSensitiveClassifier)c3.getClassifier()).setCostMatrix( WekaFacade.createCostMatrix(1.0, 1.0));
    c3.buildClassifier(data);

    Evaluation ec3 = new Evaluation(data,c1.getCostMatrix());
    ec3.evaluateModel(c3, data);

    logger.info(String.format("Cost c3=%f, c1=%f",ec3.avgCost(),ec1.avgCost()));
    assertEquals(ec3.avgCost(),ec1.avgCost(),0.0001);
    // fail!        

    logger.info(String.format("ErrorRate c3=%f, c2=%f",ec3.errorRate(),ec2.errorRate()));
    assertEquals(ec3.errorRate(),ec2.errorRate(),0.0001);
    // fail!    

    // d => cost sensitive classifier applied to j48, normal situation 
    CostSensitiveClassifier d = new CostSensitiveClassifier();
    d.setClassifier(new J48());

    d.setCostMatrix( createCostMatrix(3.0, 1.0));
    d.buildClassifier(data);

    Evaluation ed = new Evaluation(data,d.getCostMatrix());
    ed.evaluateModel(d, data);

    // c => cost sensitive classifier applied to another cost sensitive classifier, abnormal situation
    CostSensitiveClassifier c = new CostSensitiveClassifier();
    c.setClassifier(new CostSensitiveClassifier());
    ((CostSensitiveClassifier)c.getClassifier()).setClassifier(new J48());

    c.setCostMatrix( createCostMatrix(1.0, 1.0));
    ((CostSensitiveClassifier)c.getClassifier()).setCostMatrix( createCostMatrix(3.0, 1.0));
    c.buildClassifier(data);

    Evaluation ec = new Evaluation(data, c.getCostMatrix());
    ec.evaluateModel(c, data );

    // should ec average cost be the same as ed's ?
    logger.info(String.format("Cost c=%f, d=%f",ec.avgCost(),ed.avgCost()));
    assertEquals(ec.avgCost(),ed.avgCost(),0.0001);
    // fails!

}
}

Answer 1

What I have found until now

CostSensitiveClassifier has two operation modes: it may either set explicit weights on samples (by using the .weight() method) or it may resample with substitution. In my particular case, it was using the last approach.

Therefore, the class arrangement described above would resample twice the original sample. Being resample a random procedure, the results shouldn't be equal to a single resampling.