I've trained a model using PySpark and would like to compare its performance to that of an existing heuristic.
I just want to hardcode an LR model with the coefficients 0.1, 0.5, and 0.7, call .transform
on the test data to get the predictions, and compute the accuracies.
How do I hardcode a model?
Unfortunately it's not possible to just set the coefficients of a pyspark LR model. The pyspark LR model is actually a wrapper around a java ml model (see class
JavaEstimator
).So when the LR model is fit, it transfers the params from the
paramMap
to a new java estimator, which is fit to the data. All theLogisticRegressionModel
methods/attributes are just calls to the java model using the_call_java
method.Since the coefficients aren't params (you can see a comprehensive list using
explainParams
on a LR instance), you can't pass them to the java LR model that's created, and there is not a setter method.For example, for a logistic regression model
lrm
, you can see that the only setters are for the params you can set when you instantiate a pyspark LR instance:lowerBoundsOnCoefficients
andupperBoundsOnCoefficients
.Trying to set the "coefficients" attribute yields this:
So you'd have to roll your own pyspark transformer if you want to be able to provide coefficients. It would probably be easier just to calculate results using the standard logistic function as per @pault's comment.