R Logistic regression on ffdf objects

Question

I have built a logistic regression model using the glm function from the stats package. I now would like to predict the outcome of this model on a large number of values, stored in a "ffdf" object (see ff package), however I do not find how to proceed:

1. How can I create a subset of my ffdf object, in order to keep only the variables (i.e. columns) to be used in my prediction? - needed to specify as an input in the predict function

2. How should I proceed next? Which function should be used between predict(), predict.glm(), predict.bigglm() (Maybe biglm package is helpful)?

Thank you in advance for your views on this!

Best regards

UPDATE

Thank you for your feedback BondedDust.
Let me be more precise, it is indeed a coding question, aiming at performing logistic regression based on an ffdf object (learning dataset), and predict the outcome of the model for another ffdf object (test dataset).

(1/3) Learning data set: ffdf object (created with ff package).

` class(train.random.sample)` >   
[1] "ffdf"

below is the structure of the ffdf object in case of needs:

`str(train.random.sample) ` >

List of 3   
 $ virtual: 'data.frame':   27 obs. of  7 variables:   
 .. $ VirtualVmode     : chr  "integer" "integer" "integer" "integer" ...   
 .. $ AsIs             : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...   
 .. $ VirtualIsMatrix  : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...   
 .. $ PhysicalIsMatrix : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...   
 .. $ PhysicalElementNo: int  1 2 3 4 5 6 7 8 9 10 ...   
 .. $ PhysicalFirstCol : int  1 1 1 1 1 1 1 1 1 1 ...   
 .. $ PhysicalLastCol  : int  1 1 1 1 1 1 1 1 1 1 ...   
 .. - attr(*, "Dim")= int  500000 27   
 .. - attr(*, "Dimorder")= int  1 2   
 $ physical: List of 27   
 .. $ id                : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ click             : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ hour              : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ C1                : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ banner_pos        : list()   
 ..  ..- attr(*, "physical")=Class 'ff_pointer' <externalptr>    
 ..  .. ..- attr(*, "vmode")= chr "integer"   
 ..  .. ..- attr(*, "maxlength")= int 500000   
 ..  .. ..- attr(*, "pattern")= chr "ffdf"   
 ..  .. ..- attr(*, "filename")= chr "anonymized.ff"   
 ..  .. ..- attr(*, "pagesize")= int 65536   
 ..  .. ..- attr(*, "finalizer")= chr "delete"   
 ..  .. ..- attr(*, "finonexit")= logi TRUE   
 ..  .. ..- attr(*, "readonly")= logi FALSE   
 ..  .. ..- attr(*, "caching")= chr "mmnoflush"   
 ..  ..- attr(*, "virtual")= list()   
 ..  .. ..- attr(*, "Length")= int 500000   
 ..  .. ..- attr(*, "Symmetric")= logi FALSE    
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ site_id           : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ site_domain       : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ site_category     : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ app_id            : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ app_domain        : list()   
…  
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ app_category      : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ device_id         : list()   
 …   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ device_ip         : list()   
….   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ device_os         : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ device_make       : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ device_model      : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ device_type       : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ device_conn_type  : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ device_geo_country: list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
 .. $ C17               : list()   
…   
 .. .. - attr(*, "class") =  chr [1:2] "ff_vector" "ff"   
$ row.names:  NULL   
- attributes: List of 2   
 .. $ names: chr [1:3] "virtual" "physical" "row.names"   
 .. $ class: chr "ffdf"   

(2/3) Logistic regression based on learning dataset:

Objective is to learn/ predict ‘click’ outcome based on ‘baser_pos’ input

`logreg1 <- glm(click ~ banner_pos, data = train.random.sample, family = "binomial")   
summary(logreg1)` >   


Call:
glm(formula = click ~ banner_pos, family = "binomial", data = train.random.sample)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.0555  -0.6495  -0.5951  -0.5951   1.9071  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -1.641416   0.004702 -349.12   <2e-16 xxx
banner_pos   0.192534   0.007595   25.35   <2e-16 xxx
---
Signif. codes:  0 ‘xxx’ 0.001 ‘xx’ 0.01 ‘x’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 458848  on 499999  degrees of freedom
Residual deviance: 458215  on 499998  degrees of freedom
AIC: 458219

Number of Fisher Scoring iterations: 4

`class(logreg1)`>
[1] "glm" "lm" 

(3/3)Test dataset: ffdf object (created with ff package).

`class(df.test)` >   
[1] "ffdf"

Test dataset structure is identical to training dataset, with ~4.8m rows

`str(df.test)`>   

List of 3   
 $ virtual: 'data.frame':   26 obs. of  7 variables:   
 .. $ VirtualVmode     : chr  "integer" "integer" "integer" "integer" ...   
.. $ AsIs             : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...   
.. $ VirtualIsMatrix  : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...   
.. $ PhysicalIsMatrix : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...   
.. $ PhysicalElementNo: int  1 2 3 4 5 6 7 8 9 10 ...   
.. $ PhysicalFirstCol : int  1 1 1 1 1 1 1 1 1 1 ...   
.. $ PhysicalLastCol  : int  1 1 1 1 1 1 1 1 1 1 ...   
.. - attr(*, "Dim")= int  4769401 26   
.. - attr(*, "Dimorder")= int  1 2   
$ physical: List of 26   
…   

I could not succeed in predicting click outcome. I first tried to create a dataframe or ffdf object containing the banner_pos variable:

`modeldata <- df.test[["banner_pos"]]`

Then I tried to predict the outcome:

`predict.glm(object = logreg1, newdata = modeldata, type = "response")`

Error in as.data.frame.default(data) : 
  cannot coerce class "c("ff_vector", "ff")" to a data.frame

Is there something wrong in my code? Should I use other function leveraging other packages such as biglm?
Many thanks in advance for your views on that issue,
Best regards

Answer 1

Something similar as this will score your ffdf alongside your glm.

require(ff)
df.test$score <- ff(as.numeric(NA), length = nrow(df.test))
chunks <- chunk(df.test)
for(chunkrangeindex in chunks){
  df.test$score[chunkrangeindex] <- predict(object = logreg1, newdata = df.test[chunkrangeindex, ], type = "response")
}