How can I tell h2o deep learning grid to have AUC instead of residual deviance

Question

I would like to measure models performance by looking for AUC or Accuracy. In the grid search I get results with residual deviance,how can I tell h2o deep learning grid to have AUC instead of residual deviance and present the results as atable like the one attached below ?

train <- read.table(text = "target birds    wolfs     snakes
                              0        9         7 a
                              0        8         4 b
                              1        2         8 c
                              1        2         3 a
                              1        8         3 a
                              0        1         2 a
                              0        7         1 b
                              0        1         5 c
                              1        9         7 c
                              1        8         7 c
                              0        2         7 b
                              1        2         3 b
                              1        6         3 c
                              0        1         1 a
                              0        3         9 a
                              1        1         1 b ",header = TRUE)
trainHex <- as.h2o(train)

g <- h2o.grid("deeplearning",
              hyper_params = list(
                  seed = c(123456789,12345678,1234567),
                  activation = c("Rectifier", "Tanh", "TanhWithDropout", "RectifierWithDropout", "Maxout", "MaxoutWithDropout")
              ),
              reproducible = TRUE,
              x = 2:4,
              y = 1,
              training_frame = trainHex,
              validation_frame = trainHex,
              epochs = 50,
              )
g
model_ids <- g@summary_table
model_ids<-as.data.frame(model_ids)

The results table that I got:

     Hyper-Parameter Search Summary: ordered by increasing residual_deviance
             activation      seed                                                  model_ids   residual_deviance
1                Maxout  12345678 Grid_DeepLearning_train_model_R_1483217086840_112_model_10 0.07243775676256235
2                Maxout   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_16 0.10060885040861599
3     MaxoutWithDropout 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_5  0.1706496158406441
4                Maxout 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_4 0.17243125875659948
5                  Tanh 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_1 0.18326527198894926
6                  Tanh  12345678  Grid_DeepLearning_train_model_R_1483217086840_112_model_7 0.18763395264761593
7                  Tanh   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_13 0.18791531211136187
8       TanhWithDropout 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_2 0.19808063817007837
9       TanhWithDropout  12345678  Grid_DeepLearning_train_model_R_1483217086840_112_model_8 0.19815190962052193
10      TanhWithDropout   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_14 0.19832946889767458
11            Rectifier 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_0 0.20679125165086842
12    MaxoutWithDropout   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_17 0.21971759565380736
13 RectifierWithDropout 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_3 0.22337599298253263
14    MaxoutWithDropout  12345678 Grid_DeepLearning_train_model_R_1483217086840_112_model_11 0.22440661112729862
15 RectifierWithDropout   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_15  0.2284671685474275
16 RectifierWithDropout  12345678  Grid_DeepLearning_train_model_R_1483217086840_112_model_9 0.23163744415703522
17            Rectifier   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_12  0.2516917276707789
18            Rectifier  12345678  Grid_DeepLearning_train_model_R_1483217086840_112_model_6  0.2642221616447725

Answer 1

You can do this with h2o.getGrid(). Following on from your example code:

g_rmse <- h2o.getGrid(g@grid_id, "rmse")
g_rmse  #Output it

I've chosen root-MSE there. AUC is not available for your sample data: it has to be a binomial classification, and you are doing a regression.

The reason you are doing a regression is your y contains 0 and 1, so H2O has guessed it is numeric. You need to use as.factor() on that column, just after uploading it into H2O.

train <-  ...
trainHex <- as.h2o(train)
trainHex[,1] = as.factor(trainHex[,1])  #Add this

g <- ...

Then you can do this:

g_auc <- h2o.getGrid(g@grid_id, "auc", decreasing = TRUE)
g_auc

I've set it to decreasing=TRUE so that the best AUC is at the top.