CTC_Loss for Model training in Tensorflow2.0

Question

I am working on a project where in we have to use ctc_batch_cost for Loss. I defined the function that returns the CTC loss and tried to use it in 'model.compile' statement. But here, I could not figure out how to get 'y_pred'. Can someone help me in fixing this 'model.compile' statement please?

CTC loss function

    def ctc_loss_func(args):
       y_pred, labels, input_length, label_length = args
       return K.ctc_batch_cost(labels, y_pred, input_length, label_length)

As per tensorflow documentation here, we need to provide ['y_true','y_pred','input_length','label_length'].My dataframe has 10000 datapoints and my model has an output layer as model.add(Dense(78, activation='softmax'))

So I created a list of 78s with 10000 elements as input_length input_length = [78]*10000

I got the length of original words into label_length as below:

    label_length = []
    for item in y.iteritems():
       tex = item[1]
       l = len(tex)
       label_length.append(l)

I had encoded each word in the sample into a vector of 78 characters and created an array of size (10000,78). I am passing this as y_true

But how do I get the 'y_pred' before compiling the model? should I first compile and train the model with some other loss function like 'categorical_cross_entropy' to get y_pred ? If yes, Does that mean I have to compile and train my model twice. First with 'categorical_cross_entropy' and then with 'ctc_loss'

Compiling model

    model.compile(loss=ctc_loss_func(y_true,y_pred,input_length,label_length), optimizer='adam', metrics=['acc'])

Answer 1

Loss function should accept only y_true and y_pred. For example:

def foo(y_true, y_pred):
    loss = abs(y_true - y_pred) # or other logic
    return loss

So you generally can't pass four values to loss function. You have several options how to solve this issue.

1. You may write a wrapper for ctc_batch_cost if you don't care about lengths, or they are constant and hardcode them as constants or take them as dimensions of tensor:

   def ctc_loss(y_true, y_pred):
       batch_size = tf.shape(y_true)[0]
       input_length = tf.shape(y_pred)[1]
       label_length = tf.shape(y_true)[1]
   
       input_length = input_length * tf.ones(shape=(batch_size, 1), dtype="int64")
       label_length = label_length * tf.ones(shape=(batch_size, 1), dtype="int64")
       loss = K.ctc_batch_cost(y_true,y_pred,input_length,label_length)
   
       return loss
   # Now you can compile model
   model.compile(loss=ctc_loss, optimizer='adam', metrics=['acc'])
   

2. You may concatenate information about input_length to your y_true and extract this in ctc_loss function:

   def ctc_loss(y_true, y_pred):
       batch_size = tf.shape(y_true)[0]
   
       input_length = tf.shape(y_pred)[1]
       label_length = y_true[:,-1:]
       y_true = y_true[:,:-1]
       input_length = input_length * tf.ones(shape=(batch_size, 1), dtype="int64")
       label_length = label_length * tf.ones(shape=(batch_size, 1), dtype="int64")
       loss = K.ctc_batch_cost(y_true,y_pred,input_length,label_length)
   
       return loss
   

3. You may write custom keras layer put calculations there and add loss by self.addloss(calculated_ctc_loss)