I am trying to implement a loss function as the one described in 1. My problem is that when I add the L1 regularization term, I encounter the following error message :
RuntimeError: Function 'SqrtBackward0' returned nan values in its 0th output.
I have tried different codes to calculate L1 norm of matrix L :
(torch.sign(L[:, idx[0], idx[1]])*L[:, idx[0], idx[1]]).sum(dim=1).mean()
tmp[:, idx[0], idx[1]].abs().sum(dim = 1).mean()
torch.linalg.norm(tmp[:, idx[0], idx[1]], dim = 1).mean()
torch.linalg.vector_norm(tmp[:, idx[0], idx[1]], dim = 1, ord = 1).mean()
I have coded the rest of the loss function as follows : - net.likelihood.log_prob(batch).mean() + 0.001 * net.kld_() + 0.01 * (F.mse_loss(batch, mu) ** 2).mean()
This part of the loss doesn't have any problem, the training does fine with it.
However, I keep on having the mistake quoted above when I add the L1 norm. I believe this originates from the abs() function, which can also be formulated as sqrt(x²). There is a problem during the loss.backward() with the derivative of this function.
I have researched on stackoverflow and seen numerous encounter of this problem, but none with the abs() function. I wanted to know if anyone would have some piece of information about this error ?
Thanks in advance !