mse loss in pytorch geometric gives nan for loss function

Question

I am doing a regression problem using GCN with pytorch geometric. And I am getting nan loss while using mse loss. However, output tensor is not nan. Here is my model-

import torch
import torch.nn.functional as F
from torch_geometric.nn import GCNConv


class GCN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = GCNConv(data.num_node_features, 100)
        self.conv2 = GCNConv(100, 16)
        self.conv3 = GCNConv(16, data.num_node_features)
        self.linear1 = torch.nn.Linear(104,1)

    def forward(self, data):
        x, edge_index = data.x, data.edge_index

        h = self.conv1(x, edge_index)
        h = F.relu(h)
        h = F.dropout(h, training=self.training)
        h = self.conv2(h, edge_index)
        h = self.conv3(h, edge_index)
        h = self.linear1(h)
        h = h.tanh()
        return h

Here is the loop for calling the model and calculate the loss.

import torch.nn as nn
device = torch.device('cpu')
model = GCN().to(device)
model = model.double()
data = data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
model.train()

for epoch in range(3):
    optimizer.zero_grad()
    out = model(data)
    loss = F.mse_loss(out.squeeze(), data.y.squeeze())
    loss.backward()
    optimizer.step()
    print(f'Epoch: {epoch}, Loss: {loss}')

I also tried not using tanh and using softmax in the forward pass. out tensor is not null. I printed and checked it also the length of both data.y and out is same.

As I am a novice in this GCN and pytorch geometric, I am unable to solve this problem.

Answer 1

getting nan in loss can be happened for one of following reasons-

1. There is nan data in the dataset.
2. Using relu function sometimes gives nan output. (Use leaky-relu instead)
3. Sometimes zero into square_root from torch gives nan output.
4. Using wrong loss. (eg. classification loss in regression problem)