Getting socket exception on 127.0.0.1 (error code 10049) C#

Question

    {
        ipAddress = IPAddress.Parse("127.0.0.1");
        remoteEndPoint = new IPEndPoint(ipAddress, portAddress);

        socket = new Socket(ipAddress.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
        socket.Connect(remoteEndPoint);
        OnConnected();
    }

The Exception error is thrown on the line "socket.Connect(...);

SocketException: The requested address is not valid in its context." NetworkManager.StartClient () (at Assets/Scripts/Networking/NetworkManager.cs:51)

(Its a unity project, the line is socket.Connect) Is there something I'm missing? I'm sure I've had this working before in different projects. I tried binding the socket first, but I was getting the exact same error and error code.