WSARecv sometimes return "invalid handle (error no 6)" for a socket associated with an IOCP port. (C++)

Question

I'm trying to write a server which can support many clients connections simultaneously so I'm trying to do it with IOCP. So let me brief about my code flow and then I can explain my problem. First of all, server is opening a port for listening and waiting on an "accept" call for new incoming connections. For reference I have used same code as mentioned here So it accepts every new incoming connection and returns a new socket descriptor (sd), and then it marks as nonblocking with:

arg = 1;
ioctlsocket(sd, FIONBIO, &arg);

and then enable TCP_NODELAY:

level = IPPROTO_TCP;
optName = TCP_NODELAY;
value = 1;
setsockopt(sd, level, optName, (const char*)&value, sizeof(value));

thereafter associating with an IOCP port as:

CreateIoCompletionPort((HANDLE)sd, iocp_port, (DWORD)completion_key, 4);

completion_key is a class object which is nothing but a container, it contains data buffer, overlapped-buffer, query-type recv/send etc. and in last issuing a read call:

WSARecv(sd, wsabuf, 1, &bytes, &flags, overlapped, NULL);

wsabuf and overlapped are part of completion_key object.

In 90% cases it works fine i.e. when there is some incoming data available on this socket "GetQueuedCompletionStatus" gets unblocked and it has valid data. But sometimes WSARecv call returns with an error and GetLastError() returns 6 which is "invalid handle" error. I'm bit bewildered why it's happening so.

The way I'm creating an iocp port:

iocp_port = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);

and there are threads which are waiting on "GetQueuedCompletionStatus".

Answer 1

I monitored all system calls which were happening in background. WSARecv internally calls NtDeviceIoControlFile and there is an argument "Event" which is same as what is passed in lpOverlapped structure of WSARecv as hEvent. I wasn't setting hEvent to NULL, so it was taking some garbage value, when it was NULL then NtDeviceIoControlFile returned successfully and for other cases it returned "INVALID_HANDLE" error. Unfortunately, it was NULL most of the time.