Debug ideas to investigate on packets getting dropped?

Question

I recently encountered a scenario where in I find that ICMP network unreachable [type-1, code-3- meaning destination unreachable, which is correct w.r.t trace route working, refer rfc link below] packet is getting dropped only when bytes sent exceeds 'X' [certain] bytes.

** Check this RFC, section 3.1 - https://www.rfc-editor.org/rfc/rfc4443#section-3.1 **

For example [source: FreeBSD based trace route code],

i = sendto(sndsock, (char *)outpacket,outpacketlength, 0,
           (struct sockaddr *)&Dst, Dst.sin6_len); // to send packet of length "outpacketlength", from source buffer "outpacket" and to destination "Dst".
...
retval = recvmsg(rcvsock, mhdr, 0); // use recvmsg for receiving reply. 

Question

• When I malloc and send 'X' bytes of data, I get reply received in receive buffer.But, not greater than that limit. i.e, retval is always 0 when "outpacketlen" is greater 'X' bytes, even though you malloc and reset the buffer for 'X' bytes. But, with packet capture I see packet received in my host. Meaning, I receive packet but not in receive buffer. How/When is this possible [Any malformed packet info?]?

• How can I go about debugging this issue?

• Is there a tool that can use to debug such a scenario? I used "truss" utility in BSD. Are there better way of handling this?

• How should I go about investigating the packets lost between interface and application? What tools/utilities/technique would be efficient choice based on your experience?

Am I missing something? Thanks for your time and inputs. I appreciate it.

Answer 1

You need to work with Wireshark and investigate if you are sending your packets correctly. Put a breakpoint just before the sendto and see what happens.

Answer 2

Is the packet you are referring to as "being dropped between interface and application" the ICMP error message for destination unreachable? ICMP errors are usually not delivered to the application that sent the packet triggering the error. I guess the logic there is, that "regular" applications using UDP or TCP cant be assumed to prepare for receiving every kind of ICMP error out of the blue inside their application protocol stream.

You dont mention your specific platform, but I am assuming (out of my hat) FreeBSD. On that platform I think you have to use raw IP sockets, if you want to receive ICMP errors. I could be wrong, or you might be on another BSD variant, so check your IP protocol man-pages for suitable socket options.

In Linux you could receive this packet by setting the IP_RECVERR socket option and then calling recvmsg()with the flag MSG_ERRQUEUE. See e.g. Read ICMP payload from a recvmsg with MSG_ERRQUEUE flag