MSG_PROXY not working to provide/specify alternate addresses for transparent proxying

Question

I'm trying to write a transparent proxy that translates arbitrary UDP packets to a custom protocol and back again. I'm trying to use transparent proxying to read the incoming UDP packets that need translation, and to write the outgoing UDP packets that have just been reverse-translated.

My setup for the socket I use for both flavors of UDP sockets is as follows:

static int
setup_clear_sock(uint16_t proxy_port)
{
    struct sockaddr_in saddr;
    int sock;
    int val = 1;
    socklen_t ttllen = sizeof(std_ttl);

    sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP);
    if (sock < 0)
    {
        perror("Failed to create clear proxy socket");
        return -1;
    }
    if (getsockopt(sock, IPPROTO_IP, IP_TTL, &std_ttl, &ttllen) < 0)
    {
        perror("Failed to read IP TTL option on clear proxy socket");
        return -1;
    }
    if (setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val)) < 0)
    {
        perror("Failed to set reuse address option on clear socket");
        return -1;
    }
    if (setsockopt(sock, IPPROTO_IP, IP_TRANSPARENT, &val, sizeof(val)) < 0)
    {
        perror("Failed to set transparent proxy option on clear socket");
        return -1;
    }
    saddr.sin_family = AF_INET;
    saddr.sin_port = htons(proxy_port);
    saddr.sin_addr.s_addr = INADDR_ANY;
    if (bind(sock, (struct sockaddr *) &saddr, sizeof(saddr)) < 0)
    {
        perror("Failed to bind local address to clear proxy socket");
        return -1;
    }

    return sock;
}

I have two distinct, but possibly related problems. First, when I read an incoming UDP packet from this socket, using this code:

struct sock_double_addr_in
{
    __SOCKADDR_COMMON (sin_);
    in_port_t sin_port_a;
    struct in_addr sin_addr_a;
    sa_family_t sin_family_b;
    in_port_t sin_port_b;
    struct in_addr sin_addr_b;
    unsigned char sin_zero[sizeof(struct sockaddr) - __SOCKADDR_COMMON_SIZE - 8
                             - sizeof(struct in_addr) - sizeof(in_port_t)];
};


void
handle_clear_sock(void)
{
    ssize_t rcvlen;
    uint16_t nbo_udp_len, coded_len;
    struct sockaddr_in saddr;
    struct sock_double_addr_in sdaddr;
    bch_coding_context_t ctx;
    socklen_t addrlen = sizeof(sdaddr);

    rcvlen = recvfrom(sock_clear, &clear_buf, sizeof(clear_buf),
                      MSG_DONTWAIT | MSG_PROXY,
                      (struct sockaddr *) &sdaddr, &addrlen);
    if (rcvlen < 0)
    {
        perror("Failed to receive a packet from clear socket");
        return;
    }

    ....

I don't see a destination address come back in sdaddr. The sin_family_b, sin_addr_b, and sin_port_b fields are all zero. I've done a block memory dump of the structure in gdb, and indeed the bytes are coming back zero from the kernel (it's not a bad placement of the field in my structure definition).

Temporarily working around this by hard-coding a fixed IP address and port for testing purposes, I can debug the rest of my proxy application until I get to the point of sending an outgoing UDP packet that has just been reverse-translated. That happens with this code:

    ....

    udp_len = ntohs(clear_buf.u16[2]);
    if (udp_len + 6 > decoded_len)
        fprintf(stderr, "Decoded fewer bytes (%u) than outputting in clear "
                        "(6 + %u)!\n", decoded_len, udp_len);

    sdaddr.sin_family = AF_INET;
    sdaddr.sin_port_a = clear_buf.u16[0];
    sdaddr.sin_addr_a.s_addr = coded_buf.u32[4];
    sdaddr.sin_family_b = AF_INET;
    sdaddr.sin_port_b = clear_buf.u16[1];
    sdaddr.sin_addr_b.s_addr = coded_buf.u32[3];
    if (sendto(sock_clear, &(clear_buf.u16[3]), udp_len, MSG_PROXY,
               (struct sockaddr *) &sdaddr, sizeof(sdaddr)) < 0)
        perror("Failed to send a packet on clear socket");
}

and the packet never shows up. I've checked the entire contents of the sdaddr structure I've built, and all fields look good. The UDP payload data looks good. There's no error coming back from the sendto() syscall -- indeed, it returns zero. And the packet never shows up in wireshark.

So what's going on with my transparent proxying? How do I get this to work? (FWIW: development host is a generic x86_64 ubuntu 14.04 LTS box.) Thanks!

Answer 1

Alright, so I've got half an answer.

It turns out if I just use a RAW IP socket, with the IP_HDRINCL option turned on, and build the outgoing UDP packet in userspace with a full IP header, the kernel will honor the non-local source address and send the packet that way.

I'm now using a third socket, sock_output, for that purpose, and decoded UDP packets are coming out correctly. (Interesting side note: the UDP checksum field must either be zero, or the correct checksum value. Anything else causes the kernel to silently drop the packet, and you'll never see it go out. The kernel won't fill in the proper checksum for you if you zero it out, but if you specify it, it will verify that it's correct. No sending UDP with intentionally bad checksums this way.)

So the first half of the question remains: when I read a packet from sock_clear with the MSG_PROXY flag to recvfrom(), why do I not get the actual destination address in the second half of sdaddr?