C++ network sockets, SCTP and packet size

Question

I'm currently developing a server using connection-oriented SCTP to serve a small number of clients. After finishing the first prototype with a naive implementation, I'm now profiling the application to optimize. As it turns out, one of the two main consumers of CPU time is the networking part.

There are two questions about the efficiency of the application-level protocol I have implemented:

1) Packet size

Currently, I use a maximum packet size of 64 bytes. You can find many posts discussing packet sizes that are too big, but can they be too small? As SCTP allows me to read one packet at a time - similarly to UPD - while guaranteeing in-order delivery - similarly to TCP - this simplified implementation significantly. However, if I understand correctly, this will cost one syscall for each and every time that I send a packet. Does the amount of syscalls have a significant impact on performance? Would I be able to shave off a lot of CPU cycles by sending the messages in bunches in bigger packets, i.e. 1024 - 8192 bytes?

2) Reading and writing the buffers

I'm currently using memcpy to move data into and out of the application-level network buffers. I found many conflicting posts about what is more efficient, memcpy or normal assignment. I'm wondering if one approach will be significantly faster than the other in this scenario:

Option 1

void Network::ReceivePacket(char* packet)
{
    uint8_t param1;
    uint16_t param2
    uint32_t param3;

    memcpy(&param1, packet, 1);
    memcpy(&param2, packet+1, 2);
    memcpy(&param3, packet+3, 4);

    // Handle the packet here
}

void Network::SendPacket(uint8_t param1, uint16_t param2, uint32_t param3)
{
    char packet[7]

    memcpy(&packet, &param1, 1);
    memcpy(&packet+1, &param2, 2);
    memcpy(&packet+3, &param3, 4);

    // Send the packet here
}

Option 2

void Network::ReceivePacket(char* packet)
{
    uint8_t param1;
    uint16_t param2
    uint32_t param3;

    param1 = *((uint8_t*)packet);
    param2 = *((uint16_t*)packet+1);
    param3 = *((uint32_t*)packet+3);

    // Handle the packet here
}

void Network::SendPacket(uint8_t param1, uint16_t param2, uint32_t param3)
{
    char packet[7]

    *((uint8_t*)packet) = param1;
    *((uint16_t*)packet+1) = param2;
    *((uint32_t*)packet+3) = param3;

    // Send the packet here
}

The first one seems a lot cleaner to me, but I've found many posts indicating that maybe the second one is quite a bit faster.

Any kind of feedback is of course welcome.

Answer 1

As far as I know compilers optimize memcpy calls in particular so you should probably use it.

About your first question:

• Summary: Make your packet size as big as you can and avoid the possibility of having an enlowered CPU performance.

A syscall, a system call, is your OS replying or processing your request and every time your request is being executed in kernel, which is a moderate amount of work. To be honest I am not familiar with the SCTP concept, as a matter of fact I haven't dealt with socket programming since the last time I worked on some stuff and created a server via TCP. I remember the MTU for the relevant physical layer element was 1500, I also recall implementing my packet size as 1450-1460, as I was trying to get the maximum packet size underneath the 1500 cap.

So what Im saying is If I were you I would want my OS to be less active as it could so I wont get any trouble with CPU performance.

Answer 2

If you're looking to minimize the number of system calls, and you do find yourself sending and receiving multiple messages at a time, you might want to look at using the (Linux only) sendmmsg() and recvmmsg().

To use these, you would likely need to enqueue messages internally, which might add latency that would not exist otherwise.

Answer 3

I wouldn't go over 1024 for a buffer size personally. I've experienced some runtime problems when using packets over 1500, but 1024 is of course 4 to the 5th power making it wonderful to use. It is possible, but I wouldn't advise it. I would make a separate thread for receiving packets, using recvmsg() so you can use multiple streams. I've found this to work wonderfully. The main point of SCTP is multiple streams, so I would take full advantage of that. You just have to ensure that the data is put back in the correct order once everything is received (which takes some work to do).