lwip to send data bigger than 64kb

Question

i'm trying to send over lwip a RT data (4 bytes) sampled at 100kHz for 10 channels. I've understood that lwip has a timer which loops every 250ms and it cannot be changed. In this case i'm saving the RT over RAM at 100kHz and every 250ms sending the sampled data over TCP.

The problem is that i cannot go over 65535 bytes every 250ms because i get the MEM_ERR. i already increased the buffer up to 65535 but when i try to increase it more i get several error during compiling.

So my doubt is: can lwip manage buffer bigger than 16bits?

Thanks, Marco

Answer 1

Better to focus on throughput.

You neglected to show any code, describe which Xilinx board/system you're using, or which OS you're using (e.g. FreeRTOS, linux, etc.).

Your RT data is: 4 bytes * 10 channels * 100kHz --> 400,000 bytes / second.

From your lwip description, you have 65536 byte packets * 4 packets/sec --> 256,000 bytes / second.

This is too slow. And, it much slower than what a typical TCP / Ethernet link can process, so I think your understanding of the maximum rate is off a bit.

You probably can't increase the packet size.

But, you probably can increase the number of packets sent.

You say that the 250ms interval for lwip can't be changed. I believe that it can.

From: https://www.xilinx.com/support/documentation/application_notes/xapp1026.pdf we have the section: Creating an lwIP Application Using the RAW API:

4. Set up a timer to interrupt at a constant interval. Usually, the interval is around 250 ms. In the timer interrupt, update necessary flags to invoke the lwIP TCP APIs tcp_fasttmr and tcp_slowtmr from the main application loop explained previously

The "usually" seems to me to imply that it's a default and not a maximum.

But, you may not need to increase the timer rate as I don't think it dictates the packet rate, just the servicing/completion rate [in software].

A few guesses ...

Once a packet is queued to the NIC, normally, other packets may be queued asynchronously. Modern NIC hardware often has a hardware queue. That is, the NIC H/W supports multiple pending requests. It can service those at line speed without CPU intervention.

The 250ms may just be a timer interrupt that retires packet descriptors of packets completed by the NIC hardware.

That is, more than one packet can be processed/completed per interrupt. If that were not the case, then only 4 packets / second could be sent and that would be ridiculously low.

Generating an interrupt from the NIC for each packet incurs an overhead. My guess is that interrupts from the NIC are disabled. And, the NIC is run in a "polled" mode. The polling occurs in the timer ISR.

The timer interrupt will occur 4 times per second. But, will process any packets it sees that are completed. So, the ISR overhead is only 4 interrupts / second.

This increases throughput because the ISR overhead is reduced.

---

UPDATE:

Thanks for the reply, indeed is 4 bytes * 10 channels * 100kHz --> 4,000,000 bytes / second but I agree that we are quite far from the 100Mbit/s.

Caveat: I don't know lwip, so most of what I'm saying is based on my experience with other network stacks (e.g. linux), but it appears that lwip should be similar.

Of course, lwip will provide a way to achieve full line speed.

Regarding the changing of the 250ms timer period, to achieve what i want it should be lowered more than 10times which seems it is too much and it can compromise the stability of the protocol.

When you say that, did you actually try that?

And, again, you didn't post your code or describe your target system, so it's difficult to provide help.

Issues could be because of the capabilities [or lack thereof] of your target system and its NIC.

Or, because of the way your code is structured, you're not making use of the features that can make it fast.

So basically your suggestion is to enable the interrupt on each message? In this case i can send the remaining data in the ACK callback if I understood correctly. – Marco Martines

No.

The mode for interrupt on every packet is useful for low data rates where the use of the link is sparse/sporadic.

If you have an interrupt for every packet, the overhead of entering/exiting the ISR (the ISR prolog/epilog) code will become significant [and possibly untenable] at higher data rates.

That's why the timer based callback is there. To accumulate finished request blocks and [quickly] loop on the queue/chain and complete/reclaim them periodically. If you wish to understand the concept, look at NAPI: https://en.wikipedia.org/wiki/New_API

Loosely, on most systems, when you do a send, a request block is created with all info related to the given buffer. This block is then queued to the transmit queue. If the transmitter is idle, it is started. For subsequent blocks, the block is appended to the queue. The driver [or NIC card] will, after completing a previous request, immediately start a new/fresh one from the front of the queue.

This allows you to queue multiple/many blocks quickly [and return immediately]. They will be sent in order at line speed.

What actually happens depends on system H/W, NIC controller, and OS and what lwip modes you use.