How to buffer and write to disk a low latency input with C

Question

I need to write C code (not C# or C++) to get data from hardware analyzer (through 100Mb TCP/IP network) and write it to disk.

Why I say "Low latency", well, Hardware analyser have a 9KB internal buffer, this is 2ms of data storing, after that it runs into buffer overflow and I lose information packets.

My application is able to get this buffer without losing any packet, but I noticed I wasn't able to write data to the disk at this rate.

My code looks like this:

int main (int argc, char * argv [] )
{
  pthread_t th_rx; /* Thread for receiving packets */

  outputfile = fopen ("output.log", "wb");

  link_open(); // open device link

  pthread_create ( &th_rx, NULL, read_packets, 0 );

  // running loop
  fclose (outputfile);
  pthread_exit(NULL);
  link_close(); // close device link 
  return 0; 
}

static thread_result read_packets (void *arg)
{
  char  rxbuf[40960];

  while (receiving)
  {
    bytes_received = read_packet(); //read packet from device buffer
    rxbuf = extract_data(bytes_received);
    fwrite (rxbuf, 1, bytes_received, outputfile); 
  }
  return thread_return;
}

What I need here are ideas to how to do that.

• 1: don't write packet upon received, create a circular? buffer
• 2: Creating 2 thread circular buffer?

Any ideas of how to improve my code and what I can do to get stable writes?

Code examples will be very appreciated because I'm feeling lost :(

Thanks

Answer 1

As you said hw analyzer has 9 kb memory for only 2 ms. Then speed required for writing to HDD is 9*1024*500 = 4608000 b/s or 4.6 Mb/s. I believe that test computer has speed of writing at HDD at least 10 Mb/s. So the only one problem here is to write to HDD with such a big portions of data which can give enough time to do writing. For instance, buffer can be 1 sec or 4.6 Mb. Storing of such portion of data will be faster then collecting of 1 s of data from hw device and "writing thread" takes no more than 500 ms to do its job. In real program this time can differ but in any way it must be enough for single-core systems as well.
Therefore you may use two ping-pong (one is filled with data from hw, one is stored to HDD) buffers with two threads and without circular buffer. Thus you can use producer/consumer model which is well described and has a lot of examples.

---

Answer 2

You should look into asynchronous I/O. However Async-I/O APIs are operating system dependent and you didn't specify which OS you're using. Also your use of threads makes no sense. If you want to use threads (instead of async I/O) then you'd have one thread doing the packet reading and the other thread should to the writing. Putting it both into a single thread gives you no benefit.

Answer 3

As you guessed correctly, the firs thing you should do is, do not write to disk while you are reading from device.

• create circular/ ring buffer (or just list of buffers) to store data read from device. This operation will be fast enough, so that device memory does not fill up.

  • However, if device read is non-blocking you may want o decide to sleep so that your loop do not hog the cpu.

• One thread reads data from device and puts in buffer.

• In another thread, keep writing the buffer to disk.