How to read variable length data from an asynchronous tcp socket?

Question

I'm using CocoaAsyncSocket for an iOS project. I'm trying to read VarInts through an asynchronous interface. The problem is unlike something else like a String, where I can prefix a length, I don't know the length of a varint beforehand. It needs to be processed one byte at a time, but since each read operation is asynchronous other read calls may have been queued in between.

I considered reading into a buffer then processing it, say reading 5 bytes (the max length for a varint-32), and pushing extra bytes back, but that may hang unnecessarily if the varint is only 4 bytes and I'm waiting for a 5th byte to be available.

How can I do this? Also, I cannot change the protocol on the other end, to use fixed size ints.

---

Here's a snippet of code as Josh requested

- (void)readByte:(void (^)(int8_t))onComplete {
    NSUInteger size = 1;
    int32_t tag = OSAtomicAdd32(1, &_nextTag);
    dispatch_async(self.dispatchQueue, ^{
        [self.onCompleteHandlers setObject:(^void (NSData* data) {
            int8_t x = 0;
            [data getBytes:&x length:size];
            onComplete(x);
        }) forKey:[NSNumber numberWithInteger:((NSInteger) tag)]];
        [self.socket readDataToLength:size withTimeout:-1 tag:tag];
    });
}

A callback is saved in a dictionary, which is used in the delegate method socket: didReadData: withTag.

Suppose I'm reading a VarInt byte by byte:

• execute read first byte for varint
• don't know if we need to read another byte for a varint or not; that depends on the result of the first read
• (possible) read another byte for something else
• read second byte for varint, but now it's actually the 3rd byte being read

I can imagine using a flag to indicate whether or not I'm in a multipart-read, and a queue to hold reads that should be executed after the multipart-read, and I've started writing it but it's quite messy. Just wondering if there is a standard/recommended/better way to approach this problem.

Answer 1

in short there are 4 ways to know how much to read from a socket...

1. read some format that you can infer the length from like the Content-Length header... only works if the whole request can be put together before the body is sent.
2. read until some pattern: like \r\n\r\n at the end of the headers
3. read until some timeout... after you get no bytes after n seconds you flush the buffers and close the connection.
4. read until the server closes the connection... actually used to be pretty common.

these each have problems and I would probably lean in your case from using some existing protocol.

of course there is overhead to doing it that way, and you may find that you don't want to use any of that application level stuff and your requests may be like:

client>"doMath(2+5)\0"

server>"(7)\0"

but it is hard to answer your general question specifically.

edit:

So I looked into the varint base-128 issue a little more and I think really only a timeout or the server closing the connection will work, if you are writing these right at the TCP level which is horrible...