Error Correction with Python and Reed Solomon for large inputs

Question

I am currently implementing a messaging system. I want to send an error protected message to a receiver, but I am failing at the basics, i.e. calculating the error correcting codes. I use the following library for error correction.
Consider the following MWE:

from reedsolo import RSCodec

with open("imageToSend.png", "rb") as pic:
  picContent = pic.read()

correctionLength = int((len(picContent)/100)*20)
rs = RSCodec(correctionLength)

rs.encode(picContent)

As you can see I want to protect the image from 20% errors that might occur. The problem here? The encoded bytearray is empty. And my question: Is it possible to protect large files from errors, without chunking them into smaller pieces and then calculating the error correcting codes?

Answer 1

Is it possible to protect large files from errors, without chunking them into smaller pieces

Depends on the code. With bytewise RS, chunks are necessary (but this lib does the work for you).

As you can see I want to protect the image from 20% errors that might occur. The problem here?

Yes. The number isn't meant to be a percent-like thing in the first place. You should really read the examples of the lib, and get to know a bit how RS works.
The number is how many byte out of 255 should be used for error correcting. Eg. 40 means that for each 215 byte data, there will be 40 byte RS code (about 20%), and in that 255 byte it can correct up to 20 bytes error.

Finally, the LDPC principle might be something you want to look into. A bit worse than RS in correcting errors, but noch much, and it's much faster.

Addition from the comments:
If it can be corrected depends on the locations of the error, yes. If full 255-blocks are gone, it can't correct it. To make the span larger, higher-order RS codes could be used (eg. one independent block could have 65536 byte instead of 255), but a) that's again much slower than the (already slow) 255-RS, and b) The RS libs I know can't do it (yours inclded). You would have to write it yourself.

Again, LDPC could help, if it doesn't bother you that it's a completely different thing. Eg. it has no clear values how many errors are too much to correct/detect, it depends on the error pattern too. And since it's newer than RS, there are less codes/libraries online, maybe none for your case.

((Well, it's old too, but for decades nobody was interested in it, until someone realized that it's useful)).