Huffman's algorithm with stream in Node JS

Question

I have implemented Huffman's algorithm in Node JS and it looks like this:

huffman.encode(inputFilename, outputFilename)
huffman.decode(inputFilename, outputFilename)

But I want to implement it like this:

inputStream.pipe(HuffmanEncoderStream).pipe(outputStream)
outputStream.pipe(HuffmanDecoderStream).pipe(inputStream)

And the problem is I need to read content of the source file twice. Firstly to create table of frequencies and Huffman's tree and secondary to exactly encode content. So is it possible to implement this task with Transform Stream?

P.S. with decoding there no problems

Answer 1

Huffman's algorithm requires that you have all the data first in order to compute frequencies. However, nothing is stopping you from applying Huffman's algorithm to chunks of your data, which would allow streaming. If the chunks are large enough (100's of K to MBs), the overhead of transmitting a code description will be very small in comparison. If the data is homogenous, you will get about the same compression. If the data is not homogenous, your compression might even improve, since the Huffman codes would be optimized to the statistics of the data local to that chunk.