How can I take the difference of two overlapping byte streams in Python?

Question

I have an audio stream coming in from an API call which generated text-to-speech; I am currently generating overlapping chunks of consecutive texts and want to only write the difference between a stream chunk and the one preceding it (excluding the overlap), how can I do that when what I'm dealing with are byte streams and not actual audio/text streams?

Here's an example:

let's say at iteration 0, I make an API call with the text I am feeling good and that returns an audio stream which is written to a file

now at iteration 1, I make another API call with the text feeling good because I and that returns an audio stream too, but what I want to do is skip the overlapping feeling good and append the remaining part because I from the audio stream to the file. Any suggestions on how to do this when I'm working with byte streams instead of text?

here's what I have so far, which is storing the previous stream and the current one, but how can I take the difference here?

for i in sentences:
   curr_audio_stream = API_call(i) #Returns a botocore.response.StreamingBody object
   with open("file.mp3", "ab") as file:
      file.write(curr_audio_stream.read())
   prev_audio_stream = curr_audio_stream

Here's an example bunch of texts overlapping:

1. Oh, that sounds like
2. a fun game!  Let's do it! So
3. Let's do it! So do you want to
4. do you want to start by giving me
5. start by giving me a clue about the