How to attach a progress bar (e.g. tqdm) to f.write()?

239 views Asked by At
with open("./output/" + filename + ".txt", "w") as f:
    f.write(very_long_string)

I have a very long string that I have to write to file, and I would like to attach a progress bar to the write process. How could I hook the progress bar to the write process?

I've looked at a related question (?) and attempted this, but it doesn't quite work:

class tqdmIOWrapper(io.TextIOBase):
    def __init__(self, file, callback):
        self.file = file
        self.callback = callback

    def write(self, string):
        buf = self.file.write(string)
        if buf:
            self.callback(buf)
        return buf

with open("./output/" + filename + ".js", "w") as f:
    f.write("var " + filename + " = ")
    with tqdm(
        total=len(very_long_string), desc="Printing: " + filename, mininterval=1
    ) as pbar:
        wrapper = tqdmIOWrapper(f, pbar.update)
        wrapper.write(very_long_string)
        pbar.update()

The progress bar only shows after the entire string is written, and it looks like I didn't hook the progress bar to the write process at all. Here is a sample output at the console (process took way more than 1s):

Printing: test:   0%|                                                             | 0/466073757 [00:00<?, ?it/s]466073757
Printing: test: 466073758it [00:01, 396677046.13it/s]

Is there a way to hook a progress bar to a f.write() process?

another related question without answer

Edit: In reply to comment for a working but very slow one-character-by-a-time code (not preferred):

with open("./output/" + filename + ".txt", "w") as f:
    with tqdm(total=len(very_long_string)) as pbar:
        for char in very_long_string:
            f.write(char)
            pbar.update()
1

There are 1 answers

2
Kelly Bundy On

Like your 1-char-at-a-time, but instead a million at a time:

with open("./output/" + filename + ".txt", "w") as f:
    n = len(very_long_string)
    k = 10 ** 6
    with tqdm(total=n) as pbar:
        for i in range(0, n, k):
            chunk = very_long_string[i : i+k]
            f.write(chunk)
            pbar.update(len(chunk))