How to show tqdm progress in MetaFlow?

Question

When running MetaFlow Flows, tqdm progress bars do not get displayed until the final iteration, which defeats the purpose of measuring progress. Is there a way to force MetaFlow to print out tqdm updates?

Answer 1

The problem is that tqdm writes to stderr, but MetaFlow hides output to stderr. The solution requires two tricks: using a context manager to redirect tqdm outputs to a logger, and setting that logger to write to stderr.

Example:

import logging
import sys
from time import sleep

import tqdm
from metaflow import FlowSpec, step
from tqdm.contrib.logging import tqdm_logging_redirect

# stream to stdout needed because MetaFlow hides output to stderr :C
logging.basicConfig(level=logging.INFO, stream=sys.stdout)


class TQDMFlow(FlowSpec):
    @step
    def start(self):
        print("Training...")
        with tqdm_logging_redirect():  # this context manager redirects tqdm output to logging
            for _ in tqdm.tqdm(range(20)):
                sleep(0.25)
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    TQDMFlow()

I also tried redirecting output directly to stdout (without the tqdm_logging_redirect context manager or logging) using tqdm(range(n), file=sys.stdout) but that did not work.

Answer 2

crypdick's answer works for me, but the problem is that I'm also seeing info logging from other unrelated modules, which can be very noisy. Instead you can set up logging only for tqdm. I've also written a helper function you can use that will wrap up all the logic into one function:

import logging
from tqdm import tqdm
from tqdm.contrib.logging import tqdm_logging_redirect


def show_progress():
    tqdm_logger = logging.getLogger('tqdm')
    tqdm_logger.setLevel(logging.INFO)
    tqdm.pandas()
    return tqdm_logging_redirect()

Usage:

with show_progress():
     df.progress_apply(...)