Instantiating Flyte task to create workflow at runtime

Question

Is it possible to instantiate a Flyte Task at runtime so that I can create a Workflow with a variable number of Tasks and with each Task running a runtime-determined Python callable? In the documentation, I only see references to compile-time Workflows that are declaratively composed of Python function annotated with the @Task decorator.

If you can provide any existing examples in open source code or a new, small inline example, please do! Thanks!

Answer 1

The functionality I was looking for is provided by the FlyteRemote class. With this class, one can register instantiated entities, i.e. tasks, workflows, and launchplans.

Answer 2

Have you looked at dynamic workflows https://docs.flyte.org/projects/cookbook/en/stable/auto/core/control_flow/dynamics.html. Dynamic in Flyte is like JITing in a language like Java. The new workflow graph is created, compiled, verified and then executed. But the graph is created in response to the inputs and you control the shape / structure at runtime