How do you set compute/memory limits to runs in Dagster (self-hosted)?

230 views Asked by At

I have a Dagster deployment running locally - it's small enough that it can be cloned to a PC/Laptop and run to update some assets (using the local IO pickle manager).

However, the dag has some very wide sections where 50+ assets "could" run at once. There is also some narrower but compute heavy sections, where maybe only 3 assets could be materialized at once but each takes all available CPU/Memory.

The issue is that when either of these things run (the 50+ assets or the 3+ heavy assets) the PC/Laptop becomes entirely unresponsive due to resource starvation.

I know I can set concurrency limits to Dagster, but this does not fully solved my issue. Is there a way to limit available CPU/Memory for Dagster?

I tried using the QueuedRunCoordinator to limit the # of simultaneous runs, but this is not solving my issue of too many "light" assets materializing at the same time, or the few "heavy" assets gobbling up resources.

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 8
0

There are 0 answers