I have a Dagster deployment running locally - it's small enough that it can be cloned to a PC/Laptop and run to update some assets (using the local IO pickle manager).
However, the dag has some very wide sections where 50+ assets "could" run at once. There is also some narrower but compute heavy sections, where maybe only 3 assets could be materialized at once but each takes all available CPU/Memory.
The issue is that when either of these things run (the 50+ assets or the 3+ heavy assets) the PC/Laptop becomes entirely unresponsive due to resource starvation.
I know I can set concurrency limits to Dagster, but this does not fully solved my issue. Is there a way to limit available CPU/Memory for Dagster?
I tried using the QueuedRunCoordinator to limit the # of simultaneous runs, but this is not solving my issue of too many "light" assets materializing at the same time, or the few "heavy" assets gobbling up resources.
run_coordinator:
module: dagster.core.run_coordinator
class: QueuedRunCoordinator
config:
max_concurrent_runs: 8