Dagster: Preventing OOM on Multiprocessor Runs

Roel M. Hogervorst


Categories: blog Tags: dagster 100DaysToOffload fan-out TIL

I had a problem in dagster, some jobs had an out of memory error.

My setup is like this

The processes are not aware of each other and I had a beautiful fan-out pattern and so the container got out of memory.

I first tried to increase memory for peak memory usage on that job, but there is a more elegant solution. You can limit the amount of ops that run concurrently

according to the docs you can annotate a job:

        "execution": {
            "config": {
                "multiprocess": {
                    "max_concurrent": 4}
def an_interesing_job():

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com, post - 32/100

Find other posts tagged #100DaysToOffload here