Which tmpsf partition is used in singularity and how can I increase it

Question

I'm wondering how can I increase tmpfs size in singularity sif image and how it relates to tmpfs on host system. According to a post by Pawsey Centre:

Singularity has a flag for rendering containers from SIF image files ephemerally writable. --writable-tmpfs will allocate a small amount of RAM for this purpose (configured by the sys admins, by default just a bunch of MB)

On my host system I have the following tmpfs:

$ df -h | grep tmpfs
tmpfs                         13G  2,9M   13G   1% /run
tmpfs                         63G   84M   63G   1% /dev/shm
tmpfs                        5,0M     0  5,0M   0% /run/lock
tmpfs                         63G     0   63G   0% /sys/fs/cgroup

While inside container I have:

overlay                                 16M   12K   16M   1% /
...
tmpfs                                   63G   84M   63G   1% /dev/shm
tmpfs                                   16M   12K   16M   1% /.singularity.d/libs
tmpfs                                   13G  2.9M   13G   1% /run/nvidia-persistenced/socket

I can only write small files in my container (couple of KB, otherwise "no space error" is thrown) Which tmpfs does singularity use and why? How can I increase it?

Answer 1

The size of the tmpfs partition is set by the admin using the sessiondir max size config entry. It defaults to 16MB, but you can check it via sudo singularity config global --get "sessiondir max size" or check the config file directly if you don't have sufficient permissions: grep sessiondir /usr/local/etc/singularity/singularity.conf.

You can change the config value (or ask your admins to do it) to increase it to the desired size. If that's not an option, you'll need to make sure the host filesystem is mounted at locations where the data is being written. This is likely a good idea anyway if you'll be writing a lot of data. Docker also does this behind the scenes when using volumes.