BeeGFS very slow after update

210 views Asked by At

I am running a cluster with 1 head node, 40 compute nodes, 2 data and 2 meta nodes. All nodes are running Ubuntu 20.04, kernel version 5.4.0-100-generic. They are all connected via 10 Gbit/s interfaces expect for the data and meta nodes, which are connected via 40 Gbit/s.
The file system is run by BeeGFS, meaning the /home of all users is lying on the data and meta nodes and mounted via BeeGFS to any compute and the head node. We recently updated BeeGFS from version 7.21 to 7.25 but kept the whole configuration of all beegfs-services.
However, since the update we are experiencing very low data rates (~ 10 - 20 MB/s) whenever involving BeeGFS in a file operation process like e.g. copying data from a compute node to some sub directory of /home/user. This is not a problem of the physical data servers, as beegfs-ctl --storagebench shows reasonable results. Also, if I copy a large file from some compute node or the head node to the storage node without using BeeGFS, the speed is also around some 100 MB/s, which is reasonable.
I am not sure where even to start looking as the BeeGFS configs are vast. Similarly, I don't really know what of it to include here. Please let me know if you need to see some specific part of the configuration or if you have any idea where this problem could come from.
Thanks in advance for any idea or help!

0

There are 0 answers