We're using yescrypt from libc's crypt(3) function to encrypt and check credentials in our C-written application server. Sometimes the crypt(3) fails because the application gets SIGKILL from the kernel oom-killer. Here is what strace show in such a moment:
11:24:13.136472 mmap(NULL, 1075838976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1,
0) = -1 ENOMEM (Cannot allocate memory) <0.000053>
11:24:13.136579 mmap(NULL, 1073766464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fadab8ad000 <0.000016>
11:24:14.607035 +++ killed by SIGKILL +++
Why yescrypt is asking for so much memory (~1 GB) and is this somehow tuneable?
With the help of the yescrypt community (see here: https://www.openwall.com/lists/yescrypt/2024/03/20/2 ) it turned out the the parameter "count" of crypt_gensalt(3) is wrong/bad explained in its man page. It says only "...count controls the CPU time cost of the hash; the valid range for count and the exact meaning of “CPU time cost” depends on the hashing method, but larger numbers correspond to more costly hashes." It affects also the used memory in logarithmic base-2 manner. A value of 5 gives 16 MBytes while 11 gives 1++ GByte. The value is used on creation of the hash and stored in the hash for checking the hash later.