I need to do 12 requests per second to the API on Twitch. I'm currently doing this with 3 processes, each doing 4 lookups per second but sequencially. I thought I'd give curl_multi a try so only 1 process would be needed.
Well - curl_multi is not quite what I expected !!! I thought all requests would be virtualy simultanious, but apparently not !
Currently, I am using file_get_contents (sequentially), and each request takes about 200-250ms, so one process could make about 5 requests per second maximum.
I thought that by using curl_multi I could do 12 requests simultaniously. Although it IS faster, it's not fast enough to do 12/sec. 12 lookups actually take approx 1.3 seconds. I used hrtime to measure the time taken between stages of the process, and it's this...........
do {
curl_multi_exec($multihandle, $stillrunning);
curl_multi_select($multihandle);
} while($stillrunning > 0);
that is taking all the time.
Apparently cURL can have performance issues and on reading various articles, it appears that cURL has issues with the OpenSSL 3 library, and that some people have rolled back to an earlier version which used OpenSSL 1.1 . This would obviously not be the ideal solution but then I don't even know if this would solve the problem anyway.
Does anyone have any thoughts on this please ?
System: Linux Zoe2 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 PHP: Version 8.1.27 cURL: 7.81.0 SSL Version: OpenSSL/3.0.2