cURL not working sometimes and gives empty result

Question

I used cURL to get data from another website. Sometimes it shows data and sometimes empty result

Here is my Code

    function get_data($url) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        $agent=$_SERVER["HTTP_USER_AGENT"];
        curl_setopt($ch,CURLOPT_USERAGENT, $agent); 
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
         curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false);

        $data = curl_exec($ch);
        curl_close($ch);
        return $data;
    }
    $returned_content = get_data('www.example.com');
    echo $returned_content;

Answer 1

Possibly you call too many connections from your curl_init to one Ip Address, So the server blocks the connection and causes the on/off errors.

Answer 2

I ran into this issue trying to connect to an API multiple times in succession.

My answer is pretty much the same as @developer:

Possibly you call too many connections from your one Ip Address, So the server blocks the connection and causes the on/off errors.

This works for me as a work around.

Create a method that attempts the curl request a few times, adding time between each request until it gets data or gives up.

/**
 * Thanks to issues with sometimes not receiving data when making multiple requests
 * here we make a reasonable effort to get data even when a firewall or WAF or ?? maybe
 * throttling request, DDOS, or on fire
 *
 * @param Curl $ch
 *
 * @return bool|string
 */
function curlExecFireWalker($ch)
{
   //In case we fail, let's count our failures
   $wowFailCount = 0;
   do {
      $data = curl_exec($ch);
      if (!$data) {
         //Well this is awkward, we are expecting data and didn't get any :(
         $wowFailCount++;
         //Let's calculate the length of a smoke break
         $smokeBreak = pow($wowFailCount, 2);
         //Smoke 'em if you got 'em
         sleep($smokeBreak);
      }
      //Well let's not allow to many failures and get out of here before the entire thing is on fire
   } while (!$data && $wowFailCount <= 3);

   return $data;

}

Then instead of

$data = curl_exec($ch);

call the fire walker method

$data = curlExecFireWalker($ch);

Answer 3

Not receiving any content back could be due to one or more out of many different reasons and you need to figure out which.

1. Use curl_error($ch) in your code after the transfer has been performed to see if curl has told you about a problem.

2. Investigate the response headers. A HTTP transfer can be successful and not return any error but might only respond with HTTP headers and no body at all. The HTTP headers may then contain clues as to why. Perhaps it is a redirect you should follow, perhaps it requires HTTP authentication or perhaps the server indicates a temporary server glitch.

Answer 4

Curl in PHP returns null on AWS EC2 instance

I had similar issue and I fixed it by making sure versions and settings in php.ini file which is in PHP5 and Apache2 folders were same. If its not then Apache2 tries to execute versions set inside the php.ini settings. Also make sure PHP5-libcurl is installed. Its important too.

Answer 5

Solution

Make sure you have no space for any variables or in url. Like "$user_name = "Lokman Hosen"". You have to give urldecode "$user_name = "Lokman_Hosen"" or "$user_name = "Lokman%20Hosen"". You can use

urldecode()

Function to remove space. Now cURL dont allow any space.

Answer 6

Possibly the intermittent failures are due to flaky DNS servers, even if it isn't, you'll still benefit from using Google's DNS servers.. See: https://developers.google.com/speed/public-dns/docs/using