The company I am working for uses Azure Blob Storage for our filesystem and we have found that there is a very high rate of failure because of Connection Refused:
On our main PHP application running laravel in swoole we have seen 8 of these errors in the last day. And on one long running that required handling about 1000 files it failed twice in a row.
So I would estimate that the failure rate for azureBlob storage is about 0.1 -> 1 %
For the connection we are using: "name": "league/flysystem-azure-blob-storage", "version": "1.0.0",
With the retry middleware: "microsoft/azure-storage-blob": "1.1", "microsoft/azure-storage-common": "1.5.1",
And guzzle: "guzzlehttp/guzzle": "7.3.0",
We create the provider with this code:
class AzureBlobStorageServiceProvider extends ServiceProvider {
/** * Bootstrap services. * * @return void */
public function boot() {
Storage::extend('azure', function ($app, $config) { $client =
BlobRestProxy::createBlobService( "DefaultEndpointsProtocol=https;AccountName={$config['name']};AccountKey={$config['key']};" );
$retryMiddleware = RetryMiddlewareFactory::create(
RetryMiddlewareFactory::GENERAL_RETRY_TYPE, // Specifies the retry logic
10, // Number of retries
1000, // Interval
RetryMiddlewareFactory::EXPONENTIAL_INTERVAL_ACCUMULATION, // How to increase the wait interval
true // Whether to retry connection failures too, default false
);
$client->pushMiddleware($retryMiddleware);
$adapter = new AzureBlobStorageAdapter($client, $config['container']);
//$cachedAdapter = new CachedAdapter($adapter, new PredisStore(Redis::client()));
return new Filesystem($adapter, new Config([
'disable_asserts' => true,
]));
});
}
/**
* Register services.
*
* @return void
*/
public function register()
{
//
}
/**
* Get the services provided by the provider.
*
* @return array
*/
public function provides()
{
return [Filesystem::class];
}}`
Does anyone have any ideas what we can do to prevent ConnectionRefused Errors or handle them better?
The full error:
GuzzleHttp\Exception\ConnectException cURL error 7: Failed to connect to sonarprodblob01.blob.core.windows.net port 443: Connection timed out (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://sonarprodblob01.blob.core.windows.net/sonar-prod-blob-01/monitoring_results/ffda21bb-ca8f-4e02-aa5a-93886155271b
/usr/share/sonar/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:210 GuzzleHttp\Handler\CurlFactory::createRejection
/usr/share/sonar/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:158 GuzzleHttp\Handler\CurlFactory::finishError
/usr/share/sonar/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:110 GuzzleHttp\Handler\CurlFactory::finish
/usr/share/sonar/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:236 GuzzleHttp\Handler\CurlMultiHandler::processMessages
/usr/share/sonar/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:168 GuzzleHttp\Handler\CurlMultiHandler::tick
/usr/share/sonar/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute
/usr/share/sonar/vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn
/usr/share/sonar/vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending
/usr/share/sonar/vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList
/usr/share/sonar/vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending
/usr/share/sonar/vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait
/usr/share/sonar/vendor/microsoft/azure-storage-blob/src/Blob/BlobRestProxy.php:3018 MicrosoftAzure\Storage\Blob\BlobRestProxy::getBlobProperties
/usr/share/sonar/vendor/league/flysystem-azure-blob-storage/src/AzureBlobStorageAdapter.php:240 League\Flysystem\AzureBlobStorage\AzureBlobStorageAdapter::getMetadata
/usr/share/sonar/vendor/league/flysystem-azure-blob-storage/src/AzureBlobStorageAdapter.php:153 League\Flysystem\AzureBlobStorage\AzureBlobStorageAdapter::has
/usr/share/sonar/vendor/league/flysystem/src/Filesystem.php:58 League\Flysystem\Filesystem::has
/usr/share/sonar/vendor/league/flysystem/src/Filesystem.php:99 League\Flysystem\Filesystem::put
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php:249 Illuminate\Filesystem\FilesystemAdapter::put
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemManager.php:398 Illuminate\Filesystem\FilesystemManager::__call
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php:261 Illuminate\Support\Facades\Facade::__callStatic
Http/Controllers/PollerEndpointController.php:98 Sonar\Http\Controllers\PollerEndpointController::getSnmpResults
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54 Illuminate\Routing\Controller::callAction
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:45 Illuminate\Routing\ControllerDispatcher::dispatch
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/Route.php:254 Illuminate\Routing\Route::runController
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/Route.php:197 Illuminate\Routing\Route::run
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/Router.php:695 Illuminate\Routing\Router::Illuminate\Routing\{closure}
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:128 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
Http/Middleware/PollerMiddleware.php:49 Sonar\Http\Middleware\PollerMiddleware::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php:50 Illuminate\Routing\Middleware\SubstituteBindings::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php:21 Illuminate\Foundation\Http\Middleware\TransformsRequest::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php:31 Illuminate\Foundation\Http\Middleware\ConvertEmptyStringsToNull::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php:21 Illuminate\Foundation\Http\Middleware\TransformsRequest::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php:40 Illuminate\Foundation\Http\Middleware\TrimStrings::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:103 Illuminate\Pipeline\Pipeline::then
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/Router.php:697 Illuminate\Routing\Router::runRouteWithinStack
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/Router.php:672 Illuminate\Routing\Router::runRoute
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/Router.php:636 Illuminate\Routing\Router::dispatchToRoute
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Routing/Router.php:625 Illuminate\Routing\Router::dispatch
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php:166 Illuminate\Foundation\Http\Kernel::Illuminate\Foundation\Http\{closure}
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:128 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
/usr/share/sonar/vendor/laravel/nova/src/Http/Middleware/ServeNova.php:27 Laravel\Nova\Http\Middleware\ServeNova::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php:27 Illuminate\Foundation\Http\Middleware\ValidatePostSize::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php:86 Illuminate\Foundation\Http\Middleware\PreventRequestsDuringMaintenance::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
Http/Middleware/BlockRepeatedAuthenticationFailuresMiddleware.php:25 Sonar\Http\Middleware\BlockRepeatedAuthenticationFailuresMiddleware::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
Http/Middleware/MultiTenant.php:65 Sonar\Http\Middleware\MultiTenant::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
/usr/share/sonar/vendor/fideloper/proxy/src/TrustProxies.php:57 Fideloper\Proxy\TrustProxies::handle
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:167 Illuminate\Pipeline\Pipeline::Illuminate\Pipeline\{closure}
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:103 Illuminate\Pipeline\Pipeline::then
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php:141 Illuminate\Foundation\Http\Kernel::sendRequestThroughRouter
/usr/share/sonar/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php:110 Illuminate\Foundation\Http\Kernel::handle
/usr/share/sonar/vendor/swooletw/laravel-swoole/src/Server/Sandbox.php:257 SwooleTW\Http\Server\Sandbox::handleRequest
/usr/share/sonar/vendor/swooletw/laravel-swoole/src/Server/Sandbox.php:217 SwooleTW\Http\Server\Sandbox::prepareObResponse
/usr/share/sonar/vendor/swooletw/laravel-swoole/src/Server/Sandbox.php:179 SwooleTW\Http\Server\Sandbox::run
/usr/share/sonar/vendor/swooletw/laravel-swoole/src/Server/Manager.php:225 SwooleTW\Http\Server\Manager::onRequest
/usr/share/sonar/vendor/upscale/swoole-blackfire/src/ProfilerDecorator.php:54 [main]
Even if you have implemented Rate Limiting process, it still requires transientfault handling.
Properly implementing a rate limiting mechanism in your application will substantially reduce the number of transient faults.
Transient faults can occur for reasons other than TooManyRequests
Reference Link: https://learn.microsoft.com/en-us/azure/cosmos-db/rate-limiting-requests#transient-fault-handling