Website crashes during high load when using Amazon Kinesis Client, issue with number of open connections

Question

So my website crashes during high load when using the AmazonKinesisFireHoseClient to put records in kinesis.

I get no exceptions logged it just sort of dies and you need to restart the application to get it to work again, sometimes it just starts working by itself after a crash.

I know it has something to do with the number of connections that are open but I have no idea how to fix it.

this is my logger that I instantiate as a singleton

public class KinesisFirehoseLogger<T>
{
    private string streamName;

    private AmazonKinesisFirehoseClient client;

    protected ILogger logger;

    public KinesisFirehoseLogger(IKinesisLogConfig config, ILogger logger)
    {
        this.streamName = config.StreamName;
        this.logger = logger;
        this.PartitionKey = config.PartitionKey;
        AWSCredentials credentials = new BasicAWSCredentials(config.AccessKey, config.SecretKey);
        this.client = new AmazonKinesisFirehoseClient(credentials, config.Region);
    }

    public async Task<bool> WriteToKinesisAsync(T logObject)
    {
        using (var ms = new MemoryStream(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(logObject) + "\n")))
        {
            PutRecordRequest putRecordRequest = new PutRecordRequest() { DeliveryStreamName = this.streamName, Record = new Record() { Data = ms } };

            // Put record into the DeliveryStream
            return (await client.PutRecordAsync(putRecordRequest)).HttpStatusCode == HttpStatusCode.OK;
        }
    }
}

Am I supposed to use the dispose() method of the AmazonKinesisFirehoseClient somewhere ? How do I know when to call it.

Answer 1

Assuming that the client.PutRecordAsync opens the connection then it might make sense to put it in a using block as well. For example:

using (var ms = new MemoryStream(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(logObject) + "\n")))
{
    PutRecordRequest putRecordRequest = new PutRecordRequest() { DeliveryStreamName = this.streamName, Record = new Record() { Data = ms } };

    // Put record into the DeliveryStream
    using (PutRecordResponse response = await client.PutRecordAsync(putRecordRequest))
    {
        return response.HttpStatusCode == HttpStatusCode.Ok;
    }
}