APM Server transport error with elastic cloud APM and nodejs application

Question

We have installed APM on elastic cloud and configured to send application traces to Elastic cloud. We are facing some issue related to TLS. We are using default Elastic Cloud agent policy.

Kibana version: 8.8.1

Elasticsearch version: 8.8.1

APM Server version: 8.8.1

APM Agent language and version: NodeJs Agent elastic-apm-node": "elastic-apm-node": "^4.3.0"

Configuration:

var apm = require('elastic-apm-node').start({
serviceName: 'Staging-Device360',

secretToken: 'xxxxxxxxxxxxxx',

serverUrl: 'xxxxxxxxxxxxxxx',

environment: 'xxxxxxx',

});

Using this configuration we were able to trace all our applications. But still we are seeing huge error log in application related to elastic-apm. Attaching the logs for your reference. This error logs are casing storage issue on our elastic cluster.

Startup info log:

{
"log.level": "info",
"@timestamp": "2024-01-11T10:16:45.379Z",
"log.logger": "elastic-apm-node",
"ecs.version": "8.10.0",
"agentVersion": "4.3.0",
"env": {
"pid": 1,
"proctitle": "node",
"os": "linux 5.10.201-191.748.amzn2.x86_64",
"arch": "x64",
"host": "ip-10-180-13-201.ap-south-1.compute.internal",
"timezone": "UTC+00",
"runtime": "Node.js v14.21.3"
},
"config": {
"environment": {
"source": "start",
"value": "xxxxxxx"
},
"logLevel": {
"source": "default",
"value": "info",
"commonName": "log_level"
},
"serverUrl": {
"source": "start",
"value": "xxxxxxxxxxx",
"commonName": "server_url"
},
"verifyServerCert": {
"source": "start",
"value": false,
"sourceValue": "false"
},
"secretToken": {
"source": "start",
"value": "[REDACTED]",
"commonName": "secret_token"
},
"serviceName": {
"source": "start",
"value": "xxxxx",
"commonName": "service_name"
},
"serviceVersion": {
"source": "default",
"value": "1.0.0",
"commonName": "service_version"
}
},
"activationMethod": "require",
"message": "Elastic APM Node.js Agent v4.3.0"
}

Error Log:

{
"log.level": "error",
"@timestamp": "2024-01-11T10:34:38.293Z",
"log.logger": "elastic-apm-node",
"ecs.version": "8.10.0",
"message": "APM Server transport error (502): Unexpected APM Server response\nPost \"https://172.22.9.142:18768/intake/v2/events\": local error: tls: bad record MAC\n"
}
{
    "log.level": "error",
    "@timestamp": "2024-01-30T11:59:22.671Z",
    "log.logger": "elastic-apm-node",
    "ecs.version": "8.10.0",
    "message": "APM Server transport error (502): Unexpected APM Server response\nPost \"https://172.22.9.142:18987/intake/v2/events\": use of closed network connection\n"
}

Traces were coming as expected by application logs were getting bombarded with constant error logs