Implementing load protection for Fastify Node js

Question

Story:

As a developer, I wanted to implement load protection for Fastify so that the server load can be protected against server crashes.

Context:

Root cause analysis of serious performance issues can take time. In the case of a live deployed project, it’s worth adding overload protection to servers or services. The idea of overload protection is to monitor event loop delay (among other things), and respond with “503 Service Unavailable” if a threshold is passed. This allows a load balancer to failover to other instances, or in the worst case means users will have to refresh. The under-pressure module can provide this with minimum overhead for Fastify.

Problem:

The main purpose of under-pressure plugin is to protect the server against the high load. I created a fake load using autocannon and the server suddenly starts throwing this error:

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 00007FF7D342A9AC]
Security context: 0x01b3e02408d1 <JSObject>
    1: _write [000001EBE7CBA7B1] [_stream_transform.js:~169] [pc=000002737CBBD07D](this=0x02fe03573099 <LineStream map = 00000149AA79E209>,0x00f819d83789 <Uint8Array map = 000002F9BBE3F279>,0x00c0496035d9 <String[#6]: buffer>,0x02fe03575639 <JSBoundFunction (BoundTargetFunction 000003C5D37BC781)>)        
    2: ondata [000002FE03577FC9] [_stream_readable.js:~712]...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF7D28A363F napi_wrap+128063
 2: 00007FF7D2842836 v8::base::CPU::has_sse+35142
 3: 00007FF7D28434F6 v8::base::CPU::has_sse+38406
 4: 00007FF7D3059F4E v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF7D3042021 v8::SharedArrayBuffer::Externalize+833
 6: 00007FF7D2F0E57C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1436
 7: 00007FF7D2F197D0 v8::internal::Heap::ProtectUnprotectedMemoryChunks+1312
 8: 00007FF7D2F162F4 v8::internal::Heap::PageFlagsAreConsistent+3204
 9: 00007FF7D2F0BB13 v8::internal::Heap::CollectGarbage+1283
10: 00007FF7D2F0A184 v8::internal::Heap::AddRetainedMap+2452
11: 00007FF7D2F31E1F v8::internal::Factory::NewRawTwoByteString+95
12: 00007FF7D2F349CB v8::internal::Factory::NewStringFromUtf8+187
13: 00007FF7D305681A v8::String::NewFromUtf8+298
14: 00007FF7D27A9ABF node::tracing::TraceEventHelper::SetAgent+40751
15: 00007FF7D285A46D v8::internal::Malloced::operator delete+1661
16: 00007FF7D342A9AC v8::internal::SetupIsolateDelegate::SetupHeap+45852
17: 000002737CBBD07D

Expectation:

The very fastify plugin which is supposed to protect against load is breaking the fastify server, need some help with another plugin or some other way to implement load protection

Cross-posted to Github: https://github.com/fastify/under-pressure/issues/64

Answer 1

As pointed out in the issue, the config used was disabling the protection.

Anyway, I think you should evaluate if it is an external API or not.

If it is external fastify-rate-limit would be ideal to limit the request per seconds every tenant can do to your endpoints. This is good for the business too because you can provide a different threshold to each customer.

An internal API can use under-pressure as you pointed out, and the best config for my experience is:

const v8 = require('v8')

console.log(v8.getHeapStatistics())

fastify.register(require('under-pressure'), {
  maxEventLoopDelay: 200, // are you using sync method?!
  maxHeapUsedBytes: v8.getHeapStatistics().heap_size_limit,
  maxRssBytes: v8.getHeapStatistics().total_available_size
})

In this way, the plugin will adapt to use all the default or custom memory settings:

node --max-old-space-size=4096 server.js

Notice that Nodejs 12 adapt the heap space to the system it runs as explained in an official post

This update will configure the JavaScript heap size based on available memory instead of using defaults that were set by V8 for use with browsers. In previous releases, unless configured, V8 defaulted to limiting the max heap size to 700 MB or 1400MB on 32 and 64-bit platforms respectively. Configuring the heap size based on available memory ensures that Node.js does not try to use more memory than is available and terminating when its memory is exhausted. This is particularly useful when processing large data-sets. As before, it will still be possible to set — max-old-space-size to use a different limit if the default is not appropriate for your application.