Post request response time spikes

Question

in my project I need to upload images as bytes to an endpoint and then process them, within a certain time. The images have a resolution of 600x600 and weight ~700 kB per image. Images are in png format. I have created a minimal flask application with one endpoint. The application is served using a waitress.

The problem is that sometimes response time from an endpoint suddenly increases by an order of magnitude. As I have to fit in a certain amount of time, such spikes can cause a critical error.

I ran a cProfile on the post request and found out that when there is a spike two functions take more time than normal:

Normal

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2    0.000    0.000    0.000    0.000 {method 'sendall' of '_socket.socket' objects}
     1    0.004    0.004    0.004    0.004 {method 'recv_into' of '_socket.socket' objects}

Spike

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2    0.008    0.004    0.008    0.004 {method 'sendall' of '_socket.socket' objects}
     1    0.012    0.012    0.012    0.012 {method 'recv_into' of '_socket.socket' objects}

What is the cause of these spikes, and can they be resolved in some way?

Code to reproduce:

# app.py
import numpy as np
from flask import Flask, request


def create_app():
    app = Flask(__name__)

    @app.route("/endpoint")
    def endpoint():
        data = request.data
        img_arr = np.frombuffer(data, dtype=np.uint8)
        # ...
        return {"foo": "bar"}

    return app

# wsgi.py
from waitress import serve

from app import create_app


def main():
    app = create_app()
    serve(app, host="0.0.0.0", port="9009")


if __name__ == "__main__":
    main()

# send.py

from pathlib import Path

import numpy as np
import requests
from PIL import Image


def main():
    img_paths = list(Path("images").rglob("*.png"))
    ses = requests.Session()
    speeds = []
    for img_path in img_paths:
        img_arr = np.asarray(Image.open(img_path))
        response = ses.post(url='http://0.0.0.0:9009/endpoint', data=img_arr.tobytes())
        speeds.append(response.elapsed.total_seconds())


if __name__ == "__main__":
    main()

I have executed python wsgi.py and in another terminal python send.py.

Python and package versions:

python==3.8.18

Flask==3.0.2
numpy==1.24.4
pillow==10.2.0
requests==2.31.0
waitress==3.0.0

Three independent performances for 1,000 images:

First

Second

Third

I have tried to define my own socket like this:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
s.bind(("0.0.0.0" 9009))
serve(app, sockets=[s])

It did not help at all. I have little knowledge in networking.