How to make htttp request with akka stream for 10K request

Question

I build server with akka-http and akka-stream but it lost some request in 2K+ request. What I miss something or my understand for akka is wrong.

this is my code

  implicit val actorRef = ActorSystem("system", testConf)
  implicit val materializer = ActorMaterializer(ActorMaterializerSettings(actorRef).withInputBuffer(initialSize = 4, maxSize = 16))
  implicit val requestTimeout = Timeout(8 seconds)

  def response(req: HttpRequest): Future[Response] = {
    val connectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] =
      Http().outgoingConnection(host = "url").async
    for {
      res <- Source.single(req.copy(uri = s"${req.uri.path}?${req.uri.rawQueryString.get}")).via(connectionFlow).runWith(Sink.head)
      data <- res.entity.toStrict(5 second)
    } yield (data.getData().decodeString("UTF-8"), res.status.intValue())
  }

Thank you.

Answer 1

Most likely there were either timeout or server-side errors in first part of your for-comprehension therefore you got only successful responses in res.

I recommend to create flow/graph that processes your requests with withSupervisionStrategy in your materializer so you can see what exactly went wrong. Details of implementation depend on your business logic.