Sending async https request using boost/beast

Question

I had a synchronous method that send https request using http::write and than expect to read it's response using http::read.

However, in order to add timeout I had to move to async calls in my method. So I've tried to use http::async_read and http::async_write, but keep this overall flow synchronous so the method will return only once it has the https response.

here's my attempt :

class httpsClass {

  std::optional<boost::beast::ssl_stream<boost::beast::tcp_stream>> ssl_stream_;
  
  httpsClass(..) {
    // notice that ssl_stream_ is initialized according to io_context_/ctx_ 
    // that are class members that get set by c'tor args
    ssl_stream_.emplace(io_context_, ctx_); 
  }

}

std::optional<boost::beast::http::response<boost::beast::http::dynamic_body>>
httpsClass::sendHttpsRequestAndGetResponse (
    const boost::beast::http::request<boost::beast::http::string_body>
        &request) {
  try{
    boost::asio::io_context ioc;

    beast::flat_buffer buffer;
    http::response<http::dynamic_body> res;

    beast::get_lowest_layer(*ssl_stream_).expires_after(kTimeout);

    boost::asio::spawn(ioc, [&, this](boost::asio::yield_context yield) {

      auto sent = http::async_write(this->ssl_stream_.value(), request, yield);
      auto received = http::async_read(this->ssl_stream_.value(), buffer, res, yield);
    });

    ioc.run();// this will finish only once the task above will be fully executed.

    return res;
  } catch (const std::exception &e) {
    log("Error sending/receiving:{}", e.what());
    return std::nullopt;
  }
}

During trial, this method above reaches the task I assign for the internal io contexts (ioc). However, it gets stuck inside this task on the method async_write.

Anybody can help me figure out why it gets stuck? could it be related to the fact that ssl_stream_ is initialize with another io context object (io_context_) ?

Answer 1

Yes. The default executor for completion handlers on the ssl_stream_ is the outer io_context, which cannot make progress, because you're likely not running it.

My hint would be to:

• avoid making the second io_context
• also use the more typical future<Response> rather than optional<Response> (which loses the the error information)
• avoid passing the io_context&. Instead pass executors, which you can more easily change to be a strand executor if so required.

Adding some code to make it self-contained:

class httpsClass {
    ssl::context&                                       ctx_;
    std::string                                         host_;
    std::optional<beast::ssl_stream<beast::tcp_stream>> ssl_stream_;
    beast::flat_buffer                                  buffer_;

    static constexpr auto kTimeout = 3s;

  public:
    httpsClass(net::any_io_executor ex, ssl::context& ctx, std::string host)
        : ctx_(ctx)
        , host_(host)
        , ssl_stream_(std::in_place, ex, ctx_) {

        auto ep = tcp::resolver(ex).resolve(host, "https");
        ssl_stream_->next_layer().connect(ep);
        ssl_stream_->handshake(ssl::stream_base::handshake_type::client);
        log("Successfully connected to {} for {}",
            ssl_stream_->next_layer().socket().remote_endpoint(), ep->host_name());
    }

    using Request  = http::request<http::string_body>;
    using Response = http::response<http::dynamic_body>;

    std::future<Response> performRequest(Request const&);
};

Your implementation was pretty close, except for the unnecessary service:

std::future<httpsClass::Response>
httpsClass::performRequest(Request const& request) {
    std::promise<Response> promise;
    auto fut = promise.get_future();

    auto coro = [this, r = request, p = std::move(promise)] //
        (net::yield_context yield) mutable {
            try {
                auto& s = *ssl_stream_;
                get_lowest_layer(s).expires_after(kTimeout);

                r.prepare_payload();
                r.set(http::field::host, host_);

                auto sent = http::async_write(s, r, yield);
                log("Sent: {}", sent);

                http::response<http::dynamic_body> res;
                auto received = http::async_read(s, buffer_, res, yield);
                log("Received: {}", received);
                p.set_value(std::move(res));
            } catch (...) {
                p.set_exception(std::current_exception());
            }
        };

    spawn(ssl_stream_->get_executor(), std::move(coro));
    return fut;
}

Now, it is important to have the io_service run()-ning for any asynchronous operations. With completely asynchronous code you wouldn't need threads, but as you are blocking on the response you will. The easiest way is to replace io_service with a thread_pool which does the run()-ning for you.

int main() {
    net::thread_pool ioc;
    ssl::context ctx(ssl::context::sslv23_client);
    ctx.set_default_verify_paths();

    for (auto query : {"/delay/2", "/delay/5"}) {
        try {
            httpsClass client(make_strand(ioc), ctx, "httpbin.org");

            auto res = client.performRequest({http::verb::get, query, 11});

            log("Request submitted... waiting for response");
            log("Response: {}", res.get());
        } catch (boost::system::system_error const& se) {
            auto const& ec = se.code();
            log("Error sending/receiving:{} at {}", ec.message(), ec.location());
        } catch (std::exception const& e) {
            log("Error sending/receiving:{}", e.what());
        }
    }

    ioc.join();
}

As you can see this test will run two requests against https://httpbin.org/#/Dynamic_data/get_delay__delay_. The second will timeout because 5s exceeds the 3s expiration on the ssl_stream_.

Full Demo

Live On Coliru

#include <boost/asio.hpp>
#include <boost/asio/spawn.hpp>
#include <boost/beast.hpp>
#include <boost/beast/ssl.hpp>
#include <fmt/ostream.h>
#include <fmt/ranges.h>
#include <optional>
using namespace std::chrono_literals;
namespace net   = boost::asio;
namespace beast = boost::beast;
namespace http  = beast::http;
namespace ssl   = net::ssl;
using net::ip::tcp;

////// LOG STUBS
template <> struct fmt::formatter<boost::source_location> : fmt::ostream_formatter {};
template <> struct fmt::formatter<tcp::endpoint> : fmt::ostream_formatter {};
template <bool isRequest, typename... Args>
struct fmt::formatter<http::message<isRequest, Args...>> : fmt::ostream_formatter {};

static inline void log(auto const& fmt, auto const&... args) {
    fmt::print(fmt::runtime(fmt), args...);
    fmt::print("\n");
    std::fflush(stdout);
}
////// END LOG STUBS

class httpsClass {
    ssl::context&                                       ctx_;
    std::string                                         host_;
    std::optional<beast::ssl_stream<beast::tcp_stream>> ssl_stream_;
    beast::flat_buffer                                  buffer_;

    static constexpr auto kTimeout = 3s;

  public:
    httpsClass(net::any_io_executor ex, ssl::context& ctx, std::string host)
        : ctx_(ctx)
        , host_(host)
        , ssl_stream_(std::in_place, ex, ctx_) {

        auto ep = tcp::resolver(ex).resolve(host, "https");
        ssl_stream_->next_layer().connect(ep);
        ssl_stream_->handshake(ssl::stream_base::handshake_type::client);
        log("Successfully connected to {} for {}",
            ssl_stream_->next_layer().socket().remote_endpoint(), ep->host_name());
    }

    using Request  = http::request<http::string_body>;
    using Response = http::response<http::dynamic_body>;

    std::future<Response> performRequest(Request const&);
};

std::future<httpsClass::Response>
httpsClass::performRequest(Request const& request) {
    std::promise<Response> promise;
    auto fut = promise.get_future();

    auto coro = [this, r = request, p = std::move(promise)] //
        (net::yield_context yield) mutable {
            try {
                auto& s = *ssl_stream_;
                get_lowest_layer(s).expires_after(kTimeout);

                r.prepare_payload();
                r.set(http::field::host, host_);

                auto sent = http::async_write(s, r, yield);
                log("Sent: {}", sent);

                http::response<http::dynamic_body> res;
                auto received = http::async_read(s, buffer_, res, yield);
                log("Received: {}", received);
                p.set_value(std::move(res));
            } catch (...) {
                p.set_exception(std::current_exception());
            }
        };

    spawn(ssl_stream_->get_executor(), std::move(coro));
    return fut;
}

int main() {
    net::thread_pool ioc;
    ssl::context ctx(ssl::context::sslv23_client);
    ctx.set_default_verify_paths();

    for (auto query : {"/delay/2", "/delay/5"}) {
        try {
            httpsClass client(make_strand(ioc), ctx, "httpbin.org");

            auto res = client.performRequest({http::verb::get, query, 11});

            log("Request submitted... waiting for response");
            log("Response: {}", res.get());
        } catch (boost::system::system_error const& se) {
            auto const& ec = se.code();
            log("Error sending/receiving:{} at {}", ec.message(), ec.location());
        } catch (std::exception const& e) {
            log("Error sending/receiving:{}", e.what());
        }
    }

    ioc.join();
}

Live on my system: