Major slowdown using ssl and add_buffer(req->buffer_out,...)

[aflin]

First: Thanks for the library!

Details

Steps or code to reproduce the problem.

In the examples/https/example_https_server.c I added a call to evbuffer_add:

static void
http__callback_(evhtp_request_t * req, void * arg) {
    evhtp_connection_t * conn;

    evhtp_assert(req != NULL);

    conn = evhtp_request_get_connection(req);
    evhtp_assert(conn != NULL);

    htp_sslutil_add_xheaders(
        req->headers_out,
        conn->ssl,
        HTP_SSLUTILS_XHDR_ALL);
    evbuffer_add(req->buffer_out,"hello world",11); // <-- ADDED TEXT TO BODY
    return evhtp_send_reply(req, EVHTP_RES_OK);
}

Benchmarking the difference:
Before adding the body text:

$ wrk -t 24 -c 48 -d 2 --timeout 20 https://127.0.0.1:4443/
Running 2s test @ https://127.0.0.1:4443/
  24 threads and 48 connections  
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.52ms    4.40ms  69.76ms   98.63%
    Req/Sec     1.83k   348.73     4.90k    92.71%
  89175 requests in 2.10s, 9.10MB read
Requests/sec:  42484.07
Transfer/sec:      4.34MB

After adding the body text:

$ wrk -t 24 -c 48 -d 2 --timeout 20 https://127.0.0.1:4443/
Running 2s test @ https://127.0.0.1:4443/
  24 threads and 48 connections  
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    44.68ms    3.96ms 108.14ms   97.55%
    Req/Sec    43.66     10.42    60.00     78.66%
  2118 requests in 2.09s, 251.20KB read
Requests/sec:   1012.22
Transfer/sec:    120.05KB

This only occurs when enabling https.
As a quick fix, I made the following changes to htp__create_reply_

static struct evbuffer *
htp__create_reply_(evhtp_request_t * request, evhtp_res code)
{
    struct evbuffer * buf;

    ...

    evhtp_headers_for_each(request->headers_out, htp__create_headers_, buf);
    evbuffer_add(buf, "\r\n", 2);

    /* -ajf --  Apparently SSL and evbuffer combined don't like evbuffer_add_buffer.
                Performance jumps by a factor of 40 with this solution.
                Also tried evbuffer_add_reference() and evbuffer_remove_buffer()
                    to avoid a copy, but no joy.
                Data is copied at least 3 times making this a horrible hack.
    */
    size_t len=evbuffer_get_length(request->buffer_out);
    if(len)
    {
        char *add;

        add=htp__malloc_(len);
        evhtp_alloc_assert(add);
        evbuffer_copyout(request->buffer_out, add, len);
        evbuffer_add(buf, add, len);
        free(add);
    }
    return buf;
/*
    if (evbuffer_get_length(request->buffer_out)) {
        evbuffer_add_buffer(buf, request->buffer_out);
    }

    return buf;
*/
}     /* htp__create_reply_ */

Maybe I'm doing something wrong? If not, I'm guessing the problem is in libevent.
However, since I'm only tracking down the problem this far, I'm reporting it here. Hoping you can come up with a better solution than the multiple copies I'm doing.

[aflin]

Actually the problem doesn't go away with the above edit when buffer_out size is more than about 800 bytes. My guess is that performance is degraded when the buffer is fragmented and ssl is on. This is a simpler fix and seems to work with larger buffer_out sizes:

static struct evbuffer *
htp__create_reply_(evhtp_request_t * request, evhtp_res code)
{
    struct evbuffer * buf;

    ...

    evhtp_headers_for_each(request->headers_out, htp__create_headers_, buf);
    evbuffer_add(buf, "\r\n", 2);

    if (evbuffer_get_length(request->buffer_out)) {
        evbuffer_add_buffer(buf, request->buffer_out);

        if (request->conn->htp->ssl_ctx != NULL)
            evbuffer_pullup(buf,-1);
    }
    return buf;
}     /* htp__create_reply_ */