Seemingly random non ascii characters in response body when using APISIX plugin to rewrite response

Question

I am trying to create a data-mask/redact plugin for APISIX in Lua. I have tried writing a custom plugin based on the following article and I have also tried using the out-of-the-box plugin response-rewrite provided by APISIX. In both cases I am receiving the response turned into complete gibberish, in the following manner: Postman response screenshot The expected response would be a json like: { "success": {}, "timeStamp": "2023-12-01 10:03:12.0", "version": "21.2.0.0" }

The code being used to filter response:

function _M.body_filter(conf, ctx)
    if conf.filters then

        local body = core.response.hold_body_chunk(ctx)
        if not body then
            return
        end
        
        local err
        for _, filter in ipairs(conf.filters) do
            body, _, err = re_gsub(body, filter.regex, filter.replace, "jo")

            if err ~= nil then
                core.log.error("regex \"" .. filter.regex .. "\" substitutes failed:" .. err)
            end
        end

        ngx.arg[1] = body
        return
    end

end

I tried to log the response body:

        local body = core.response.hold_body_chunk(ctx)
        if not body then
            return
        end
        core.log.error(body)

the log also shows random non ascii characters, so the problem is not occurring during substitution, rather it's happening in the way we get response body itself.

Answer 1

Since core.response.hold_body_chunk receives the raw response body, it does receive cases that contain non-alphabets and symbols.

For example, when gzip is used between upstream and APISIX, it will receive the compressed raw response body, APISIX will not decompress automatically for you.

You need to check if this is the case (including but not limited to gzip)