How to save hex string to png file in spiffs to create image

Question

I'm trying to send an image file from a node.js server across a TCP/IP connection. I converted the image file to a hexadecimal string using fs.createReadStream and received the hex string as expected on the client side. Now I need to figure out how to reconstruct the image on the client side using the hex string.

Node.js code:

function newStream(res, imageFile){
    var readStream = fs.createReadStream(imageFile);
    readStream.on('data', chunk => {
        res.send(chunk.toString('hex'));
    });
}

Client side code (in C) used on an ESP32 development board to save data to a png file in spiffs:

#define MAX_HTTP_OUTPUT_BUFFER 512
int max_buff = 512;

static void http_native_request(void)
{
    char output_buffer[MAX_HTTP_OUTPUT_BUFFER] = {0};
    int content_length = 0;
    int track_length = 0;

    esp_http_client_config_t config = {
        .url = "http://192.168.1.122/api?file=file01",
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);

    // GET Request
    esp_http_client_set_method(client, HTTP_METHOD_GET);
    esp_err_t err = esp_http_client_open(client, 0);
    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Failed to open HTTP connection: %s", esp_err_to_name(err));
    } else {
        content_length = esp_http_client_fetch_headers(client);
        track_length = content_length;
        if (content_length < 0) {
            ESP_LOGE(TAG, "HTTP client fetch headers failed");
        } else {
            do {
            int data_read = esp_http_client_read_response(client, output_buffer, max_buff);
            output_buffer[max_buff] = '\0';
            create_file_app(output_buffer); //saves output_buffer to a newly created png file
            if (data_read >= 0) {
                track_length -= data_read;
                if (max_buff > track_length){
                    max_buff = track_length;
                }
            } else {
                ESP_LOGE(TAG, "Failed to read response");
            }
            } while (
                track_length>0
            );
        }
    }
    esp_http_client_close(client);
}

Hex string from server looks like this (has the signature of a png file):

89504e470d0a1a0a0000000d494844520000006400000064080600 ... f03b8c85cc0643044ae0000000049454e44ae426082

My research indicates that I need to convert this hex string into binary data on the client side. But when I do that (code for conversion not shown here), the png file created by create_file_app function simply displays the binary string (which correctly corresponds to the hex string) received by client as opposed to displaying the image that I thought I downloaded from the server. How do I save this hex data so I get the image I was expecting when I open the png file created on the client side? Or is there C library that can help with this?

Edit 1:

My code for sending image data from the node.js server as is, without converting to hex or another format:

function newStream(res, imageFile){
    var readStream = fs.createReadStream(imageFile);
    readStream.on('data', chunk => {
        res.send(chunk);
    });
}

Here's what the node.js console shows for "chunk":

<Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00 64 00 00 00 64 08 06 00 00 00 70 e2 95 54 00 00 00 06 62 4b 47 44 00 ff 00 ff 00 ff a0 bd a7 ... >

On the client side, here's ESP's code for esp_http_client_read_response():

int esp_http_client_read_response(esp_http_client_handle_t client, char *buffer, int len)
{
    int read_len = 0;
    while (read_len < len) {
        int data_read = esp_http_client_read(client, buffer + read_len, len - read_len);
        if (data_read <= 0) {
            return read_len;
        }
        read_len += data_read;
    }
    return read_len;
}

Esp_http_client_read_response calls esp_http_client_read (again, this was published by ESP) - here's the code for that:

int esp_http_client_read(esp_http_client_handle_t client, char *buffer, int len)
{
    esp_http_buffer_t *res_buffer = client->response->buffer;

    int rlen = ESP_FAIL, ridx = 0;
    if (res_buffer->raw_len) {
        int remain_len = client->response->buffer->raw_len;
        if (remain_len > len) {
            remain_len = len;
        }
        memcpy(buffer, res_buffer->raw_data, remain_len);
        res_buffer->raw_len -= remain_len;
        res_buffer->raw_data += remain_len;
        ridx = remain_len;
    }
    int need_read = len - ridx;
    bool is_data_remain = true;
    while (need_read > 0 && is_data_remain) {
        if (client->response->is_chunked) {
            is_data_remain = !client->is_chunk_complete;
        } else {
            is_data_remain = client->response->data_process < client->response->content_length;
        }

        if (!is_data_remain) {
            break;
        }
        int byte_to_read = need_read;
        if (byte_to_read > client->buffer_size_rx) {
            byte_to_read = client->buffer_size_rx;
        }
        errno = 0;
        rlen = esp_transport_read(client->transport, res_buffer->data, byte_to_read, client->timeout_ms);
        ESP_LOGD(TAG, "need_read=%d, byte_to_read=%d, rlen=%d, ridx=%d", need_read, byte_to_read, rlen, ridx);

        if (rlen <= 0) {
            if (errno != 0) {
                esp_log_level_t sev = ESP_LOG_WARN;
                /* On connection close from server, recv should ideally return 0 but we have error conversion
                 * in `tcp_transport` SSL layer which translates it `-1` and hence below additional checks */
                if (rlen == -1 && errno == ENOTCONN && client->response->is_chunked) {
                    /* Explicit call to parser for invoking `message_complete` callback */
                    http_parser_execute(client->parser, client->parser_settings, res_buffer->data, 0);
                    /* ...and lowering the message severity, as closed connection from server side is expected in chunked transport */
                    sev = ESP_LOG_DEBUG;
                }
                ESP_LOG_LEVEL(sev, TAG, "esp_transport_read returned:%d and errno:%d ", rlen, errno);
            }
            if (rlen < 0 && ridx == 0) {
                return ESP_FAIL;
            } else {
                return ridx;
            }
        }
        res_buffer->output_ptr = buffer + ridx;
        http_parser_execute(client->parser, client->parser_settings, res_buffer->data, rlen);
        ridx += res_buffer->raw_len;
        need_read -= res_buffer->raw_len;

        res_buffer->raw_len = 0; //clear
        res_buffer->output_ptr = NULL;
    }

    return ridx;
}

Here's my code for create_file_app() which saves retrieved data in a spiffs file:

void create_file_app(char *buffer)
{
    ESP_LOGI(TAG, "Opening file");
    FILE* f = fopen("/spiffs/hello1.png", "a+");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for writing");
        return;
    }
    fprintf(f, buffer);
    fclose(f);
}

Answer 1

Instead of calling fprintf() you need to call fwrite().

You receive the binary bytes with esp_http_client_read_response(). Store the returned length.

Then use this function to create the file:

void create_file_app(char *buffer, size_t length)
{
    ESP_LOGI(TAG, "Opening file");
    FILE* f = fopen("/spiffs/hello1.png", "wb");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for writing");
        return;
    }
    fwrite(buffer, 1, length, f);
    fclose(f);
}

Why is fprintf() the wrong function?

Because it works only with C-strings. These are sequences of characters, that means of type char[]. The length of such a string is determined by a special character, '\0', which has commonly the value 0. And the received bytes have a lot of zeroes in them. This character marks the end of the string, independent of the size of the character array.

Arrays in C don't "carry" their length with them, if you hand their address to other functions. You need to provide the length separately. Other languages do this in other ways.