Only the first MQTT message from ESP32 to AWS IoT Core succeeds – what am I missing?

Question

I am using an ESP32-CAM to capture an image and transmit that image in Base64 to AWS IoT Core.

I'm using Pubsubclient, and the connection is sound.

When I compile my code, the first message transmits successfully and is complete. Subsequent messages are truncated, even after a reset of the device. The only message that transmits successfully is the first message upon compile and flash.

Here is my publishMessage() function:

void publishMessage() {
  camera_fb_t * fb = NULL;
  fb = esp_camera_fb_get();  
  if(!fb) {
    Serial.println("Camera capture failed");
    return;
  }  

  char *input = (char *)fb->buf;

//AWS Lambda Actions need base64 encoded data, jsonified
  char output[base64_enc_len(3)];
  String image;
  for (int i=0;i<fb->len;i++) {
    base64_encode(output, (input++), 3);
    if (i%3==0) image += String(output);
  }
  
  String imageFile = "{\"image\":\""+image+"\"}";
  uint32_t fbLen=imageFile.length();

  if(!client.connected()) {
    Serial.println("Connecting to AWS IOT");

    while (!client.connect(THINGNAME)) {
      Serial.print(".");
      delay(100);
    }

    if (!client.connected()) {
      Serial.println("AWS IoT Timeout!");
      return;
    }
  }

  boolean res = client.beginPublish(AWS_IOT_PUBLISH_TOPIC, fbLen, false);
  Serial.println("publish began " + String(res));

  for (uint32_t n=0;n<fbLen;n=n+2048) {
    String str = "";
    if (n+2048<fbLen) {
      str = imageFile.substring(n, n+2048);
      client.write((uint8_t*)str.c_str(), 2048);
    }
    else if (fbLen%2048>0) {
      uint32_t remainder = fbLen%2048;
      str = imageFile.substring(n, n+remainder);
      client.write((uint8_t*)str.c_str(), remainder);
    }
    Serial.println(String(n));
  }  

  Serial.println("publish written " + String(res));

  res = client.endPublish();
  Serial.println("publish ended " + String(res));

  esp_camera_fb_return(fb);
}

As far as I can tell, I am handling memory allocation properly, but I suspect this may be where the problem lies.

The truncated message contains a significant portion of the image, but there seems to be white space added in, and, of course, it is missing the closing quotation marks & curly brace.

What am I missing?

I used setBufferSize to accommodate the total message size. I tried chunking the image into 1 byte chunks as well as 64000 byte chunks. The same result occurs. I have searched for any information as to why the initial message succeeds, but all subsequent messages fail. That's the most perplexing part of the issue.

Answer 1

See: https://github.com/knolleary/pubsubclient/issues/791

PubSubClient's buildHeader function takes a uint16_t payload length. This restricts payloads to 64kb or less.

Edits to the function definitions for both buildHeader and write in PubSubClient.h and PubSubClient.cpp allow for larger payloads.

Before:

boolean write(uint8_t header, uint8_t* buf, uint16_t length);
size_t buildHeader(uint8_t header, uint8_t* buf, uint16_t length);

After:

boolean write(uint8_t header, uint8_t* buf, uint32_t length);
size_t buildHeader(uint8_t header, uint8_t* buf, uint32_t length);