Unable to send HTTP POST payload of size greater than 1449 characters with SIM800L+ESP32

Question

So I am sending data to my php script hosted on 000WebHost via HTTP POST on my ESP32+SIM800L set up. I am recording sensor data at 800Hz and storing it in an character array like: a[]=3&a[]=5&a[]=8... which becomes my payload array for the POST request.

For some reason I can only send 161 values which is a Content Length of roughly 1449.

The code is a bit lengthy so I have reduced it here: I am using the TinyGSM Library

//Start GSM:
 SerialAT.begin(115200, SERIAL_8N1, MODEM_RX, MODEM_TX);
 SerialMon.println("Initializing modem...");


modem.init();
  SerialMon.print(F("Connecting to "));
   SerialMon.print(apn);
   if (!modem.gprsConnect(apn, gprsUser, gprsPass)) {
      SerialMon.println(" fail");
      delay(10000);
      return;
    }
   SerialMon.println(" success");

   if (modem.isGprsConnected()) {
      SerialMon.println("GPRS connected");
  }

  SerialMon.print("Connecting to ");
  SerialMon.println(server);
  if (!client.connect(server, port)) {
    SerialMon.println(" fail");
    delay(10000);
    return;
  }
  SerialMon.println(" success");


// Record Sensor values for one second
//Send the character array to the httpPost function:
if (client.connect(server,port)) {
  Serial.println("connected");
  client.println("POST /upload.php? HTTP/1.1");
  client.println("Host:  epiblastic-reactor.000webhostapp.com");
  client.println("User-Agent: TTGO-TCALL/1.0");
  client.println("Content-Type: application/x-www-form-urlencoded;");
  client.print("Content-Length: ");
  client.println(String(acceldata).length());
  client.println();
  client.println(acceldata);
  

  uint32_t timeout = millis();
  while (client.connected() && millis() - timeout < 10000L) {
    // Print available data
    while (client.available()) {
      char c = client.read();
      SerialMon.print(c);
      timeout = millis();
    }

If I send more than 161 values the Serial Monitor prints:

#Available: 0 on 1

continuously.

What is going wrong? Can Someone please help me?

Answer 1

The MTU of an Ethernet is typically at 1500, but for SIM800, I believe it is set to 1460 bytes(You can check the value using AT+CIPSEND?). The TCP/HTTP client that TinyGSM used does not seems to break the data longer than one MTU into chunks, so it will be user's responsibility to do it.

Here is an example on how to send data in multiple chunks.

Code has been updated from initial input

#define ONE_CHUNK 1024  //you can change it as long as it is <= MTU

int payload_length = acceldata.length(); //assuming your data is already a String object
client.print("Content-Length: ");
client.println(payload_length);
client.println();

if (payload_length < ONE_CHUNK) {  //less than ONE_CHUNK, send it
  client.println(acceldata);
}
else {  // more than ONE_CHUNK, break it into number of chunks
  int chunks = payload_length / ONE_CHUNK;
  for(int i = 0; i < chunks; i++){
    client.print(acceldata.substring(i*ONE_CHUNK, (i*ONE_CHUNK)+ONE_CHUNK));
  }
  int last_chunk = payload_length - chunks * ONE_CHUNK;
  if (last_chunk) {
    client.println(acceldata.substring(payload_length-last_chunk, payload_length - 1)); 
  } 
}

Answer 2

Nothing obvious, you might want to check your TinyGSM configuration and maybe enable debug output from that library.

Anyway. There are always the usual suspects:

Firstly you're not using flow control on the serial link between the ESP32 and the modem. This means that the modem's serial buffers could overflow if you're pumping more data to it that it can send to the Internet. The good solution is to enable serial flow control (hardware would be best, software does the trick) on both sides. A poor solution is to send data more slowly, e.g. 1400 bytes every few seconds.

Secondly check that there's nothing anomalous happening on the server side, i.e. server closing the connection.

Then you can check that - whatever the object client is (your code doesn't show where it's created) - it actually permits you to call println() with large amounts of data.

As for your data amount, there's nothing unusual about sending 6-7 KiB of data via a HTTP request. The problem arises only if you want to send that much every second, through a link that has less throughput - obviously it doesn't have the capacity to do that. GPRS uploads are slow.