The application is running on AppEngine/Go.
Data is stored in both DataStore and BigQuery.
To save to BigQuery, create a Push task in Task Queues From AppEngine(/job/csv_upload).However, multiple [context canceled] occurred during data loading and eventually the process was killed.
Process terminated because the request deadline was exceeded. Please ensure that your HTTP server is listening for requests on 0.0.0.0 and on the port defined by the PORT environment variable. (Error code 123)
Despite the failure, tasks were being Pushed to Cloud Tasks.
And Cloud Tasks executes the task and AppEngine (/job/bigquery_taskqueue_request) is called to save the data to the table with a DML insert.
When AppEngine was configured as a Load Balancing backend service, infinite retries occurred in Cloud Tasks.The logs showed a BigQuery concurrency error.
[googleapi: Error 400: Resources exceeded during query execution: Too many DML statements outstanding against table XXX:XXXXX, limit is 20, resourcesExceeded]
This error occurred when 2,000 data items were stored in a batch process. Before installing the Load Balancing, the error had occurred in the same situation, but the error was resolved over time and all data was saved to the BigQuery table.
before
after
Process diagram after load balancer is installed
I expect to know two things
・Clarification of why Cloud Tasks caused infinite retries with the Load Balancing in place.
・Steps on how to avoid infinite retries.