I have a spring boot application running on GCP Compute Engine, this application reads from Google Pub/Sub and after computation writes to Firestore and BigQuery. Another process also reads from BigQuery then computes its results. Once a week BigQuery operations stops and after at least 24 hours operations returns to normal. In these problematic situations no exceptions or timeout errors can be found.
Code for inserting line to BQ is below:
private static void rowInsert(InsertAllRequest.Builder builder) {
try{
log.info("Inserting record to BQ");
InsertAllResponse response = BQO.getInstance().insertAll(builder.build());
log.info("Insert all response: {}", response.toString());
if (response.hasErrors()) {
StringBuilder sb = new StringBuilder();
for (Map.Entry<Long, List<BigQueryError>> entry : response.getInsertErrors().entrySet()){
for (BigQueryError error : entry.getValue()){
sb.append(error.getMessage()).append(", ");
}
}
log.error("Error inserting record {}", sb);
}
} catch (BigQueryException e){
log.error("Error inserting record {}", e.getMessage());
log.error("Error inserting record {}", e);
} catch (Exception e){
log.error("Unexpected exception row insert:" + e.getMessage(), e);
throw e;
}
}
In the execution "Inserting record to BQ" log is written after that I cannot see "Insert all response:" log. It seems write operation never finishes. After 24~30 hours new operations executes correctly, and old operations never returns.
Also in this time slots I cannot read from bigquery. When I try to read from BigQuery I got StackOverflowError with the log: java.base/java.util.Collections$UnmodifiableCollection.isEmpty(Collections.java:1033) ~[na:na]
Until 11th of October this code was working flawlessly, after this date I started to get errors above. I have upgraded my spring boot to 2.7.17 and my spring-cloud-gcp to 3.7.3. No changes occured.