I am writing a CLI to tail the GCP logs.
I am using the HTTP GET method to read the logs here: https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list
Issue: There are log entries missing.
When using pagination (using pageToken and nextPageToken) and constantly polling the last page I can get the latest records. This is fine when the log-write-rate is low.
But when the log-write-rate is high, there are log entries missing using this polling technique.
If I query the same time period but in the past, those missing log entries show up.
It seems that at a certain log-write-rate the GCP logging server will leave out some log entries from pagination pages, seemingly skipping over them and never including them again in the same series of paginated pages.
This is an issue as I could be polling for a particular log query but the result set is incomplete as log entries on the leading edge of time are randomly missing.
Question: What is the correct way to poll this end point without missing log entires when they are being written at a high rate?