I have recently upgraded monitoring agent to OPS agent and logs are flooded with following error:
08T11:31:25.681Z#011error#011exporterhelper/queued_retry.go:361#011Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors#011{"error": "failed to export time series to GCM: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: gce_instance{zone:us-central1-c,instance_id:<>} timeSeries[0-199]: agent.googleapis.com/cpu/load_15m{}\nerror details: name = Unknown desc = total_point_count:200 success_point_count:148 errors:{status:{code:3} point_count:52}; failed to export time series to GCM: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: gce_instance{zone:us-central1-c,instance_id:<>} timeSeries[0-199]: agent.googleapis.com/processes/cpu_time{command_line:apache2 -DFOREGROUND,pid:3045127,owner:www-data,process:all,command:apache2,user_or_syst:syst}\nerror details: name = Unknown desc = total_point_count:200 success_point_count:132 errors:{status:{code:3} point_count:68}; failed to export time series to GCM: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: gce_instance{zone:us-central1-c,instance_id:<>} timeSeries[0-199]: agent.googleapis.com/disk/bytes_used{state:used,device:/dev/loop3}\nerror details: name = Unknown desc = total_point_count:200 success_point_count:117 errors:{status:{code:3} point_count:83}; failed to export time series to GCM: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: gce_instance{instance_id:<>,zone:us-central1-c} timeSeries[0-136]: agent.googleapis.com/disk/operation_count{device:loop9,direction:write}\nerror details: name = Unknown desc = total_point_count:137 success_point_count:31 errors:{status:{code:3} point_count:106}", "errorCauses": [{"error": "failed to export time series to GCM: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: gce_instance{zone:us-central1-c,instance_id:<>} timeSeries[0-199]: agent.googleapis.com/cpu/load_15m{}\nerror details: name = Unknown desc = total_point_count:200 success_point_count:148 errors:{status:{code:3} point_count:52}"}, {"error": "failed to export time series to GCM: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: gce_instance{zone:us-central1-c,instance_id:<>} timeSeries[0-199]: agent.googleapis.com/processes/cpu_time{command_line:apache2 -DFOREGROUND,pid:3045127,owner:www-data,process:all,command:apache2,user_or_syst:syst}\nerror details: name = Unknown desc = total_point_count:200 success_point_count:132 errors:{status:{code:3} point_count:68}"}, {"error": "failed to export time series to GCM: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: gce_instance{zone:us-central1-c,instance_id:<>} timeSeries[0-199]: agent.googleapis.com/disk/bytes_used{state:used,device:/dev/loop3}\nerror details: name = Unknown desc = total_point_count:200 success_point_count:117 errors:{status:{code:3} point_count:83}"}, {"error": "failed to export time series to GCM: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: gce_instance{instance_id:<>,zone:us-central1-c} timeSeries[0-136]: agent.googleapis.com/disk/operation_count{device:loop9,direction:write}\nerror details: name = Unknown desc = total_point_count:137 success_point_count:31 errors:{status:{code:3} point_count:106}"}]}
Note: I have replaced in log snippet above my instance id with <>.
After getting this following 2 jsonPayload in logs again and again:
message: "Dec 8 11:37:25 helpcenter otelopscol[3059431]: go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1"
message: "Dec 8 11:37:25 helpcenter otelopscol[3059431]: #011/root/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:61"
I have followed this to remove and reinstall the OPS agent, I also checked this error is not mentioned in the link: https://cloud.google.com/logging/docs/agent/ops-agent/troubleshooting
Please suggest what needs to be done to fix this.
I have replicated your issue, and as a workaround, you should restart your VM instance.