Java Google AppEngine Managed VMs: What logs are obtainable through the Logging API?

Question

I like that I can use the Logs API (described here: https://cloud.google.com/appengine/docs/java/logs/) to programatically access and display app & request logs as I see fit--it's great.

Now that I'm using Managed VMs on AppEngine you can see on the Admin Console Logs Viewer that there are a ton of additional logs--including in my case a custom log which I found I could include in the viewer (decribed here: https://cloud.google.com/appengine/docs/managed-vms/custom-runtimes#logging).

My question is: Is there any way I can use the Logs API (or other pipelines already built?) to access these logs? My Managed VM module includes several components which could produce logs that I want to view:

1. App logs -- I can get these! No problem here.
2. Custom log files created by background processes I kick off in _ah/start (like "my_custom_1.log" in the screenshot)
3. STDERR & STDOUT from my background processes
4. Relevant Managed VM logs (e.g. for when an instance was restarted due to bad health... other system events like normal restarts?)

Basically I want "the total picture" at the instance level. Anyone tried to tame Managed VMs in this way with success? I'm not looking forward to rolling my own solution. And I wouldn't even know where to start on the problem of capturing STDERR and STDOUT. Any help appreciated.

Answer 1

There is a difference between App Engine logging and Google Cloud logging. Some of the Managed VM logs go to both, but much of it only goes to cloud logging.

Until recently there was not an API to read Cloud logs, only to write them. However, there is a new v2 beta API: https://cloud.google.com/logging/docs/api/introduction_v2

To do things at an instance level, entries in Cloud logging should have metadata set to denote which VM they came from. Both of these values seem to vary on logs from my VMs:

• compute.googleapis.com/resource_name
• compute.googleapis.com/resource_id