COS: Google Container-Optimized OS: How do I record process CPU usage over time?

Question

I have a VM running Google Container-Optimized OS (COS).

It has been running fine for 6 weeks, then suddenly yesterday CPU spiked to 100% for 12 hours, and disk read IOP/Throughput also stayed high during this time.

What can I run on the VM to record exactly which process is causing this?

Because COS is locked down, it seems I cannot install psacct.

What are my options?

Answer 1

See Debugging node issues using toolbox

Although Container-Optimized OS does not include a package manager, you can use the pre-installed toolbox utility to install any additional packages or tools you require. Using /usr/bin/toolbox is the preferred method for installing and running one-off debugging tools.

Once you've invoked the toolbox utility to start the shell, you can use apt-get inside the resulting container to install packages.

Answer 2

You can check the Node Problem Detector agent to monitor the system health of COS instances

You can check here the list of supported Disk, Throughput and CPU metrics, Do note that some of the metrics are still on Alpha and Beta phase