A Virtual Machine that we use as a self host agent runned out of space on it's data disk resulting in failure to start agent services and running pipelines. The result was the agent in an idle state, not communicating to Azure DevOps Pipeline service. Extending Disk size of VM agent would help resolve this issue but would increase costs, instead a temporary solution was to delete older build logs from the _diag folder.
Looking inside the VM agent directory we can see the following folders:
Is there any tool that Azure provides to deal with this kind of problem?
Is the "_diag" folder the right one that needs the contents deleted?
Is there any kind of automation that can be used, for example in the pipelines?
Is the Maintenance job functionality of azure a solution for this problem?
You can use maintenance jobs to cleanup the working folders for your builds. Working folders are getting reused on each pipeline run, but if you have multiple pipelines that are not getting used often * maintenance jobs will free you space on the virtual machine.
Maintenance jobs will not remove logs from your agents. However I do not believe the _diag folder consumes much space on the agent, I suggest to focus on cleaning up artifacts and setting mj.
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops&tabs=yaml%2Cbrowser
A detailed explanation for the setup can be found on my article:
https://blog.geralexgr.com/docker/maintenance-jobs-for-build-agents-explained-azure-devops