Extract pod logs during skaffold deploy

Question

We run some automated tests against our app that's deployed locally (to a CI build agent) using k3s and skaffold.

How can we view (stdout or file) the pod logs when we run skaffold deploy --status-check? (We've had errors during deployment that we can't see unless we ssh onto the build agent.)

The following, when run after the deployment was successful, captures logs, but this doesn't work if we run it before running skaffold deploy, presumably because it doesn't pick up any pods that are started after running this command:

kubectl logs \
    --namespace x \
    --follow \
    --ignore-errors \
    --max-log-requests 50 \
    --all-containers \
    --prefix \
    --timestamps \
    --selector release=y >"k8s.log" 2>&1 &

The output from skaffold deploy --status-check looks something like this:

LAST DEPLOYED: Tue Oct 13 18:14:03 2020
NAMESPACE: smoke
STATUS: deployed
REVISION: 1
TEST SUITE: None
Waiting for deployments to stabilize...
 - smoke:deployment/smoke-assets is ready. [7/8 deployment(s) still pending]
 - smoke:deployment/smoke-b is ready. [6/8 deployment(s) still pending]
 - smoke:deployment/smoke-c is ready. [5/8 deployment(s) still pending]
 - smoke:deployment/smoke-d: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - smoke:deployment/smoke-e: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - smoke:deployment/smoke-f: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - smoke:deployment/smoke-g: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - smoke:deployment/smoke-h: waiting for rollout to finish: 0 of 2 updated replicas are available...
 - smoke:deployment/smoke-d is ready. [4/8 deployment(s) still pending]
 - smoke:deployment/smoke-g is ready. [3/8 deployment(s) still pending]
 - smoke:deployment/smoke-e is ready. [2/8 deployment(s) still pending]
 - smoke:deployment/smoke-h: waiting for rollout to finish: 1 of 2 updated replicas are available...
 - smoke:deployment/smoke-h is ready. [1/8 deployment(s) still pending]
1/8 deployment(s) failed
 - smoke:deployment/smoke-f: running [kubectl --context k3d-smoke-tests-cluster rollout status deployment smoke-f --namespace smoke --watch=false]
 - stdout: ""
 - stderr: "error: deployment \"smoke-f\" exceeded its progress deadline\n"
 - cause: exit status 1
 - smoke:deployment/smoke-f failed. Error: running [kubectl --context k3d-smoke-tests-cluster rollout status deployment smoke-f --namespace smoke --watch=false]
 - stdout: ""
 - stderr: "error: deployment \"smoke-f\" exceeded its progress deadline\n"
 - cause: exit status 1.

Answer 1

I found that stern can cope with new pods being added to the namespace.

So, running:

stern -n namespace . >my.log &

before the skaffold deploy does the trick.

Answer 2

@Graham Russell --status-check should logs for the pod which failed if they exist. The only think i could think of is the pod container terminated and was cleaned up before skaffold could grab the logs.

There is currently no way to stream logs to std out before all the deployments stabilize.

Can you paste the current status-check output