TechQA.

image attestation using Kyverno not working

79 views Asked by At

I am writing a simple test to verify that Kyverno is able to block images without attestation from being deployed in k3s cluster.

https://github.com/whoissqr/cg-test-keyless-sign

I have the following ClusterPolicy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-image-keyless
spec:
  validationFailureAction: Enforce
  failurePolicy: Fail
  webhookTimeoutSeconds: 30
  rules:
    - name: check-image-keyless
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - imageReferences:
        - "ghcr.io/whoissqr/cg-test-keyless-sign"
        attestors:
        - entries:
          - keyless:
              subject: "https://github.com/whoissqr/cg-test-keyless-sign/.github/workflows/main.yml@refs/heads/main"
              issuer: "https://token.actions.githubusercontent.com"
              rekor:
                url: https://rekor.sigstore.dev

and the following pod yaml

apiVersion: v1
kind: Pod
metadata:
  name: cg
  namespace: app
spec:
  containers:
    - image: ghcr.io/whoissqr/cg-test-keyless-sign
      name: cg-test-keyless-sign
      resources: {}

And, I purposely commented out the image cosign step in Github action so that the cosign verify failed as expected, but the pod deployment to k3s is still succeeded. What am I missing here?

name: Publish and Sign Container Image

on:
  schedule:
    - cron: '32 11 * * *'
  push:
    branches: [ main ]
    # Publish semver tags as releases.
    tags: [ 'v*.*.*' ]
  pull_request:
    branches: [ main ]

jobs:
  build:

    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Install cosign
        uses: sigstore/[email protected]
          
      - name: Check install!
        run: cosign version
        
      - name: Setup Docker buildx
        uses: docker/setup-buildx-action@v2

      - name: Log into ghcr.io
        uses: docker/login-action@master
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push container image
        id: push-step
        uses: docker/build-push-action@master
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:latest

      - name: Sign the images with GitHub OIDC Token
        env:
          DIGEST: ${{ steps.push-step.outputs.digest }}
          TAGS: ghcr.io/${{ github.repository }}
          COSIGN_EXPERIMENTAL: "true"
        run: |
          echo "dont sign image"
          # cosign sign --yes "${TAGS}@${DIGEST}"
        
      - name: Verify the images
        run: |
          cosign verify ghcr.io/whoissqr/cg-test-keyless-sign \
             --certificate-identity https://github.com/whoissqr/cg-test-keyless-sign/.github/workflows/main.yml@refs/heads/main \
             --certificate-oidc-issuer https://token.actions.githubusercontent.com | jq

      - name: Create k3s cluster
        uses: debianmaster/actions-k3s@master
        id: k3s
        with:
          version: 'latest'
          
      - name: Install Kyverno chart
        run: |
          helm repo add kyverno https://kyverno.github.io/kyverno/
          helm repo update
          helm install kyverno kyverno/kyverno -n kyverno --create-namespace

      - name: Apply image attestation policy
        run: |
          kubectl apply -f ./k3s/policy-check-image-keyless.yaml
          
      - name: Deploy pod to k3s
        run: |
          set -x
          # kubectl get nodes
          kubectl create ns app
          sleep 20
          # kubectl get pods -n app
          kubectl apply -f ./k3s/pod.yaml
          kubectl -n app wait --for=condition=Ready pod/cg
          kubectl get pods -n app
          kubectl -n app describe pod cg
          kubectl get polr -o wide

      - name: Install Kyverno CLI
        uses: kyverno/[email protected]
        with:
          release: 'v1.9.5'
          
      - name: Check policy using Kyverno CLI
        run: |
          kyverno version
          kyverno apply ./k3s/policy-check-image-keyless.yaml --cluster -v 10

in the GH action console

+ kubectl apply -f ./k3s/pod.yaml
pod/cg created
+ kubectl -n app wait --for=condition=Ready pod/cg
pod/cg condition met
+ kubectl get pods -n app
NAME   READY   STATUS    RESTARTS   AGE
cg     1/1     Running   0          12s

and the kyverno CLI output has

I0225 10:00:31.650505    6794 common.go:424]  "msg"="applying policy on resource" "policy"="check-image-keyless" "resource"="app/Pod/cg"
I0225 10:00:31.652646    6794 context.go:278]  "msg"="updated image info" "images"={"containers":{"cg-test-keyless-sign":{"registry":"ghcr.io","name":"cg-test-keyless-sign","path":"whoissqr/cg-test-keyless-sign","tag":"latest"}}}
I0225 10:00:31.654017    6794 utils.go:29]  "msg"="applied JSON patch" "patch"=[{"op":"replace","path":"/spec/containers/0/image","value":"ghcr.io/whoissqr/cg-test-keyless-sign:latest"}]
I0225 10:00:31.659697    6794 mutation.go:39] EngineMutate "msg"="start mutate policy processing" "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "startTime"="2024-02-25T10:00:31.659674165Z"
I0225 10:00:31.659737    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.659815    6794 rule.go:286] autogen "msg"="generating rule for cronJob" 
I0225 10:00:31.659834    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.659940    6794 mutation.go:379] EngineMutate "msg"="finished processing policy" "kind"="Pod" "mutationRulesApplied"=0 "name"="cg" "namespace"="app" "policy"="check-image-keyless" "processingTime"="249.225µs"
I0225 10:00:31.659966    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660040    6794 rule.go:286] autogen "msg"="generating rule for cronJob" 
I0225 10:00:31.660059    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660153    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660218    6794 rule.go:286] autogen "msg"="generating rule for cronJob" 
I0225 10:00:31.660236    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660337    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660402    6794 rule.go:286] autogen "msg"="generating rule for cronJob" 
I0225 10:00:31.660421    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660648    6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"pods","singularName":"pod","namespaced":true,"version":"v1","kind":"Pod","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["po"],"categories":["all"]} "kind"="Pod"
I0225 10:00:31.660729    6794 imageVerify.go:121] EngineVerifyImages "msg"="processing image verification rule" "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "ruleSelector"="All"
I0225 10:00:31.660889    6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"daemonsets","singularName":"daemonset","namespaced":true,"group":"apps","version":"v1","kind":"DaemonSet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ds"],"categories":["all"]} "kind"="DaemonSet"
I0225 10:00:31.661037    6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"deployments","singularName":"deployment","namespaced":true,"group":"apps","version":"v1","kind":"Deployment","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["deploy"],"categories":["all"]} "kind"="Deployment"
I0225 10:00:31.661184    6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"jobs","singularName":"job","namespaced":true,"group":"batch","version":"v1","kind":"Job","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"]} "kind"="Job"
I0225 10:00:31.661327    6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"statefulsets","singularName":"statefulset","namespaced":true,"group":"apps","version":"v1","kind":"StatefulSet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sts"],"categories":["all"]} "kind"="StatefulSet"
I0225 10:00:31.661465    6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"replicasets","singularName":"replicaset","namespaced":true,"group":"apps","version":"v1","kind":"ReplicaSet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rs"],"categories":["all"]} "kind"="ReplicaSet"
I0225 10:00:31.661606    6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"replicationcontrollers","singularName":"replicationcontroller","namespaced":true,"version":"v1","kind":"ReplicationController","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rc"],"categories":["all"]} "kind"="ReplicationController"
I0225 10:00:31.661789    6794 validation.go:591] EngineVerifyImages "msg"="resource does not match rule" "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "reason"="rule autogen-check-image-keyless not matched:\n 1. no resource matched"
I0225 10:00:31.661938    6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"cronjobs","singularName":"cronjob","namespaced":true,"group":"batch","version":"v1","kind":"CronJob","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["cj"],"categories":["all"]} "kind"="CronJob"
I0225 10:00:31.662056    6794 validation.go:591] EngineVerifyImages "msg"="resource does not match rule" "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "reason"="rule autogen-cronjob-check-image-keyless not matched:\n 1. no resource matched"
I0225 10:00:31.662091    6794 imageVerify.go:83] EngineVerifyImages "msg"="processed image verification rules" "applied"=0 "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "successful"=true "time"="1.748335ms"
I0225 10:00:31.662113    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662189    6794 rule.go:286] autogen "msg"="generating rule for cronJob" 
I0225 10:00:31.662208    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662302    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662368    6794 rule.go:286] autogen "msg"="generating rule for cronJob" 
I0225 10:00:31.662385    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662481    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662544    6794 rule.go:286] autogen "msg"="generating rule for cronJob" 
I0225 10:00:31.662577    6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"

thanks!

==== edit 02/25/2024 8:24 PM =====

I noticed one small typo, and added tag 'latest' to image reference in the policy yaml, and now the 'kyverno apply' step failed as expected


policy check-image-keyless -> resource app/Pod/cg failed: 
1. check-image-keyless: failed to verify image ghcr.io/whoissqr/cg-test-keyless-sign:latest: .attestors[0].entries[0].keyless: no matching signatures:
....
pass: 0, fail: 1, warn: 0, error: 0, skip: 98 
Error: Process completed with exit code 1.

However, my 'kubectl apply -f ./k3s/pod.yaml' statement in previous step still proceed without error and pod are still created and running.

why?

==== 2nd edit =====

we need to add the following to the policy

background: false
kubernetes k3s attestations kyverno
Original Q&A
1

There are 1 answers

0
sqr On BEST ANSWER 
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-image-keyless
spec:
  validationFailureAction: Enforce
  failurePolicy: Fail
  background: false
  webhookTimeoutSeconds: 30
  rules:
    - name: check-image-keyless
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - imageReferences:
        - "ghcr.io/whoissqr/cg-test-keyless-sign:latest"
        attestors:
        - entries:
          - keyless:
              subject: "https://github.com/whoissqr/cg-test-keyless-sign/.github/workflows/main.yml@refs/heads/main"
              issuer: "https://token.actions.githubusercontent.com"
              rekor:
                url: https://rekor.sigstore.dev

pod yaml

apiVersion: v1
kind: Pod
metadata:
  name: cg
  namespace: app
spec:
  containers:
    - image: ghcr.io/whoissqr/cg-test-keyless-sign:latest
      name: cg-test-keyless-sign
      resources: {}

Github action

name: Publish and Sign Container Image

on:
  schedule:
    - cron: '32 11 * * *'
  push:
    branches: [ main ]
    # Publish semver tags as releases.
    tags: [ 'v*.*.*' ]
  pull_request:
    branches: [ main ]

jobs:
  build:

    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Install cosign
        uses: sigstore/[email protected]
          
      - name: Check install!
        run: cosign version
        
      - name: Setup Docker buildx
        uses: docker/setup-buildx-action@v2

      - name: Log into ghcr.io
        uses: docker/login-action@master
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push container image
        id: push-step
        uses: docker/build-push-action@master
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:latest

      - name: Sign the images with GitHub OIDC Token
        env:
          DIGEST: ${{ steps.push-step.outputs.digest }}
          TAGS: ghcr.io/${{ github.repository }}
          COSIGN_EXPERIMENTAL: "true"
        run: |
          echo "dont sign image"
          # cosign sign --yes "${TAGS}@${DIGEST}"
        
      - name: (optional) Verify the images
        run: |
          cosign verify ghcr.io/whoissqr/cg-test-keyless-sign \
             --certificate-identity https://github.com/whoissqr/cg-test-keyless-sign/.github/workflows/main.yml@refs/heads/main \
             --certificate-oidc-issuer https://token.actions.githubusercontent.com | jq

      - name: Create k3s cluster
        uses: debianmaster/actions-k3s@master
        id: k3s
        with:
          version: 'latest'
          
      - name: Install Kyverno chart
        run: |
          helm repo add kyverno https://kyverno.github.io/kyverno/
          helm repo update
          helm install --atomic kyverno kyverno/kyverno -n kyverno --create-namespace
          sleep 10

      - name: Apply image attestation policy
        run: |
          kubectl apply -f ./k3s/policy-check-image-keyless.yaml

      - name: Deploy pod to k3s
        if: always() 
        run: |
          kubectl create ns app
          kubectl apply -f ./k3s/pod.yaml
          kubectl -n app wait --for=condition=Ready pod/cg
          kubectl get pods -n app

      - name: (optional) Install Kyverno CLI
        if: always() 
        uses: kyverno/[email protected]
        with:
          release: 'v1.9.5'
          
      - name: (optional) Dry run policy using Kyverno CLI
        if: always() 
        run: |
          kyverno version
          # kyverno apply ./k3s/policy-check-image-keyless.yaml --cluster -v 10
          kubectl get clusterpolicies -o yaml | kyverno apply - --resource ./k3s/pod.yaml -v 10

Related Questions in KUBERNETES

Related Questions in K3S

Related Questions in ATTESTATIONS

Related Questions in KYVERNO

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Trending Questions