~/makariy
ENRU
← All articles

6 min read

Kyverno: a PodSecurityPolicy replacement, or something more?

PodSecurityPolicy was removed in Kubernetes 1.25. Here is Kyverno as a replacement — validate, mutate and generate policies, on the actual manifests we run in production.

Originally published at

Hello!

Have you heard of Kyverno, and what it is for? In this article I will explain what it does and show, with examples, how we use it.

My name is Makariy Balashov, I am an SRE at Ak Bars Digital. Our team builds and maintains the infrastructure the development teams run on.

So what is this Kyverno of yours?

Kyverno is a policy engine built specifically for Kubernetes.

Kyverno lets cluster administrators manage environment-specific configuration independently of workload configuration, and enforce configuration best practices across their clusters. You can use it to scan existing resources for best-practice violations, or to enforce them by blocking or mutating API requests.

A short list of things Kyverno does well:

  • policies are Kubernetes resources (plain YAML manifests);

  • it can validate, mutate or generate k8s resources;

  • container image verification for software supply chain security;

  • image metadata verification;

  • configuration synchronisation across namespaces;

  • blocking non-compliant resources through admission control, or reporting policy violations instead;

  • policy-as-code with familiar tooling such as git and kustomize;

  • a steadily growing library of ready-made policies (244 at the time of writing).

The alternatives worth knowing about are OPA/Gatekeeper, Kubewarden and jsPolicy.

There is plenty of comparison material online — for example, this write-up compares Kyverno and Gatekeeper.

Why did we need it?

PodSecurityPolicy was deprecated in Kubernetes 1.21 and removed outright in 1.25.

That left us with the question of how to carry on. We tested several options and settled on Kyverno. It made us very happy and covered everything we needed from PSP.

Against the alternatives, Kyverno stood out for how much it does out of the box and for its simple vocabulary — ordinary YAML — which means new team members get productive with it far faster.

How does it work?

Kyverno runs as a dynamic admission controller in the Kubernetes cluster. It receives validating and mutating admission webhooks from kube-apiserver, then applies the matching policies and returns results that either enforce admission policy or reject the request.

What do we use it for?

Initially we planned to use Kyverno purely as a PodSecurityPolicy replacement. We started by rewriting our existing PSP manifests as Kyverno policy manifests with Validate rules.

Validate Resources

For instance, we wanted to forbid privileged containers straight away, because a privileged pod can reach host resources and kernel capabilities.

The policy we use 
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged-containers
  annotations:
    policies.kyverno.io/title: Disallow Privileged Containers
    policies.kyverno.io/category: Pod Security Standards (Baseline)
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    kyverno.io/kyverno-version: 1.6.0
    kyverno.io/kubernetes-version: "1.22-1.23"
    policies.kyverno.io/description: >-
      Privileged mode disables most security mechanisms and must not be allowed. This policy
      ensures Pods do not call for privileged mode.
spec:
  validationFailureAction: audit
  background: true
  rules:
    - name: privileged-containers
      match:
        any:
        - resources:
            kinds:
              - Pod
      validate:
        message: >-
          Privileged mode is disallowed. The fields spec.containers[*].securityContext.privileged
          and spec.initContainers[*].securityContext.privileged must be unset or set to `false`.
        pattern:
          spec:
            =(ephemeralContainers):
              - =(securityContext):
                  =(privileged): "false"
            =(initContainers):
              - =(securityContext):
                  =(privileged): "false"
            containers:
              - =(securityContext):
                  =(privileged): "false"

Mutating Resources

Next we decided to set the securityContext we want on every resource created in our cluster. Here is what we ended up with:

Policy that adds a securityContext to resources 
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-securitycontext
  annotations:
    policies.kyverno.io/title: Add Default securityContext
    policies.kyverno.io/category: Sample
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      A Pod securityContext entry defines fields such as the user and group which should be used to run the Pod.
      Sometimes choosing default values for users rather than blocking is a better alternative to not impede
      such Pod definitions. This policy will mutate a Pod to set default securityContext
spec:
  background: false
  rules:
  - name: add-default-securitycontext-containers
    match:
      any:
      - resources:
          kinds:
          - Pod
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: In
        value:
          - CREATE
          - UPDATE
    mutate:
      foreach:
      - list: "request.object.spec.containers"
        patchStrategicMerge:
          spec:
            securityContext:
              runAsUser: 10001
              runAsGroup: 10001
              fsGroup: 10001
              seccompProfile:
                type: RuntimeDefault
            containers:
            - name: "{{ element.name }}"
              securityContext:
                runAsUser: 10001
                runAsGroup: 10001
                capabilities:
                  drop:
                    - ALL
                runAsNonRoot: true
                allowPrivilegeEscalation: false
                privileged: false
                seccompProfile:
                  type: RuntimeDefault
  - name: add-default-securitycontext-initContainers
    match:
      any:
      - resources:
          kinds:
          - Pod
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: In
        value:
          - CREATE
          - UPDATE
      - key: "{{ request.object.spec.initContainers[] || '' | length(@) }}"
        operator: GreaterThanOrEquals
        value: 1
    mutate:
      foreach:
      - list: "request.object.spec.initContainers"
        patchStrategicMerge:
          spec:
            securityContext:
              runAsUser: 10001
              fsGroup: 10001
              seccompProfile:
                type: RuntimeDefault
            initContainers:
            - name: "{{ element.name }}"
              securityContext:
                runAsUser: 10001
                runAsGroup: 10001
                capabilities:
                  drop:
                    - ALL
                runAsNonRoot: true
                allowPrivilegeEscalation: false
                privileged: false
                seccompProfile:
                  type: RuntimeDefault

Generate Resources

The obvious use for generate policies is copying a private registry credentials secret (dockerconfigjson) into every namespace.

We found another use for it: copying the secret holding the self-signed root certificate that cert-manager generates into every namespace, so another policy can then inject it into containers. Note synchronize — it lets Kyverno automatically update the contents of every resource it created whenever the source changes. In our case, that means the certificate is refreshed in every secret after cert-manager reissues it.

Copying a secret across namespaces 
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: clone-cert
  annotations:
    policies.kyverno.io/title: Clone Certificate's secret
    policies.kyverno.io/category: Cert-Manager
    policies.kyverno.io/subject: Certificate, Namespace
    policies.kyverno.io/description: >-
      Clone certificate's secret to every namespace
spec:
  background: true
  rules:
  - name: clone-cert
    match:
      any:
      - resources:
          kinds:
          - Namespace
    exclude:
      any:
      - resources:
          namespaces:
          - "kube-system"
          - "kube-public"
          - "default"
    generate:
      synchronize: true
      apiVersion: v1
      kind: Secret
      name: base-cert
      namespace: "{{request.object.metadata.name}}"
      clone:
        namespace: cert-manager
        name: base-cert
The policy that performs the injection 
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-certificates-volume
  annotations:
    policies.kyverno.io/title: Add Certificates as a Volume
    policies.kyverno.io/category: Sample
    policies.kyverno.io/subject: Pod,Volume
    kyverno.io/kyverno-version: 1.5.2
    kyverno.io/kubernetes-version: "1.21"
    policies.kyverno.io/minversion: 1.5.0
    pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,Job,StatefulSet
    policies.kyverno.io/description: >-
      In some cases you would need to trust custom CA certificates for all the containers of a Pod.
      It makes sense to be in a Secret so that you can automount them by only setting an annotation.
      This policy adds a volume to all containers in a Pod containing the certificate if the annotation
      called `inject-certs` with value `enabled` is found.
spec:
  background: true
  rules:
  - name: add-ssl-certs
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - "kube-system"
          - "kube-public"
          - "default"
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: In
        value:
          - CREATE
          - UPDATE
    mutate:
      foreach:
      - list: "request.object.spec.containers"
        patchStrategicMerge:
          spec:
            containers:
            - name: "{{ element.name }}"
              volumeMounts:
              - name: ssl-certs
                mountPath: /etc/ssl/certs
            volumes:
            - name: ssl-certs
              secret:
                secretName: base-cert

Verify Image

Kyverno together with Sigstore Cosign can also verify and sign container images. We are only just rolling these features out. A collection of articles on the topic is available in the k8s (in)security Telegram channel.

Conclusion

Kyverno is excellent because it lets us govern our clusters consistently, at the cluster level, without writing real code. Non-compliant resources can be blocked with an error message that actually reaches the user. Misconfigured resources can be fixed on the fly, and new resources can be created dynamically. Managing Kubernetes clusters with this tool has been a genuinely pleasant experience, and we will keep using it.