Flux Multitenant Policies
In this guide we will cover how to use the Kubernetes built-in validating admission engine to enforce policies on multi-tenant clusters managed by Flux.
Kubernetes Validating Admission Policy
Starting from Kubernetes 1.30, the Kubernetes API server comes with a built-in validating admission controller that allows defining custom policies using the Common Expression Language (CEL).
In a GitOps setup, the Kubernetes ValidatingAdmissionPolicy
and ValidatingAdmissionPolicyBinding
resources are defined by cluster admins in the fleet repository. Using Flux dependsOn
feature,
the cluster admins can ensure that the policies are applied before any other resources are reconciled,
thus enforcing the policies on all tenant's namespaces and workloads.
Restricting Access to Flux Sources
One common policy is to restrict the access to external Flux sources such as Git repositories, OCI registries and Helm repositories which are controlled by tenants.
Define the Allow List
Assuming that cluster admins want to restrict the access to a specific GitHub organization,
first, they need to define ConfigMap
in the flux-system
namespace
with the list of allowed Flux sources:
apiVersion: v1
kind: ConfigMap
metadata:
name: allowlist
namespace: flux-system
labels:
fluxcd.controlplane.io/role: "policy"
data:
sources: >-
oci://ghcr.io/controlplaneio-fluxcd/charts/
https://github.com/controlplaneio-fluxcd/
ssh://[email protected]/controlplaneio-fluxcd/
The sources
list contains the allowed URL prefixes separated by a new line for
tenant-owned Flux sources e.g. GitRepository
, OCIRepository
and HelmRepository
.
Define the Admission Policy
Next, the cluster admins need to define a ValidatingAdmissionPolicy
resource that
verifies the source URL against the allow list:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: "source.policy.fluxcd.controlplane.io"
annotations:
policy.fluxcd.controlplane.io/role: |
Restrict Flux access to Git repositories, OCI registries and Helm repositories,
based on an allowlist defined in a ConfigMap stored in the flux-system namespace.
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [ "source.toolkit.fluxcd.io" ]
apiVersions: [ "*" ]
operations: [ "CREATE", "UPDATE" ]
resources: [ "gitrepositories", "ocirepositories", "helmrepositories" ]
matchConditions:
- name: "exclude-source-controller-finalizer"
expression: >
request.userInfo.username != "system:serviceaccount:flux-system:source-controller"
paramKind:
apiVersion: v1
kind: ConfigMap
variables:
- name: url
expression: object.spec.url
- name: sources
expression: params.data.sources.split(' ')
validations:
- expression: >
variables.sources.exists_one(prefix, variables.url.startsWith(prefix))
messageExpression: >
"Source " + variables.url + " is not allowed, must be one of " + variables.sources.join(", ")
reason: Invalid
Define the Admission Policy Binding
Finally, the cluster admins need to define a ValidatingAdmissionPolicyBinding
resource
that binds the policy to all tenant namespaces and references the ConfigMap
with the allow list:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: tenant-sources
spec:
policyName: "source.policy.fluxcd.controlplane.io"
validationActions: [ "Deny" ]
paramRef:
name: allowlist
namespace: flux-system
parameterNotFoundAction: "Deny"
matchResources:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- flux-system
- kube-system
With the above policy in place, any tenant trying to create or update a Flux source that is not listed in the allow list will receive a validation error and the operation will be denied.
Applying the Policies
The cluster admins can apply the policies using a dedicated Flux Kustomization
that gets reconciled
before the tenant's resources.
Repository structure:
├── clusters
│ └── production
│ ├── policies-sync.yaml
│ └── tenants-sync.yaml
├── policies
│ ├── allowlist.yaml
│ └── policies.yaml
└── tenants
├── team1
└── team2
The policies-sync.yaml
manifest configures Flux to reconcile the policies resources first:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: policies
namespace: flux-system
spec:
path: "./policies"
prune: true
wait: true
The tenants-sync.yaml
manifest configures Flux to reconcile the tenants resources after the policies:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: tenants
namespace: flux-system
spec:
dependsOn:
- name: policies
path: "./tenants"
prune: true
Testing the Policy
If a tenant adds an HelmRepository
manifest to their repository that tries to
pull Helm charts from a registry that is not in the allow list, for example:
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: podinfo
namespace: apps
spec:
type: oci
url: oci://ghcr.io/stefanprodan/charts/
The admission controller will deny the creation of the HelmRepository
and the tenant will receive
an alert from Flux about the policy violation:
The helmrepository "podinfo" is invalid:
ValidatingAdmissionPolicy 'source.policy.fluxcd.controlplane.io' with binding 'tenant-sources' denied request:
Source oci://ghcr.io/stefanprodan/charts/ is not allowed, must be one of oci://ghcr.io/controlplaneio-fluxcd/charts/, https://github.com/controlplaneio-fluxcd/, ssh://[email protected]/controlplaneio-fluxcd/
Restricting Access to Container Registries
Another common policy is to restrict the access to container registries for workloads running in tenant namespaces.
Assuming that cluster admins want to restrict the access to specific container registries,
they can add the allowed registry prefixes to the ConfigMap
in the flux-system
namespace:
apiVersion: v1
kind: ConfigMap
metadata:
name: allowlist
namespace: flux-system
labels:
fluxcd.controlplane.io/role: "policy"
data:
registries: >-
ghcr.io/controlplaneio-fluxcd/
709825985650.dkr.ecr.us-east-1.amazonaws.com/controlplane/
sources: >-
omitted for brevity
Next, the cluster admins need to define a ValidatingAdmissionPolicy
resource that
verifies the containers image URL against the allow list:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: "registry.policy.fluxcd.controlplane.io"
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [ "apps" ]
apiVersions: [ "v1" ]
operations: [ "CREATE", "UPDATE" ]
resources: [ "deployments", "statefulsets", "daemonsets" ]
paramKind:
apiVersion: v1
kind: ConfigMap
variables:
- name: registries
expression: params.data.registries.split(' ')
validations:
- expression: >
object.spec.template.spec.containers.all(
container,
variables.registries.exists(
prefix, container.image.startsWith(prefix)
)
)
messageExpression: >
"Container image is not allowed, must be one of " + variables.registries.join(", ")
reason: Invalid
- expression: >
!has(object.spec.template.spec.initContainers) ||
object.spec.template.spec.initContainers.all(
container,
variables.registries.exists(
prefix, container.image.startsWith(prefix)
)
)
messageExpression: >
"Init container image is not allowed, must be one of " + variables.registries.join(", ")
reason: Invalid
Finally, the cluster admins need to define a ValidatingAdmissionPolicyBinding
resource
that binds the policy to all tenant namespaces and references the ConfigMap
with the allow list:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: tenant-registries
spec:
policyName: "registry.policy.fluxcd.controlplane.io"
validationActions: [ "Deny" ]
paramRef:
name: allowlist
namespace: flux-system
parameterNotFoundAction: "Deny"
matchResources:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- flux-system
- kube-system
With the above policy in place, any tenant trying to create or update a workload that uses a container image from a registry that is not listed in the allow list will receive a validation error and the operation will be denied.
The deployments "test-deployment" is invalid:
ValidatingAdmissionPolicy 'registry.policy.fluxcd.controlplane.io' with binding 'tenant-registries' denied request:
Init container image is not allowed, must be one of ghcr.io/controlplaneio-fluxcd/, 709825985650.dkr.ecr.us-east-1.amazonaws.com/controlplane/
Conclusions
In this guide we've shown how to use the Kubernetes native validating admission policies to enforce policies on multi-tenant clusters managed by Flux. Extending the examples above, cluster admins can define policies to enforce best practices, security and compliance requirements across all tenant namespaces and workloads.
The new Kubernetes validating admission policies offer an alternative to admission webhooks such as OPA Gatekeeper or Kyverno. Admission webhooks are single points of failure and can introduce latency in the API server response time, blocking or slowing down the reconciliation of the cluster state performed by Flux. Using the built-in policy engine, eliminates the maintenance overhead of running and managing custom admission webhooks and makes policy enforcement more reliable and efficient.