Skip to content

ResourceSet CRD

ResourceSet is a declarative API for generating a group of Kubernetes objects based on a matrix of input values and a set of templated resources.

Example

The following example shows a ResourceSet that generates an application instance consisting of a Flux HelmRelease and OCIRepository for each tenant with a specific version and replica count.

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: podinfo
  namespace: default
  annotations:
    fluxcd.controlplane.io/reconcile: "enabled"
    fluxcd.controlplane.io/reconcileEvery: "30m"
    fluxcd.controlplane.io/reconcileTimeout: "5m"
spec:
  commonMetadata:
    labels:
      app.kubernetes.io/name: podinfo
  inputs:
    - tenant: "team1"
      app:
       version: "6.7.x"
       replicas: 2
    - tenant: "team2"
      app:
       version: "6.6.x"
       replicas: 3
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1beta2
      kind: OCIRepository
      metadata:
        name: podinfo-<< inputs.tenant >>
        namespace: default
      spec:
        interval: 10m
        url: oci://ghcr.io/stefanprodan/charts/podinfo
        ref:
          semver: << inputs.app.version | quote >>
    - apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: podinfo-<< inputs.tenant >>
        namespace: default
      spec:
        interval: 1h
        releaseName: podinfo-<< inputs.tenant >>
        chartRef:
          kind: OCIRepository
          name: podinfo-<< inputs.tenant >>
        values:
          replicaCount: << inputs.app.replicas | int >>

You can run this example by saving the manifest into podinfo.yaml.

1. Apply the ResourceSet on the cluster:

kubectl apply -f podinfo.yaml

2. Wait for the ResourceSet to reconcile the generated resources:

kubectl wait resourceset/podinfo --for=condition=ready --timeout=5m

3. Run kubectl get resourceset to see the status of the resource:

$ kubectl get resourceset
NAME      AGE   READY   STATUS
podinfo   59s   True    Reconciliation finished in 52s

4. Run kubectl describe resourceset to see the reconciliation status conditions and events:

$ kubectl describe resourceset podinfo
Status:
  Conditions:
    Last Transition Time:  2024-09-24T09:58:53Z
    Message:               Reconciliation finished in 52s
    Observed Generation:   1
    Reason:                ReconciliationSucceeded
    Status:                True
    Type:                  Ready
Events:
  Type    Reason          Age   From           Message
  ----    ------          ----  ----           -------
  Normal  ApplySucceeded  72s   flux-operator  HelmRelease/default/podinfo-team1 created
                                               HelmRelease/default/podinfo-team2 created
                                               OCIRepository/default/podinfo-team1 created
                                               OCIRepository/default/podinfo-team2 created
  Normal  ReconciliationSucceeded  72s  flux-operator  Reconciliation finished in 52s

5. Run kubectl events to see the events generated by the flux-operator:

kubectl events --for resourceset/podinfo

6. Run kubectl delete to remove the ResourceSet and its generated resources:

kubectl delete resourceset podinfo

Writing a ResourceSet spec

As with all other Kubernetes config, a ResourceSet needs apiVersion, kind, metadata.name and metadata.namespace fields. The name of a ResourceSet object must be a valid DNS subdomain name. A ResourceSet also needs a .spec section.

Inputs configuration

The .spec.inputs field is optional and specifies a list of input values to be used in the resources templates.

An input value is a key-value pair of strings and structs, where the key is the input name which can be referenced in the resource templates using the << inputs.name >> syntax.

Example of static inputs:

spec:
  inputs:
   - tenant: team1
     role: restricted
   - tenant: team2
     role: privileged

The .spec.inputsFrom field is optional and specifies a list of ResourceSetInputProvider objects that provide dynamic input values to the ResourceSet.

Example of dynamic inputs generated from GitHub Pull Requests:

spec:
  inputsFrom:
    - apiVersion: fluxcd.controlplane.io/v1
      kind: ResourceSetInputsProvider
      name: podinfo-pull-requests

At runtime, the operator will fetch the input values every time the ResourceSetInputProvider reconciler detects a change in the upstream source.

When both .spec.inputs and .spec.inputsFrom are set, the resulting inputs are the union of the two.

Resources configuration

The .spec.resources field is optional and specifies the list of Kubernetes resource to be generated and reconciled on the cluster.

The resources can be templated using the << inputs.name >> syntax. The templating engine is based on Go text template. The << >> delimiters are used instead of {{ }} to avoid conflicts with Helm templating and allow ResourceSets to be included in Helm charts.

Example of templated resources:

spec:
  inputs:
    - tenant: team1
      role: admin
    - tenant: team2
      role: cluster-admin
  resources:
    - apiVersion: v1
      kind: Namespace
      metadata:
        name: << inputs.tenant >>
    - apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: flux
        namespace: << inputs.tenant >>
    - apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: flux
        namespace: << inputs.tenant >>
      subjects:
        - kind: ServiceAccount
          name: flux
          namespace: << inputs.tenant >>
          roleRef:
            kind: ClusterRole
            name: << inputs.role >>
            apiGroup: rbac.authorization.k8s.io

The above example will generate a Namespace, ServiceAccount and RoleBinding for each tenant with the specified role.

Templating functions

The templating engine supports slim-sprig functions.

It is recommended to use the quote function when templating strings to avoid issues with special characters e.g. << inputs.version | quote >>.

When templating integers, use the int function to convert the string to an integer e.g. << inputs.replicas | int >>.

When templating booleans, use the bool function to convert the string to a boolean e.g. << inputs.enabled | bool >>.

When using integer or boolean inputs as metadata label values, use the quote function to convert the value to a string e.g. << inputs.enabled | quote >>.

When templating nested fields, use the toYaml and nindent functions to properly format the string e.g.:

spec:
  inputs:
    - tenant: team1
      layerSelector:
        mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
        operation: copy
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1beta2
      kind: OCIRepository
      metadata:
        name: << inputs.tenant >>
      spec:
        layerSelector: << inputs.layerSelector | toYaml | nindent 4 >>

In addition to the slim-sprig functions, a slugify function is available to normalize a string for use in a Kubernetes label value e.g. << inputs.tenant | slugify >>.

Resource deduplication

The flux-operator deduplicates resources based on the apiVersion, kind, metadata.name and metadata.namespace fields.

This allows defining shared resources that are applied only once, regardless of the number of inputs.

Example of a shared Flux source:

spec:
  inputs:
    - tenant: "team1"
      replicas: "2"
    - tenant: "team2"
      replicas: "3"
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1beta2
      kind: OCIRepository
      metadata:
        name: podinfo
        namespace: default
      spec:
        interval: 10m
        url: oci://ghcr.io/stefanprodan/charts/podinfo
        ref:
          semver: '*'
    - apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: podinfo-<< inputs.tenant >>
        namespace: default
      spec:
        interval: 1h
        releaseName: podinfo-<< inputs.tenant >>
        chartRef:
          kind: OCIRepository
          name: podinfo
        values:
          replicaCount: << inputs.replicas | int >>

In the above example, the OCIRepository resource is created only once and referred by all HelmRelease resources.

Conditionally resource exclusion

To exclude a resource based on input values, the fluxcd.controlplane.io/reconcile annotation can be set to disabled on the resource metadata. This will prevent the resource from being reconciled by the operator.

Example of excluding a resource based on an input value:

spec:
  inputs:
    - tenant: "team1"
    - tenant: "team2"
  resources:
    - apiVersion: v1
      kind: Namespace
      metadata:
        name: << inputs.tenant >>
    - apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: flux
        namespace: << inputs.tenant >>
        annotations:
          fluxcd.controlplane.io/reconcile: << if eq inputs.tenant "team1" >>enabled<< else >>disabled<< end >>

In the above example, the ServiceAccount resource is generated only for the team1 tenant.

Resources template

The .spec.resourcesTemplate field is optional and offers an alternative to the .spec.resources. The .spec.resourcesTemplate is a single string that contains the multi-document YAML of the resources definitions. This field can be used for complex templating scenarios with the trade-off of reduced readability.

Note that when both .spec.resources and .spec.resourcesTemplate are set, the resulting resources are the union of the two. If duplicate resources are defined in both fields, the resources from .spec.resources take precedence.

Example of a template containing conditional and repeated blocks:

spec:
  inputs:
    - bundle: addons
      decryption: false
      components:
        - ingress-nginx
        - cert-manager
    - bundle: apps
      decryption: true
      components:
        - frontend
        - backend
  resourcesTemplate: |
    ---
    apiVersion: source.toolkit.fluxcd.io/v1beta2
    kind: OCIRepository
    metadata:
      name: << inputs.bundle >>
      namespace: flux-system
    spec:
      interval: 10m
      url: oci://registry.example.com/<< inputs.bundle >>
    <<- range $component := inputs.components >>
    ---
    apiVersion: kustomize.toolkit.fluxcd.io/v1
    kind: Kustomization
    metadata:
      name: << $component >>
      namespace: flux-system
    spec:
      interval: 1h
      prune: true
      <<- if inputs.decryption >>
      decryption:
        provider: sops
        secretRef:
          name: << inputs.bundle >>-sops
      <<- end >>
      sourceRef:
        kind: OCIRepository
        name: << inputs.bundle >>
      path: ./<< $component >>
    <<- end >>

The above example generates two OCIRepository resources (one for each bundle) and four Kustomization resources (one for each component in each bundle).

Common metadata

The .spec.commonMetadata field is optional and specifies common metadata to be applied to all resources.

It has two optional fields:

  • labels: A map used for setting labels on an object. Any existing label will be overridden if it matches with a key in this map.
  • annotations: A map used for setting annotations on an object. Any existing annotation will be overridden if it matches with a key in this map.

Example common metadata:

spec:
  commonMetadata:
    labels:
      app.kubernetes.io/name: podinfo
    annotations:
      fluxcd.controlplane.io/prune: disabled

In the above example, all resources generated by the ResourceSet will not be pruned by the garbage collection process as the fluxcd.controlplane.io/prune annotation is set to disabled.

Dependency management

.spec.dependsOn is an optional list used to refer to Kubernetes objects that the ResourceSet depends on. If specified, then the ResourceSet is reconciled after the referred objects exist in the cluster.

A dependency is a reference to a Kubernetes object with the following fields:

  • apiVersion: The API version of the referred object (required).
  • kind: The kind of the referred object (required).
  • name: The name of the referred object (required).
  • namespace: The namespace of the referred object (optional).
  • ready: A boolean indicating if the referred object must have the Ready status condition set to True (optional, default is false).
  • readyExpr: A CEL expression that evaluates to a boolean indicating if the referred object is ready (optional).

Example of conditional reconciliation based on the existence of CustomResourceDefinitions and the readiness of a ResourceSet:

spec:
  dependsOn:
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: helmreleases.helm.toolkit.fluxcd.io
      ready: true
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: servicemonitors.monitoring.coreos.com
    - apiVersion: fluxcd.controlplane.io/v1
      kind: ResourceSet
      name: cluster-addons
      namespace: flux-system
      ready: true

Note that is recommended to define dependencies on CustomResourceDefinitions if the ResourceSet deploys Flux HelmReleases which contain custom resources.

When the dependencies are not met, the flux-operator will reevaluate the requirements every five seconds and reconcile the ResourceSet when the dependencies are satisfied. Failed dependencies are reported in the ResourceSet Ready status condition, in log messages and Kubernetes events.

CEL readiness expressions

The readyExpr field allows for more complex readiness checks and can be used for gating the reconciliation of a ResourceSet based on the evaluation of the CEL expression.

The expression is evaluated in the context of the referred object and has access to all the fields of the object, including the status conditions and the status subfields. The expression must evaluate to a boolean value, any syntax or runtime errors will be reported in the ResourceSet status conditions.

Example readiness expression:

spec:
  dependsOn:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: my-cluster
      namespace: dev
      ready: true
      readyExpr: |
        metadata.generation == status.observedGeneration &&
        status.controlPlaneReady == true
    - apiVersion: v1
      kind: Secret
      name: my-gate
      namespace: dev
      ready: true
      readyExpr: |
        string(base64.decode(data.gate)) == 'opened'

For testing the CEL expressions, you can use the CEL playground.

Reconciliation configuration

The reconciliation of behaviour of a ResourceSet can be configured using the following annotations:

  • fluxcd.controlplane.io/reconcile: Enable or disable the reconciliation loop. Default is enabled, set to disabled to pause the reconciliation.
  • fluxcd.controlplane.io/reconcileEvery: Set the reconciliation interval used for drift detection and correction. Default is 1h.
  • fluxcd.controlplane.io/reconcileTimeout: Set the reconciliation timeout including health checks. Default is 5m.

Health check configuration

The .spec.wait field is optional and instructs the flux-operator to perform a health check on all applied resources and waits for them to become ready. The health check is disabled by default and can be enabled by setting the .spec.wait field to true.

The health check is performed for the following resources types:

  • Kubernetes built-in kinds: Deployment, DaemonSet, StatefulSet, PersistentVolumeClaim, Service, Ingress, CustomResourceDefinition.
  • Flux kinds: HelmRelease, OCIRepository, Kustomization, GitRepository, etc.
  • Custom resources that are compatible with kstatus.

By default, the wait timeout is 5m and can be changed with the fluxcd.controlplane.io/reconcileTimeout annotation, set on the ResourceSet object.

Role-based access control

The .spec.serviceAccountName field is optional and specifies the name of the Kubernetes ServiceAccount used by the flux-operator to reconcile the ResourceSet. The ServiceAccount must exist in the same namespace as the ResourceSet and must have the necessary permissions to create, update and delete the resources defined in the ResourceSet.

On multi-tenant clusters, it is recommended to use a dedicated ServiceAccount per tenant namespace with the minimum required permissions. To enforce a ServiceAccount for all ResourceSets, the --default-service-account=flux-operatorflag can be set in the flux-operator container arguments. With this flag set, only the ResourceSets created in the same namespace as the flux-operator will run with cluster-admin permissions.

Garbage collection

The operator performs garbage collection of the resources previously generated by a ResourceSet that are no longer present in the current revision. Garbage collection is also performed when a ResourceSet object is deleted, triggering a removal of all Kubernetes objects previously applied on the cluster.

The garbage collection process removes stale resources in stages, first it deletes the Flux custom resources and waits for the Flux Kustomizations and HelmReleases to be finalized by the controllers. After the Flux resources are removed (max wait is one minute), the operator proceeds with the deletion of the remaining Kubernetes objects. This ensures that the Flux controllers have a chance to clean up the resources they manage before the operator deletes the Kubernetes ServiceAccount and RoleBinding used by Flux impersonation.

After the garbage collection process is completed, the operator issues a Kubernetes event containing the list of removed resources and the duration of the cleanup.

The garbage collection is enabled by default and can be disabled for certain resources by setting the fluxcd.controlplane.io/prune annotation to disabled. To fully disable the garbage collection for a ResourceSet, the annotation must be set on all resources using the .spec.commonMetadata field.

ResourceSet Status

Conditions

A ResourceSet enters various states during its lifecycle, reflected as Kubernetes Conditions. It can be reconciling while applying the resources on the cluster, it can be ready, or it can fail during reconciliation.

The ResourceSet API is compatible with the kstatus specification, and reports Reconciling and Stalled conditions where applicable to provide better (timeout) support to solutions polling the ResourceSet to become Ready.

Reconciling ResourceSet

The flux-operator marks a ResourceSet as reconciling when it starts the reconciliation of the same. The Condition added to the ResourceSet's .status.conditions has the following attributes:

  • type: Reconciling
  • status: "True"
  • reason: Progressing | reason: ProgressingWithRetry

The Condition message is updated during the course of the reconciliation to report the action being performed at any particular moment such as building manifests, detecting drift, etc.

The Ready Condition's status is also marked as Unknown.

Ready ResourceSet

The flux-operator marks a ResourceSet as ready when the resources were built and applied on the cluster and all health checks are observed to be passing.

When the ResourceSet is "ready", the flux-operator sets a Condition with the following attributes in the ResourceSet’s .status.conditions:

  • type: Ready
  • status: "True"
  • reason: ReconciliationSucceeded

Failed ResourceSet

The flux-operator may get stuck trying to reconcile and apply a ResourceSet without completing. This can occur due to some of the following factors:

  • The dependencies are not ready.
  • The templating of the resources fails.
  • The resources are invalid and cannot be applied.
  • Garbage collection fails.
  • Running health checks fails.

When this happens, the flux-operator sets the Ready Condition status to False and adds a Condition with the following attributes to the ResourceSet’s .status.conditions:

  • type: Ready
  • status: "False"
  • reason: DependencyNotReady | BuildFailed | ReconciliationFailed | HealthCheckFailed

The message field of the Condition will contain more information about why the reconciliation failed.

While the ResourceSet has one or more of these Conditions, the flux-operator will continue to attempt a reconciliation with an exponential backoff, until it succeeds and the ResourceSet is marked as ready.

Inventory status

In order to perform operations such as drift detection, garbage collection, upgrades, etc., the flux-operator needs to keep track of all Kubernetes objects that are reconciled as part of a ResourceSet. To do this, it maintains an inventory containing the list of Kubernetes resource object references that have been successfully applied and records it in .status.inventory. The inventory records are in the format Id: <namespace>_<name>_<group>_<kind>, V: <version>.

Example:

Status:
  Inventory:
    Entries:
      Id: default_podinfo__ServiceAccount
      V:  v1
      Id: default_podinfo__Service
      V:  v1
      Id: default_podinfo_apps_Deployment
      V:  v1