Flux Instance CRD

FluxInstance is a declarative API for the installation, configuration and automatic upgrade of the Flux distribution.

A single custom resource of this kind can exist in a Kubernetes cluster with the name flux that must be created in the same namespace where the flux-operator is deployed.

Example

The following example shows a FluxInstance custom resource that installs the upstream Flux distribution with all available components, and configures the flux-operator to automatically upgrade Flux to the latest stable version:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
  name: flux
  namespace: flux-system
  annotations:
    fluxcd.controlplane.io/reconcile: "enabled"
    fluxcd.controlplane.io/reconcileEvery: "1h"
    fluxcd.controlplane.io/reconcileTimeout: "3m"
spec:
  distribution:
    version: "2.x"
    registry: "ghcr.io/fluxcd"
  components:
    - source-controller
    - kustomize-controller
    - helm-controller
    - notification-controller
    - image-reflector-controller
    - image-automation-controller
  cluster:
    type: kubernetes
    multitenant: false
    networkPolicy: true
    domain: "cluster.local"
  storage:
    class: "standard"
    size: "10Gi"
  commonMetadata:
    labels:
      app.kubernetes.io/name: flux
  kustomize:
    patches:
      - target:
          kind: Deployment
          name: "(kustomize-controller|helm-controller)"
        patch: |
          - op: add
            path: /spec/template/spec/containers/0/args/-
            value: --concurrent=10
          - op: add
            path: /spec/template/spec/containers/0/args/-
            value: --requeue-dependency=5s

You can run this example by saving the manifest into fluxinstance.yaml.

1. Apply the resource on the cluster:

kubectl apply -f fluxinstance.yaml

1. Run kubectl get fluxinstance to see the status of the resource:

 $ kubectl -n flux-system get fluxinstance
 NAME   AGE   READY   STATUS                           REVISION
 flux   59s   True    Reconciliation finished in 52s   v2.3.0@sha256:4cc5babdb1279ad0177bf513292deadbfa3f7b7c3da0be7fa53b39ab434f7219

1. Run kubectl describe fluxinstance to see the reconciliation status components, conditions and events:

   $ kubectl -n flux-system describe fluxinstance flux
   Status:
     Components:
       Digest:      sha256:161da425b16b64dda4b3cec2ba0f8d7442973aba29bb446db3b340626181a0bc
       Name:        source-controller
       Repository:  ghcr.io/fluxcd/source-controller
       Tag:         v1.3.0
       Digest:      sha256:48a032574dd45c39750ba0f1488e6f1ae36756a38f40976a6b7a588d83acefc1
       Name:        kustomize-controller
       Repository:  ghcr.io/fluxcd/kustomize-controller
       Tag:         v1.3.0
       Digest:      sha256:a67a037faa850220ff94d8090253732079589ad9ff10b6ddf294f3b7cd0f3424
       Name:        helm-controller
       Repository:  ghcr.io/fluxcd/helm-controller
       Tag:         v1.0.1
       Digest:      sha256:c0fab940c7e578ea519097d36c040238b0cc039ce366fdb753947428bbf0c3d6
       Name:        notification-controller
       Repository:  ghcr.io/fluxcd/notification-controller
       Tag:         v1.3.0
       Digest:      sha256:aed795c7a8b85bca93f6d199d5a14bbefaf925ad5aa5316b32a716cfa4070d0b
       Name:        image-reflector-controller
       Repository:  ghcr.io/fluxcd/image-reflector-controller
       Tag:         v0.32.0
       Digest:      sha256:ab5097213194f3cd9f0e68d8a937d94c4fc7e821f6544453211e94815b282aa2
       Name:        image-automation-controller
       Repository:  ghcr.io/fluxcd/image-automation-controller
       Tag:         v0.38.0
     Conditions:
       Last Transition Time:  2024-06-03T12:20:57Z
       Message:               Reconciliation finished in 52s
       Observed Generation:   1
       Reason:                ReconciliationSucceeded
       Status:                True
       Type:                  Ready
     Last Applied Revision:    v2.3.0@sha256:4cc5babdb1279ad0177bf513292deadbfa3f7b7c3da0be7fa53b39ab434f7219
     Last Attempted Revision:  v2.3.0@sha256:4cc5babdb1279ad0177bf513292deadbfa3f7b7c3da0be7fa53b39ab434f7219
   Events:
     Type    Reason                   Age    From             Message
     ----    ------                   ----   ----             -------
     Normal  Progressing              6m20s  flux-controller  Installing revision v2.3.0@sha256:4cc5babdb1279ad0177bf513292deadbfa3f7b7c3da0be7fa53b39ab434f7219
     Normal  ReconciliationSucceeded  5m9s   flux-controller  Reconciliation finished in 52s
   

2. Run kubectl logs on the flux-operator pod to see the reconciliation logs:

   kubectl -n flux-system logs deployment/flux-operator
   

3. Run kubectl events to see the events generated by the flux-operator:

   kubectl -n flux-system events --for FluxInstance/flux
   

4. Run kubectl delete to remove the FluxInstance resource and to uninstall Flux without affecting any Flux-managed workloads:

   kubectl -n flux-system delete FluxInstance/flux
   

Writing a FluxInstance spec

As with all other Kubernetes config, a FluxInstance needs apiVersion, kind, and metadata fields. The name of a FluxInstance object must be a valid DNS subdomain name.

A FluxInstance also needs a .spec section.

Distribution configuration

The .spec.distribution field is required and specifies the Flux distribution to install.

Example using the upstream Flux distribution:

spec:
  distribution:
    version: "2.x"
    registry: "ghcr.io/fluxcd"

Distribution version

The .spec.distribution.version field is required and specifies the version of the Flux distribution to install. The version field value must be a valid semver range or an exact version.

Example using a semver range to configure the automatic upgrade to the latest Flux minor version:

spec:
  distribution:
    version: "2.x"

Example using a semver range to configure the automatic upgrade to the latest Flux patch version of the 2.3 series:

spec:
  distribution:
    version: "2.3.x"

Example using an exact version to install a specific Flux version:

spec:
  distribution:
    version: "2.3.0"

Distribution registry

The .spec.distribution.registry field is required and specifies the container registry where the Flux distribution images are pulled from.

Example using the upstream Flux distribution registry:

spec:
  distribution:
    version: "2.x"
    registry: "ghcr.io/fluxcd"

Distribution image pull secret

The .spec.distribution.imagePullSecret field is optional and specifies the name of the Kubernetes secret that contains the credentials to pull the Flux distribution images from a private registry.

Example using the ControlPlane enterprise registry:

spec:
  distribution:
    version: "2.3.x"
    registry: "ghcr.io/controlplaneio-fluxcd/distroless"
    imagePullSecret: "flux-enterprise-auth"

The image pull secret must be created in the same namespace where the FluxInstance is deployed and must be of type kubernetes.io/dockerconfigjson.

Example generating a secret for the ControlPlane enterprise registry:

kubectl create secret docker-registry flux-enterprise-auth \
  --namespace flux-system \
  --docker-server=ghcr.io \
  --docker-username=flux \
  --docker-password=$ENTERPRISE_TOKEN

Distribution artifact

The .spec.distribution.artifact field is optional and specifies the OCI artifact URL containing the Flux distribution manifests. When specified, the operator will pull the artifact on a regular interval to determine the latest Flux version available including CVE patches and hotfixes.

Example using the official distribution artifact:

spec:
  distribution:
    version: "2.x"
    registry: "ghcr.io/fluxcd"
    artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"

Components configuration

The .spec.components field is optional and specifies the list of Flux components to install.

When not specified, the operator will install the default set of components for the Flux distribution:

spec:
  components:
    - source-controller
    - kustomize-controller
    - helm-controller
    - notification-controller

Cluster configuration

The .spec.cluster field is optional and specifies the Kubernetes cluster configuration.

Example using the OpenShift cluster configuration:

spec:
  cluster:
    type: openshift
    multitenant: true
    tenantDefaultServiceAccount: "flux"
    networkPolicy: true
    domain: "cluster.local"

Cluster type

The .spec.cluster.type field is optional and specifies the type of the Kubernetes cluster. This field is used to enable specific configuration for AKS, EKS, GKE and OpenShift clusters.

The supported values are kubernetes (default), openshift, aks, eks and gke.

Cluster multitenant

The .spec.cluster.multitenant field is optional and specifies whether to enable Flux multi-tenancy lockdown.

The .spec.cluster.tenantDefaultServiceAccount is optional and specifies the default service account used by Flux when reconciling Kustomization and HelmRelease resources found in the tenant namespaces.

Cluster network policy

The .spec.cluster.networkPolicy field is optional and specifies whether to restrict network access to the Flux namespace from other namespaces. By default, network policy is enabled.

Cluster domain

The .spec.cluster.domain field is optional and specifies the cluster internal domain name. By default, the domain is set to cluster.local.

Storage configuration

The .spec.storage field is optional and specifies the persistent storage for Flux internal artifacts. When specified, the operator will create a persistent volume claim named source-controller with the specified storage class and size and mount it to the Flux source-controller /data volume.

Storage class

The .spec.storage.class field is required and specifies the storage class to use for the persistent volume claim.

Example using the standard storage class:

spec:
  storage:
    class: "standard"
    size: "10Gi"

Storage size

The .spec.storage.size field is required and specifies the size of the persistent volume claim.

Sharding configuration

The .spec.sharding field is optional and specifies the sharding configuration for the Flux controllers.

Example:

spec:
  sharding:
    key: "sharding.fluxcd.io/key"
    shards:
      - "shard1"
      - "shard2"

For each shard, the operator will create a separate set of controllers, e.g.:

$ kubectl -n flux-system get deployments -l app.kubernetes.io/part-of=flux
NAME
source-controller
source-controller-shard1
source-controller-shard2
kustomize-controller
kustomize-controller-shard1
kustomize-controller-shard2
helm-controller
helm-controller-shard1
helm-controller-shard2

Note that only the source-controller, kustomize-controller and helm-controller controllers support sharding.

To assign a resource to a specific shard, add the sharding.fluxcd.io/key label with the shard value, e.g.: sharding.fluxcd.io/key: shard1.

Sharding key

The .spec.sharding.key field is optional and specifies the sharding key label to use for the Flux controllers. By default, the key is set to sharding.fluxcd.io/key.

Shards

The .spec.sharding.shards field is required and specifies the list of sharding values to use for the Flux controllers.

Common metadata

The .spec.commonMetadata field is optional and specifies common metadata to be applied to all Kubernetes resources part of the Flux instance.

It has two optional fields:

• labels: A map used for setting labels on an object. Any existing label will be overridden if it matches with a key in this map.
• annotations: A map used for setting annotations on an object. Any existing annotation will be overridden if it matches with a key in this map.

Example common metadata:

spec:
  commonMetadata:
    labels:
      app.kubernetes.io/name: flux
    annotations:
      toolkit.fluxcd.io/tenant: sre-team

Kustomize patches

The .spec.kustomize.patches field is optional and specifies the Kustomize patches to apply to the Flux controllers.

Example:

spec:
  kustomize:
    patches:
      - target:
          kind: Deployment
          name: "(kustomize-controller|helm-controller)"
        patch: |
          - op: add
            path: /spec/template/spec/containers/0/args/-
            value: --concurrent=10
          - op: add
            path: /spec/template/spec/containers/0/args/-
            value: --requeue-dependency=5s

Reconciliation configuration

The reconciliation behaviour can be configured using the following annotations:

• fluxcd.controlplane.io/reconcile: Enable or disable the reconciliation loop. Default is enabled, set to disabled to pause the reconciliation.
• fluxcd.controlplane.io/reconcileEvery: Set the reconciliation interval. Default is 1h.
• fluxcd.controlplane.io/reconcileTimeout: Set the reconciliation timeout. Default is 5m.

Sync configuration

The .spec.sync field is optional and specifies the Flux sync configuration. When set, a Flux source and a Flux Kustomization are generated to sync the cluster state with the source repository.

The Flux objects are created in the same namespace where the FluxInstance is deployed using the namespace name as the Flux source and Kustomization name. The naming convention matches the one used by flux bootstrap to ensure compatibility with upstream, and to allow transitioning a bootstrapped cluster to a FluxInstance managed one.

Sync fields:

• kind: The source kind, supported values are GitRepository, OCIRepository and Bucket.
• url: The URL of the source repository, can be a Git repository HTTP/S or SSH address, an OCI repository address or a Bucket endpoint.
• ref: The source reference, can be a Git ref name e.g. refs/heads/main, an OCI tag e.g. latest or a Bucket name.
• path: The path to the source directory containing the kustomize overlay or plain Kubernetes manifests to sync from.
• pullSecret: The name of the Kubernetes secret that contains the credentials to pull the source repository. This field is optional.
• interval: The sync interval. This field is optional, when not set the default is 1m.
• name: The name of the generated Flux source and Kustomization objects. This field is optional, when not set the default is the FluxInstance namespace name. Note that this field is considered immutable, and cannot be changed after the FluxInstance is created.

Sync from Git over HTTP/S

Example:

spec:
  sync:
    kind: GitRepository
    url: "https://gitlab.com/my-group/my-fleet.git"
    ref: "refs/heads/main"
    path: "clusters/my-cluster"
    pullSecret: "git-token-auth"

If the source repository is private, the Kubernetes secret must be created in the same namespace where the FluxInstance is deployed, and have the following format:

apiVersion: v1
kind: Secret
metadata:
  name: git-token-auth
  namespace: flux-system
type: Opaque
stringData:
  username: "git-username"
  password: "git-token"

To generate the secret with the Flux CLI:

flux create secret git git-token-auth \
  --namespace flux-system \
  --url=https://gitlab.com/my-group/my-fleet.git \
  --username=git-username \
  --password=git-token

Sync from Git over SSH

Example:

spec:
  sync:
    kind: GitRepository
    url: "ssh://[email protected]/my-org/my-fleet.git"
    ref: "refs/heads/main"
    path: "clusters/my-cluster"
    pullSecret: "git-ssh-auth"

If the source repository is private, the Kubernetes secret must be created in the same namespace where the FluxInstance is deployed, and have the following format:

apiVersion: v1
kind: Secret
metadata:
  name: git-ssh-auth
  namespace: flux-system
type: Opaque
stringData:
  identity: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----    
  known_hosts: |
    github.com ecdsa-sha2-nistp256 AAAA...  

To generate the secret with the Flux CLI:

flux create secret git git-ssh-auth \
  --namespace flux-system \
  --url=ssh://[email protected]/my-org/my-fleet.git \
  --private-key-file=my-private.key

Sync from OCI over HTTP/S

Example:

spec:
  sync:
    kind: OCIRepository
    url: "oci://ghcr.io/my-org/my-fleet-manifests"
    ref: "latest"
    path: "clusters/my-cluster"
    pullSecret: "oci-token-auth"

If the container registry is private, the Kubernetes secret must be created in the same namespace where the FluxInstance is deployed, and be of type kubernetes.io/dockerconfigjson:

apiVersion: v1
kind: Secret
metadata:
  name: oci-token-auth
  namespace: flux-system
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: "base64-encoded-docker-config"

To generate the secret with the Flux CLI:

flux create secret oci oci-token-auth \
  --namespace flux-system \
  --url=ghcr.io \
  --username=ghcr-username \
  --password=ghcr-token

Sync from S3-compatible storage over HTTP/S

Example:

spec:
  sync:
    kind: Bucket
    url: "minio.my-org.com"
    ref: "my-bucket-fleet"
    path: "clusters/my-cluster"
    pullSecret: "bucket-auth"

If the Bucket is private, the Kubernetes secret must be created in the same namespace where the FluxInstance is deployed, and have the following format:

apiVersion: v1
kind: Secret
metadata:
  name: bucket-auth
  namespace: flux-system
type: Opaque
stringData:
  accesskey: "my-accesskey"
  secretkey: "my-secretkey"

Resources migration configuration

The .spec.migrateResources field is optional and instructs the operator to migrate the Flux custom resources stored in Kubernetes etcd to the latest API version as specified in the Flux CRDs. The migration runs after the Flux distribution is upgraded from a minor version to another and only when a new API version is introduced.

By default, the field value is set to true. Note that disabling the migration may result in upgrade failures due to deprecated API versions being removed in future Flux releases.

FluxInstance Status

Conditions

A FluxInstance enters various states during its lifecycle, reflected as Kubernetes Conditions. It can be reconciling while applying the resources on the cluster, it can be ready, or it can fail during reconciliation.

The FluxInstance API is compatible with the kstatus specification, and reports Reconciling and Stalled conditions where applicable to provide better (timeout) support to solutions polling the Kustomization to become Ready.

Reconciling FluxInstance

The flux-operator marks a FluxInstance as reconciling when it starts the reconciliation of the same. The Condition added to the FluxInstance's .status.conditions has the following attributes:

• type: Reconciling
• status: "True"
• reason: Progressing | reason: ProgressingWithRetry

The Condition message is updated during the course of the reconciliation to report the action being performed at any particular moment such as building manifests, detecting drift, etc.

The Ready Condition's status is also marked as Unknown.

Ready FluxInstance

The flux-operator marks a FluxInstance as ready when the Flux configuration was built and applied on the cluster and all health checks are observed to be passing.

When the FluxInstance is "ready", the flux-operator sets a Condition with the following attributes in the FluxInstance’s .status.conditions:

• type: Ready
• status: "True"
• reason: ReconciliationSucceeded

Failed FluxInstance

The flux-operator may get stuck trying to reconcile and apply a FluxInstance without completing. This can occur due to some of the following factors:

• The distribution artifact is not accessible.
• The distribution version is not available.
• The kustomization of the Flux components fails to build.
• Garbage collection fails.
• Running health checks fails.

When this happens, the flux-operator sets the Ready Condition status to False and adds a Condition with the following attributes to the FluxInstance’s .status.conditions:

• type: Ready
• status: "False"
• reason: ArtifactFailed | BuildFailed | HealthCheckFailed | ReconciliationFailed

The message field of the Condition will contain more information about why the reconciliation failed.

While the FluxInstance has one or more of these Conditions, the flux-operator will continue to attempt a reconciliation with an exponential backoff, until it succeeds and the FluxInstance is marked as ready.

Components status

In order to provide visibility into the Flux components that are installed, the flux-operator records the status of each component in the .status.components field, including the image repository, tag and digest.

Example:

Status:
  Components:
    Digest:      sha256:161da425b16b64dda4b3cec2ba0f8d7442973aba29bb446db3b340626181a0bc
    Name:        source-controller
    Repository:  ghcr.io/fluxcd/source-controller
    Tag:         v1.3.0
    Digest:      sha256:48a032574dd45c39750ba0f1488e6f1ae36756a38f40976a6b7a588d83acefc1
    Name:        kustomize-controller
    Repository:  ghcr.io/fluxcd/kustomize-controller
    Tag:         v1.3.0

Inventory status

In order to perform operations such as drift detection, garbage collection, upgrades, etc., the flux-operator needs to keep track of all Kubernetes objects that are reconciled as part of a FluxInstance. To do this, it maintains an inventory containing the list of Kubernetes resource object references that have been successfully applied and records it in .status.inventory. The inventory records are in the format Id: <namespace>_<name>_<group>_<kind>, V: <version>.

Example:

Status:
  Inventory:
    Entries:
      Id: flux-system_source-controller__ServiceAccount
      V:  v1
      Id: flux-system_source-controller__Service
      V:  v1
      Id: flux-system_source-controller_apps_Deployment
      V:  v1

Last applied revision

.status.lastAppliedRevision is the last revision of the Flux distribution that was successfully applied to the cluster.

The revision is in the format <version>@sha256:<digest>.

The version is the Flux distribution exact semver version that was applied to the cluster.

The digest is the SHA256 hash of the Flux distribution manifests and customisations that was applied to the cluster.

Last attempted revision

.status.lastAttemptedRevision is the last revision of the Flux distribution that was attempted to be applied to the cluster.

Example:

Status:
  Last Applied Revision:    v2.3.0@sha256:4cc5babdb1279ad0177bf513292deadbfa3f7b7c3da0be7fa53b39ab434f7219
  Last Attempted Revision:  v2.3.0@sha256:4cc5babdb1279ad0177bf513292deadbfa3f7b7c3da0be7fa53b39ab434f7219

FluxInstance Metrics

The Flux Operator exports metrics for the FluxInstance resource. These metrics are refreshed every time the operator reconciles the instance.

Metrics:

flux_instance_info{uid, kind, name, exported_namespace, ready, suspended, registry, revision}

Labels:

• uid: The Kubernetes unique identifier of the resource.
• kind: The kind of the resource (e.g. FluxInstance).
• name: The name of the resource (e.g. flux).
• exported_namespace: The namespace where the resource is deployed (e.g. flux-system).
• ready: The readiness status of the resource (e.g. True, False or Unkown).
• reason: The reason for the readiness status (e.g. Progressing, BuildFailed, HealthCheckFailed, etc.).
• suspended: The suspended status of the resource (e.g. True or False).
• registry: The container registry used by the instance (e.g. ghcr.io/fluxcd).
• revision: The Flux revision installed by the instance (e.g. v2.3.0@sha256:75aa209c6a...).

Example:

flux_instance_info{
   exported_namespace="flux-system",
   kind="FluxInstance",
   name="flux",
   ready="True",
   reason="ReconciliationSucceeded",
   registry="ghcr.io/fluxcd",
   revision="v2.3.0@sha256:75aa209c6a2e25b97114ccf092246d02ab4363bc136edefc239d2a88da882b63",
   suspended="False",
   uid="16ca7202-9319-445b-99d0-617c25bda182"
}