How ACM policies and Helm+Kustomize are used to configure OpenShift?

ACM Policies and Helm+Kustomize are used to configure Openshift clusters with ApplicationSets

Here in this blog, we are going to discuss how ACM policies and Helm+Kustomize are used to configure OpenShift clusters with the ApplicationSets.

There are numerous articles about GitOps that explain how to configure numerous clusters or environments using ArgoCD and Kustomize. They demonstrate several good kustomization techniques, however, when you try to apply them in your company, you’ll undoubtedly run into the following “How can I” problems:

Progressively add new clusters?
How can I quickly replace variables using templating?
Any clusters that have any additional custom configurations?
The configuration I wish to add to each cluster should be specified.

Perhaps you also read a fascinating article about making a kustomize plugin. Organizations, however, do not want to rely on an unsupported bespoke plugin. And a pretty wonderful combination—ApplicationSets + Helm + kustomize—is the answer to all of these queries.

To configure the clusters, create an ApplicationSet that automatically builds Helm applications.

Utilize Helm templates to create all of the configuration templates.
In order to limit which templates can be used in which cluster, define conditional blocks in each template.

Make a set of kustomize applications that automatically generate them so that you can add any overlay.

Using patch and kustomize, add additional setups

Note:

An alternative would be to create merely an ApplicationSet with a Kustomize+Helm plugin, which would duplicate the kustomization overlay across all Cluster applications. Therefore, it is preferable to have an ApplicationSet for each cluster’s Kustomization as well as an ApplicationSet for all configurations based on templates.

Cluster-config ApplicationSet

We can automate the creation of Argo CD Applications using an ApplicationSet, which will serve as placeholders for each of our cluster setups.

To configure the clusters using Helm templates, we’ll use the following ApplicationSet:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
 name: cluster-config
 namespace: openshift-gitops
spec:
 generators:
   - matrix:
       generators:
         - git:
             repoURL: https://github.com/albertogd/gitops-for-organizations.git
             revision: main
             files:
               - path: "clusters/**/conf.yaml"
         - list:
             elements:
               - template: openshift-machine-config
               - template: openshift-ingress
               - template: openshift-marketplace
 template:
   metadata:
     name: "config-{{cluster.environment}}-{{cluster.name}}-{{template}}"
     labels:
       environment: "{{cluster.environment}}"
       cluster: "{{cluster.fqdn}}"
       region: "{{cluster.region}}"
       cloud: "{{cluster.cloud}}"
   spec:
     project: default
     source:
       repoURL: https://github.com/albertogd/gitops-for-organizations.git
       targetRevision: main
       path: base/config/{{template}}
       helm:
         valueFiles:
           - /conf/{{cluster.environment}}/conf.yaml
           - /clusters/{{cluster.environment}}/{{cluster.fqdn}}/conf.yaml
     destination:
       server: "{{cluster.address}}"
     syncPolicy: {}

Generator

There are several generators, but we’ll combine git and list to demonstrate the strength and adaptability of applicationset generators.

We obtain all of the conf.yaml items using the git generator. They each specify a cluster.
We define all of the argo Applications we have for each cluster using the list generator. Every form of configuration will develop into an application.

We produce each cluster’s complete list of elements using the matrix generator. The cluster salamanca.pre.redhat.com, for instance, will have three Argo Applications:

The cluster’s openshift-machine-config application
The cluster’s openshift-ingress application
The cluster’s openshift marketplace application

Template

In the template section, we define the Applications that will be created by the ApplicationSet:

metadata:
- name: ArgoCD Application name config-{{cluster.environment}}-{{cluster.name}} -{{template}}
spec:
- source:
  - path: Helm chart path base/config/{{template}}
  - helm: type of the application
    - valueFiles: configuration files with the parameters of the cluster
      - /conf/{{cluster.environment}}/conf.yaml
      - /clusters/{{cluster.environment}}/{{cluster.fqdn}}/conf.yaml
  - repoURL: URL of the git repository
- destination:
  - server: the API address of the cluster {{cluster.address}}

Cluster-config-overlay ApplicationSet

In order to apply the overlay configuration (all configuration not dependent on templates) for this solution, Kustomize will be utilized, and this is the ApplicationSet that will be used:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
 name: cluster-config-overlays
 namespace: openshift-gitops
spec:
 generators:
 - git:
     repoURL: https://github.com/albertogd/gitops-for-organizations.git
     revision: main
     files:
       - path: "clusters/**/conf.yaml"
 template:
   metadata:
     name: "config-{{cluster.environment}}-{{cluster.name}}-overlay"
     labels:
       environment: "{{cluster.environment}}"
       cluster: "{{cluster.fqdn}}"
       region: "{{cluster.region}}"
       cloud: "{{cluster.cloud}}"
   spec:
     project: default
     source:
       repoURL: https://github.com/albertogd/gitops-for-organizations.git
       targetRevision: main
       path: clusters/{{cluster.environment}}/{{cluster.fqdn}}/overlay
     destination:
       server: "{{cluster.address}}"
     syncPolicy: {}

Generator

We obtain all of the conf.yaml items using the git generator. For each cluster, each of them defines an application.

Template

The Applications that the ApplicationSet will produce are defined in the template:

metadata:
- name: ArgoCD Application name config-{{cluster.environment}}-{{cluster.name}}-overlay
spec:
- source:
  - path: cluster overlay path clusters/{{cluster.environment}}/{{cluster.fqdn}}/overlay
  - repoURL: URL of the git repository
- destination:
  - server: the API address of the cluster {{cluster.address}}

Git repository structure

3 primary directories are as follows:

base: This directory houses all of the Helm templates. It has two directories inside:
- config: For each Chart, we make a directory. (ArgoCD Application). The machine-config, oauth, and the example
- provision: We only use one application for OpenShift provisioning. The application mentioned in the first section of provisioning Openshift clusters is this one.
clusters: The orchestrator will write the objects conf.yaml and provision.yaml in the subdirectory as described in the first section. We’ll maintain the settings of our personal ACM cluster for the acm-hub:
- applications: The first Argo, app-argocd, synchronises the cluster configuration.
- applicationsets: 3 applicationsets
  - cluster-config-overlays.yaml: cluster-config ApplicationSet for kustomize applications
  - cluster-config.yaml: cluster-config ApplicationSet for Helm applications
  - cluster-provisioning.yaml: Cluster provisioning ApplicationSet for provisioning ACM clusters
- argocd: An instance of ArgoCD
- gitops-cluster: Objects to register ManagedClusters in ArgoCD with gitops-cluster.
  - GitOpsCluster.yaml: GitOpsCluster object to register the ManagedClusters to ArgoCD
  - ManagedClusterSetBinding.yaml: Binding for the ClusterSet
  - Placement.yaml: Placement to select ManagedClusters
- policies: policies used to configure the ManagedClusters
conf: For each environment, we will maintain the default values in this section (conf.yaml and provision.yaml). (dev, pre and pro).

├── base
│   ├── config
│   |   ├── openshift-ingress
│   |   |   ├── Chart.yaml
│   |   |   └── templates
│   |   |       ├── ingress.yaml
│   |   |       └── service.yaml
│   |   ├── openshift-machine-config
│   |   |   ├── Chart.yaml
│   |   |   └── templates
│   |   |       ├── 05-worker-kernelarg-selinuxpermissive.yaml
│   |   |       └── master-kubeletconfig.yaml
│   |   └── openshift-marketplace
│   |       ├── Chart.yaml
│   |       └── templates
│   |           ├── local-certified-operators.yaml
│   |           └── local-community-operators.yaml
│   └── provision
│       └── openshift-provisioning
│           ├── Chart.yaml
│           └── templates
│               ├── clusterdeployment.yaml
│               ├── klusteraddonconfig.yaml
│               ├── machinepool.yaml
│               ├── managedcluster.yaml
│               ├── managedclusterinfo.yaml
│               └── namespace.yaml
├── clusters
│   ├── acm-hub
│   │   ├── applications
│   |   |   ├── app-argocd.yaml
│   |   |   └── kustomization.yaml
│   │   ├── applicationsets
│   |   |   ├── cluster-config-overlays.yaml
│   |   |   ├── cluster-config.yaml
│   |   |   ├── cluster-provisioning.yaml
│   |   |   └── kustomization.yaml
│   │   ├── argocd
│   |   |   ├── argocd.yaml
│   |   |   └── kustomization.yaml
│   │   ├── gitops-cluster
│   |   |   ├── GitOpsCluster.yaml
│   |   |   ├── ManagedClusterSetBinding.yaml
│   |   |   ├── Placement.yaml
│   |   |   └── kustomization.yaml
│   │   └── policies
│   |       └── 4.11
│   │           ├── odf-operator
│   │           |   ├── binding-odf-operator-411.yaml
│   │           |   ├── kustomization.yaml
│   │           |   ├── placement-odf-operator-411.yaml
│   │           |   └── policy-odf-operator-411.yaml
│   │           └── upgrade-ocp
│   │               ├── binding-upgrade-cluster-411.yaml
│   │               ├── kustomization.yaml
│   │               ├── placement-upgrade-cluster-411.yaml
│   │               └── policy-upgrade-cluster-411.yaml
│   ├── dev
│   │   └── zamora.dev.redhat.com
│   |       ├──  provision.yaml
│   |       ├──  conf.yaml
│   |       └──  overlay
│   |            ├── kustomization.yaml
│   |            └── project.yaml
│   └── pre
│       └── salamanca.pre.redhat.com
│           ├──  provision.yaml
│           ├──  conf.yaml
│           └──  overlay
│   
└── conf
    ├── dev
    │   ├── conf.yaml
    │   └── provision.yaml
    └── pre
        ├── conf.yaml
        └── provision.yaml

Initial ArgoCD application

Bootstrapping all the configuration can be done in a variety of ways, such as by utilizing a policy or an Argo Application. If we have another cluster using ArgoCD, we may use OpenShift GitOps to synchronize all the configurations from that cluster. However, in our method, we’re assuming that we’re starting from scratch and that all we have is the ACM hub cluster. In this instance, we must manually create the initial application. This initial application will then be synchronized with ArgoCD after that.

$ oc apply -f clusters/acm-hub/applications/app-argocd.yaml

Why 2 ApplicationSets instead of using a custom plugin of Helm + Kustomize

Let’s start by talking about the feature we want before going into detail regarding Helm + Kustomize:

Base manifest templates:
- A universal manifest that will be used by all clusters (i.e. machine-config template)
- Has the ability to include/exclude any base manifest (i.e. exclude oauth template)
Custom manifests per cluster:
- Ability to add any custom manifest to a particular cluster (i.e. add an Ingress).
Default values per environment:
- Have a default configuration file for each environment with default values. (i.e. development configuration)
Custom values per cluster
- Each cluster should have a unique configuration file
Avoid using any unsupported or custom plugins

Although Kustomize is a tool for templating, adding Kustom manifests is really simple with it. Yes, a Kustomize plugin can be added to handle the templating. But keep in mind that this is “GitOps for organizations,” hence we need something that is OOB supported. Furthermore, Kustomize is not an OOB templating tool. Helm, however, is a templating tool. We have a very strong templating engine with Helm, however unlike Kustomize, we are not as free to build unique manifests.

Why then don’t we leverage Helm’s templating functionality with one ApplicationSet and Kustomize’s customization flexibility with another ApplicationSet?

Helm Templates

A particularly effective tool for templating is Helm. Using the vars defined in each cluster’s (clusters//conf.yaml) and the default environment’s (config//conf.yaml) configuration files, we will define templates for all the objects we need to build. Environment variables won’t matter; the cluster variables do.

Kustomize

Using kustomize, we can add any Kustomization we desire for each cluster. Any custom items we want to add will be added using the folder overlay.

Cluster Lifecycle with ACM policies

We’ll use ACM policies, which are located in the policies folder in the git repository under clusters/acm-hub, for the cluster lifecycle.

└── clusters
    └── acm-hub
        └── policies
            └── 4.11
                ├── odf-operator
                |   ├── binding-odf-operator-411.yaml
                |   ├── kustomization.yaml
                |   ├── placement-odf-operator-411.yaml
                |   └── policy-odf-operator-411.yaml
                └── upgrade-ocp
                    ├── binding-upgrade-cluster-411.yaml
                    ├── kustomization.yaml
                    ├── placement-upgrade-cluster-411.yaml
                    └── policy-upgrade-cluster-411.yaml

ACM applies the policies that are synced with ArgoCD and stored in the git repository.

There are numerous policy examples for Configuration Management, Security Assessment and Authorization, Audit and Accountability, and Access Control in this repository.

Operators

ArgoCD installation of operators can be challenging because it isn’t designed to handle dependencies. There are some solutions, such as using a Job. But with this solution, we’re employing ACM policies, which is a far better strategy. We can force any cluster to have the operators we want by using an ACM policy.

A policy similar to the one used to install the Compliance Operator can be used to deploy other operators.