A redhat distributed continuous integration workflow for F5

Here in this blog, we are going to learn about redhat distributed continuous integration workflow for F5.

The primary DCI building components are shown in the diagram below:

It is outside the purview of this page to explain DCI in detail; for more details, check the official documentation or this article. Generally speaking, DCI can track resources that are deployed. These are called components, which can apply to any kind of artifact (file, package, URL, etc.) that is used during the deployment process. Components can even include the RHOCP version that was used to build a cluster or the F5 product version that was deployed in it.

Red Hat provides the majority of this information through DCI. In this case, F5’s duties include hardware management, pipeline logic-based RHOCP installation deployment via an Ansible-based agent, multiple RHOCP deployment options (including Assisted Installer and IPI), and hardware configuration (baremetal, virtualized nodes).

Specifically, the environment examined for F5 testing and validation of CGAM and SPK comprises of a few Red Hat laboratories that are reachable through F5, configured as follows:

• On demand, several RHOCP clusters of three worker and three master nodes can be built and used concurrently.
• Based on a disconnected environment and IPI deployment, each cluster has a provisioner computer to facilitate the RHOCP installation, and a jumphost machine to centrally manage DCI automation and access to external resources.
• Source code that includes the Ansible playbooks and pipeline configurations needed to set up the intended scenario

The graphic below illustrates the description that was given above:

You might be asking yourself how Ansible can assist in achieving the required automation in a situation like this. For this, an agent hosted in the partner’s lab is utilized, and DCI presently supports the following two types of agents:

• dci-openshift-agent: Utilize various Red Hat-supported deployments to install any RHOCP version.
• dci-openshift-app-agent: Utilize RHOCP to deploy workloads, test them, and certify them in accordance with Red Hat certification processes.

For each agent, specific responsibilities and playbooks are supplied to aid with implementation. In addition to typical Ansible modules, they utilize the following resources to facilitate automation:

• To support the many RHOCP deployment types that Red Hat supports, DCI imports Ansible projects. An instance of F5 testing and validation is the IPI installation, which is accessible to the general public via the openshift-kni/baremetal-deploy repository.
• Using Ansible’s Kubernetes collection, you may define particular Ansible jobs to control the lifespan of RHOCP resources and communicate with them during a deployment.

The agents adhere to a well-defined workflow that divides up the various Ansible activities that must be completed throughout the infrastructure that requires configuration. The workflow for the dci-openshift-agent, which F5 uses to perform the DCI automation, is shown below. It differentiates between actions that are executed on the jumphost (nearly all phases) and those that are executed in the provisioner (only its configuration and the RHOCP installation using the chosen method):

Pre-run, install, tests, post-run, success, and teardown are the stages that are made up of generic activities that are provided directly by the agent and partner hooks, which are optional codes that partners can supply to carry out specific deployments and setups. In this instance, SPK and CGAM are deployed, configured, and tested correctly via partner hooks.

The definition of the previously stated pipelines governs the right tasks that are executed at each stage and ultimately automates the deployment of various jobs with the desired configuration. In the end, pipelines are just YAML files with all the settings and parameters required for the continuous integration system to reach the intended outcome, including:

• The Ansible configuration’s agent and hooks playbooks’ locations
• Network setup
• Operators of RHOCP to initiate (or not)
• Things that will be used (both for RHOCP and F5-other partners)

Keeping all of these resources in mind, a proper execution of each component as a whole yields:

• If an OpenShift cluster with the required configuration is available
• Conducting tests to assess the implemented setup

An execution report in graphical form. A variety of data, such as the deployment’s actual execution and analytics utilizing the F5’s component coverage to display the proportion of successfully completed jobs for every SPK version evaluated on DCI, are shown. The following shows the component coverage for the SPK component in debug DCI jobs (not for production environments) using RHOCP 4.12.

Test scenarios and methods for SPK deployment

This is a condensed list of the steps that are used to install and configure SPK during the install stage; they are all implemented using Ansible in the F5 partner’s hooks. RHOCP 4.14 uses this setup, which is based on SPK v1.8.2:

• Utilize the F5 SPK custom CRD and pipeline configuration variables.
• Give RHOCP nodes labels
• Establish a webhook configuration.
• Establish CoreDNS settings.
• Configure the SPK Cert Manager.
• Create a setup for SPK Fluentd.
• Construct an SPK DSSM setup.
• Establish the SPK CWC setup.
• Establish SPK instances, including SPK Data and DNS46.
• Make a Webhook for SPK Conversion.

A test pod (mttool) is IPv4 exclusively in an example test environment with a single stack IPv4 RHOCP cluster.However, the “Hello World” FQDN is an IPv6 address and a AAAA record in the External DNS. In order to deliver the DNS46 service in this instance, a different SPK (SPK DNS46) is also present in the flow.

A test pod client (mttool) is generated in the tests hook. This pod transfers all of its external DNS to CoreDNS via webhook. All inquiries are sent through the CoreDNS pods to SPK DNS46. The client pods submit a DNS A query to Hello World since they need to communicate with it. A query is forwarded to External DNS via SPK DNS46. But as was before said, External DNS only has AAAA records for Hello World in IPv6.

As a result, the External DNS returns an empty response. The DNS query for Hello World is resent by SPK DNS46 as a AAAA record. IPv6 address is returned by the External DNS.

Using the DNS46 function, SPK DNS46 maps IPv6 to IPv4 in the SPK DSSM database and sends the mapping’s matching IPv4 address.

The client test pod is sending TCP packets to the SPK Data instance on port 80 of that IPv4.

To determine the corresponding IPv6 address from the mapping in DSSM produced by SPK DNS46 above, SPK Data employs its NAT46 feature.

The TCP packet is routed using SPK data to port 80 of the IPv6 Virtual Service, and then it is sent to the IPv4 HTTP pod.

Test cases and procedures for deploying Carrier-Grade Aspen Mesh

A number of jobs have been put in place as F5 partner hooks to run the CGAM automation, much like SPK. The list that follows uses CGAM 1.14.6-am5 and is based on RHOCP 4.14:

• Utilize pipeline configuration variables
• Extract the installer for Aspen Mesh, then create certificates.
• Install the Istio base chart under the namespace Istio-system.
• Make sure CRDs are set up.
• Install the Istio-CNI chart in the namespace kube-system.
• Install the ISID chart in the ISID-system namespace.
• Deploy the istio-ingress chart in the istio-system namespace.
• If necessary, install Citadel add-ons and DNS Controller.

The following figure shows the testing scenario for the CGAM validation:

In this situation, three different kinds of test cases are executed:

• Verify the gateway: Check to see if the Aspen Mesh sidecar container is running on a pod that has been configured with Istio. To confirm the connectivity, the host IP of the pod and the gateway service’s NodePort are utilized.
• Inter-namespace verify: Confirm that the namespaces being tested are in contact with one another.
• Dual stack test: Lastly, confirm that the data and control planes are ready for dual stack and that dual-stack DNS and TCP processes are operating from various network nodes.

Overcoming obstacles and findings

In conclusion, based on the platform automation configured as a result of the continuous integration process permitted by Red Hat Distributed CI, we can see that we have achieved the targeted configuration and automation capabilities we sketched at the beginning of our collaborative work. Red Hat and F5 are moving forward more quickly than ever thanks to this work, which is addressing some significant obstacles like:

• Checking the most recent version of the F5 solution and validating it on various OpenShift versions
• Parameterizing specific configurations in order to tailor each deployment (not only for OpenShift operators, common utilities, and network capabilities, but also for 5G components).
• Introducing updates and changes with ease, to be evaluated in later deployments

It is true that it took some effort to accomplish this in order to get the suggested platform automation configured the way that was wanted. This explains why less than 50% of the DCI jobs launched with each SPK component in Figure 7 have a reported success rate.

We are in the process of testing and validating, which is why we have a significant number of failed tests. Under these unfavorable circumstances, DCI finds problems early on in the deployment of F5 or RHOCP solutions, producing insightful logs and comments for Red Hat and F5 developers. It takes time to accomplish all of this, but all of this information is used to suggest a fix for the problem and enhance the automation and pipelines. The solution is tried again while this work is being done, yielding unfavorable outcomes until the problem is eventually fixed.

In order to eventually achieve an error-free automated deployment, it is implied by this low success rate that Red Hat and F5 are still working to improve their solutions through testing and troubleshooting procedures like promptly examining new releases, finding bugs, and resolving them. The subsequent objectives of this collaboration are to reduce the time it takes to find a stable solution when issues are identified, hence increasing the success rate of the F5 product’s DCI components.