Boosting Edge Reliability with Two-Node OpenShift and Fencing

In this blog, we will learn how to boost edge reliability with two-node OpenShift and fencing.

As organizations expand their use of edge computing, they are moving applications and data processing closer to the locations where data is produced. This approach can reduce latency and improve application responsiveness, but it also introduces a new set of infrastructure challenges.

Retail stores, manufacturing facilities, telecommunications sites, and other distributed environments often need highly available applications at locations where space, power, hardware, and IT resources are limited. Deploying infrastructure across hundreds or thousands of sites makes every additional server a significant operational consideration.

Traditionally, a three-node OpenShift cluster has been the preferred approach for maintaining quorum and high availability. A third node provides the additional vote required to maintain cluster consistency when one node becomes unavailable. However, deploying that additional server at every edge location can substantially increase hardware, energy, deployment, and maintenance costs.

This challenge has created demand for smaller cluster configurations that can still provide a reliable availability model.

Red Hat has addressed this requirement with two-node OpenShift with fencing, a deployment model designed specifically for edge environments where maintaining a compact infrastructure footprint is important.

An alternative known as two-node OpenShift with arbiter uses a lightweight third system to help maintain quorum. While this reduces the amount of infrastructure required at the primary site, it still represents a three-member architecture.

Two-node OpenShift with fencing takes a different approach. It provides a true two-node architecture, using hardware-based fencing to maintain cluster consistency and protect against split-brain conditions.

With OpenShift 4.22, this capability is generally available for supported deployments.

Why Two-Node OpenShift Matters at the Edge

Edge locations are fundamentally different from centralized data centers.

A typical data center can accommodate multiple servers, redundant power systems, dedicated networking equipment, storage infrastructure, and specialized IT personnel. A remote retail outlet, factory floor, telecom site, or other edge location may have considerably less room and fewer resources.

When the number of locations grows into the hundreds or thousands, infrastructure requirements multiply quickly.

For example, deploying three servers at 1,000 locations means maintaining approximately 3,000 servers. Reducing that requirement to two servers per location brings the number down to approximately 2,000 servers.

The savings extend beyond the initial hardware purchase. Organizations can also reduce:

• Power consumption
• Physical space requirements
• Cooling requirements
• Hardware maintenance
• Replacement costs
• Transportation and deployment expenses
• Remote-site operational complexity

This makes a two-node architecture particularly attractive for edge deployments where cost and physical footprint are major considerations.

The challenge, however, is maintaining cluster consistency when there are only two nodes.

Understanding the Two-Node OpenShift Architecture

Two-node OpenShift with fencing is designed around two physical servers rather than three.

The current architecture is supported on x86 bare-metal systems and can also be used with Red Hat OpenShift Virtualization. This allows organizations to deploy both traditional container workloads and virtual machines on the same compact OpenShift environment.

At the heart of the architecture is the integration of OpenShift with high-availability technologies provided by the Red Hat Enterprise Linux High Availability Add-On.

Two important components are:

• Corosync — responsible for cluster communication and failure detection.
• Pacemaker — responsible for managing resources and initiating recovery and fencing actions.

Together, these components provide the mechanisms required to safely operate an OpenShift cluster with only two nodes.

Instead of relying on a third node to provide the deciding vote, the architecture uses fencing to establish which system is allowed to continue operating.

Storage Flexibility

Storage is another important consideration for edge environments.

Two-node OpenShift with fencing supports node-local storage configurations, including solutions based on Logical Volume Manager.

Organizations can also use shared or distributed software-defined storage solutions from supported partners. Examples include:

• LINBIT, using DRBD technology
• Arctera InfoScale
• IBM Storage Fusion Data Foundation, based on Ceph

The fencing mechanism plays an important role in storage protection. When a node failure occurs, storage I/O can be stopped until the affected node has been successfully isolated.

This prevents a failed or disconnected server from continuing to modify data while the surviving node attempts to recover the workloads.

Why Fencing Is Critical in a Two-Node Cluster

The biggest challenge with a two-node cluster is the absence of a third member to resolve a communication failure.

Consider two servers that are operating normally. Both are communicating with each other and participating in the cluster.

Now imagine the network connection between them suddenly disappears.

Node 1 can no longer see Node 2.

At the same time, Node 2 can no longer see Node 1.

Neither node can immediately determine whether the other server has actually failed or whether only the communication path has been interrupted.

If both systems assume they are the only active node, they could independently begin managing workloads and accessing shared resources.

This creates a split-brain condition.

Split-brain is particularly dangerous for applications that maintain state. Two instances of the same database, for example, could independently accept transactions and modify data. Reconciling those conflicting changes later could be extremely difficult and potentially result in data loss or corruption.

Fencing prevents this situation by making sure that only one side of the cluster can continue operating.

How Hardware Fencing Works

Two-node OpenShift with fencing relies on the server’s Baseboard Management Controller (BMC).

The BMC must support the Redfish API, which provides a standardized way to remotely control and manage server hardware.

The process works roughly as follows.

First, Corosync detects that communication with a node has been lost. At this stage, the cluster does not necessarily know whether the server itself has failed.

Pacemaker then takes responsibility for resolving the uncertainty.

Instead of simply assuming that the unreachable server has stopped, Pacemaker initiates a fencing operation through the BMC.

The BMC receives a ForceRestart request through Redfish. This forces the affected server to shut down and restart without performing a normal graceful shutdown.

The purpose is not simply to reboot the machine.

The important objective is to establish certainty that the isolated node is no longer running workloads.

Once fencing has successfully completed, the surviving node can safely continue the recovery process.

The previously uncertain node is now known to be inactive.

This also ensures that its etcd instance is no longer participating in the cluster.

Restoring etcd Quorum

The OpenShift control plane relies heavily on etcd,etcdwhich stores critical cluster state.

When one node disappears from a two-member etcd cluster, quorum is lost. As a result, control-plane components cannot continue normal operation.

After the failed node has been successfully fenced, the remaining node can safely reorganize etcd into a single-member configuration.

This restores quorum on the surviving server.

Once quorum is available:

1. The control plane can become operational again.
2. The scheduler can resume its responsibilities.
3. Missing workloads can be restarted when sufficient resources are available.
4. Applications can continue operating from the surviving node.

The key point is that the surviving node does not make this transition until the other node has been successfully fenced.

That sequence is what protects the cluster from having two active versions of the environment running independently.

Avoiding Simultaneous Fencing

A natural question is whether both servers could detect the same communication failure and attempt to fence each other.

The architecture uses different timing delays for the two nodes.

When a failure is detected, the nodes do not necessarily begin fencing at exactly the same instant. The configured delays create a preferred sequence, reducing the possibility that both systems will simultaneously initiate fencing against one another.

This helps establish a single surviving side of the cluster and prevents competing recovery operations.

What Happens When a Node Fails?

To understand the recovery process, consider a two-node OpenShift cluster consisting of Node 1 and Node 2.

The failure and recovery sequence can be divided into five stages.

Stage 1: Normal Operation

Initially, both nodes are healthy.

Node 1 and Node 2 communicate normally and etcd operate as a two-member cluster.

The OpenShift control plane, API server, scheduler, and workloads are functioning normally.

From an application perspective, the environment is operating in its normal high-availability state.

Stage 2: Node Failure

Now assume Node 1 experiences a hardware failure and suddenly goes offline.

The remaining node can no longer communicate with Node 1.

Because the etcd cluster has lost one of its members, quorum is no longer available. This affects the OpenShift control plane and API services.

However, the cluster does not immediately fence Node 1.

This delay is intentional.

A temporary network interruption should not automatically cause an aggressive recovery action. Corosync first monitors the situation and records the loss of communication.

During this period, the cluster enters a degraded state.

Workloads that were running exclusively on Node 1 may no longer be available. For example, a virtual machine or single-instance application hosted on that server could experience an outage.

Workloads already running on Node 2 can continue operating.

Stage 3: Fencing Is Triggered

If Node 1 remains unreachable for the required failure-detection period, Corosync moves the node into a failure state.

Pacemaker then starts the fencing procedure.

Using the BMC, Pacemaker sends the appropriate Redfish command to Node 1.

The server is forcibly restarted, ensuring that any workloads that may still be running on it are terminated.

This step is essential because the cluster cannot safely assume that an unreachable node is actually powered off.

While fencing is taking place, some control-plane services may remain unavailable.

However, workloads already operating on Node 2 can continue serving traffic.

Stage 4: The Surviving Node Regains Quorum

The next phase begins only after fencing has completed successfully.

The fencing system must receive confirmation that the operation succeeded.

Once Node 1 is confirmed to be safely isolated, Node 2 can take over the cluster state.

etcd is converted from a two-member configuration to a single-member configuration.

Node 2 therefore regains quorum.

The OpenShift control plane becomes functional again, although the cluster is still operating in a degraded state because only one server is available.

The scheduler can also resume.

If Node 2 has enough CPU, memory, and other required resources, workloads that were previously hosted on Node 1 can be rescheduled and restarted.

Stage 5: Node 1 Returns

Eventually, the hardware issue affecting Node 1 is resolved, and the server is powered back on.

OpenShift detects the returning node and begins reintegrating it into the cluster.

The node initially participates as a etcd learner rather than immediately becoming a full member.

During this process, it receives the cluster transactions that occurred while it was offline.

After synchronization is complete, Node 1 can become a full member again.

The environment returns to its normal two-node configuration.

Because replaying and synchronizing etcd data can generate considerable disk activity, the storage subsystem needs to provide strong I/O performance. For this reason, SSD-class storage or better is required for these deployments.

Once synchronization is finished, the control plane is fully operational, and the cluster returns to its normal high-availability state.

Handling More Complex Failure Scenarios

Fencing provides an effective mechanism for dealing with individual node failures and communication problems. However, certain failure combinations create situations where the cluster cannot safely determine which node contains the authoritative state.

These cases may require administrator involvement.

Scenario 1: Complete Site Power Loss

Imagine that both servers lose power at the same time.

Later, only one server comes back online.

The returning server cannot immediately start workloads because it cannot determine whether the other node contains newer cluster information.

Automatically starting workloads could create a risk of inconsistent state if the other server were actually still operating somewhere.

There are two possible recovery approaches.

Option 1: Bring Both Nodes Back Together

The preferred approach is to restore communication with the second server.

Once both systems are online and can communicate, they can determine the correct cluster state.

The node with the outdated state can then synchronize with the current member through the learner recovery process.

Option 2: Administrative Recovery

If the second server cannot immediately return, an administrator can connect through SSH and verify its condition.

The administrator must establish that the other server has remained offline throughout the outage.

This manual confirmation provides the necessary confidence to proceed with recovery.

For this reason, reliable SSH access should be configured as part of the deployment process. Public keys can be supplied during installation to make administrative access available when required.

One important limitation is that oc debug node/nodeXXX cannot be used in this situation because the OpenShift control plane is not yet operational.

Scenario 2: Failure of the Surviving Node

A different challenge occurs when the cluster has already lost one node and successfully fenced it.

At this point, the remaining node is operating alone.

Now suppose the surviving server also reboots unexpectedly.

The cluster cannot automatically determine whether the previously failed server is still offline or has somehow become active again.

Without that information, restarting workloads could potentially create conflicting cluster states.

The recovery process therefore follows the same basic principle as a cold-boot scenario.

Either:

• Bring the second server back online and allow both systems to establish their state, or
• Have an administrator verify the status of the other node through SSH before continuing.

This additional verification is necessary because data consistency must take priority over automatically restoring services.

Why This Architecture Fits Edge Deployments

The requirements of edge computing are driving organizations toward smaller and more efficient infrastructure designs.

A traditional three-node cluster provides a straightforward quorum model, but deploying three full servers at every remote location can become expensive when infrastructure is distributed across a large number of sites.

Two-node OpenShift with fencing offers another option.

It combines:

• A smaller physical footprint
• Hardware-based fencing
• Corosync-based failure detection
• Pacemaker-based recovery
• OpenShift container workloads
• Support for OpenShift Virtualization
• Flexible storage architectures
• Protection against split-brain conditions
• Automatic recovery for supported node-failure scenarios

The result is an architecture designed to provide high availability while keeping the edge infrastructure footprint as small as possible.

Final Thoughts

Edge computing requires a different approach to infrastructure design than centralized data centers.

Organizations need highly available platforms, but they also need to consider the realities of remote deployments: limited physical space, constrained power availability, higher maintenance costs, and the sheer number of locations that must be supported.

A three-node architecture can address quorum requirements effectively, but maintaining three physical systems at every site may not always be economically practical.

Two-node OpenShift with fencing provides an alternative by using hardware-level isolation to eliminate the need for a dedicated third node.

Corosync identifies failures, Pacemaker coordinates recovery, and the BMC-based fencing mechanism ensures that an uncertain node is forcibly isolated before the surviving node resumes cluster operations.

The architecture does not eliminate every possible failure scenario. Complete site outages and subsequent failures of the surviving node can still require administrator intervention. However, for common single-node failures, fencing provides a controlled recovery mechanism while protecting cluster and storage consistency.

For organizations deploying OpenShift across large numbers of edge locations, this approach can help balance availability, infrastructure cost, physical footprint, and operational resilience—making two-node OpenShift with fencing an important architecture to consider for modern edge environments.