Using add-ons to extend Red Hat Unified Kernel Images

I examine the add-ons that allow for a safer extension of the UKI kernel command line in this blog.

Unified Kernel Image (UKI): What is it?

The Linux kernel serves as the fundamental element of every Linux operating system. It serves as a mediator between the hardware and the processes that operate on it, all of which are invisible to the user. It is also in charge of scheduling programs so that memory and CPU are distributed among them in a way that ensures security, isolation, and other factors.

The Initramfs is a component of the Linux boot process. It includes every piece of software needed to set up a system in order to load the root filesystem, which is where all user data is stored. For instance, the user needs to input a password in order to decrypt the root drive. Because the rootfs are encrypted, the software and modules that request the password and decrypt the disk cannot be located there; instead, they are located in the initramfs.

A UKI is a UEFI binary that includes the Red Hat signature, Linux kernel image, initramfs, and kernel command line. We enhance the SecureBoot process by integrating initramfs and the kernel command line, ensuring that all components are delivered collectively.

Confidential virtual machines (CVM) are not and should not be the only use of a UKI. In a conventional framework, the initramfs of the target system must be constructed subsequent to the distribution of the kernel by Red Hat. The advantages of shipping a Red Hat-provided pre-built infrastructure in a UKI are as follows:

• Faster build process: The kernel installation is quicker because the target system does not need to install any tools because it does not need to build its own initramfs.
• Enhanced initramfs support: Red Hat is now able to manage the provided UKI with greater efficiency, as the pre-constructed initramfs contained within is consistent across all installations.

The advantages of UKI extensions are therefore not limited to CVMs.

Although UKI successfully assists in demonstrating that no malicious actor changed any of the boot chain’s objects, the idea has certain drawbacks because it lessens the boot process’s flexibility. Specifically:

• When the UKI is being developed, the operating system vendor is required to create the initialization files. This suggests that there is no way to add more kernel modules to the final system because the list of ones that are included in the initramfs is fixed and constant.
•  Additionally, the kernel command line is made constant. For instance, using the root= option on the kernel command line makes it impossible to set the desired root volume. This necessitates a different method for determining the root volume.

The purpose of this page is to describe the UKI Addons, which are Red Hat Enterprise Linux 9.4’s upstream solution to the aforementioned limitations.

The purpose of this page is to describe the UKI Addons, which are Red Hat Enterprise Linux 9.4’s upstream solution to the aforementioned limitations.

Kernel command line additions are my main area of interest. The development of initramfs extensions is still ongoing, thus that will be covered in a later post.

I’m paying particular attention to the boot scenario that follows:

We start with the UEFI firmware and use the shim bootloader to run the kernel after calling systemd-stub to extract and unpack the pertinent parts of the UKI.

Using the kernel command line to extend

There are many features and choices available in the Linux kernel that may be turned on and adjusted at will. The kernel command line is the most effective approach for the user or vendor to specify what they wish to enable during the subsequent boot.

The Linux kernel allows users to verify the current kernel command line by accessing the /proc/cmdline file. This is a plain text file that contains the current command line used by the kernel.

If the kernel fails to recognize a command, and the command does not include a period (.) or is placed after a hyphen (-), the arguments will be forwarded to init and disregarded by the kernel, as stated in the Linux documentation.

It is essential to ensure that the kernel command line is constructed in such a manner that it is transmitted directly to the kernel and, if applicable, to the init process. An attacker could exploit this vulnerability or inject custom kernel parameters into the UKI system to open a backdoor and gain access.

Root Kernel Command Line Example

As an example, consider the kernel command line option root=. The kernel documentation clarifies that it is utilized for configuring the root filesystem.

Suppose an attacker manages to change the root= option and attach an additional volume with a malicious OS image (either loaded with malware or simply with a backdoor or a known root password setup). Upon startup, the UKI kernel is provided with the root=exploited parameter, resulting in the booting of the compromised operating system image. The measured boot phase is used to verify the authenticity of the boot chain, but it is fooled into not noticing this change.

This is especially possible if only PCR7 is used since PCR7 only records the certificates used during the boot process. Volume changes are not detected by this measurement. For additional details regarding PCRs and the measurement of boots, please refer to the earlier article authored by Vitaly Kuznetsov

How the kernel command line works in UKI

UKI is just a PE binary file consisting of multiple parts. The main parts of UKI are:

• linux: contains the Linux kernel
• cmdline: contains the kernel command line
• initrd: contains initramfs

There are other parts, such as .sbat, but for simplicity, I will focus on the list above for now. I will discuss .sbat later. For a complete list of parts, see the UAPI group specification.

The role of .cmdline in UKI is simply to transmit the command line in plain text, which systemd-stub unpacks and injects into the kernel. Obviously, in a CVM environment, the kernel command line must be carefully worded to avoid unwanted parameters and prevent attackers from exploiting these parameters to gain access to the UKI system.

Once a UKI is created and signed, its contents cannot be changed, including the command line. Therefore, there is usually a very small UKI .cmdline section that usually only contains console options.This ensures optimal security; however, it compromises flexibility as there is no longer a means to modify the kernel command line.

But a static UKI is essentially useless. Think about the different parameters required for different instance sizes in public clouds, debug options that kernel developers need to add, etc.

This is why UKI plugins are so important.

Who creates UKI add-ons?

Add-ons can be created at different levels of the virtualization stack:

• Providers: Provide optional features or debugging tools for your UKI. The advantage of this is that the keys are already added to the SecureBoot database, so no further steps are required after installing the add-on. On the other hand, add-ons must be carefully crafted to avoid malfunctions or security issues.
• UKI users are encouraged to incorporate personalized options and develop unique add-ons to enhance the hardware and software functionalities within a UKI system. To be trusted, users must add custom keys to the SecureBoot database. This can be achieved using a MOK (Machine Owner Key) and mokutil.
• Virtual hosts: Virtual host administrators can also add custom options if trusted. It is prohibited in public cloud environments due to the lack of trustworthiness associated with cloud providers. As a result, the keys are not added to the system’s SecureBoot database.

Current techniques exist for extending the kernel command line, which include the use of SMBIOS parameters and the EFI Shell, among other options. Nevertheless, these parameters are susceptible to emulation by the hypervisor, which lacks trustworthiness.

For instance, the SMBIOS data is supplied by the hypervisor, rendering it unreliable. In QEMU, it is quite straightforward to input SMBIOS table entries into the virtual machine:

The EFI shell functions as an application operating within the guest context; however, it disrupts user interaction on an emulated console device.

Consequently, all such functionalities have been disabled by system-stub and are disregarded automatically.

Safely augmenting a UKI kernel command line with UKI addons is now possible.

Starting from upstream systemd version 254 and RHEL systemd-252-31, systemd-stub is capable of reading UKI addons and employing them to improve the .cmdline section of a UKI.

How is this achieved? It utilizes the SecureBoot UEFI mechanism, which allows the caller to authenticate a PE section by checking its signature against trusted keys stored in the SecureBoot database of the system. If the signature is included in the trusted database, the verification is successful; otherwise, it fails. This process enables the extension of the UKI kernel command line while maintaining security integrity.

What exactly is a UKI addon? In essence, it is a PE binary that contains a very limited number of sections. It may include all the sections found in a UKI, with the exception of .linux and .initrd (as including these would classify it as a UKI). The .cmdline section constitutes the fundamental component of an addon.

Systemd-stub is configured to search for addons in designated locations within the EFI System Partition (ESP), where the utilized UKI and its corresponding addons are expected to be found. The designated mount point for the EFI System Partition (ESP) is identified as $BOOT/EFI, where the definition of $BOOT is established by the specifications set forth by the UAPI group. In the context of RHEL, $BOOT is equivalent to /boot/efi.

The criteria for identifying an addon are as follows:

1. The addon is required to be a PE binary.
2. The filename of the addon must end with the extension .addon.efi.
3. If the addon is intended for use exclusively when the UKI named $UKI_NAME is booted, it should be placed in $BOOT/EFI/$UKI_NAME.extra.d/ (these addons are classified as local addons). Conversely, if the addon is applicable to any UKI booted, it should be located in $BOOT/loader/addons/ (these are referred to as global addons).

If all these criteria are met, systemd-stub selects the addon and performs a verification. Upon verifying the authenticity of the signature key, the addon is accessed, and its .cmdline section is added to the current .cmdline content of the UKI.

All addons provided in RHEL are signed with the same RHEL key that is used for signing the UKI. For further details regarding the key utilized by RHEL, please refer to a previous blog post.

Utilizing ukify for the generation of UKI addons

Ukify was formally integrated into the RHEL systemd package beginning with version 252-20.This tool proves to be advantageous for creating a UKI, as it offers greater versatility compared to conventional tools such as objcopy and dracut. In addition to generating a UKI addon, ukify also allows for inspection of these addons, enabling users to view the contents of the text sections within a UKI and its addons.

The ukify application can create a signed UKI or UKI addon using either pesign (which is included in RHEL) or sbsign. Furthermore, it provides the capability to inspect the addon, which is particularly useful since a PE binary is not in plain text format. For more information on its features and usage, please consult the ukify documentation.

The tools specifically designed for RHEL include kernel-bootcfg and kernel-addons, both of which are components of the virt-firmware package. The former facilitates the addition, removal, and updating of a Unified Kernel Image (UKI), while the latter performs similar functions at the level of UKI addons.

Secure Boot Advanced Targeting (SBAT)

In the event that a vendor releases a new UKI addon and subsequently discovers an error or identifies that the command line is vulnerable to attacks, several approaches can be considered to address this issue:

1. It is impractical and unfeasible for Red Hat to alter the key used for signing the addons, as using the same key for both UKIs and addons would render all previously generated UKIs and addons invalid.
2. Include the addon hash in a blocklist: This method is also not feasible for Red Hat and carries a risk of errors.
3. Adjust the attestation process: This would involve filtering out the hash corresponding to the blocklisted addon and rejecting it. This approach is unfeasible and susceptible to mistakes.
4. Implement SBAT regulations: This entails recognizing the approved generation number and systematically rejecting all add-ons that possess a lower generation number. This is the only viable solution, as it enables Red Hat to reject addons without necessitating a change to the signing key or hardcoding any information related to the addon content.

A .sbat section is included in both a UKI and its addons. This section consists of text, and as specified in the SBAT documentation, each line adheres to the format of component, generation, vendor, package, package version, and URL. The critical aspect is the component,generation tuple.

Please observe the various elements involved:

• sbat: This refers to the SBAT mechanism itself. The version number is incremented when the mechanism encounters bugs or security vulnerabilities.
•  systemd: The upstream version of systems.
•  systemd.rhel refers to the version of systemd that is tailored for Red Hat Enterprise Linux (RHEL).
•  linux denotes the original version of the Linux kernel.
• linux.rhel: The downstream version of the Linux kernel for RHEL.
• kernel-uki-virt.rhel: The RHEL UKI, which does not have an upstream counterpart.

Mapping these components is straightforward. For instance, in the systemd.rhel component mentioned earlier:

•  Component = systemd.rhel
• Generation = 1
• Vendor = Red Hat Enterprise Linux
• Package = systems
• Package version = 252-22.el9
• Url = mailto:secalert@redhat.com

For the mechanism to function correctly, identical tuples must be present in the shim and are validated at startup against the UKI components, immediately following signature verification. If the shim generation for a corresponding component is greater, the binary will be rejected, even if it is signed by a key that exists in the SecureBoot database.

To illustrate, consider that the shim contains the following SBAT variables:

An addon is identified during the boot process with the following .sbat:

In this scenario, all (component, version) pairs are in agreement, thus the addon is deemed acceptable provided it is signed by a trusted key. However, if an issue with the addon arises later, the shim version is updated to 2:

At this juncture, any other addon with my_addon version set to 1 is automatically rejected, irrespective of whether the signature is trusted.

Shipping RHEL UKI Addons

RHEL currently distributes UKIs within the kernel-uki-virt sub-RPM. Additionally, a new sub-RPM, kernel-uki-virt-addons, is provided, which includes the RHEL signed and supported add-ons. Upon installation, these add-ons are automatically placed in the /usr/lib/modules/(uname -r)/ directory, which serves as the designated root filesystem location for storing non-installed UKIs and add-ons. This arrangement is intended to prevent the ESP from becoming cluttered with unnecessary add-ons and to conserve the limited available space.

Tools such as kernel add-ons assist users in installing the selected add-ons in the appropriate ESP locations, ensuring their utilization at the next boot.

To illustrate, in order to install the FIPS enabler addon (provided that the kernel-uki-virt-addons.rpm has been previously installed):

In this instance, the addon path is specified as /usr/lib/modules/(uname -r)/vmlinuz-virt.efi.extra.d/fips-enable-virt.fedora.x86_64.addon.efi, while ‘My_UKI’ represents the title of the UKI as designated during its creation. The kernel-addons tool offers a variety of options for installing, deleting, updating, and listing addons. To view all available flags, it is advisable to install the tool and execute the command kernel-addons –help.

Summary

RHEL UKIs enhance security by integrating the kernel, initramfs, and command line. Addons for UKIs, which are signed by RHEL, help maintain these security advantages while allowing for the flexibility associated with traditional methods, where users can modify artifacts post-installation. A range of tools is available to facilitate the creation, inspection, and installation of addons in the EFI System Partition (ESP), as well as the ability to revoke potentially harmful addons, including those signed by a trusted certificate. In summary, RHEL UKIs play a crucial role in SecureBoot and Confidential Computing.