Scaling software development is a perennial problem. As product lines grow, maintenance costs for device manufacturers can increase dramatically when software is ported and reused from one product to the next. This can balloon into major issues like security vulnerability duplication, increased testing, and difficulty getting new products up and running.
Challenges and Solutions
This challenge multiplies when we consider that a modern embedded Linux distribution (commonly shortened to “distro”) can be made up of hundreds of independent software packages.
To help offset these challenges we ideally want to avoid copy-pasting code and instead reuse as much common code between product lines as possible. We can leverage Yocto project features to reuse system, kernel and bootloader recipes between similar products. We can even go as far as running the exact same software binaries on different hardware platforms.
Overview
When considering opportunities for software reuse between hardware product lines, it’s helpful to take a holistic look at all the software sources which compose the product’s functionality. We can logically break them up into the following software groups.
Bootloader
The software which handles initializing the hardware at boot, typically compiled small enough to fit in processor SRAM, and often involves multiple stages. At its most basic the bootloader simply configures the processor and DDR memory, then copies the kernel code into memory for execution, but it can be responsible for much more. The source code varies by processor vendor but the u-boot bootloader is common (often a vendor provided fork of u-boot).
Kernel
The Linux kernel and related binaries. Handles configuring and managing hardware peripherals and provides interfaces for userspace. Like u-boot many vendors provide their own vendor maintained kernel that is rebased regularly on the Linux kernel.
Userspace
The large variety of software that encapsulates userspace: init systems, libraries, application software, etc. Generally where most of a product’s value is derived from and where you will want to expend most of your software effort to focus on creating the best product.
Attempting to commonize these groups across product lines often comes with various pitfalls that we will highlight as we go along.
Common Meta Layer
Before we touch on each software group and how we can commonize them, let’s make a quick pit stop and discuss how Yocto enables this code reuse. Software recipes within Yocto projects are organized within meta layers, and an individual product will usually make use of several meta layers to bring in different hardware and software support as illustrated in the below diagram.
The above image is from Bootlin’s free “Yocto Project and OpenEmbedded system development training” PDF.
Normally each product using the Yocto project would have its own high level meta layer where it can create a custom image recipe and override any lower-level recipes that it needs to. When we are working to share the bootloader, kernel, and userspace software recipes between a product line, we recommend making a common meta layer that all the products can include and use. While we don’t go into the details about how to create a new meta layer in this article, the Yocto project has some great existing documentation that describes that process here.
Common Bootloader
The bootloader handles initializing the hardware at boot and preparing the system to handoff execution to the kernel. For this article, we’re going to focus primarily on u-boot when discussing bootloaders, but these fundamentals often transfer to other common bootloaders.
When we talk about creating a common bootloader, it can be commonized by reusing the same source code to compile multiple binary images for each product, or we can go a step further and use the same compiled binary on multiple products.
How can products effectively share bootloader source code?
Yocto has great built in support for handling sharing the source by reusing the bootloader recipe. It’s as simple as creating a machine configuration file in your products’ common meta layer and setting the PREFERRED_PROVIDER_virtual/bootloader. For example if you had multiple products using the Freescale provided u-boot-fslc you would include the following line in their machine configurations:
PREFERRED_PROVIDER_virtual/bootloader = “u-boot-fslc”.
And it’s that simple, now if you need different u-boot configurations for device drivers on different products you can use machine overrides to add config fragments that will apply on top of the u-boot defconfig, or just configure Yocto to patch in a complete defconfig for each individual machine.
How can products effectively share a bootloader binary?
Sometimes it is useful to run the same bootloader binary across many products. The Raspberry Pi bootloader is a great example of this, their closed source bootloader can run on all of the many RPI variants which greatly simplifies distribution and maintenance. Support for this requires the hardware to have some method of identifying a board type like an EEPROM containing a board ID. At boot time, the bootloader can read the board ID, then choose which Device Tree Binary (DTB) to load with the Linux kernel.
Potential Pitfall:
The bootloader is one of the most hardware dependent pieces of software for an embedded system. Because of that, any differences in processor or memory between different products can make it difficult or even impossible to create a common bootloader.
Recommended Solution:
The only real option here is to plan ahead when designing new products by attempting to use the same SoMs and SoCs between products. And if that is not an option for your products, then the good news is that bootloaders are usually pretty simple so duplication between products can still be relatively minimal.
Common Kernel
The Linux kernel handles initializing any hardware/peripherals that the bootloader did not, mounting the filesystems, starting userspace init, and providing interfaces for userspace applications to access hardware. It is made up of the following components:
- Linux Kernel: The binary image for the kernel, can be a uImage, zImage, or Image file.
- Device Tree Binary: The binary form of the device tree structure. Used by the kernel to know which hardware peripherals are available on this board
- Initramfs: A small temporary root filesystem mounted in RAM that provides an early userspace to provide utilities for initializing and mounting the primary root filesystem.
How can products effectively share kernel source code?
The following will be very reminiscent of what we just did for u-boot because Yocto uses a lot of the same idioms between u-boot and linux recipes. First off, to use the same kernel recipe for your products’ common meta layer, set the PREFERRED_PROVIDER_virtual/kernel for each machine. For example if you had multiple products using the Freescale provided linux-fslc linux kernel you would include the following line in their machine configurations:
PREFERRED_PROVIDER_virtual/kernel = “linux-fslc”.
Then by taking advantage of the kernel Kconfig system, we can gate any product specific drivers behind the conditional CONFIG_* options. Just like u-boot you can have Yocto swap out those configs at product build time by using machine overrides to add config fragments that will apply on top of the defconfig, or just configure Yocto to patch in a complete defconfig for each individual machine.
The big advantage of doing all of this is now you have one place to patch any security vulnerabilities for all your products. So when an exploit or bug gets discovered for the kernel you can apply the fix in one place and then deploy the update to all your products much faster.
How can products effectively share a kernel binary?
To ease distribution, you may want to run the same compiled kernel binary for all of your products, this is possible assuming the product line is using the same instruction set architecture (arm32, arm64, x86, risc-v, etc).
For example: If you have one core kernel team that regularly releases N Linux distributions to N product teams, the scaling can quickly get out of hand for that one kernel team to keep up as N increases with new product releases.
By commonizing into one binary you can greatly reduce the number of release variations that need to be tested, maintained, and released. To accomplish this at run time the kernel binary will have to be compiled with a superset of driver support for all the different board types. Then the bootloader will need to know what board it is booting on and pass the kernel a board appropriate device tree.
We will have to pack up multiple device tree binaries, one for each distinct product alongside our common kernel:
From the above image you might have noticed that the kernel and associated binaries are being stored in a FIT Image. This stands for Flattened uImage Tree (FIT), and is basically a container for all the kernel components. This FIT type is designed for this use case and includes configuration nodes that define valid combinations to boot from; all the bootloader has to do is specify which product configuration to boot with. FIT images are not strictly necessary for this, the bootloader could just pick a device tree and pass it to the kernel, but generally FIT images are the right tool to use here (and come with a plethora of other advantages).
If you want to learn more about FIT images, Andrew Murray has written a great article that talks about their design and uses here.
Potential Pitfall:
Creating one binary kernel for several products isn’t always possible. Sometimes you have conflicting config options that just can’t be compiled in simultaneously. An example would be a legacy product using the display framebuffer (FB) drivers, and a more modern product using the newer Direct Rendering Manager (DRM) drivers for graphical displays. Those two options are just incompatible with each other because you can’t have more than one display manager (and wouldn’t really want to).
Recommended Solution:
Legacy products are often pretty resistant to a change as large as migrating to the newer display drivers because changes like that can have big upstream ramifications for preexisting application software that depends on those FB interfaces. So if you find yourself in a situation like this where you just have to have multiple kernel binaries, but still want to create one binary image to rule them all, you can look to FIT images for a helping hand. FIT images are very flexible and can contain multiple kernel binaries:
Common Userspace
Userspace is where all the really interesting software lives, it’s where your users will interact with your device interfaces and get most of the value from your product. It is made up of the following components:
- Init: This is process ID 1, the first process the kernel will start that is responsible for starting all other user processes. There are a variety of Linux init systems available but most modern embedded systems run systemd or sysvinit.
- System Libraries: This is a bit of a catch-all term, but it really encapsulates every shared library, the shell, and every utility that is necessary for the system to boot and provide dependencies for the application software.
- Application SW: This is the custom software that handles all the product functionality and requirements.
Most of the userspace software is stored in the device’s root filesystem partition (RFS), and application software is often stored in its own filesystem partition that can be updated separately.
How can products effectively share userspace source code?
Yocto starts with your product’s image recipe to determine which userspace software to compile and include in the root filesystem. To help avoid source and recipe duplication between product images you can create a common image recipe that products can “require” in their image recipe, then add any product specific software on top of it in the product’s image recipe. Easy right?
How can products effectively share their userspace binary code?
Where things can get complicated is if you want to load the same RFS binary on multiple products. This can simplify distribution but in exchange you’ll have to consider all of your product line’s application dependencies with the following questions:
- Are any of the products using a different instruction set architecture?
- Can the application software for the entire product line run with the same software dependencies?
- If they can, are there conflicting shared library version requirements?
- Will including every software library that any product might need balloon your storage requirements beyond your hardware’s physical capacity?
If you answer no to all of these questions, then creating one RFS binary will just require packaging a superset of all the application dependencies in your common Yocto image.
But if you answered yes, then this is where software containers like Docker can save the day. Each application will be packaged up in a container that can be independently updated and stored on the application filesystem partition. This allows application developers to package up all their unique dependencies with their application and not burden the operating system with them. Docker is outside the scope of this article but we do have an article from Brian Bender introducing containers in embedded Linux.
Conclusion
Expending the effort to commonize your bootloader, kernel, and userspace software can pay dividends in avoiding the maintenance, distribution, and security issues that naturally arise with scaling, and provide you the flexibility to get new products out the door quickly. So if you find yourself contemplating expanding device product lines and struggling with wrangling all their duplicated software sources then hopefully this article provided you with the strategies to tackle that challenge with confidence.
Looking for assistance with Long-Term BSP Maintenance?
Maintaining your Operating System (OS) and Board Support Pack (BSP) to stay ahead of security threats can be complex, time-consuming, and even expensive if you do it yourself. If you’re looking for a Linux OS/BSP Maintenance service that is less than half the cost of a junior engineer and that can free up your resources to work on next-gen products, consider our Long-Term OS and BSP Maintenance solution.
With Long-Term OS and BSP Maintenance by Timesys, you can take the complex, time-consuming work of Linux OS and BSP maintenance off your plate for the full ten, or more, year lifecycle of your device. Our subscription service provides long-term security updates and maintenance and is available for Yocto Project, Buildroot, and Timesys Factory build systems.
Simplify Security with Timesys
With more than 20 years of embedded development experience, Timesys is a pioneer and industry leader in open source software security, development tools, and engineering services and consulting, spanning the embedded software market. At Timesys, our goal is to keep you ahead of the next security threat, help you design security into your devices and keep them secure while in the field—AND get your products to market faster.
How? By simplifying the security process with a comprehensive suit of products and services that offer end-to-end device security, development, testing, and maintenance throughout the entire product lifecycle – from Software Bill of Materials (SBOM) management, CVE monitoring and remediation, to security feature implementation services, remote test automation and access infrastructure, and more.
Contact us today to begin your journey to a more simplified, secure product.
About Timesys
Timesys has extensive experience with embedded system development and lifecycle management. Timesys has been instrumental in working with global leader semiconductor manufacturers with smart, quick, and quality solutions for highly complex systems with accelerated product innovation and multiple product variants.
