The “upstream first” strategy is only half of the story for embedded devices
A recent blog post by Kees Cook, a Linux security expert and Google security engineer, illustrates the challenges in maintaining the security of the Linux kernel. One of the main takeaways from the blog is: “If you’re not using the latest kernel, you don’t have the most recently added security defenses (including bug fixes).”
Going the “upstream first” route is the absolute best way of keeping the kernel secure. However, it is only part of the story. The challenges faced by device manufacturers running on Linux on embedded devices is vastly different.
Most device manufacturers rely on the semiconductor vendor-provided Linux kernel as part of the Board Support Package (BSP), in particular for non-x86 devices. In the interest of going to market early, the semiconductor vendors typically freeze a version of the upstream kernel and add their patches to support their System On Chip (SoC) / Processor.
Now the device manufacturers are stuck on a version of the Linux kernel that no longer receives security or bug fixes, whereas the fixes are made available in both the stable and long term supported (LTS) upstream kernel releases. In order to get these fixes, the device manufacturer now has to apply the semiconductor vendor patches (varies from 10s to 10000s) on top of the upstream kernel, which tend to result in conflicts that are difficult to resolve. Once the conflicts are resolved, testing the updated kernel poses a whole another set of challenges where all the sub-systems and drivers need to be retested to ensure they did not break anything.
Now imagine doing this over and over every week, because on an average that is how frequently a new version of the LTS kernel is released!
In an ideal world, all of the SoC specific patches are upstreamed in a timely fashion by the semiconductor vendors so that the device manufacturers can use the upstream kernel as-is to get security/bug fixes. Unfortunately, the reality is far from it.
Upstreaming patches is a time consuming process and semiconductor vendors are typically in a rush to support new processors/variants while fixing bugs in their older BSP releases. While many semiconductor vendors actively contribute back their SoC specific patches to the upstream kernel, only a subset makes it to the upstream kernel. Hence the next BSP release from the SoC vendor ends up with a combination of patches supporting old and new processors/variants, resulting in the number of vendor patches in the frozen vendor kernel being more or less the same…and the cycle repeats with each release.
As a device manufacturer, it is not obvious if all of the patches for your processor have been upstreamed by the vendor, so the safest starting point for the Linux kernel is the vendor-provided kernel. This means that “upstream first” is not an option for most device manufacturers.
Another aspect to consider is maintaining the security of open source software that runs in user space. A Linux based embedded device comprises both the Linux kernel and a large number of user space libraries/utilities typically built using a buildsystem such as Yocto/Buildroot. Many of the critical vulnerabilities that are remotely exploited in IoT devices are often in open source user space software (e.g.: openssl, curl, openssh, etc.). So it is equally important to keep these software up to date with the latest and greatest security fixes.
Unlike the Linux kernel where the API compatibility to user space is maintained, only a handful of user space libraries provide LTS releases where API compatibility is maintained. The implication is that everytime you update a 3rd party library, you might have to update your application and re-test the whole system, which makes security maintenance all the more cumbersome. Further upgrading one package might result in a cascading chain of dependencies that need to be updated, sometimes all the way down to the toolchain and/or buildsystem.
Many embedded devices are expected to be supported for a period of 10 years or more. For device manufacturers without in-house Linux expertise, resources, and the right set of test automation tools, Linux OS security maintenance becomes challenging, expensive and takes the focus away from their value add. So unless you are an extremely well-staffed organization, the simple advice of updating to the latest and greatest version to get security fixes is no easy feat. The easiest solution — which also happens to be the most economical — is to engage companies who can bridge your resource gap by maintaining Linux OS security for you.
– – – – – – – –
At Timesys, we have the tools and expertise to maintain Linux OS security for the full lifecycle of your device. Learn more.
About Timesys
Timesys has extensive experience with embedded system development and lifecycle management. Timesys has been instrumental in working with global leader semiconductor manufacturers with smart, quick and quality solutions for highly complex systems with accelerated product innovation and multiple product variants.
