After Notification: The Next Steps
In a previous blog, we covered how Timesys handles security monitoring and notification of open source software vulnerabilities, how to generate reports on demand for the current state of a Yocto or Factory build on the desktop, and how to view, generate, and subscribe for reports on the web. If you missed it, now would be a good time to catch up before reading this post, because the next steps cover what to do with the information contained in those reports. Specifically, you may have the following questions:
- What should I fix?
- Where do I find the fixes?
- How do I apply fixes to my build?
We’ll start by explaining the meaning of the subcategories of “Unfixed” CVEs and the “Vector” column in the reports, and then break down each of the above questions. Along the way, you’ll see how the solutions offered by Timesys can save you countless hours spent searching for patches, applying them to your build, and dealing with conflicts that arise when upgrading.
Interpreting the Report
First, anything that is marked “Fixed” is already patched. We want you to know that these CVEs were relevant to a particular version of a recipe in your build, but your configuration already includes a patch to fix or mitigate the issue.
Interpreting the report gets more complicated with the items that aren’t fixed, because we want to highlight the different cases where you can do something about it. Here are the subcategories and their definitions:
- Unfixed — An “Unfixed” CVE is one known to affect this component at the current version, and at this time no fix is readily available locally or from Timesys.
- Unapplied — An unfixed CVE that is marked as “Patch not Applied” is one with a fix available in your copy of the meta-timesys-security Yocto layer or your local Factory, but it is not being applied in your build configuration.
- Upgrade Available — An unfixed CVE that is marked as “Upgrade Available” is one that can be fixed by upgrading your local meta-timesys-security Yocto layer or Desktop Factory to the latest version, which contains the patch or update.
For Factory, the fixes are built in to the recipes as default patch lists. An unapplied fix means that your build’s patch list differs from the current default which includes fixes you aren’t using. Running “make menuupdate” and accepting the package’s updates will take care of that. Likewise, updating the engine itself brings in the new default patch list containing the fixes reported as fixed in a newer Factory version, which can then be applied through the same update mechanism.
For Yocto, an unapplied patch means that you are already using meta-timesys-security, and it has a patch for the CVE, but for one reason or another it isn’t currently applied. If the fix is available in an upgrade, it means your copy of meta-timesys-security is out-of-date and that updating it will pull in new patches for that recipe. There is more discussion about how the patches are handled later in this post.
What Should I Fix?
You may be surprised by the large number of CVEs in the report. Of course, the older the build, the more vulnerabilities you’re likely to find as the package versions are probably lagging behind what is available upstream. This is especially true with the kernel, where you could see hundreds of CVEs relevant to some versions.
The good news is, you might not need to fix everything. While the listed CVEs are all relevant to the package itself, they may not all actually be relevant to how you use that component. For example, many of the kernel CVEs may be for drivers or features that you don’t include in your product’s kernel configuration. Another piece of software may leak data through an API that you don’t enable or expose.
Sifting through every CVE to determine it’s direct effect on your product takes effort that may be largely unavoidable. Usually, Timesys doesn’t have knowledge of what your proprietary application is really doing with the platform (BSP Maintenance Services aside). You can start with the report on the web and use the direct links to detailed CVE information to understand what exactly it impacts, along with the “Vector” column to understand how it might be exploitable. The vectors are defined in CVSSv2 as follows¹:
- Local — The attacker must either have physical access to the vulnerable system (e.g. firewire attacks) or a local account (e.g. a privilege escalation attack).
- Adjacent Network — The attacker must have access to the broadcast or collision domain of the vulnerable system (e.g. ARP spoofing, bluetooth attacks).
- Network — The vulnerable interface is working at layer 3 or above of the OSI Network stack. These types of vulnerabilities are often described as remotely exploitable (e.g. a remote buffer overflow in a network service).
This may seem like a lot of work, but consider how your time is being spent. At this stage you are using the knowledge about your product to make import decisions about it. The starting point, the list of vulnerabilities and exposures tailored to a specific build which Timesys has provided, represents a huge savings in time and effort for your team that would have been spent sifting through the busy and noisy security atmosphere just to get to this point.
Finally, consider the stage your product is in. It may be reasonable to accept all updates, keep moving to the latest versions, and take all available patches as you go until you hit a certain point where things get pinned down.
Where Do I Find the Fixes?
Now that you know which CVEs to fix, you need to obtain the patches or updates that take care of each issue. Not surprisingly, there are a few ways to do this. First of all, your build engine’s maintainers might have already found it for you! Check for Factory updates or for newer versions of the Yocto layers that provide vulnerable recipes. Unfortunately, most projects hit a point where huge upgrades aren’t really an option, and you’ll need to backport specific patches to your existing version. That’s where things start to get tricky.
If there is a fix in a newer version of the Factory or Yocto layer, it may not be preventatively difficult to backport that fix yourself, as long as you’re handy with git and know how to handle patch fuzz (the offsets that occur when trying to apply a patch from one context in a slightly different one). You may run into trouble if the patch doesn’t apply cleanly, or your team doesn’t have experience with creating bbappend files to modify Yocto recipes, etc.
If there is no fix in a later version that you can grab, you can start hunting. Our reports link to the CVE details in the NIST National Vulnerability Database, where there is often a reference section with useful links to bug reports and fixes (which may even include a “Patch” tag to help you sort out which links to visit). Find the website for the software package itself and look for an announcement, or scan its changelog / source history for fixes. It might still take some work to apply the proposed fix to the version you have. If information is still lacking, you can scan mailing lists (security lists, yocto project lists, individual project lists) for announcements, search for blog posts, and find websites devoted to specific vulnerabilities (usually with catchy names and logos!). They might point to which version of a package to update to, or directly to a patch for a current version.
This is where the Timesys patching offering for Yocto shines. For many commonly used recipes and everything found in some of the core images, we maintain patches for CVEs backwards across the previous two Yocto releases in addition to the current release. For individual CVEs, the report links straight to the commit in meta-timesys-security that provides the patch file. Getting the patches is as easy as dropping the layer into your BSP and inheriting a class in your config.
How Do I Apply Fixes to My Build?
Once you’ve identified the CVEs that matter and found the patches you need, how do you actually apply them? What concerns are there?
In Factory, new default patch lists are obtained by upgrading to a newer release of the build engine. Then, updates can be reviewed and accepted by running “make menuupdate”. As long as you can take all the patches as tested by Timesys, that’s it! For web builds, the equivalent is forking your existing build with the box checked to upgrade to the latest Factory, then accepting updates and saving on the “summary” page of the wizard. To apply patches selectively, you can use the default patch list as a guide, but set the variable in the workorder to only the files you want to include. For custom patches, you should add a local patch to your source directory and include it in the Workorder. These patches are applied after the default patch list that Timesys has tested. In either case, you may need to modify the patch lists to get them to cleanly apply (because patch order matters!). For most packages in Factory, CVEs are fixed by bumping the package to the latest version. If you take that upgrade, you don’t have to worry about patch lists, unless you have customizations in which case you’d need to identify local patches to drop or forward-port if they’re still needed.
In Yocto, when not using our patching services that provide meta-timesys-security, you would create a bbappend file for each recipe to add your patch or patches to the “SRC_URI” variable, thus telling bitbake to include your patch along with the recipe’s other files. Likely you’d be maintaining these bbappends in your own custom metalayer, and manually updating them over time when you do BSP updates.
Timesys has implemented meta-timesys-security in a way that automates most of this work from your team’s perspective, and you don’t have to upgrade Poky, or maintain your own fork of it to modify specific recipes yourself. Just keep meta-timesys-security up to date and the fixes will keep showing up! Not constantly modifying the layers which provide these recipes themselves has another benefit too. When it is time for a major BSP update, you won’t have a tangled web of conflicts to resolve or changes to forward-port. In fact, meta-timesys-security will automatically detect when the providing layer includes a CVE fix, and defer to that layer’s fix by removing our patch dynamically to prevent a conflict. This happens for each build without human intervention. We take the time to find and backport patches, and to append the recipes across multiple Yocto releases, so that you don’t have to. And they are contained in a layer that we maintain for you too, kept separate from your custom layers and all the layers maintained by others so that there is no added maintenance headache if you upgrade components of your BSP.
More Things to Come
We’re continuing to develop meta-timesys and meta-timesys-security to add features and enhance the user experience. We already offer drop-in meta layer solutions that save time finding and fixing CVEs, avoid maintenance headaches associated with backporting patches and modifying recipes, and automatically handle complicated situations like recipe versions offset by git revisions. But we have more helpful features on the way, including the kernel CVE filtering based on kernel config and whitelisting of CVE IDs that you don’t want to fix (at least not yet).
It can be overwhelming to find and apply patches for CVEs, then track the current status in your build (Is it applied or not? What did that patch fix?) when you’re doing it yourself. Of course, that’s kind of the point of this blog; you don’t have to do it yourself! The previous post about notification stressed all the time needed to keep up with CVEs and not miss something important. Timesys has a solution for that, through easy on-demand and periodic by-subscription reports. This post shows the immense effort needed to do something with that information by finding, applying, and maintaining the patches. Timesys also has a solution for that through its patch notification service!
Your feedback is always welcome! Let us know where your team finds pain points, has good or bad experiences, or needs more help.
Does doing the patching still seem like too much? Do you want us to do it for you? Find out how we can work with you to lighten the load by applying and testing updates through our BSP Lifecycle Maintenance Service.
Timesys has extensive experience with embedded system development and lifecycle management. Timesys has been instrumental in working with global leader semiconductor manufacturers with smart, quick and quality solutions for highly complex systems with accelerated product innovation and multiple product variants.