Monitoring and managing vulnerabilities in embedded Linux devices presents a unique set of needs that traditional IT vulnerability tools fail to address and result in wasted efforts chasing false positives and inefficiencies due to cumbersome workflows. After evaluating multiple IT cybersecurity tools, we at Timesys ended up creating a vulnerability management tool called Vigiles, which is optimized for embedded devices. This blog aims to share the lessons learned and how the right tool can bring your security maintenance cost down while improving the security posture of the device.
Generating an SBOM
Software Composition Analysis (SCA) tools generate a software bill of materials (SBOM), which is a list of software components installed on your product, and then create a vulnerability report by comparing the components against a known vulnerability list. There are a few different ways of generating the SBOM as discussed below:
Build System based
The best option for generating the SBOM is to extract the component information from the build system. Build systems have all the information required to generate an SBOM along with additional metadata that can help improve the accuracy of vulnerability reports. The metadata consists of:
- Configurations enabled in the software (e.g., drivers enabled in the Linux kernel)
- List of vulnerabilities already addressed based on applied patches
- Hardware platform information
This additional metadata can be used by the vulnerability monitoring tool to reduce false positives by reporting “unfixed” vulnerabilities applicable to “your hardware platform” based on “enabled configurations.” In components like the Linux kernel, this can reduce the number of vulnerabilities by up to 75%, saving you investigation and remediation time.
The two popular build systems supported by semiconductor vendors are Yocto and Buildroot, so your ideal tool of choice should integrate with these build systems. With Vigiles, we created open source software to generate an SBOM with vulnerability data that directly integrates with Yocto (meta-timesys) and Buildroot (vigiles-buildroot) to address the needs of embedded systems.
Traditional IT approach
Binary scanners: Generate an SBOM by analyzing uploaded target binary images based on file signatures. This works fairly well where the SBOM and/or build sources are not available. However, below are the drawbacks of such an approach:
- False positives: The metadata (configurations enabled, patches applied, etc.) available in build systems is not part of the binary image. So if upstream Yocto or Buildroot has patched a vulnerability, a binary scanner shows it as “unfixed.” Similarly if a vulnerability affected only a driver or a platform not applicable to your product, they would still be reported.
- Missed vulnerabilities: Sometimes the SBOM cannot be accurately generated because the component and/or version cannot be determined based on a binary signature (e.g., a newer or older piece of software that is not cataloged in the scanner database).
Example tool: Synopsis Blackduck
Source scanners: Generate an SBOM by analyzing source code and the scanner typically integrates with version control software or application package managers (e.g., npm, nuget, python pip, godep, etc.). This works fairly well for application software that pulls in a handful of open source components, but it falls apart when it comes to working with build systems like Yocto and Buildroot for the following reasons:
- The target device SBOM can not be easily generated. In order to generate a custom Linux distribution (OS), build systems pull in thousands of open source packages. Some packages are needed by the host machine to build the toolchain, while a few others are build-time dependencies and only a subset of packages end up getting installed on the target device. Source scanners are not able to identify the installed packages and end up reporting all software as part of the SBOM. Manually specifying which sources to monitor is tedious and impractical.
- If you did manage to generate an SBOM, the source scanner approach still suffers from the false positives since it can not collect build-artifacts such as patches applied by build system, kernel configuration, etc.
Example tool: Whitesource
- Network scanners such as Tenable Nessus, Rapid7 InsightVM, nmap, etc. can only determine software running based on open ports on the device and cannot generate a complete SBOM and vulnerability report.
- Querying the live target device: If the device uses a package manager, it can be used to generate an SBOM. This is typically the case when using standard Linux distributions like RedHat, Debian, Ubuntu, and their corresponding security trackers can be used to monitor vulnerabilities. This is outside the scope of this discussion.
Note that while standard Linux distributions have security trackers for vulnerability monitoring, the attack surface (number of lines of code) is much much larger than a custom embedded Linux image based on Yocto/Buildroot/Timesys Factory. One of the security best practices recommended by standards such as NIST 8259A and ETSI EN 303 645 is to reduce the attack surface.
SBOM generation comparison
– based on patches applied, configurations and hardware info
Once an SBOM is generated, the next step is to create a list of vulnerabilities affecting the components. Most tools use the National Vulnerability Database (NVD) maintained by the National Institute of Standards and Technology (NIST) to report vulnerabilities, which are called Common Vulnerabilities and Exposures (CVE) in the database. In this section we dive into the need for more accurate curated data than that which is reported in the NVD.
The way vulnerability reporting typically works is a security researcher finds a vulnerability, reserves a CVE-ID and does a responsible disclosure allowing the maintainer of the affected product to investigate and patch the vulnerability if possible. The vulnerability then appears in the NVD, and the NVD maintainers analyze the vulnerability and add metadata such as Common Platform Enumeration (CPE) which allows automated tools to parse which CVE affects which software components and versions in a standardized way. While this sounds great, the actual implementation has multiple weaknesses as outlined below.
- Missed CVEs and false positives: In our analysis, at least 40% of the NVD CVE entries have incorrect CPE data, resulting in missed CVEs or false positives. A missed CVE compromises the security of a device because the device’s exposure to the CVE is basically unreported. A false positive leads to wasted analysis time because the CVE appears to affect a device, which leads device maintainers to expend time investigating only to find it does not. The reasons for the inaccuracies are as follows:
- Inconsistent naming (e.g., same package named differently for different CVEs)
- Typos (e.g., mistakes in entering the package name or affected version)
- Incorrect/incomplete analysis (e.g., not all affected versions of the software are listed, etc.)
- Outdated information (e.g., when a patch gets backported to say LTS releases of kernel, the affected range of the package for that CVE is not updated)
- No version information (e.g., only package name without version information is entered)
- Delays in reporting: This is the time delay from when the vulnerability becomes public (news, mailing list, etc.), to when NVD analyzes the CVE for your tool to actually report the vulnerability against your product. The average delay in 2018 was on the order of months, leaving you in the dark about critical vulnerabilities if your tool solely relied on NVD.
Find more information in our analysis presented at the Embedded Linux Conference.
The way we addressed these vulnerability data issues with Vigiles:
- Multiple sources: Adding multiple sources of vulnerability information such as Ubuntu, Debian, RedHat, SoC advisories, security bulletins and mailing lists.
- Curated data: We have a dedicated security research team that analyzes the data from the above sources and fixes any inaccuracies, specifically in the context of packages available in Yocto and Buildroot. Additionally we have intelligent curation algorithms that figure out if a vulnerability is backported to Long Term Stable (LTS) versions of software by analyzing git commits that fix the vulnerability.
The result: Up to 40% improvement in accuracy over NVD and up to 4 weeks earlier notification.
When evaluating tools, verify that claims of “lowest false positives” and “enhanced coverage” are in the context of your usage.
Ease of monitoring
In order to improve efficiency when monitoring vulnerabilities, the features to look for in a tool, which Vigiles offers, are:
- Alerts: Ability to trigger alerts of new vulnerabilities that affect your SBOM.
- Comparison: Highlighting what changed between scans or SBOM manifests. This way you can focus on the changes between scans or builds.
- Advanced Search: Ability to search for CVEs by package name and version, or by ID to get in-depth Timesys curated vulnerability information.
Intelligent applicability filters
Now that you have a list of vulnerabilities affecting based on your SBOM which ones are truly applicable? Adding intelligence to tools can cut up to 75% of the time spent chasing false positives. In Vigiles, we offer:
- Configuration-based filter: While the Linux kernel itself typically has hundreds of vulnerabilities, only a quarter of them are applicable based on the drivers/modules being used. Tools that take into account the Linux kernel configuration and filter out ones not applicable can reduce your analysis time by up to four times. The same applies to other software components like U-Boot.
- Platform-based filters: Some vulnerabilities are OS/Platform specific (e.g., Android, Windows, Linux). Platform filters enable you to filter out the ones not applicable to your platform.
- Hardware architecture filters: Vulnerabilities can be processor specific (e.g., x86, ARM, PowerPC, etc.) and can be similarly filtered out.
- Attack vector filters: Vulnerability data typically accompanies how the vulnerability can be exploited. Example: Remotely exploitable (Network), Local network exploitable like LAN or Bluetooth (Adjacent), Local user exploitable (Local) or exploitable via physical media like USB (Physical). IoT devices might care only about network attack vectors whereas air-gapped industrial devices might care only about physical attack vectors.
- Status filters: When using build systems such as Yocto or Buildroot, many vulnerability fixes are backported and already fixed by the maintainers. Status filters lets you filter out patched vulnerabilities and focus on the ones you need to address.
All of these applicability features are extremely useful for reducing the effort needed to manage vulnerabilities in embedded devices, and the features are unique to Vigiles.
The next step in the process is to determine risk to your product and deciding if the vulnerability needs to be addressed immediately or can be scheduled as part of standard security release. This process is referred to as vulnerability triage. To aid the effort, many tools including Vigiles, offer the features below:
- CVSS filter: The Common Vulnerability Scoring System (CVSS) is industry standard for assessing the severity of vulnerabilities. One can filter based on severity, for example High and Critical, to address those vulnerabilities first.
- Custom scores: Option to specify and filter based on custom severity or risk score for vulnerabilities.
- Custom notes: Ability to add custom vulnerability analysis notes.
- Whitelisting: Ability to ignore vulnerabilities that are not important or applicable. Lets you focus on just the important vulnerabilities in the report.
- Import/share triage data: Option to import or share triage data (notes/whitelist/scores) across platforms to avoid duplicate analysis effort.
Once you have a list of vulnerabilities that need to be addressed, the next step is to check if there are fixes available. Vigiles helps by providing accurate remediation information:
- Suggested fix: The minimum version to which the software needs to be upgraded in order to remediate the vulnerability. In the case of the Linux kernel this feature additionally specifies the minimum LTS version of the kernel in which the vulnerability is fixed.
- Reference links: Information regarding available patches, mitigation and exploits. In the case of the Linux kernel and U-Boot we provide a link to the mainline commit containing the fix and the config option that results in the vulnerability.
Workflow and product lifecycle management
Most tools are geared toward security engineers enforcing IT policies. Having a tool additionally geared towards developers can reduce the maintenance burden of engineers actually addressing vulnerabilities and maintaining product releases. Some of the Vigiles features we developed to aid this are:
- Product management: Ability to create products and folders (for releases, product variations, etc.) to which SBOM manifest can be uploaded from CI. This permits seamless integration of security maintenance into your existing Software Development Lifecycle (SDLC) process.
- Collaboration: Ability to collaborate with other team members and global security teams. This is typically used to share products or reports and to triage collaboratively.
- Comparison report: Allows you track vulnerability and SBOM differences between various releases and across products. This is particularly helpful for generating compliance documents and release notes in addition to tracking the security posture of a device over time.
- Download reports: Ability to create executive summaries or export data for creating custom dashboards (.csv, .pdf, .xls)
- APIs for integration: Ability to integrate with other SDLC tools such as issue tracker (e.g., Jira)
- Support: In addition to tool support, does the vendor provide access to security engineers who can assist with any vulnerability related questions?
Vulnerability monitoring for embedded systems has unique needs that are not addressed by traditional IT or application SCA tools. By using the best tool suited for the job, one can save up to 75% in developer time in investigating false positives and in security maintenance tasks while improving the product security. Tools geared towards developers, such as Vigiles, can help ease the pain of managing the ever-growing number of vulnerabilities in embedded and IoT devices. Even if your organization already has a generic tool for vulnerability monitoring, a specialized complementary tool for embedded Linux developers produces significant maintenance cost savings because of the unique requirements of embedded Linux vulnerability management.
- If you are a Yocto or Buildroot user, try out the Vigiles SBOM Manager. Register here and download meta-timesys or vigiles-buildroot to get started.
- Download our free SCA for Embedded Tool Evaluation Checklist as a guide to pick the best tool for your product needs.
While Vigiles lets you monitor and manage vulnerabilities yourself, a completely hands-off approach is available too. Check our BSP Security Lifecycle Maintenance service where we handle the periodic security updates for you!
Timesys has extensive experience with embedded system development and lifecycle management. Timesys has been instrumental in working with global leader semiconductor manufacturers with smart, quick and quality solutions for highly complex systems with accelerated product innovation and multiple product variants.