Login   |   1.866.392.4897 |   sales@timesys.com        English Japanese German French Korean Chinese (Simplified) Chinese (Traditional)
Evaluating vulnerability tools for embedded Linux devices

Evaluating vulnerability tools for embedded Linux devices

 

 

Monitoring and managing vulnerabilities in embedded Linux devices presents a unique set of needs that traditional IT vulnerability tools fail to address and result in wasted efforts chasing false positives and inefficiencies due to cumbersome workflows. After evaluating multiple IT cybersecurity tools, we at Timesys ended up creating a vulnerability management tool called Vigiles, which is optimized for embedded devices. This blog aims to share the lessons learned and how the right tool can bring your security maintenance cost down while improving the security posture of the device.

Generating an SBOM

Software Composition Analysis (SCA) tools generate a software bill of materials (SBOM), which is a list of software components installed on your product, and then create a vulnerability report by comparing the components against a known vulnerability list. There are a few different ways of generating the SBOM as discussed below:

Build System based

The best option for generating the SBOM is to extract the component information from the build system. Build systems have all the information required to generate an SBOM along with additional metadata that can help improve the accuracy of vulnerability reports. The metadata consists of:

  • Configurations enabled in the software (e.g., drivers enabled in the Linux kernel)
  • List of vulnerabilities already addressed based on applied patches
  • Hardware platform information

This additional metadata can be used by the vulnerability monitoring tool to reduce false positives by reporting “unfixed” vulnerabilities applicable to “your hardware platform” based on “enabled configurations.” In components like the Linux kernel, this can reduce the number of vulnerabilities by up to 75%, saving you investigation and remediation time.

The two popular build systems supported by semiconductor vendors are Yocto and Buildroot, so your ideal tool of choice should integrate with these build systems. With Vigiles, we created open source software to generate an SBOM with vulnerability data that directly integrates with Yocto (meta-timesys) and Buildroot (vigiles-buildroot) to address the needs of embedded systems.

Traditional IT approach

Binary scanners: Generate an SBOM by analyzing uploaded target binary images based on file signatures. This works fairly well where the SBOM and/or build sources are not available. However, below are the drawbacks of such an approach:

  • False positives: The metadata (configurations enabled, patches applied, etc.) available in build systems is not part of the binary image. So if upstream Yocto or Buildroot has patched a vulnerability, a binary scanner shows it as “unfixed.” Similarly if a vulnerability affected only a driver or a platform not applicable to your product, they would still be reported.
  • Missed vulnerabilities: Sometimes the SBOM cannot be accurately generated because the component and/or version cannot be determined based on a binary signature (e.g., a newer or older piece of software that is not cataloged in the scanner database).

Example tool: Synopsis Blackduck

Source scanners: Generate an SBOM by analyzing source code and the scanner typically integrates with version control software or application package managers (e.g., npm, nuget, python pip, godep, etc.). This works fairly well for application software that pulls in a handful of open source components, but it falls apart when it comes to working with build systems like Yocto and Buildroot for the following reasons:

  • The target device SBOM can not be easily generated. In order to generate a custom Linux distribution (OS), build systems pull in thousands of open source packages. Some packages are needed by the host machine to build the toolchain, while a few others are build-time dependencies and only a subset of packages end up getting installed on the target device. Source scanners are not able to identify the installed packages and end up reporting all software as part of the SBOM. Manually specifying which sources to monitor is tedious and impractical.
  • If you did manage to generate an SBOM, the source scanner approach still suffers from the false positives since it can not collect build-artifacts such as patches applied by build system, kernel configuration, etc.

Example tool: Whitesource

Other methods:

  • Network scanners such as Tenable Nessus, Rapid7 InsightVM, nmap, etc. can only determine software running based on open ports on the device and cannot generate a complete SBOM and vulnerability report.
  • Querying the live target device: If the device uses a package manager, it can be used to generate an SBOM. This is typically the case when using standard Linux distributions like RedHat, Debian, Ubuntu, and their corresponding security trackers can be used to monitor vulnerabilities. This is outside the scope of this discussion.

Note that while standard Linux distributions have security trackers for vulnerability monitoring, the attack surface (number of lines of code) is much much larger than a custom embedded Linux image based on Yocto/Buildroot/Timesys Factory. One of the security best practices recommended by standards such as NIST 8259A and ETSI EN 303 645 is to reduce the attack surface.

SBOM generation comparison

Feature
Build system based
Binary scan
Source scan
SBOM generation accuracy
Best
Good
Poor
Vulnerability metadata for generating accurate reports
– based on patches applied, configurations and hardware info
Best
Poor
Good
Integration into the developer workflow
Best
Poor
Poor

Vulnerability monitoring

Once an SBOM is generated, the next step is to create a list of vulnerabilities affecting the components. Most tools use the National Vulnerability Database (NVD) maintained by the National Institute of Standards and Technology (NIST) to report vulnerabilities, which are called Common Vulnerabilities and Exposures (CVE) in the database. In this section we dive into the need for more accurate curated data than that which is reported in the NVD.

Vulnerability data

The way vulnerability reporting typically works is a security researcher finds a vulnerability, reserves a CVE-ID and does a responsible disclosure allowing the maintainer of the affected product to investigate and patch the vulnerability if possible. The vulnerability then appears in the NVD, and the NVD maintainers analyze the vulnerability and add metadata such as Common Platform Enumeration (CPE) which allows automated tools to parse which CVE affects which software components and versions in a standardized way. While this sounds great, the actual implementation has multiple weaknesses as outlined below.

  • Missed CVEs and false positives: In our analysis, at least 40% of the NVD CVE entries have incorrect CPE data, resulting in missed CVEs or false positives. A missed CVE compromises the security of a device because the device’s exposure to the CVE is basically unreported. A false positive leads to wasted analysis time because the CVE appears to affect a device, which leads device maintainers to expend time investigating only to find it does not. The reasons for the inaccuracies are as follows:
    • Inconsistent naming (e.g., same package named differently for different CVEs)
    • Typos (e.g., mistakes in entering the package name or affected version)
    • Incorrect/incomplete analysis (e.g., not all affected versions of the software are listed, etc.)
    • Outdated information (e.g., when a patch gets backported to say LTS releases of kernel, the affected range of the package for that CVE is not updated)
    • No version information (e.g., only package name without version information is entered)
  • Delays in reporting: This is the time delay from when the vulnerability becomes public (news, mailing list, etc.), to when NVD analyzes the CVE for your tool to actually report the vulnerability against your product. The average delay in 2018 was on the order of months, leaving you in the dark about critical vulnerabilities if your tool solely relied on NVD.
CVE false positives

Find more information in our analysis presented at the Embedded Linux Conference.

The way we addressed these vulnerability data issues with Vigiles:

  • Multiple sources: Adding multiple sources of vulnerability information such as Ubuntu, Debian, RedHat, SoC advisories, security bulletins and mailing lists.
  • Curated data: We have a dedicated security research team that analyzes the data from the above sources and fixes any inaccuracies, specifically in the context of packages available in Yocto and Buildroot. Additionally we have intelligent curation algorithms that figure out if a vulnerability is backported to Long Term Stable (LTS) versions of software by analyzing git commits that fix the vulnerability.

The result: Up to 40% improvement in accuracy over NVD and up to 4 weeks earlier notification.

When evaluating tools, verify that claims of “lowest false positives” and “enhanced coverage” are in the context of your usage.

Ease of monitoring

In order to improve efficiency when monitoring vulnerabilities, the features to look for in a tool, which Vigiles offers, are:

  • Alerts: Ability to trigger alerts of new vulnerabilities that affect your SBOM.
  • Comparison: Highlighting what changed between scans or SBOM manifests. This way you can focus on the changes between scans or builds.
  • Advanced Search: Ability to search for CVEs by package name and version, or by ID to get in-depth Timesys curated vulnerability information.

Intelligent applicability filters

Now that you have a list of vulnerabilities affecting based on your SBOM which ones are truly applicable? Adding intelligence to tools can cut up to 75% of the time spent chasing false positives. In Vigiles, we offer:

  • Configuration-based filter: While the Linux kernel itself typically has hundreds of vulnerabilities, only a quarter of them are applicable based on the drivers/modules being used. Tools that take into account the Linux kernel configuration and filter out ones not applicable can reduce your analysis time by up to four times. The same applies to other software components like U-Boot.
  • Platform-based filters: Some vulnerabilities are OS/Platform specific (e.g., Android, Windows, Linux). Platform filters enable you to filter out the ones not applicable to your platform.
  • Hardware architecture filters: Vulnerabilities can be processor specific (e.g., x86, ARM, PowerPC, etc.) and can be similarly filtered out.
  • Attack vector filters: Vulnerability data typically accompanies how the vulnerability can be exploited. Example: Remotely exploitable (Network), Local network exploitable like LAN or Bluetooth (Adjacent), Local user exploitable (Local) or exploitable via physical media like USB (Physical). IoT devices might care only about network attack vectors whereas air-gapped industrial devices might care only about physical attack vectors.
  • Status filters: When using build systems such as Yocto or Buildroot, many vulnerability fixes are backported and already fixed by the maintainers. Status filters lets you filter out patched vulnerabilities and focus on the ones you need to address.

All of these applicability features are extremely useful for reducing the effort needed to manage vulnerabilities in embedded devices, and the features are unique to Vigiles.

Triaging

The next step in the process is to determine risk to your product and deciding if the vulnerability needs to be addressed immediately or can be scheduled as part of standard security release. This process is referred to as vulnerability triage. To aid the effort, many tools including Vigiles, offer the features below:

  • CVSS filter: The Common Vulnerability Scoring System (CVSS) is industry standard for assessing the severity of vulnerabilities. One can filter based on severity, for example High and Critical, to address those vulnerabilities first.
  • Custom scores: Option to specify and filter based on custom severity or risk score for vulnerabilities.
  • Custom notes: Ability to add custom vulnerability analysis notes.
  • Whitelisting: Ability to ignore vulnerabilities that are not important or applicable. Lets you focus on just the important vulnerabilities in the report.
  • Import/share triage data: Option to import or share triage data (notes/whitelist/scores) across platforms to avoid duplicate analysis effort.

Remediation

Once you have a list of vulnerabilities that need to be addressed, the next step is to check if there are fixes available. Vigiles helps by providing accurate remediation information:

  • Suggested fix: The minimum version to which the software needs to be upgraded in order to remediate the vulnerability. In the case of the Linux kernel this feature additionally specifies the minimum LTS version of the kernel in which the vulnerability is fixed.
  • Reference links: Information regarding available patches, mitigation and exploits. In the case of the Linux kernel and U-Boot we provide a link to the mainline commit containing the fix and the config option that results in the vulnerability.

Workflow and product lifecycle management

Most tools are geared toward security engineers enforcing IT policies. Having a tool additionally geared towards developers can reduce the maintenance burden of engineers actually addressing vulnerabilities and maintaining product releases. Some of the Vigiles features we developed to aid this are:

  • Product management: Ability to create products and folders (for releases, product variations, etc.) to which SBOM manifest can be uploaded from CI. This permits seamless integration of security maintenance into your existing Software Development Lifecycle (SDLC) process.
  • Collaboration: Ability to collaborate with other team members and global security teams. This is typically used to share products or reports and to triage collaboratively.
  • Comparison report: Allows you track vulnerability and SBOM differences between various releases and across products. This is particularly helpful for generating compliance documents and release notes in addition to tracking the security posture of a device over time.
  • Download reports: Ability to create executive summaries or export data for creating custom dashboards (.csv, .pdf, .xls)
  • APIs for integration: Ability to integrate with other SDLC tools such as issue tracker (e.g., Jira)
  • Support: In addition to tool support, does the vendor provide access to security engineers who can assist with any vulnerability related questions?

Conclusion

Vulnerability monitoring for embedded systems has unique needs that are not addressed by traditional IT or application SCA tools. By using the best tool suited for the job, one can save up to 75% in developer time in investigating false positives and in security maintenance tasks while improving the product security. Tools geared towards developers, such as Vigiles, can help ease the pain of managing the ever-growing number of vulnerabilities in embedded and IoT devices. Even if your organization already has a generic tool for vulnerability monitoring, a specialized complementary tool for embedded Linux developers produces significant maintenance cost savings because of the unique requirements of embedded Linux vulnerability management.

Next Steps

  • If you are a Yocto or Buildroot user, try out the free version of Vigiles. Register here and download meta-timesys or vigiles-buildroot to get started.
  • Download our free SCA for Embedded Tool Evaluation Checklist as a guide to pick the best tool for your product needs.

While Vigiles lets you monitor and manage vulnerabilities yourself, a completely hands-off approach is available too. Check our BSP Security Lifecycle Maintenance service where we handle the periodic security updates for you!

 

Akshay Bhat is CTO, Embedded Products and Services at Timesys. He oversees the strategic direction of Timesys’ technology roadmap. With more than 16 years of industry experience with embedded systems software development and security, Akshay’s focus is on Timesys solutions that transform the software development lifecycle for embedded and enable the development of embedded system products with stronger security. Akshay has authored numerous embedded Linux and industry articles and delivered several embedded systems security presentations. He received his MS in Electrical Engineering from NYU Polytechnic University.

About Timesys

Timesys has extensive experience with embedded system development and lifecycle management. Timesys has been instrumental in working with global leader semiconductor manufacturers with smart, quick and quality solutions for highly complex systems with accelerated product innovation and multiple product variants.

Best practices for triaging Common Vulnerabilities & Exposures (CVEs) in embedded systems: Top Three Questions Answered

Best practices for triaging Common Vulnerabilities & Exposures (CVEs) in embedded systems: Top Three Questions Answered

 

 

Keeping embedded system products secure requires ongoing, constant monitoring and management of Common Vulnerabilities and Exposures (CVEs) throughout the production lifecycle.

With the constant flood of CVEs reported each week, you need to have a process for understanding the exposure of your embedded system devices to cybersecurity exploits. It is important to see how CVEs apply to your product so you can quickly address the vulnerabilities that pose the greatest risk.

Continue reading “Best practices for triaging Common Vulnerabilities & Exposures (CVEs) in embedded systems: Top Three Questions Answered” »

Vulnerability management and triaging

Vulnerability management and triaging

 

 

With 300+ vulnerabilities being reported weekly in the US National Vulnerability Database (NVD), it is more challenging than ever to maintain the security of open source and third-party software used in embedded system products. One common approach to tackle the problem is to adopt a risk-based vulnerability management strategy in which vulnerabilities that pose the highest risk to your organization are remediated first. This blog outlines how to establish such an process as part of your software development lifecycle while keeping the maintenance cost and risk of exposure low.

Continue reading “Vulnerability management and triaging” »

Webinar with NXP: CVE triage best practices for efficient vulnerability mitigation

Webinar with NXP: CVE triage best practices for efficient vulnerability mitigation

 

 

Securing your embedded system devices is no longer just a final step in product development.

Security today must be a continuous process, a focus at every stage of your software development, release and maintenance cycles.

That’s because today’s vulnerability environment is radically different from the past. Hundreds of vulnerabilities that may or may not affect your products come to light every week.

Continue reading “Webinar with NXP: CVE triage best practices for efficient vulnerability mitigation” »

Vigiles Quick Start … because securing your products doesn’t have to be hard

Vigiles Quick Start … because securing your products doesn’t have to be hard

 

 

There is an old saying among cybersecurity vulnerability management practitioners:

The “good guys” have to get it right every time.

The “bad guys” have to get it right only once.

That means that the “good guys” — the security professionals whose mission is to keep corporate data safe — need to monitor, analyze and respond to every vulnerability that puts their systems, users and data at risk.

Continue reading “Vigiles Quick Start … because securing your products doesn’t have to be hard” »

Assessing CVE exposure and mitigation effort to reduce embedded system product cybersecurity risk

Assessing CVE exposure and mitigation effort to reduce embedded system product cybersecurity risk

 

 

Embedded system security has never been more important. Rising cyberattacks against embedded systems are increasingly putting devices and customers at risk of compromise.

Continue reading “Assessing CVE exposure and mitigation effort to reduce embedded system product cybersecurity risk” »

BSP Lifecycle Maintenance: Top Three Questions Answered

BSP Lifecycle Maintenance: Top Three Questions Answered

 

 

Security is critical throughout the lifetime of an embedded system. Continually changing threat environments, new deployment modes and third-party software updates mean that the BSP software for an embedded system device can no longer remain static and “frozen.”

Continue reading “BSP Lifecycle Maintenance: Top Three Questions Answered” »

Vigiles Enhancements: Software Composition Analysis (SCA) & CVE Mitigation for Stronger Embedded System Security

Vigiles Enhancements: Software Composition Analysis (SCA) & CVE Mitigation for Stronger Embedded System Security

 

 

A giant list of vulnerabilities does little to help you bring more secure products to market.

What matters is how you filter the list, triage the vulnerabilities, and mitigate the ones that pose the greatest risk.

That’s why the new enhancements to our Timesys Vigiles Security Monitoring & Management Service will enable you to develop more secure embedded system products today and maintain stronger product security throughout their lifecycles.

Continue reading “Vigiles Enhancements: Software Composition Analysis (SCA) & CVE Mitigation for Stronger Embedded System Security” »

Stop Chasing Vulnerability Ghosts: Why efficient vulnerability detection is essential to medical device security

Stop Chasing Vulnerability Ghosts: Why efficient vulnerability detection is essential to medical device security

 

 

To detect and correct vulnerabilities, eliminate false positives and prioritize the rest

Every embedded system device maker should want to make security a selling point, preventing breaches or exploits, not an embarrassment if a security problem occurs.

But nothing equals the liability and regulatory risk associated with medical devices.

Continue reading “Stop Chasing Vulnerability Ghosts: Why efficient vulnerability detection is essential to medical device security” »

Medical Devices:  Automated Vulnerability Monitoring for Streamlined FDA Security Compliance

Medical Devices: Automated Vulnerability Monitoring for Streamlined FDA Security Compliance

 

 

Reducing cybersecurity risks to medical devices is essential. Regulators like the US Food and Drug Administration (FDA) have made improving medical device security a critical focus in recent years.

This means many in the medical device manufacturing community are now rethinking how the software components of their products are secured throughout their product lifecycles.

Continue reading “Medical Devices: Automated Vulnerability Monitoring for Streamlined FDA Security Compliance” »

Click to Hide Advanced Floating Content

Timesys Vigiles™
Vulnerability Management

 

 

Click to Hide Advanced Floating Content

Reduce Embedded System
Cybersecurity Risk