With 300+ vulnerabilities being reported weekly in the US National Vulnerability Database (NVD), it is more challenging than ever to maintain the security of open source and third-party software used in embedded system products. One common approach to tackle the problem is to adopt a risk-based vulnerability management strategy in which vulnerabilities that pose the highest risk to your organization are remediated first. This blog outlines how to establish such an process as part of your software development lifecycle while keeping the maintenance cost and risk of exposure low.
Risk based vulnerability management process
At a high level, the process involves 3 steps:
- Monitor: The first step is to obtain a list of security vulnerabilities affecting your product based on your software bill of materials (SBOM). This can be done by manually checking for Common Vulnerabilities and Exposures (CVE) reported in NVD/security bulletins OR using source composition analysis (SCA) tools / binary scanners. Once a component is deemed as affected, the next step is to verify if the version being used is affected using Common Platform Enumeration (CPE) or reviewing vulnerability data against code. The more accurate the source of vulnerability information, the better the security coverage and fewer false positives you’ll have to deal with.
- Triage: In this step, the applicability of the vulnerability is analyzed in the context of your product and then prioritized based on the risk it poses. The vulnerabilities can be categorized as critical / high / medium / low / not applicable for appropriate action to be taken.
- Remediate: After triage, it is important to have a process in place to remediate vulnerabilities in a timely manner. The typical remediation steps might be to patch the vulnerability or upgrade to a newer version of software containing the fix. If the fix is not yet available, a workaround should be implemented to mitigate the same. An example security response process might be:
- Critical risk vulnerabilities are scheduled to be fixed immediately as a part of emergency security release.
- High/Medium risk vulnerabilities are addressed in the next scheduled security release. (e.g., Monthly or quarterly updates)
- Low risk vulnerabilities: are scheduled as part of a product’s standard maintenance cycle. (e.g., Yearly releases)
Figure: Risk based vulnerability management process workflow
Triage: Deep Dive and Best Practices
The goal is to classify vulnerabilities based on risk and applicability with the least amount of effort while ensuring an adequate level of security assessment.
Vulnerability risk assessment using CVSS
The Common Vulnerability Scoring System (CVSS) is a free and open industry standard for assessing the severity of computer system security vulnerabilities, which in turn can help prioritize which vulnerabilities to remediate first. CVSS scores range from 0-10, with 9 or above deemed critical, 7 to 9 deemed high, 4 to 7 deemed medium and below 4 deemed low severity.
The CVSS scoring system is composed of 3 metrics, and each metric group has several sub-components. When vulnerabilities are analyzed, each of the components can be given a value from a predefined set as per the CVSS specification.
- Base metrics (required): represent characteristics of a vulnerability that are constant over time and across user environments. It is composed of exploitability metrics (how easy it is to exploit) and impact metrics (consequence of being exploited in terms of Confidentiality, Integrity and Availability, the “CIA Triad”). NVD analysts enter this information for every CVE.
Note: This metric is independent of real world exploits being available or any counter measures you may have in your product.
- Temporal metrics (optional): represent characteristics of a vulnerability that may change over time but not across user environments (e.g., availability of exploits). These metrics are optional for CVSS score calculation and can be filled out by your security team or obtained as part of some commercial vulnerability scanners.
- Environmental metrics (optional): represent characteristics of a vulnerability that are relevant and unique to a user’s environment (e.g., Is data confidentiality important in your product? Does your product have countermeasures like firewalls?) These metrics are specific to your product and can be optionally filled out to get a more accurate severity assessment.
Figure: CVSS score metric groups overview
Triaging using CVSS
The easy way out: CVSS score consisting of just the base metrics from NVD can be used to prioritize vulnerabilities with the least amount of effort. Using just the base metrics as the sole source of security risk assessment does not take into account any defense/countermeasures built in to your system, your product/business requirements for the CIA Triad, if the affected code is reachable in your system, theoretical vs. practical risk based on availability of exploit, etc. So while it is a quick and easy way to prioritize vulnerabilities, it is not an accurate depiction of cyber risk or exposure for your product.
Custom CVSS score: The accuracy of a CVSS score can be improved by plugging in the temporal and environmental metrics into the CVSS calculator to get a severity score applicable to you. However, this is a tedious effort and is not practical without a dedicated security team with an understanding of your product security requirements. As an alternate, one could identify components that are critical to the system and only assess vulnerabilities for those. Where possible, one should leverage the work done by the OSS community and major Linux distributions that make their vulnerability analysis public via security trackers.
To justify the effort, one needs to ask themselves: Do I need to adhere to regulatory compliance (PCI-DSS, HIPAA etc.)? Can there be significant collateral damage as a result of exposure?
Vulnerability applicability via intelligent filters
While CVSS score provides a quick first pass on categorizing the severity of a CVE, there are additional ways to further narrow down ones that are not applicable to your product.
Attack vector filter
One of the base metrics for exploitability that is available as part of CVE data is the attack vector. This determines the context in which the vulnerability can be exploited:
- Network: “Remotely exploitable” attacks (e.g., from the Internet)
- Adjacent: Attacks must be launched from the same shared physical (e.g., Bluetooth) or logical (e.g., local IP subnet) network, or from within a secure or otherwise limited administrative domain (e.g., VPN).
- Local: The vulnerable component is not bound to the network stack and the attacker’s path is via read/write/execute capabilities (e.g., keyboard, console, valid SSH credentials, user interaction via social engineering techniques, etc.).
- Physical: The attack requires the attacker to physically touch or manipulate the vulnerable component (e.g., via inserting a USB device).
If your product is an air-gapped device with no user input, e.g., a standalone industrial embedded device, filtering on the “Physical” attack vector can typically rule out 90% of the CVEs!
While a component in your product might be vulnerable, the product might not use the module. For example, if your product is running the Linux kernel, based on the drivers/subsystems being used, many vulnerabilities are not applicable. Vulnerability tools that take the product Linux kernel configuration into account can substantially reduce (up to 4X) the burden in triaging CVEs by filtering out ones that are not applicable.
- Platform based filters: Some vulnerabilities are OS/Platform specific (e.g., Android, Windows, Linux). Platform filters enable you to filter out the ones not applicable to your platform.
- Hardware Architecture filters: Vulnerabilities can be processor specific (e.g., x86, ARM, PowerPC, etc.) and can be similarly filtered out.
- Status filters: When using embedded Linux based build systems such as Yocto or Buildroot, many vulnerability fixes are backported and already fixed. Similarly, when using distributions like RedHat, Debian, Ubuntu, CentOS, AWS Amazon Linux, vulnerabilities are already patched by upstream maintainers. So having tools that can extract this information from build systems and package managers can greatly reduce reviewing vulnerabilities that are already fixed.
Figure: Risk based vulnerability triage/prioritization example for an IoT device
Vulnerability Remediation Strategies
It is apparent that the triaging process requires a lot of effort. For some, it might be easier to upgrade to the latest version of the software and patch everything without analyzing vulnerabilities, which also is a low-risk strategy. However, for legacy products or for products where the testing effort and/or field upgrades/re-certification efforts are high, the triaging effort makes more sense. It is prudent to build “Secure By Design” products upfront to make vulnerabilities difficult to exploit. This includes reducing the attack surface, designing secure boot/chain of trust, hardening the system, protecting data using encryption, establishing access control policies, etc.
Cost and risk reduction using tools
Vulnerability management is a complex enough process to warrant dedicated tools that seamlessly integrate into your SDLC. Tools help you be efficient and lower the associated cost and risk. Some key aspects to consider when looking for tools are:
- Automatic SBOM generation and vulnerability reporting
- Accuracy of vulnerability data (multiple sources of vulnerability data, early notification, fewer false positives)
- Intelligent filtering (Attack Vector, CVSS, Configuration, Platform, etc.)
- Build system / Package manager integration (report only vulnerabilities for installed components, ability to detect patched vulnerabilities)
- Risk score (ideally based on ML algorithms)
- Triage customization (ability to set custom notes, CVSS score, whitelist/ignore packages and/or CVEs)
- Remediation assistance (links to patches, mitigation, exploits, suggest fixes, automated pull requests)
- Notify / highlight new CVEs, compare reports
- CI/CD/Bug Tracker integration
Managing vulnerabilities is an increasingly complex and burdensome process. There are more vulnerabilities that have to be examined, and there is less time to determine the threat posed by any given vulnerability. Having a well-defined proactive, risk-based vulnerability management process is key to maintaining the security posture of your product/organization. To keep the cost of security maintenance low and reduce risk, one should leverage the best tools available that are optimized for your use case.
- Sign up for a free version of Vigiles to start monitoring vulnerabilities in your product. Register here.
- Let us help you establish a vulnerability management program and provide a free vulnerability report for your product. Click here.
Timesys has extensive experience with embedded system development and lifecycle management. Timesys has been instrumental in working with global leader semiconductor manufacturers with smart, quick and quality solutions for highly complex systems with accelerated product innovation and multiple product variants.