To detect and correct vulnerabilities, eliminate false positives and prioritize the rest
Every embedded system device maker should want to make security a selling point, preventing breaches or exploits, not an embarrassment if a security problem occurs.
But nothing equals the liability and regulatory risk associated with medical devices.
In the U.S., the FDA is telling the industry medical devices must be as secure as possible at the outset and provide for the necessity of applying security patches.
Because hospitals and other medical facilities often don’t apply patches or updates even when they are available, the pressure for device makers to maximize the security of every product shipped keeps climbing.
One overwhelming challenge with embedded system security in general, and particularly medical device security, is keeping up with the steady stream of vulnerabilities discovered in the open source software that are the fundamental building blocks of today’s embedded systems.
The volume of security issues is so overwhelming that conscientious developers easily can exhaust themselves chasing what I call ghost vulnerabilities — phantom issues that are not actually relevant to your system or device.
Focus is essential. You need a methodology that separates the serious and relevant issues from the trivial and irrelevant ones — and tackles the real embedded device security problems that could be a problem.
No one wants to see headlines about an egregious breach attached to their brand or products. How much worse would it be to slip up because your organization wasted time and money solving the wrong problems?
Open source and vulnerability tracking
Using Linux for embedded device development offers many advantages. One is that the open source software community ruthlessly scrutinizes software, trying to prevent security flaws.
Yet new vulnerabilities continue to emerge. An average week sees the publication of more than 300 new Common Vulnerability and Exploits (CVEs) in the National Vulnerability Database maintained by the US National Institute of Standards and Technology.
CVE security issues are found in everything from the Linux kernel to commonly used libraries, utilities, and applications. That’s not a knock against open source. Proprietary systems are even more likely to have vulnerabilities, perhaps even more obvious and easily exploited ones, because they do not undergo the same scrutiny. For example, one vulnerability that prompted an FDA alert in 2019 was related to a commercial software library no longer supported by the original developer.
Still, 300 CVE security vulnerability alerts per week is a heck of a big firehose to drink from. Tracking each issue manually, as the maintainers of the major Linux distributions do, is impractical for most of us. In addition to the vulnerabilities, you must track the fixes, including those that have been backported, or retrofit, onto earlier releases.
Upgrading everything to the latest release can be impractical because API changes in newer releases may make them incompatible with shipping software or firmware. Incremental updates to the Linux kernel are released at a pace of about one every five days on average, and it’s impossible to ship a stable product with development and test cycles of less than five days.
Fortunately, when a critical medical device CVE is identified in a popular open source software product, the maintainers often go the extra mile to fix it in earlier releases, in addition to the latest one.
Unfortunately, backported fixes typically aren’t documented in the records in CVE databases. Instead, the official CVE publication will state that the flaw applies to every release prior to a given version number, ignoring the point releases to earlier versions.
In particular, the Linux kernel team does a fantastic job of backporting fixes, but their work is not reflected in the NVD repository of CVE data.
This is one of several data quality issues that can lead an organization to chase ghosts — for example, retooling software to work with a newer version of a library only because developers were unaware of a backported fix.
Errors and inconsistencies in the CVE documents also can lead to important vulnerabilities being missed.
For example, different documents might refer to the same module as “arm-trusted-firmware,” “arm_trusted_firmware,” or “trusted-firmware-a.”
A typo might turn version 2.2.3 into 2.23, or vice versa. There are enough of these errors to confuse any scripts you might devise to sort through the data.
Open source initiatives address some of these issues. For example, the Yocto Project, which assists with the creation of custom embedded Linux distributions, provides a mechanism for mapping between the names and versions used in the recipe for compiling a distribution and the NVD identifiers. That is very useful and worthwhile, but not perfect. Some CVEs are missed, particularly in software created with older versions of the Yocto tools.
And while the latest cve-update-db Yocto tool allows for more precise queries and analysis, my team has found it can also lead to more false positives.
Writing your own scripts to deal with these issues is a lot of work. And having struggled with these issues ourselves, my colleagues have developed a solution that can work for everyone.
Timesys Vigiles, a basic version of which is available for free, analyzes your embedded software product with Software Composition Analysis (SCA) to isolate the specific CVEs that are relevant to the Software Bill of Materials (SBOM) of your medical device product.
Vigiles then helps you prioritize the most critical vulnerabilities and fix them in the most efficient way. Equally important, the tool helps you whitelist the CVEs that you determine are not relevant or important. That way, they will not continue to be flagged every time you do the analysis. Paid versions of Vigiles also support team collaboration around triaging, prioritizing, resolving vulnerabilities and conducting CVE mitigation.
This focus on embedded system security and development best practices is why the top 30 medical device manufacturers employ Timesys as part of their embedded system product programs.
For example, Roshy J. Francis, Chief Technology Officer of Diagnostic Cardiology for GE Healthcare says, “We chose to partner with Timesys in the development of our new portfolio of medical devices to ensure that they stay secure throughout their lifecycle. Our customers globally face strict information security requirements combined with a heightened threat environment when deploying these devices within their enterprise.”
The importance of triage and prioritization
Building up a CVE vulnerabilities list is only part of the battle. Triage and prioritization are essential to CVE remediation because no organization has the time, energy, or money to chase everything at once.
Triage is of course an essential component of emergency medical care when time and resources are in short supply. The same basic principle applies to vulnerability remediation of your medical devices that those health care professionals may use. Your resources and time should be applied to the security issues that matter most, so analysis and prioritization is an important first step in the security maintenance process.
In an analysis of one older NXP i.MX Rocko release from the Yocto Project, we find 658 unfixed CVEs. However, if you filter that down by severity to just those rated critical, the number drops to 239. If you further cut it to just those that have a network attack vector — meaning they could be remotely exploited — that leaves you with 158 to worry about
If a product includes any network connectivity at all, I recommend fixing anything that could be remotely exploited. In a pinch, you could focus on just the 33 for which there are known exploits.
The Vigiles vulnerability list dashboard identifies the attack vector for a particular CVE affecting your product, enabling you to quickly assess if a particular vector is relevant to your product configuration and deployment mode.
In addition, Vigiles and its tools for vulnerability analysis and triage can be a great way to enhance your overall product security processes. It permits developer-driven security monitoring and maintenance of products after release and throughout their lifecycles. Vigiles’s open source vulnerability tracking and mitigation functionality complements application level software inventorying tools from vendors like Blackduck (Synopsys) and WhiteSource.
That’s because Vigiles is optimized for embedded systems and aligns to the processes for building and maintaining embedded systems using open source components. Vigiles allows CVE filtering based on Linux kernel config and u-boot config, reducing any CVEs that would relate to features not being used by your product. On an average, this reduces the triage burden on these packages by a factor of 4X.
If you use Yocto, Vigiles reports only those vulnerabilities for packages and libraries installed on your target system, which saves time by not requiring you to review the myriad of vulnerabilities in host/build dependency packages.
Further, Vigiles will automatically track and report CVE fixes that are already fixed in your Yocto build, eliminating manual data entry. Vigiles also includes support for reporting CPU/SoC vulnerabilities.
To be clear, I am not claiming Vigiles does all the work for you, only that it makes the process of maintaining embedded Linux security and other open source components much more manageable. There is no magic bullet. You can reduce the likelihood and frequency of problems by designing in security from the beginning. A good design reduces the “attack surface” by factoring out software libraries that are not necessary. If you don’t include them, you won’t need to fix vulnerabilities in them.
Put in the work up front to make your product as secure as you know how to make it. Then, when issues crop up, make sure you focus your energy on the truly dangerous monster open source software vulnerabilities, not the imaginary ghosts.
In medicine, triage is an essential element of identifying serious, life-threatening conditions, separating them from trivial complaints, and prioritizing problems that fall somewhere in between. The challenge of medical device security is not so different. Timesys is here to help you with that process of diagnosis and decision-making.
Timesys has extensive experience with embedded system development and lifecycle management. Timesys has been instrumental in working with global leader semiconductor manufacturers with smart, quick and quality solutions for highly complex systems with accelerated product innovation and multiple product variants.