Introducing Federacy

An open source, decentralized platform
for security research and vulnerability management

tldr; here is a summary of our thinking. join us on reddit to discuss.

We’ve been spending a lot of time lately thinking about the challenges of securing companies and applications in a world where we all rely on open source software. It is our belief that if we work together, in an open coalition, we can build better security infrastructure and tooling, and materially impact the degree to which we are vulnerable.

The backstory

After we left Twitter, a couple of us started hacking on the concept of an automated vulnerability metascanner. We contributed to an open source project, vuls, by adding local execution, reducing privileges required, and improving scanning accuracy. Then we wrote and open sourced tooling around it and made a hosted API and site for analyzing the resulting data. When CoreOS released Clair, a scanner for containers, we began building around it as well, and as the container ecosystem advanced we added support to our metascanner for Docker containers, Kubernetes, and Google Container Registry.

We were thinking about turning the metascanner into a business, and as we started productionizing our infrastructure we found that the Kubernetes helm chart for nginx lacked support for the Role Based Access Control (RBAC) feature added in Kube 1.6. While contributing support for RBAC, we took a deeper look at Kubernetes' access policy, and began working on the Docker Scanning Special Interest Group (SIG). We produced a working specification for tracking contents and vulnerabilities in Docker containers, utilizing Notary and a docker registry. This has since been rolled into Google's Grafeas project, and the policy engine component is the Kritis project.

It also led us deep into research on the state of vulnerabilities in public Docker images, which presented a larger problem for our metascanner:

Automated scanning isn’t very useful if the vulnerability database is incomplete, inaccurate, or untimely.

The big idea

Since then, we’ve spent a lot of time considering the best approach to making the vulnerability reporting ecosystem more robust.

We want to help secure the Internet by building infrastructure and tooling that allows us to collectively focus significantly more resources on security within open source software. We think leaning too heavily on the current CVE/NVD ecosystem, which is insufficiently funded by the Department of Homeland Security, has left the community unnecessarily vulnerable.

We think the first step is to build an open protocol and repository for security vulnerabilities with the initial objective being to make vulnerability reporting and remediation faster. When we were building the metascanner, we regularly found public information about a vulnerability while our scanner was blind to it.

We think the second step will be to build a platform that allows companies to directly fund security research within the open source software they rely upon. This would create significantly higher bounties, which could be more fairly distributed, and could attract many more talented researchers to spend time on open source projects. We want to make contributing information and research to the public repository substantially more profitable than alternatives, such as selling it to brokers or nefarious actors.

Using staking and reputation systems, we think we can provide more sophistication around consensus and conflict resolution, especially around the severity and surface area of vulnerabilities. Combining attestations and Google Kritis, we think we can enable companies to build security policy around the opinions of specific and trusted sources.

Here is a summary of our current thinking on some of the problems and potential solutions we’re currently looking at.

Do you think we’re on the right track? Some of our ideas are probably a bit audacious, and to be frank, while we have some perspective as end users of security research, and software authors, we’re novices compared to so many of the incredibly talented security researchers that are out there. We’d really value help thinking through the ramifications of different design choices as well as whether we're focused on the right problems to tackle.

Ultimately, we hope to design an open protocol, open source infrastructure, and an openly governed organization in order to form a strong coalition of security-minded people and companies, with the objective of improving the state of security.


While this is just one tiny step in what will be a very long journey, we want to give credit where credit is due:

ant and kris, sounding boards who always offer earnest, insightful feedback.

nick, who was a partner-in-crime in building MoPub, and repeated his magic on the metascanner.

aj, without whom our open source contributions would not have been possible, has been a mentor for us for over seven years.

jericho and steve, who inspired me to work on this project, shared integral information openly.

kurt, allen, art, giants upon whose shoulders we're standing, thought-leaders and progress-makers who have invested an incredible amount of time into these challenges.

will, larry cashdollar, and all of the other great security researchers, who help us keep our software and systems safe, and without whom none of this work would be useful.

david, for organizing the Docker Scanning SIG, which provided a venue for great brainstorming and conversations, all of which he led and contributed to.

Container Scanning Specification

Defining Contents, Licenses and Vulnerabilities

Since June, we've been working on a public specification to standardize container scanning with the Docker Scanning SIG, led by David Lawrence, Director of Security at Moby, including Black Duck Software, CoreOS, Google and others. It is heavily influenced by the Linux Foundation's Software Package Data Exchange (SPDX) and Open Container Initiative (OCI), NYU's The Update Framework (TUF), and Docker Notary.

We've taken contributions from dozens of people and released several revisions, one of which was adopted by Google, IBM and others with the release of Grafeas.

Below is a subsequent revision we've proposed which is being considered by the Scanning SIG and resolves some of the previous limitations with flexibility.

Kubernetes nginx ingress chart

Role Based Access Control (RBAC)

Before Kubernetes 1.6, most deployments used very permissive ABAC policies, including omnipotent service accounts. Because we were self-hosting our Kube 1.6, we had issues with some of the charts we were trying to use, including nginx-ingress, which had not had less permissive RBAC added.

We researched suggestions for RBAC permissions and implemented the consensus using patterns emerging in other charts, and took in suggestions and contributions from several people, especially Michael Goodness.

nginx-ingress RBAC Pull Request

Docker Image Vulnerability Research

24% of latest Docker images have significant vulnerabilities

Conversations: Hacker News, Twitter, LinkedIn

Subreddits: r/sysadmin, r/security, r/docker, r/hacking, r/ubuntu, r/opensource, r/commandline, r/vrd, r/netsec


Understanding the vulnerability landscape in the container ecosystem is critical to our mission of securing the world, so we decided to put our technology to work to answer a key question: what is the current state of vulnerabilities in official Docker repositories?

The official Docker repositories consist mostly of images for open source and commercial software, are generally managed by the author or vendor, and contain the most widely used images in the Docker community. The 20 most popular images have been pulled tens of millions of times each. Increasingly, they are used as the base for distributed systems replacing some of the functionality of 3rd generation configuration management software.

Docker takes security seriously; this project focuses on the wider community and practices. While this article focuses on identifying problems, the next will propose some solutions.

On February 6th, we scanned 91 of the 133 official Docker repositories. This is every repository with a ‘latest’ tagged image consisting of a major linux distribution and a functional package manager. We used an open source scanner (vuls) that we modified, and then analyzed the data using tools we’ve built for our company, Federacy. Unfortunately, this removed alpine and static binaries because vuls currently supports neither. Scoring is CVSSv2 and priority follows the standard.

New to Docker? We wrote an addendum for you.


Percent of docker images vulnerable 55060281837e39550505a8543b0a5391acb558c1a127e3f3c53942227f82aab0

24% of Docker images tested have significant vulnerabilities (high or medium priority). 10.53% have high priority vulnerabilities.

Images vulnerable by distribution 816acdbdb06e21c1151b5751c4896462ec24b6fd8516e2f7a1360fe5b4e1ba79

Ubuntu images have significantly more vulnerabilities than Debian images. Debian is the least vulnerable major distribution. It has fewer moderate and high priority vulnerabilities (8%) than Ubuntu (27%). RHEL-based distribution sample size was very small (4% of all images).

Docker images by base distribution c4915981d8907ebb91da8904fd16b503bbfed666b89bc8104bdc9a2671c57cbc

Debian is the most widely used distribution — RHEL, the least, by far. Debian is the dominant base distribution among official docker repositories, and accounts for over 79% of latest-tagged images. Ubuntu accounts for ~16%, and RHEL-based distributions the remaining ~4%.

Average no high priority vulns 8a3e20b2fcaa30ed68a736ab925c1ce8b561f6c918b6ee4060e4ed551f3f8663

Older releases of Debian and Ubuntu have significantly more vulnerabilities. Correlating indicators are: a higher average CVSS score, more packages installed, and more updatable packages.

The number of high priority vulnerabilities in Debian 8.6 and Ubuntu 14.04 was considerably higher than in the new releases (8.7 and 16.04) and Average CVSS (v2) score shows the same.

Packages installed by release 1872fe8e9f73e8358da828c189f641a613c207a776598d882ca093356e2e293e

New distribution releases had fewer packages installed and updatable. While this is not a direct indicator of vulnerabilities, it means a smaller surface area, and fewer unapplied updates.

The most common vulnerability was SSL Death Alert (CVE-2016–8610) This is a DoS-able vulnerability affecting software compiled against GnuTLS, OpenSSL, and NSS, including nginx but excluding httpd.

An attacker can flood vulnerable software with malformed SSL ALERT messages, because a new thread is not allocated to parse them. IP-based firewalls and deep packet inspection can be used to mitigate the vulnerability.

Most common vulnerability per distribution

Debian: CVE-2016–7056 (86% of images). An openssl vulnerability that could result in the compromise of a private key. However, it requires local access and the use of cache timing attacks during multiple signatures, so while a severe vulnerability, it is unlikely to impact most people. Affects most linux distributions because openssl is widely used.

Ubuntu: CVE-2016–8610 (93% of images). Described above.

RHEL: CVE-2016–1248 (50% of images). A vim vulnerability that could result in code execution and privilege escalation. Affects any system with vim, and requires opening a malicious file.

Most common high priority vulnerabilities

CVE-2016–4658 (5% of images). A libxml2 vulnerability that is misleading. Affects most linux distributions, not just Apple products as stated in the CVE, and despite being rated moderately is a significant vulnerability for people who make libxml functionality accessible remotely, which includes nokogiri and most open source software that parses XML/HTML.

CVE-2016–1252 (5% of images). A vulnerability in how the apt package manager parses signed repositories could lead to altered packages and resulting code execution. Affects Debian 8.7 and Ubuntu 16.10, 16.04, and 14.04.


As engineers we have a responsibility to our users and other stakeholders. Being compromised isn’t just embarrassing, it can cause major damage to the trust in our brands. It can also have serious financial implications, as evidenced most recently by the $350m haircut Yahoo took on their acquisition price.

Managing known vulnerabilities is the first step towards a strong security posture. If we’re not updating our systems, and keeping an eye on emerging vulnerabilities that are yet to be patched upstream, we’re basically leaving the front door wide open.

The challenge, of course, is how to best roll out updates. Many startups have invested in CI/CD systems for their code, but many have not implemented a similar process for building images or validating third-party images. Likewise, very few startups run automated vulnerability scanning in production. At Federacy, we’re trying to make this so simple and useful that it’s a no-brainer for every startup.

Next Steps

1. More Images. Add more sources of images, especially cloud (AWS, Digital Ocean, Google Compute, etc).

2. More Scanners. Add clair and other proprietary scanners.

3. More Time (Series). It would be interesting to see how quickly images become vulnerable after being deployed.

4. More Automation. Fully automate generation of charts and data so they are always current.

Addendum: How to Scan Docker Images and Mitigate Vulnerabilities

Scanning Docker Images became significantly easier in 2016, and we now have a number of options. Docker Hub,, and others offer scanning for repositories they host and there are now open source scanners (clair and vuls). At Federacy, we're supporting the latter approach by making the open source scanners easier to use and connecting them to a better vulnerability database.

Mitigating vulnerabilities in Docker images is a bit more difficult, but here are a few suggestions: install package updates in your image build process. If you can, automate updating packages on your image while it is running. Add vulnerability analysis to your image build process. Use Alpine or a similar distribution. Build a static binary image.

Thanks to ant, fujin, kris, nick, btm, and william for reading drafts of this and making it much better.