By Kenton Varda - 28 Feb 2017
A few days ago, Cloudflare disclosed a security bug in their services which may have leaked secrets from sites which use Cloudflare. Sandstorm uses Cloudflare as a cache in front of our web site and Sandstorm Oasis, and so some of our users have asked if this problem affects them.
TL;DR: If you uploaded or downloaded a grain backup (zip file) between September 22, 2016 and February 18, 2017 (more so between Feb 13 to Feb 18), there is a very remote chance that some portion of the content of your backup could have been leaked to a third party, due to a bug in Cloudflare. We suspect, but cannot say for sure, that no such leakage occurred. We believe that no data other than grain backups could have leaked.
We believe that Sandstorm services are mostly unaffected (with the exception of grain backups, described below). We intentionally divide our services such that public, static content is served from different hosts from dynamic/private content. We only enable Cloudflare in front of the public, static content (to provide caching and faster delivery). Moreover, the public, static hosts avoid storing sensitive credentials in cookies.
Specifically, we have Cloudflare enabled on the following hosts:
We chose this design because we rely on Cloudflare only for caching purposes. Since dynamic data cannot be cached anyway, it made sense to form direct connections for that data rather than pass through a proxy, both to reduce our TCB and to improve performance. On the other hand, this means we cannot fully take advantage of Cloudflare’s security features such as its Web Application Firewall and DDoS protection, but we felt comfortable with this trade-off.
That said, we’ve identified one notable exception to our design: When you download a backup of a grain via the Sandstorm Oasis UI, or upload a backup to be restored, the zip file is transmitted through the main Oasis host, proxied through Cloudflare. This seems to be an oversight; it would make sense for these requests to go through the DDP host (I’ve prepared a pull request to fix this). These transfers are authorized via short-lived tokens, so there’s no risk of Oasis credentials being leaked. However, the contents of a .zip file could possibly have been leaked by Cloudflare.
Cloudflare reports that the bug existed between 2016-09-22 and 2017-02-18, with the greatest potential impact being in the last four days of that period. If you uploaded or downloaded a grain backup in that time, it could have been leaked. However, Cloudflare reports that the incidence of such leaks was extremely small, and each instance would have leaked data from an essentially random website served by Cloudflare, of which there are many (Cloudflare handles some 10% of all internet traffic). Given the probabilities, it is likely that no Sandstorm grain backup data was actually leaked. There is also so far no evidence that anyone was actively exploiting this bug.
With all that said, if you are worried about the possibility that your data was leaked, the proper course of action depends on the app and grain involved. Sandstorm is explicitly designed to keep credentials out of the hands of apps, so usually the only sensitive information in a grain backup is information that you gave to it yourself. For example, if you had an Etherpad document where you kept a list of passwords (and you downloaded a backup of that document recently), you may want to change those passwords. Otherwise, there is nothing to do.
Bugs like this probably exist in a lot of software. The Cloudflare bug is notable because:
It’s important to keep in mind that security is not binary. Security is about risk management. The Cloudflare bug did not change anyone’s security level from “secure” to “insecure” – instead, it increased the likelihood of any particular person becoming the victim of a real attack from, say, 5.26% to 5.28% (numbers made up to illustrate point). Had Cloudflare been slow to respond once the vulnerability was found, this risk may have shot up. Fortunately, though, they were anything but.
Now, consider the use of password-based authentication. Nearly every service on the web uses it. However, it is demonstrably horribly insecure: Not only do people regularly use bad passwords or reuse the same password across sites, but passwords stored in a database (even if hashed) make it much easier for attackers to escalate a read-only database snapshot into full access privileges (possibly even to sites other than the one attacked). By requiring the user to receive an authentication code by e-mail each time they log in from a new location, a service can hugely reduce their users’ risk of compromise – probably by an order of magnitude! By this measure, bare password-based authentication should be considered a vulnerability, and a much more serious one than Cloudflare’s bug.
And yet, people accept passwords because they are more convenient than other mechanisms. (Well, many people do. Sandstorm does not implement passwords.) So, we admit that security is not absolute, but rather a trade-off which needs to be judged against other factors. We need to weigh the risks of our infrastructure being insecure against the value that infrastructure brings – and Cloudflare certainly brings a lot of value. That said, with good design, it’s often possible to reduce and contain risks without sacrificing too much utility. This is one of the core goals of Sandstorm – to introduce an infrastructure model and security practices which mitigate risks.
By Kenton Varda - 06 Feb 2017
Most people know Sandstorm as an open source, community-driven project aiming to enable self-hosting of cloud services and to make it possible for open source web apps to compete with today’s cloud services.
Many people also know that Sandstorm is a for-profit startup, with a business model centered on charging for enterprise-oriented features, such as LDAP and SAML single-sign-on integration, organizational access control policies, and the like. This product was called “Sandstorm for Work”; it was still open source, but official builds hid the features behind a paywall. Additionally, we planned eventually to release a scalable version of Sandstorm for big enterprise users, based on the same tech that powers Sandstorm Oasis, our managed hosting service.
As an open source project, Sandstorm has been successful: We have a thriving community of contributors, many developers building and packaging apps, and thousands of self-hosted servers running in the wild. This will continue.
However, our business has not succeeded. To date, almost no one has purchased Sandstorm for Work, despite hundreds of trials and lots of interest expressed. Only a tiny fraction of Sandstorm Oasis users choose to pay for the service – enough to cover costs, but not much more.
We attribute this failure to two main problems:
As a product, Sandstorm is still alpha-quality. As you may know, Sandstorm is pioneering a paradigm shift in the way servers operate, which we call fine-grained containerization. The benefits of this technology are huge, but a lot of engineering work is needed to make it work smoothly. Although we’ve built a working product that many people – including ourselves – use for real work every day, many limitations still exist that make Sandstorm feel rough and incomplete compared to alternatives. We know how to fix these problems, but we need more time to do so.
We underestimated, in classic fashion, the challenge of enterprise sales. As a company of engineers and geeks, we knew that enterprise sales would be outside of our comfort zone, but we didn’t realize just how much work it would be to learn, and how much time we needed to be successful at it. By the end, we were only scratching the surface of how to generate leads and get in the door at big companies; we never managed to do a “real” sales call. In retrospect, we should have hired a business development expert much earlier on. We also should have raised more money and planned for a longer runway.
We plan to publish a more complete postmortem in a subsequent blog post.
Unfortunately, Sandstorm the business has now run out of money, and we have been unable to raise more.
The Project Lives On
Although it will no longer be our full-time job, Sandstorm will continue as an open source project. We still strongly believe in Sandstorm’s long-term vision and cannot abandon it. I personally will continue to lead Sandstorm’s technical development: reviewing and merging pull requests, pushing releases, and developing new features. We will continue to operate Sandstorm Oasis – your data there is safe. Meanwhile, we will make it easier for our extended community to be involved in core development and decision-making. Jade will be in contact with individual community members to appoint community leaders and grant them the authority to handle a variety of community organizing functions, from App Market approval to organizing meetups.
Ironically, the pace of development may not even be hurt much. Over the past year, the Sandstorm team has spent a great deal of our time on enabling our business, e.g. building a payment mechanism, processing customers, marketing, and the like. I personally have spent far too much time on fundraising, sales, and other deal-making rather than on coding. With this shift in direction, we can now focus strictly on building out the core platform, getting more done with less time.
Immediate Action Items
As a result of this change, the following has happened today:
We have pushed a release which removes the Sandstorm for Work paywall. Sandstorm for Work features (like LDAP/SAML integration) are now available to all self-hosted Sandstorm servers at no cost. The product name “Sandstorm for Work” has been retired; there is now only Sandstorm. (Your server should already have received the update. If not, SSH in and type sudo sandstorm update. Don’t have a server yet? Install Sandstorm now.)
We have open sourced Blackrock, an alternate back-end for Sandstorm which is able to scale out across a cluster of machines. We have long used this technology to power Sandstorm Oasis, but had hesitated to release it for fear that it would harm our business model. We no longer have a business model to protect, so the code can now be set free.
We have added the Sandstorm Technology Roadmap to the Sandstorm repo, where you can learn about everything Sandstorm has built and plans to build. The roadmap is full of projects and features marked “TODO”, so you can see what needs to be built.
We’ve written a new contributing guide with a list of projects you can work on. See if there’s anything that interests you!
No changes are expected to Sandstorm Oasis nor Sandcats.io. Both services will continue to operate going forward.
How you can help
Want to see Sandstorm succeed? Then, contribute!
If you’d like to help build Sandstorm, sign up on the sandstorm-dev mailing list and join us on IRC at #sandstorm on Freenode. There is lots to do, from hardcore systems hacking to community organizing to tasks that require nothing but enthusiasm. For ideas on how you can help, check out the community page and the contributing guide.
If you have more money than time and would like to support the project financially, the best way to do it is to sign up for a paid account on Oasis.
By Drew Fisher - 01 Dec 2016
Is there a feature or bug holding you back from deploying Sandstorm at your company or for your team? Do you need the Sandstorm core devs to prioritize a feature? Sandstorm Solutions can fix that.
There’s always way more on our roadmap than time and people to do it. We receive requests from potential customers around the world who would love to use Sandstorm, but need a particular planned but unimplemented feature, or hit a bug that affects them in particular. We now offer Sandstorm feature prioritization to our customers.
We’re happy to adjust our priorities to support these folks, so we figure we should state that clearly and make sure it’s something everyone can take advantage of, not just the people who inquire.
So here’s the deal with Sandstorm Solutions: if you want a Sandstorm feature prioritized, or a particular bug fixed, we can do it. The way this works is:
Need something different? We’re happy to talk about how we can help you succeed - drop us a line at solutions@sandstorm.io.
By Kenton Varda - 17 Nov 2016
Sandstorm is about making it easy to run a personal server. But we also offer Sandstorm Oasis, a service which runs your Sandstorm server for you.
Contradiction?
Actually, no: Even if you run your own server, Oasis benefits you. Oasis is important because it makes it possible for anyone to use Sandstorm’s library of open source apps, even if they really don’t want to run their own server. A larger audience means that more and better apps will become available. Indeed, after we launched Oasis last year, the rate of new apps becoming available on Sandstorm spiked.
That benefits self-hosters, because those same apps can be used on your private server, too.
In fact, we at Sandstorm don’t necessarily think “the cloud” is a bad idea. What we believe is that you should have the freedom to choose what makes sense for you. But that choice is moot if the particular app you need to use is only available in the cloud – we need the same apps to be available everywhere.
Oasis has now been running reliably for over a year. The Sandstorm team uses Oasis every day to get our own work done. I am composing this blog post in Etherpad, while organizing my task list in Wekan, chatting with teammates in Rocket.Chat, and syncing files with Davros.
Here are just some of the things we’ve changed since Oasis was launched:
We’ve so far kept Oasis labeled “beta”, mostly because, as engineers, we always feel like there’s so much more to do. But, that will always be true – no good software project is ever “done”. With Oasis being used for so much real work, the time has come to remove the “beta” label.
Oasis will officially emerge from beta on November 27. We wanted to give advance notice of this change because it affects our paying users: we will no longer be waiving your subscription fee as we have during the beta period. For backers of our Indiegogo campaign who opted for free hosting as a perk, the timer on your service will start now (hey, you got an extra free year!). For the rest, your next monthly invoice will be charged from your credit card. Your subscription payments help support further development of Sandstorm and packaging more apps. Thank you for your support!
By Kenton Varda - 10 Nov 2016
As of a couple weeks ago – October 23, 2016 – Sandstorm can now be installed on systems with:
This means that Sandstorm can now be installed on Red Hat Enterprise Linux (RHEL) 7, as well as its cousin CentOS 7, both of which use kernel version 3.10.
It also means that Sandstorm can now be installed on Arch Linux, which has historically shipped kernels compiled with CONFIG_USER_NS=n.
So if you previously couldn’t install Sandstorm because you were using one of these distros, now you can!
For the technically curious…
Sandstorm uses the Linux kernel’s “namespaces” feature as part of setting up the secure sandboxes in which apps run. Normally, creating namespaces requires root privileges, because these features could be used to escalate privileges. However, using “user namespaces”, a process that does not have root privileges can create a special kind of namespace in which other namespaces are (ostensibly) safe to use. Hence, it allows unprivileged processes to create sandboxes.
For security reasons, most of Sandstorm does not run with root privileges. Because of this, it has long relied on user namespaces to allow it to set up sandboxes. At the time the Sandstorm project started, it looked like user namespaces would soon be broadly available across Linux distros, so this seemed like a reasonable strategy.
Unfortunately, this has not been entirely true in practice. The enterprise-oriented RHEL and CentOS distros have long release cycles. Today, they still use kernel version 3.10, which is nearly three years old. Because user namespaces still had many problems in this kernel version, they were disabled by default and are today available only with a boot flag. Meanwhile, some faster-moving distros like Arch have chosen to keep user namespaces disabled even with newer kernel versions due to security concerns: the user namespaces feature has been the source of many local privilege escalation exploits in Linux. Although these vulnerabilities can’t be exploited by Sandstorm apps, such frequent vulnerabilities are problematic for servers which rely on user account separation for security outside of Sandstorm.
Even as it became apparent that Sandstorm’s use of user namespaces was preventing it from being used on some distros, we were hesitant to try other approaches. It seemed like the only way to solve the problem would be to employ a setuid-root binary to set up sandboxes when user namespaces were not available. setuid-root binaries are inherently risky – if not written exactly correctly, it could open its own privilege escalation vulnerability. Also, it would require a major refactoring of Sandstorm internals to move the supervisor into its own binary.
But a couple weeks ago, I realized suddenly that a different idea would work. The Sandstorm server normally starts up as root, but then runs several child processes under a regular user account. Most of Sandstorm’s business logic is in a node.js web server. That process talks via Cap’n Proto RPC to a “back-end” daemon written in C++, which in turn launches app sandboxes. This back-end daemon is hand-coded in C++, with the core logic all living in a single file.
Because of this design, it turned out to be relatively easy to pass superuser privileges down through the back-end, while still keeping them away from the web server. Specifically, the back-end can execute with its effective UID set to a normal user account, but its real UID being root. Then, when it comes time to start a sandbox, it can promote itself back to root to do the work.
This turned out to take only a couple hours to implement. In retrospect, the design seems obvious, and I wish I’d thought of it sooner!
There is a minor downside: If a vulnerability allows an attacker to cause the back-end to execute arbitrary code, that code could claim the superuser privileges, whereas before it would be limited to the Sandstorm server UID. This risk is probably small because the back-end is a relatively simple program that only speaks directly to other trusted programs (although it speaks indirectly to potentially-malicious actors). Nevertheless, if user namespaces are available, then Sandstorm will avoid handing root privileges to the back-end at all, continuing to operate as it did historically.
Existing Sandstorm users need not take any action. Your servers will continue to operate exactly as they always have.
But if you’ve been held back from installing Sandstorm before because it wouldn’t work on your distro, you should try again now!