• Day 1
• Day 2
• Day 3

What the two theorems mean

The FLP theorem states that in an asynchronous network where messages may be delayed but not lost, there is no consensus algorithm that is guaranteed to terminate in every execution for all starting conditions, if at least one node may fail-stop.

The CAP theorem states that in an asynchronous network where messages may be lost, it is impossible to implement a sequentially consistent atomic read / write register that responds eventually to every request under every pattern of message loss.

Without ever having tackled the problem formally, I had speculated that they might be equivalent, based on a few observations. First, consensus and serialisable atomic objects are very closely related. Both involve causing a set of nodes to come to some agreement about shared state. In the case of consensus, it’s the value proposed, in the case of atomic objects it’s the order and identity of any operations performed. Secondly, the failure modes in both theorems appeared similar. CAP deals with ‘partitions’, which is the non-delivery of a subset of messages, and FLP deals with ‘faulty nodes’, which are nodes that only take a finite number of steps in any execution, steps which include message receipt. It seemed likely that both failure models could be used to describe the other. Thirdly, both problems deal with safety and liveness properties in distributed systems in the context of failures.

I wrote an answer up quickly to the question, and formulated a proof sketch typical for these kind of questions: if two problems are equivalent, then a solution to one is a solution to the other, and vice versa. My answer was, like so many informal arguments of this nature, concise, convenient, convincing and wrong.

The Importance Of Context

Emin Gun Sirer pointed out was what wrong with my argument. One direction of the equivalence still looks good – a solution to CAP could be used to formulate a solution to the FLP problem. The problem is in the other direction – with my assertion that an FLP solution could be used to solve CAP. The distinction arises when we consider how each theorem treats nodes that aren’t receiving the messages that are being sent to them. In FLP, such nodes are failed, and exempt from having to achieve consensus. In CAP, such nodes are only partitioned. Here’s the difference: a CAP solution requires that any live node be able to correctly serve requests, even if it has not received any messages. So a partitioned node in FLP does not have to achieve consensus, since it is considered failed, but the same node in CAP must – somehow – keep up with the activity of the rest of the system.

To reinforce this point, let’s take a look at how Gilbert and Lynch proved their formalisation of the CAP theorem. They show a simple, intuitive result. If there is a permanent partition between two disjoint subsets of nodes in the system, call them and , then a write to can never cause any different execution to occur in (because the only way that can influence is to send a message). But might have to respond to a read request after the initial write to . To be sequentially consistent, must return the value of the write. It can’t, because it can never learn of the write. CAP does not identify partition with failure. FLP does, at least in the single node case.

This blows my argument out of the water, because I was relying on equal treatment of nodes that were completely partitioned from all others in both settings. But that’s not the case. So it is not obvious that a solution to FLP can provide a solution to CAP.

Proving Yourself Wrong

It would be unsatisfying only to establish that my original argument was flawed, because doing so doesn’t actually settle the original question: can CAP and FLP be considered equivalent? Instead of trying to prove the positive result, armed with a bit more clarity about the difference between the two theorems, let’s try and prove the negative; in particular that a solution to the FLP theorem will not provide a solution to the CAP theorem.

A brief aside about the validity of this proof technique: you might reasonably be uncomfortable with me reasoning about ‘positive’ solutions to the CAP or FLP theorem, since respected researchers have already established in peer reviewed articles that no solution to either exists. However, it’s still reasonable to effectively fantasise about what would be true if positive solutions existed, and indeed it’s fundamental to posing questions about equivalence. To appeal to a more commonly considered problem – computer scientists spend a lot of time considering what would happen if a positive solution to was found, even though it’s very possible (and usually considered likely) that no positive solution exists.

So let’s pretend we have this magical black box, which will provide a solution to the FLP problem. That is, if every node in the system runs the algorithm in this box, then non-trivial consensus will always be reached in finite time at all non-faulty nodes, even where there is a single faulty node in the system, under all initial conditions and all network behaviours. Could we use this algorithm to solve CAP?

To prove otherwise, we’re going to use a very similar argument to the one Gilbert and Lynch made, which is a proof by contradiction – we’re going to assume that the algorithm can solve CAP, and then show an example of where it could not.

In the following, I’m going to use ‘achieves consensus’ as the target problem to solve. To do that, I need to quickly argue that any solution to consensus (ignoring the possibility of failures) can be used to implement an atomic sequentially consistent object. This is well established in the literature – the construction is called a distributed finite state machine. It works by having any node (i.e. one that receives a request) propose a value (i.e. the next write operation to execute, and its result). A round of consensus is performed, where the nodes either agree to the ‘next operation’ proposal, or vote it down. If they vote it down, the proposer tries again. There are solutions (such as Paxos) which ensure that every valid proposal is eventually accepted. An object implemented in this way is necessarily sequentially consistent, since all requests are ordered by the process of consensus deciding the next operation to execute. Read requests may be served locally without performing consensus, since all nodes know about the ‘most recent’ write operation.

Let’s set up our network as follows. There will be one distinguished node, , which is the ‘failing’ node. It does not receive any messages from the rest of the system, and therefore takes a finite number of steps and then stops. We’ll call the rest of the network . In order to be a solution to CAP, must itself decide on a value, since it may receive a read request to which it must respond in finite time. We’ll show that even with a solution to FLP, there must be some execution where does not know the right answer.

(Note – this is an extremely simple argument to make informally, but I’m hoping to show how one might think about this and more difficult problems more formally).

Imagine that a write() operation is initiated by a client of some node in , followed by read issued at , all the time using the FLP algorithm we were given. Then, in order to be a correct CAP solution, F must respond to the read with the value . To do that, it takes some finite series of steps , and then takes no further action, because it is receiving no messages to spur it on.

Now imagine that instead, with the same exact initial conditions, a write() operation is initiated, again by a client of some node in . Again, to be correct, must respond to a subequent read with the value . To do so, it takes another series of steps . However, and must be exactly the same sequence. Why? Because the behaviour of a node is governed the algorithm it is executing, the initial conditions, and the messages it receives. The first two are exactly the same in both executions, and since no messages are being delivered, so is the third. Therefore, the value that has decided upon by the final step of both and must be the same value. But for correctness, is required to return different values in each execution. This is a contradiction, hence the FLP-solving algorithm cannot be a solution to CAP, which was our initial assumption.

At this point you might be concerned – doesn’t FLP guarantee that all nodes agree on the same value? That is, if we had a magical FLP-solver, wouldn’t that guarantee that saw the same value as every other node in ? The key observation here is that, from the perspective of FLP, has failed. From the perspective of CAP, it has not, and must continue to participate correctly in certain activities of the system. So the algorithm is correctly solving consensus as defined by FLP, but the requirements of CAP are too strong.

If consensus cannot be achieved even with an FLP solution in CAP, we cannot use it to construct any kind of solution to CAP.

What We Have Learnt

This result (if I’ve made no mistakes) is arguably more interesting than if the original equivalence assertion was true. To be an interesting impossibility result in distributed systems, it’s usually true that you want to place the fewest restrictions possible on the environment in which you’re establishing that result. The idea is that you give your result every chance to be wrong, by allowing the environment few restrictions in the tricks it can pull to overcome the obstacles you’re constructing. Then, if your result still turns out to be true, you know you have proved something really quite strong. (There way ‘strong’ and ‘weak’ are often used can be confusing – weak assumptions lead to strong impossibility results and vice versa).

So FLP, with its strictly weaker restrictions – all messages are eventually delivered, faulty nodes don’t have to achieve consensus – is by this definition a stronger result than CAP, which allows messages to be lost forever and forces partitioned nodes to participate in the system. It’s therefore much more surprising – and the authors in the original paper remark on this – because it’s so much more unexpected.

Conversely, CAP appears much more humdrum. Is it really surprising to anyone that it’s impossible to maintain sequentially consistent state in a distributed system if nodes cannot talk to each other? It’s not too far away from discussing what’s possible in a network that can experience total node failure – nothing at all, which is why all papers disregard it as a consideration. The utility of CAP comes from telling systems designers that they must be prepared to work in a world where availability or consistency are compromised, but is rather cataclysmic about the situations in which they might be threatened.

It would be more interesting to relax one or two assumptions – what if partitions are only temporary? What if we allow the client to participate (i.e. a client that has seen a write is allowed to convey that fact to a replica that it issues a read from)? What if partitioned nodes were excluded from participation? If we can show CAP to hold in these circumstances, it greatly restricts our ability to design correct, available systems that experience much weaker failures. That would be interesting.

I figured, since I get a bunch of e-mail from students through this site, that it might be worth sharing my answer:

Take it, if you’re serious about distributed systems, and here’s why:

Systems ‘thinking’ is terribly important. One way to learn that well is to read papers and to discuss them. This hones the ability to critically think about computer systems, about the value of certain lines of work and about the meaning (or lack of) of performance results. In the absence of a formalised way of judging systems work (i.e. through a proof) you need to develop your own judgment and taste. This is a very good way to do so.

Not only that, the topics in the course are practically very relevant. Distributed systems aren’t any different from other systems topics in the sense that they interact with the network, disk and CPU in much the same way. To give you an example, distributed resource management (i.e. Mesos or YARN) which is an incredibly relevant ‘distributed’ systems topic right now needs to understand local scheduling policies, the relative characteristics of disk and memory and the interaction between CPU and the memory hierarchy to begin to be effective.

Thinking of distributed systems as an abstraction over a collection of nodes works best only when you are thinking about distributed algorithms, but it doesn’t work if you’re an engineer. If you’re mathematically minded, and you’re not that keen on actually implementing what you come up with instead of proving something about it, then this might not be the course for you (although I’d *still* suggest you consider it strongly).

But if you like to build things, and you like to *really* understand how they work, you’re going to need to know all of this stuff and more.

(And if I’m not a big enough draw – perish the thought – there are many, many other interesting sessions. In particular, Josh will be talking about Crunch, and Sarah will be giving both introductory and advanced Hadoop classes – both people I work with, and both fantastic speakers!)

Since the Dynamo paper appeared and really popularised eventual consistency, the debate has focused on a fairly binary treatment of its merits. Either you can’t afford to be wrong, ever, or it’s ok to have your reads be stale for a potentially unbounded amount of time. In fact, the suitability of eventual consistency is dependent partly on the distribution of stale reads; that is the speed of quiescence of a system immediately after a write. If the probability of a ever seeing a stale read due to consistency delays can be reduced to smaller than the probability of every machine in the network simultaneously catching fire, we can probably make use of eventual consistency.

Looking at many designed systems (where there is little more than conventional wisdom on how to choose R and W), it’s clear that an analytical model relating system parameters to distributions of behaviour is sorely needed. PBS is a good step in that direction. It would be good to see the work extended to handle a treatment of failure distributions (although a good failure model is hard to find!). The reply latencies of write and read replicas are modelled exponentially distributed CDFs, but in reality there’s a more significant probability of the reply latency becoming infinite. Once that distribution is correctly modelled, PBS should be able to run simulations against it with no change.

A great use for this tool would be to enter some operational parameters, such as the required consistency probability, max number of nodes, availability requirements and maximum request latency, and have PBS suggest some points in the system design space that would meet these requirements with high probability. As the size of the R / W quora get larger, the variance on the request latencies gets larger, but the resilience to failures increases as does the likelihood of fresh reads. For full credit, PBS could additionally model a write / read protocol (i.e. 2-phase commit) which has different consistency properties. As Daniel Abadi discusses, when things are running well different consistency guarantees trade off between latency and the strength of consistency.

Nice work PBS team!

• checking a leader election protocol for correctness
• cleaning up some code in an open source project
• designing a specialised messaging system
• working with one of our outstanding interns on the very cool project he’s taking on over the summer
• cheerleading as Cloudera pushed another open-source project out of the nest.

We have more interesting work than we can possibly do. We need great engineers to come solve some really fascinating problems. I wanted to take advantage of the fact that this blog is currently getting a lot of traffic to put this message in front of a lot of eyeballs: you should really consider coming to work at Cloudera.

We’re hiring pretty aggressively in several engineering areas, but two in particular that I can say a lot about:

Distributed systems engineers. Engineering jobs where you are doing what we would call ‘systems’ work are few and far between. They exist, but they’re not common. One thing I have found, speaking to engineers at other companies, that a job that encourages specialisation in distributed systems is even less easily found, especially at an exciting company with plenty of really smart people. At Cloudera we need people to work on a variety of large distributed systems – execution frameworks, distributed filesystems, data ingest pipelines, schedulers, coordination systems, query processing and more. The problems you would work on are fundamental and challenging, and the people you would get to work with know the systems inside and out. The need to scale is real – we have customers who have thousands of machines in their clusters, and we write software to run across every single core.

Applications engineers. Think managing one of those clusters is easy? It isn’t. The monitoring and lifecycle management challenge posed by Apache Hadoop and other systems can be enormously complex. Each cluster produces a vast amount of data, and your job as an application engineer is to extract the right signals from that data and bring it to operations engineers through clear, meaningful visualisations so that they can make sense of all that information. We’re looking for full-stack developers, as happy writing a server process to aggregate and chomp through log files as they are building a reusable heatmap control to properly visualise a multi-dimensional distribution.

The perks at Cloudera are good – we have offices in San Francisco and Palo Alto, we have communal lunches brought in every day, there’s a subsidised gym membership, you can buy books and we’re pretty light on procedural overhead. But if you’re a good fit, you’ll probably be attracted because the problems are real, interesting and hard, and the people you’ll work with to solve them are smart, knowledgable and a whole load of fun.

If you’re interested, drop me a line or send me your resume at henry AT cloudera DOT com, and I’ll happily tell you anything you want to know about this place.

Another conclusion from the article is that STM performs best when there is little contention for transactions between threads. Again, that should really be a given – all reasonable concurrency primitives have high throughput when there is little contention but high parallelism. (A lot of work has gone into making this a very fast case (since it is the most common) for locking, see e.g. biased locking schemes in the Hotspot JVM).

Bryan Cantrill (previously of Fishworks, now of Joyent) rips on transactional memory more eloquently than I ever could. STM is a declarative solution to thread safety, which I like, but no more declarative really than synchronised blocks – and Cantrill points out the elephant in the room that the CACM article seemed to ignore: doing IO inside transactions is hugely problematic (because how precisely do you roll back a network packet?).

A recent paper at SOSP 2009 called Operating System Transactions attacked this problem, although not from the viewpoint of STM, but to provide atomicity and isolation for situations where bugs arise from the separation between reads, and writes that depend on that read (Time Of Check To Time Of Use – TOCTTOU). Perhaps there’s an overlap between this paper and STM approaches, but it’s not clear whether the workloads inside an operating system’s system call layer are general enough to map onto typical user-space STM work.

I think the article makes a point worth making again, and makes it fairly well – that CAP is really about P=> ~(C & A). A couple of things I want to call out though, after a rollicking discussion on Hacker News.

“For a distributed (i.e., multi-node) system to not require partition-tolerance it would have to run on a network which is guaranteed to never drop messages (or even deliver them late) and whose nodes are guaranteed to never die. You and I do not work with these types of systems because they don’t exist.”

This is a bit strong, at least theoretically. Actually all you need to not require partition-tolerance is to guarantee that your particular kryptonite failure pattern never occurs. Many protocols are robust to a dropped message here or there. A quorum system requires a fairly dramatic failure (one node completely partitioned) before one side of the partition has to occur. In practice, of course, these failures happen more often than we would like, which is why we worry about fault-tolerant properties of distributed algorithms.

Therefore the paragraph on failure probabilities is less powerful. It’s not always a problem if a single failure occurs, and therefore you shouldn’t immediately worry about sacrificing availability or consistency as soon as one node starts running slowly. CAP only establishes the existence of a failure pattern that torpedoes any distributed implementation of an atomic object, not its high probability.

I’m no great fan of the ubiquity of the CAP theorem – it’s a solid impossibility result which appeals to the theorist in me, but it doesn’t capture every fundamental tension in a distributed system. For example: we make our systems distributed across more than one machine usually for reasons of performance and to eliminate a single point of failure. Neither of these motivations are captured verbatim by the CAP theorem. There’s more to designing distributed systems!

In this, I agree with Stonebraker; it’s the erroneous representation of ‘partition tolerance’ that I found very strange. I’ve been a good deal more forceful in private about this than I have in public

Summer of Code is a great program – providing stipends to students and more importantly connecting them with mentors in open source projects. ZooKeeper has a number of interesting projects to get started on.

ZooKeeper is a distributed coordination platform on which you can build the distributed equivalents of many traditional concurrent primitives like locks, queues and barriers. It’s heavily used in the real world – Yahoo! use it extensively, and many other major companies rely on it. The other committers and I are actively looking to increase participation in the project – there is loads of really interesting work left to do. If consensus protocols, distributed systems, scalability, fault-tolerance and performance are your thing, this is certainly the project for you.

If you have any questions at all, drop me a line at henry at apache dot org.