CouchDB - JSON and B-trees and REST, oh my!

[pozorvlak]

I've been learning about the NoSQL database CouchDB, mainly from the Definitive Guide, but also from the Coursera Introduction to Data Science course and through an informative chat with necaris, who has used it extensively at Esplorio. The current draft of the Definitive Guide is rather out-of-date and has several long-open pull requests on GitHub, which doesn't exactly inspire confidence, but CouchDB itself appears to be actively maintained. I have yet to use CouchDB in anger, but here's what I've learned so far:

• CouchDB is, at its core, an HTTP server providing append-only access to B-trees of versioned JSON objects via a RESTful interface. Say what now? Well, you store your data as JavaScript-like objects (which allow you to nest arrays and hash tables freely); each object is indexed by a key; you access existing objects and insert new ones using the standard HTTP GET, PUT and DELETE methods, specifying and receiving data in JavaScript Object Notation; you can't update objects, only replace them with new objects with the same key and a higher version number; and it's cheap to request all the objects with keys in a given range.


• The JSON is not by default required to conform to any particular schema, but you can add validation functions to be called every time data is added to the database. These will reject improperly-formed data.


• CouchDB is at pains to be RESTful, to emit proper cache-invalidation data, and so on, and this is key to scaling it out: put a contiguous subset of (a consistent hash of) the keyspace on each machine, and build a tree of reverse HTTP proxies (possibly caching ones) in front of your database cluster.


• CouchDB's killer feature is probably master-to-master replication: if you want to do DB operations on a machine that's sometimes disconnected from the rest of the cluster (a mobile device, say), then you can do so, and sync changes up and down when you reconnect. Conflicts are flagged but not resolved by default; you can resolve them manually or automatically by recording a new version of the conflicted object. Replication is also used for load-balancing, failover and scaling out: you can maintain one or more machines that constantly replicate the master server for a section of keyspace, and you can replicate only a subset of keyspace onto a new database when you need to expand.


• CouchDB doesn't guarantee to preserve all the history of an object, and in particular replications only seem to send the most recent version; I think this precludes Git-style three-way merge from the conflicting versions' most recent common ancestor (and forget about Darcs-style full-history merging!).


• The cluster-management story isn't as good as for some other systems, but there are a couple of PaaS offerings.


• Queries/views and non-primary indexes are both handled using map/reduce. If you want to index on something other than the primary key - posts by date, say - then you write a map query which emits (date, post) pairs. These are put into another B-tree, which is stored on disk; clever things are done to mark subtrees invalid as new data comes in, and changes to the query result or index are calculated lazily. Since indices are stored as B-trees, it's cheap to get all the objects within a given range of secondary keys: all posts in February, for instance.


• CouchDB's reduce functions are crippled: attempting to calculate anything that isn't a scalar or a fixed-size object is considered Bad Form, and may cause your machine(s) to thrash. AFAICT you can't reduce results from different machines by this mechanism: CouchDB Lounge requires you to write extra merge functions in Twisted Python.


• Map, reduce and validation functions (and various others, see below) are by default written in JavaScript. But CouchDB invokes an external interpreter for them, so it's easy to extend CouchDB with a new query server. Several such have been written, and it's now possible to write your functions in many different languages.


• There's a very limited SQL view engine, but AFAICT nothing like Hive or Pig that can take a complex query and compile it down into a number of chained map/reduce jobs. The aforementioned restrictions on reduce functions mean that the strategy I've been taught for expressing joins as map/reduce jobs won't work; I don't know if this limitation is fundamental. But it's IME pretty rare to require general joins in applications: usually you want to do some filtering or summarisation on at least one side.


• CouchDB can't quite make up its mind whether it wants to be a database or a Web application framework. It comes by default with an administration web app called Futon; you can also use it to store and execute code for rendering objects as HTML, Atom, etc. Such code (along with views, validations etc) is stored in special JSON objects called "design documents": best practice is apparently to have one design document for each application that needs to access the underlying data. Since design documents are ordinary JSON objects, they are propagated between nodes by replications.


• However, various standard webapp-framework bits are missing, notably URL routing. But hey, you can always use mod_rewrite...


• There's a tool called Erica (and an older one called CouchApp) which allows you to sync design documents with more conventional source-code directories in your filesystem.


• CouchDB is written in Erlang, and the functional-programming influence shows up in other places: most types of user-defined function are required to be free of side-effects, for instance. Then there's the aforementioned uses of lazy evaluation and the append-only nature of the system as a whole. You can extend it with your own Erlang code or embed it into an Erlang application, bypassing the need for HTTP requests.

tl;dr if you've ever thought "data modelling and synchronisation are hard, let's just stick a load of JSON files in Git" (as I have, on several occasions), then CouchDB is probably a good fit to your needs. Especially if your analytics needs aren't too complicated.

Comments

[andrewducker]

Sounds like a more featureful version of memcached. Which is probably great for some people, and will drive others insane :->

[necaris]

Not quite. CouchDB doesn't keep anything in memory by itself -- it lets the OS handle filesystem caching, and everything goes to the on-disk indexes. So it would be unspeakably slow for most of memcached's use cases.

Interestingly, Couchbase, which is what we're using at Esplorio these days, is a fusion of memcached and CouchDB...

[pozorvlak]

it lets the OS handle filesystem caching, and everything goes to the on-disk indexes.

Like Varnish? Makes sense - modern operating systems are AIUI pretty good at filesystem caching!

[necaris]

Haha, yes, another very cool tool that we're starting to use in our stack :-)

AIUI Varnish mmaps some files and lets the OS decide how much of it will actually be in memory, rather than doing its own cache-handling? CouchDB doesn't mmap IIRC but that's about right.

[pozorvlak]

As necaris notes, the big difference is that CouchDB is disk-backed. But I believe memcached was a big influence on most subsequent NoSQL systems, so it's probably had some influence on CouchDB.

[necaris]

Couple of things to add:

• CouchDB-as-web-app-framework isn't quite as limited as you might thing. URL rewriting, for instance, is handled just fine, and there are tools for app development and distribution (like Kanso).


• CouchDB's brand of map/reduce is pretty limited, but a reduce function that produces non-scalar results is perfectly doable. It's only considered Bad Form because it's tricky to get right, and it's terribly easy to actually *accumulate* more data in your reduce result and end up with thrashing machines and giant view indexes and other Bad Things.

Quibbles aside (as you note, I have used CouchDB a lot in the last few months), great post!

[pozorvlak]

Neat, thanks! I suspect the URL-rewriting has been added since the "Definitive Guide" was last updated :-)

A few points...

[wilhol]

It's true that the definitive guide is a little dated now. A couple of points to pick up on:

Document conflicts: replication will pass conflicted document revisions (see http://wiki.apache.org/couchdb/Replication_and_conflicts for more detail and a comparison with Git). There is a limit on the number of conflicted revisions that can be stored for each document though (configurable, but I think 1000 is the default).

Clustering: The dynamo-based clustering layer used at Cloudant (BigCouch) is becoming a core part of CouchDB (see https://blogs.apache.org/couchdb/entry/welcome_bigcouch).

re, CouchApps, the intent here is to provide an easy way to distribute data visualisation / browsing apps alongside the data itself, i.e. you can replicate data and the tool to view it between CouchDB instances. They just provide a way to serve up static assets so, for most cases, it's no substitute for a proper web-application tier. As previously mentioned, there is now support for URL rewriting and CORS.

It's possibly worth mentioning CouchDB-Lucene (https://github.com/rnewson/couchdb-lucene) as well. This provides full text search - handy for many use cases where map-reduce is not a good fit. GeoCouch (https://github.com/couchbase/geocouch/) is also nice for GeoSpatial indexes.

Also, CouchBase is far closer to MemBase than CouchDB. The original plan was to use CouchDB as a persistence layer but, as I understand it, this was abandoned. At this point they should be considered totally distinct, the naming being an unfortunately historical artefact.