First, back to the B-trees! We explained that the B-tree that backs the key-sorted view result is built only once, when you first query a view, and all subsequent queries will just read the B-tree instead of executing the map function for all documents again. What happens, though, when you change a document, add a new one, or delete one? Easy: CouchDB is smart enough to find the rows in the view result that were created by a specific document. It marks them invalid so that they no longer show up in view results. If the document was deleted, we’re good — the resulting B-tree reflects the state of the database. If a document got updated, the new document is run through the map function and the resulting new lines are inserted into the B-tree at the correct spots. New documents are handled in the same way. The B-tree is a very efficient data structure for our needs, and the crash-only design of CouchDB databases is carried over to the view indexes as well.
To add one more point to the efficiency discussion: usually multiple documents are updated between view queries. The mechanism explained in the previous paragraph gets applied to all changes in the database since the last time the view was queried in a batch operation, which makes things even faster and is generally a better use of your resources.
On to more complex map functions. We said “find documents by any value or structure that resides in them.” We already explained how to extract a value by which to sort a list of views (our date field). The same mechanism is used for fast lookups. The URI to query to get a view’s result is /database/_design/designdocname/_view/viewname. This gives you a list of all rows in the view. We have only three documents, so things are small, but with thousands of documents, this can get long. You can add view parameters to the URI to constrain the result set. Say we know the date of a blog post. To find a single document, we would use /blog/_design/docs/_view/by_date?key="2009/01/30 18:04:11" to get the “Biking” blog post. Remember that you can place whatever you like in the key parameter to the emit() function. Whatever you put in there, we can now use to look up exactly — and fast.
Note that in the case where multiple rows have the same key (perhaps we design a view where the key is the name of the post’s author), key queries can return more than one row.
We talked about “getting all posts for last month.” If it’s February now, this is as easy as /blog/_design/docs/_view/by_date?startkey="2010/01/01 00:00:00"&endkey="2010/02/00 00:00:00". The startkey and endkey parameters specify an inclusive range on which we can search.
To make things a little nicer and to prepare for a future example, we are going to change the format of our date field. Instead of a string, we are going to use an array, where individual members are part of a timestamp in decreasing significance. This sounds fancy, but it is rather easy. Instead of:
{
"date": "2009/01/31 00:00:00"
}
we use:
{
"date": [2009, 1, 31, 0, 0, 0]
}
Our map function does not have to change for this, but our view result looks a little different:
Table 2. New view results:
| Key | Value |
|---|---|
| [2009, 1, 15, 15, 52, 20] | “Hello World” |
| [2009, 2, 17, 21, 13, 39] | “Biking” |
| [2009, 1, 30, 18, 4, 11] | “Bought a Cat” |
And our queries change to /blog/_design/docs/_view/by_date?startkey=[2010, 1, 1, 0, 0, 0]&endkey=[2010, 2, 1, 0, 0, 0]. For all you care, this is just a change in syntax, not meaning. But it shows you the power of views. Not only can you construct an index with scalar values like strings and integers, you can also use JSON structures as keys for your views. Say we tag our documents with a list of tags and want to see all tags, but we don’t care for documents that have not been tagged.
{
...
tags: ["cool", "freak", "plankton"],
...
}
{
...
tags: [],
...
}
function(doc) {
if(doc.tags.length > 0) {
for(var idx in doc.tags) {
emit(doc.tags[idx], null);
}
}
}
This shows a few new things. You can have conditions on structure (if(doc.tags.length > 0)) instead of just values. This is also an example of how a map function calls emit() multiple times per document. And finally, you can pass null instead of a value to the value parameter. The same is true for the key parameter. We’ll see in a bit how that is useful.
To retrieve view results in reverse order, use the descending=true query parameter. If you are using a startkey parameter, you will find that CouchDB returns different rows or no rows at all. What’s up with that?
It’s pretty easy to understand when you see how view query options work under the hood. A view is stored in a tree structure for fast lookups. Whenever you query a view, this is how CouchDB operates:
If you specify descending=true, the reading direction is reversed, not the sort order of the rows in the view. In addition, the same two-step procedure is followed.
Say you have a view result that looks like this:
| Key | Value |
|---|---|
| 0 | “foo” |
| 1 | “bar” |
| 2 | “baz” |
Here are potential query options: ?startkey=1&descending=true. What will CouchDB do? See #1 above: it jumps to startkey, which is the row with the key 1, and starts reading backward until it hits the end of the view. So the particular result would be:
| Key | Value |
|---|---|
| 1 | “bar” |
| 0 | “foo” |
This is very likely not what you want. To get the rows with the indexes 1 and 2 in reverse order, you need to switch the startkey to endkey: endkey=1&descending=true:
| Key | Value |
|---|---|
| 2 | “baz” |
| 1 | “bar” |
Now that looks a lot better. CouchDB started reading at the bottom of the view and went backward until it hit endkey.
We use an array key here to support the group_level reduce query parameter. CouchDB’s views are stored in the B-tree file structure. Because of the way B-trees are structured, we can cache the intermediate reduce results in the non-leaf nodes of the tree, so reduce queries can be computed along arbitrary key ranges in logarithmic time. See Figure 1, “Comments map function”.
In the blog app, we use group_leve``l reduce queries to compute the count of comments both on a per-post and total basis, achieved by querying the same view index with different methods. With some array keys, and assuming each key has the value ``1:
["a","b","c"]
["a","b","e"]
["a","c","m"]
["b","a","c"]
["b","a","g"]
the reduce view:
function(keys, values, rereduce) {
return sum(values)
}
returns the total number of rows between the start and end key. So with startkey=["a","b"]&endkey=["b"] (which includes the first three of the above keys) the result would equal 3. The effect is to count rows. If you’d like to count rows without depending on the row value, you can switch on the rereduce parameter:
function(keys, values, rereduce) {
if (rereduce) {
return sum(values);
} else {
return values.length;
}
}
Note
JavaScript function about could be effectively replaced by builtin _count one.
This is the reduce view used by the example app to count comments, while utilizing the map to output the comments, which are more useful than just 1 over and over. It pays to spend some time playing around with map and reduce functions. Futon is OK for this, but it doesn’t give full access to all the query parameters. Writing your own test code for views in your language of choice is a great way to explore the nuances and capabilities of CouchDB’s incremental MapReduce system.
Anyway, with a group_level query, you’re basically running a series of reduce range queries: one for each group that shows up at the level you query. Let’s reprint the key list from earlier, grouped at level 1:
["a"] 3
["b"] 2
And at group_level=2:
["a","b"] 2
["a","c"] 1
["b","a"] 2
Using the parameter group=true makes it behave as though it were group_level=exact, so in the case of our current example, it would give the number 1 for each key, as there are no exactly duplicated keys.
We briefly talked about the rereduce parameter to your reduce function. We’ll explain what’s up with it in this section. By now, you should have learned that your view result is stored in B-tree index structure for efficiency. The existence and use of the rereduce parameter is tightly coupled to how the B-tree index works.
Consider the map result are:
"afrikan", 1
"afrikan", 1
"chinese", 1
"chinese", 1
"chinese", 1
"chinese", 1
"french", 1
"italian", 1
"italian", 1
"spanish", 1
"vietnamese", 1
"vietnamese", 1
Example 1. Example view result (mmm, food)
When we want to find out how many dishes there are per origin, we can reuse the simple reduce function shown earlier:
function(keys, values, rereduce) {
return sum(values);
}
Figure 2, “The B-tree index” shows a simplified version of what the B-tree index looks like. We abbreviated the key strings.
The view result is what computer science grads call a “pre-order” walk through the tree. We look at each element in each node starting from the left. Whenever we see that there is a subnode to descend into, we descend and start reading the elements in that subnode. When we have walked through the entire tree, we’re done.
You can see that CouchDB stores both keys and values inside each leaf node. In our case, it is simply always 1, but you might have a value where you count other results and then all rows have a different value. What’s important is that CouchDB runs all elements that are within a node into the reduce function (setting the rereduce parameter to false) and stores the result inside the parent node along with the edge to the subnode. In our case, each edge has a 3 representing the reduce value for the node it points to.
Note
In reality, nodes have more than 1,600 elements in them. CouchDB computes the result for all the elements in multiple iterations over the elements in a single node, not all at once (which would be disastrous for memory consumption).
Now let’s see what happens when we run a query. We want to know how many “chinese” entries we have. The query option is simple: ?key="chinese". See Figure 3, “The B-tree index reduce result”.
CouchDB detects that all values in the subnode include the “chinese” key. It concludes that it can take just the 3 values associated with that node to compute the final result. It then finds the node left to it and sees that it’s a node with keys outside the requested range (key= requests a range where the beginning and the end are the same value). It concludes that it has to use the “chinese” element’s value and the other node’s value and run them through the reduce function with the rereduce parameter set to true.
The reduce function effectively calculates 3 + 1 on query time and returns the desired result. The next example shows some pseudocode that shows the last invocation of the reduce function with actual values:
function(null, [3, 1], true) {
return sum([3, 1]);
}
Now, we said your reduce function must actually reduce your values.  Consider the following map result and reduce function. This time we want to get a list of all the unique labels in our view:
"abc", "afrikan"
"cef", "afrikan"
"fhi", "chinese"
"hkl", "chinese"
"ino", "chinese"
"lqr", "chinese"
"mtu", "french"
"owx", "italian"
"qza", "italian"
"tdx", "spanish"
"xfg", "vietnamese"
"zul", "vietnamese"
We don’t care for the key here and only list all the labels we have. Our reduce function removes duplicates:
function(keys, values, rereduce) {
var unique_labels = {};
values.forEach(function(label) {
if(!unique_labels[label]) {
unique_labels[label] = true;
}
});
return unique_labels;
}
This translates to Figure 4, “An overflowing reduce index”.
We hope you get the picture. The way the B-tree storage works means that if you don’t actually reduce your data in the reduce function, you end up having CouchDB copy huge amounts of data around that grow linearly, if not faster with the number of rows in your view.
CouchDB will be able to compute the final result, but only for views with a few rows. Anything larger will experience a ridiculously slow view build time. To help with that, CouchDB since version 0.10.0 will throw an error if your reduce function does not reduce its input values.
Map functions are side effect–free functions that take a document as argument and emit key/value pairs. CouchDB stores the emitted rows by constructing a sorted B-tree index, so row lookups by key, as well as streaming operations across a range of rows, can be accomplished in a small memory and processing footprint, while writes avoid seeks. Generating a view takes O(N), where N is the total number of rows in the view. However, querying a view is very quick, as the B-tree remains shallow even when it contains many, many keys.
Reduce functions operate on the sorted rows emitted by map view functions. CouchDB’s reduce functionality takes advantage of one of the fundamental properties of B-tree indexes: for every leaf node (a sorted row), there is a chain of internal nodes reaching back to the root. Each leaf node in the B-tree carries a few rows (on the order of tens, depending on row size), and each internal node may link to a few leaf nodes or other internal nodes.
The reduce function is run on every node in the tree in order to calculate the final reduce value. The end result is a reduce function that can be incrementally updated upon changes to the map function, while recalculating the reduction values for a minimum number of nodes. The initial reduction is calculated once per each node (inner and leaf) in the tree.
When run on leaf nodes (which contain actual map rows), the reduce function’s third parameter, rereduce, is false. The arguments in this case are the keys and values as output by the map function. The function has a single returned reduction value, which is stored on the inner node that a working set of leaf nodes have in common, and is used as a cache in future reduce calculations.
When the reduce function is run on inner nodes, the rereduce flag is true. This allows the function to account for the fact that it will be receiving its own prior output. When rereduce is true, the values passed to the function are intermediate reduction values as cached from previous calculations. When the tree is more than two levels deep, the rereduce phase is repeated, consuming chunks of the previous level’s output until the final reduce value is calculated at the root node.
A common mistake new CouchDB users make is attempting to construct complex aggregate values with a reduce function. Full reductions should result in a scalar value, like 5, and not, for instance, a JSON hash with a set of unique keys and the count of each. The problem with this approach is that you’ll end up with a very large final value. The number of unique keys can be nearly as large as the number of total keys, even for a large set. It is fine to combine a few scalar calculations into one reduce function; for instance, to find the total, average, and standard deviation of a set of numbers in a single function.
If you’re interested in pushing the edge of CouchDB’s incremental reduce functionality, have a look at Google’s paper on Sawzall, which gives examples of some of the more exotic reductions that can be accomplished in a system with similar constraints.
View functions specify a key and a value to be returned for each row. CouchDB collates the view rows by this key. In the following example, the LastName property serves as the key, thus the result will be sorted by LastName:
function(doc) {
if (doc.Type == "customer") {
emit(doc.LastName, {FirstName: doc.FirstName, Address: doc.Address});
}
}
CouchDB allows arbitrary JSON structures to be used as keys. You can use complex keys for fine-grained control over sorting and grouping.
The following clever trick would return both customer and order documents. The key is composed of a customer _id and a sorting token. Because the key for order documents begins with the _id of a customer document, all the orders will be sorted by customer. Because the sorting token for customers is lower than the token for orders, the customer document will come before the associated orders. The values 0 and 1 for the sorting token are arbitrary.
function(doc) {
if (doc.Type == "customer") {
emit([doc._id, 0], doc);
} else if (doc.Type == "order") {
emit([doc.customer_id, 1], doc);
}
}
It maybe be convenient to store date attributes in a human readable format (i.e. as a string), but still sort by date. This can be done by converting the date to a number in the emit() function. For example, given a document with a created_at attribute of 'Wed Jul 23 16:29:21 +0100 2013', the following emit function would sort by date:
emit(Date.parse(doc.created_at).getTime(), doc);
Alternatively, if you use a date format which sorts lexicographically, such as "2013/06/09 13:52:11 +0000" you can just
emit(doc.created_at, doc);
and avoid the conversion. As a bonus, this date format is compatible with the JavaScript date parser, so you can use new Date(doc.created_at) in your client side JavaScript to make date sorting easy in the browser.
If you need start and end keys that encompass every string with a given prefix, it is better to use a high value unicode character, than to use a 'ZZZZ' suffix.
That is, rather than:
startkey="abc"&endkey="abcZZZZZZZZZ"
You should use:
startkey="abc"&endkey="abc\ufff0"
This section is based on the view_collation function in view_collation.js:
// special values sort before all other types
null
false
true
// then numbers
1
2
3.0
4
// then text, case sensitive
"a"
"A"
"aa"
"b"
"B"
"ba"
"bb"
// then arrays. compared element by element until different.
// Longer arrays sort after their prefixes
["a"]
["b"]
["b","c"]
["b","c", "a"]
["b","d"]
["b","d", "e"]
// then object, compares each key value in the list until different.
// larger objects sort after their subset objects.
{a:1}
{a:2}
{b:1}
{b:2}
{b:2, a:1} // Member order does matter for collation.
// CouchDB preserves member order
// but doesn't require that clients will.
// this test might fail if used with a js engine
// that doesn't preserve order
{b:2, c:2}
Comparison of strings is done using ICU which implements the Unicode Collation Algorithm, giving a dictionary sorting of keys. This can give surprising results if you were expecting ASCII ordering. Note that:
You can demonstrate the collation sequence for 7-bit ASCII characters like this:
require 'rubygems'
require 'restclient'
require 'json'
DB="http://<IP_ADDRESS>:5984/collator"
RestClient.delete DB rescue nil
RestClient.put "#{DB}",""
(32..126).each do |c|
RestClient.put "#{DB}/#{c.to_s(16)}", {"x"=>c.chr}.to_json
end
RestClient.put "#{DB}/_design/test", <<EOS
{
"views":{
"one":{
"map":"function (doc) { emit(doc.x,null); }"
}
}
}
EOS
puts RestClient.get("#{DB}/_design/test/_view/one")
This shows the collation sequence to be:
` ^ _ - , ; : ! ? . ' " ( ) [ ] { } @ * / \ & # % + < = > | ~ $ 0 1 2 3 4 5 6 7 8 9
a A b B c C d D e E f F g G h H i I j J k K l L m M n N o O p P q Q r R s S t T u U v V w W x X y Y z Z
Take special care when querying key ranges. For example: the query:
startkey="Abc"&endkey="AbcZZZZ"
will match “ABC” and “abc1”, but not “abc”. This is because UCA sorts as:
abc < Abc < ABC < abc1 < AbcZZZZZ
For most applications, to avoid problems you should lowercase the startkey:
startkey="abc"&endkey="abcZZZZZZZZ"
will match all keys starting with [aA][bB][cC]
The query startkey=["foo"]&endkey=["foo",{}] will match most array keys with “foo” in the first element, such as ["foo","bar"] and ["foo",["bar","baz"]]. However it will not match ["foo",{"an":"object"}]
The _all_docs view is a special case because it uses ASCII collation for doc ids, not UCA:
startkey="_design/"&endkey="_design/ZZZZZZZZ"
will not find _design/abc because ‘Z’ comes before ‘a’ in the ASCII sequence. A better solution is:
startkey="_design/"&endkey="_design0"
To squeeze a little more performance out of views, you can specify "options":{"collation":"raw"} within the view definition for native Erlang collation, especially if you don’t require UCA. This gives a different collation sequence:
1
false
null
true
{"a":"a"},
["a"]
"a"
Beware that {} is no longer a suitable “high” key sentinel value. Use a string like "\ufff0" instead.
If your map function emits an object value which has {'_id': XXX} and you query view with include_docs=true parameter, then CouchDB will fetch the document with id XXX rather than the document which was processed to emit the key/value pair.
This means that if one document contains the ids of other documents, it can cause those documents to be fetched in the view too, adjacent to the same key if required.
For example, if you have the following hierarchically-linked documents:
[
{ "_id": "11111" },
{ "_id": "22222", "ancestors": ["11111"], "value": "hello" },
{ "_id": "33333", "ancestors": ["22222","11111"], "value": "world" }
]
You can emit the values with the ancestor documents adjacent to them in the view like this:
function(doc) {
if (doc.value) {
emit([doc.value, 0], null);
if (doc.ancestors) {
for (var i in doc.ancestors) {
emit([doc.value, Number(i)+1], {_id: doc.ancestors[i]});
}
}
}
}
The result you get is:
{
"total_rows": 5,
"offset": 0,
"rows": [
{
"id": "22222",
"key": [
"hello",
0
],
"value": null,
"doc": {
"_id": "22222",
"_rev": "1-0eee81fecb5aa4f51e285c621271ff02",
"ancestors": [
"11111"
],
"value": "hello"
}
},
{
"id": "22222",
"key": [
"hello",
1
],
"value": {
"_id": "11111"
},
"doc": {
"_id": "11111",
"_rev": "1-967a00dff5e02add41819138abb3284d"
}
},
{
"id": "33333",
"key": [
"world",
0
],
"value": null,
"doc": {
"_id": "33333",
"_rev": "1-11e42b44fdb3d3784602eca7c0332a43",
"ancestors": [
"22222",
"11111"
],
"value": "world"
}
},
{
"id": "33333",
"key": [
"world",
1
],
"value": {
"_id": "22222"
},
"doc": {
"_id": "22222",
"_rev": "1-0eee81fecb5aa4f51e285c621271ff02",
"ancestors": [
"11111"
],
"value": "hello"
}
},
{
"id": "33333",
"key": [
"world",
2
],
"value": {
"_id": "11111"
},
"doc": {
"_id": "11111",
"_rev": "1-967a00dff5e02add41819138abb3284d"
}
}
]
}
which makes it very cheap to fetch a document plus all its ancestors in one query.
Note that the "id" in the row is still that of the originating document. The only difference is that include_docs fetches a different doc.
The current revision of the document is resolved at query time, not at the time the view is generated. This means that if a new revision of the linked document is added later, it will appear in view queries even though the view itself hasn’t changed. To force a specific revision of a linked document to be used, emit a "_rev" property as well as "_id".
| Author: | Christopher Lenz |
|---|---|
| Date: | 2007-10-05 |
| Source: | http://www.cmlenz.net/archives/2007/10/couchdb-joins |
Just today, there was a discussion on IRC how you’d go about modeling a simple blogging system with “post” and “comment” entities, where any blog post might have N comments. If you’d be using an SQL database, you’d obviously have two tables with foreign keys and you’d be using joins. (At least until you needed to add some denormalization).
But what would the “obvious” approach in CouchDB look like?
A simple approach would be to have one document per blog post, and store the comments inside that document:
{
"_id": "myslug",
"_rev": "123456",
"author": "john",
"title": "My blog post",
"content": "Bla bla bla …",
"comments": [
{"author": "jack", "content": "…"},
{"author": "jane", "content": "…"}
]
}
Note
Of course the model of an actual blogging system would be more extensive, you’d have tags, timestamps, etc etc. This is just to demonstrate the basics.
The obvious advantage of this approach is that the data that belongs together is stored in one place. Delete the post, and you automatically delete the corresponding comments, and so on.
You may be thinking that putting the comments inside the blog post document would not allow us to query for the comments themselves, but you’d be wrong. You could trivially write a CouchDB view that would return all comments across all blog posts, keyed by author:
function(doc) {
for (var i in doc.comments) {
emit(doc.comments[i].author, doc.comments[i].content);
}
}
Now you could list all comments by a particular user by invoking the view and passing it a ?key="username" query string parameter.
However, this approach has a drawback that can be quite significant for many applications: To add a comment to a post, you need to:
Now if you have multiple client processes adding comments at roughly the same time, some of them will get a HTTP 409 Conflict error on step 3 (that’s optimistic concurrency in action). For some applications this makes sense, but in many other apps, you’d want to append new related data regardless of whether other data has been added in the meantime.
The only way to allow non-conflicting addition of related data is by putting that related data into separate documents.
Using this approach you’d have one document per blog post, and one document per comment. The comment documents would have a “backlink” to the post they belong to.
The blog post document would look similar to the above, minus the comments property. Also, we’d now have a type property on all our documents so that we can tell the difference between posts and comments:
{
"_id": "myslug",
"_rev": "123456",
"type": "post",
"author": "john",
"title": "My blog post",
"content": "Bla bla bla …"
}
The comments themselves are stored in separate documents, which also have a type property (this time with the value “comment”), and in addition feature a post property containing the ID of the post document they belong to:
{
"_id": "ABCDEF",
"_rev": "123456",
"type": "comment",
"post": "myslug",
"author": "jack",
"content": "…"
}
{
"_id": "DEFABC",
"_rev": "123456",
"type": "comment",
"post": "myslug",
"author": "jane",
"content": "…"
}
To list all comments per blog post, you’d add a simple view, keyed by blog post ID:
function(doc) {
if (doc.type == "comment") {
emit(doc.post, {author: doc.author, content: doc.content});
}
}
And you’d invoke that view passing it a ?key="post_id" query string parameter.
Viewing all comments by author is just as easy as before:
function(doc) {
if (doc.type == "comment") {
emit(doc.author, {post: doc.post, content: doc.content});
}
}
So this is better in some ways, but it also has a disadvantage. Imagine you want to display a blog post with all the associated comments on the same web page. With our first approach, we needed just a single request to the CouchDB server, namely a GET request to the document. With this second approach, we need two requests: a GET request to the post document, and a GET request to the view that returns all comments for the post.
That is okay, but not quite satisfactory. Just imagine you wanted to added threaded comments: you’d now need an additional fetch per comment. What we’d probably want then would be a way to join the blog post and the various comments together to be able to retrieve them with a single HTTP request.
This was when Damien Katz, the author of CouchDB, chimed in to the discussion on IRC to show us the way.
Obvious to Damien, but not at all obvious to the rest of us: it’s fairly simple to make a view that includes both the content of the blog post document, and the content of all the comments associated with that post. The way you do that is by using complex keys. Until now we’ve been using simple string values for the view keys, but in fact they can be arbitrary JSON values, so let’s make some use of that:
function(doc) {
if (doc.type == "post") {
emit([doc._id, 0], doc);
} else if (doc.type == "comment") {
emit([doc.post, 1], doc);
}
}
Okay, this may be confusing at first. Let’s take a step back and look at what views in CouchDB are really about.
CouchDB views are basically highly efficient on-disk dictionaries that map keys to values, where the key is automatically indexed and can be used to filter and/or sort the results you get back from your views. When you “invoke” a view, you can say that you’re only interested in a subset of the view rows by specifying a ?key=foo query string parameter. Or you can specify ?startkey=foo and/or ?endkey=bar query string parameters to fetch rows over a range of keys.
It’s also important to note that keys are always used for collating (i.e. sorting) the rows. CouchDB has well defined (but as of yet undocumented) rules for comparing arbitrary JSON objects for collation. For example, the JSON value ["foo", 2] is sorted after (considered “greater than”) the values ["foo"] or ["foo", 1, "bar"], but before e.g. ["foo", 2, "bar"]. This feature enables a whole class of tricks that are rather non-obvious...
See also
With that in mind, let’s return to the view function above. First note that, unlike the previous view functions we’ve used here, this view handles both “post” and “comment” documents, and both of them end up as rows in the same view. Also, the key in this view is not just a simple string, but an array. The first element in that array is always the ID of the post, regardless of whether we’re processing an actual post document, or a comment associated with a post. The second element is 0 for post documents, and 1 for comment documents.
Let’s assume we have two blog posts in our database. Without limiting the view results via key, startkey, or endkey, we’d get back something like the following:
{
"total_rows": 5, "offset": 0, "rows": [{
"id": "myslug",
"key": ["myslug", 0],
"value": {...}
}, {
"id": "ABCDEF",
"key": ["myslug", 1],
"value": {...}
}, {
"id": "DEFABC",
"key": ["myslug", 1],
"value": {...}
}, {
"id": "other_slug",
"key": ["other_slug", 0],
"value": {...}
}, {
"id": "CDEFAB",
"key": ["other_slug", 1],
"value": {...}
},
]
}
Note
The ... placeholder here would contain the complete JSON encoding of the corresponding document
Now, to get a specific blog post and all associated comments, we’d invoke that view with the query string:
?startkey=["myslug"]&endkey;=["myslug", 2]
We’d get back the first three rows, those that belong to the myslug post, but not the others. Et voila, we now have the data we need to display a post with all associated comments, retrieved via a single GET request.
You may be asking what the 0 and 1 parts of the keys are for. They’re simply to ensure that the post document is always sorted before the the associated comment documents. So when you get back the results from this view for a specific post, you’ll know that the first row contains the data for the blog post itself, and the remaining rows contain the comment data.
One remaining problem with this model is that comments are not ordered, but that’s simply because we don’t have date/time information associated with them. If we had, we’d add the timestamp as third element of the key array, probably as ISO date/time strings. Now we would continue using the query string ?startkey=["myslug"]&endkey=["myslug", 2] to fetch the blog post and all associated comments, only now they’d be in chronological order.
This is a collection of some common SQL queries and how to get the same result in CouchDB. The key to remember here is that CouchDB does not work like an SQL database at all and that best practices from the SQL world do not translate well or at all to CouchDB. This chapter’s “cookbook” assumes that you are familiar with the CouchDB basics such as creating and updating databases and documents.
How you would do this in SQL:
CREATE TABLE
or:
ALTER TABLE
How you can do this in CouchDB?
Using views is a two-step process. First you define a view; then you query it. This is analogous to defining a table structure (with indexes) using CREATE TABLE or ALTER TABLE and querying it using an SQL query.
Defining a view is done by creating a special document in a CouchDB database. The only real specialness is the _id of the document, which starts with _design/ — for example, _design/application. Other than that, it is just a regular CouchDB document. To make sure CouchDB understands that you are defining a view, you need to prepare the contents of that design document in a special format. Here is an example:
{
"_id": "_design/application",
"_rev": "1-C1687D17",
"views": {
"viewname": {
"map": "function(doc) { ... }",
"reduce": "function(keys, values) { ... }"
}
}
}
We are defining a view viewname. The definition of the view consists of two functions: the map function and the reduce function. Specifying a reduce function is optional. We’ll look at the nature of the functions later. Note that viewname can be whatever you like: users, by-name, or by-date are just some examples.
A single design document can also include multiple view definitions, each identified by a unique name:
{
"_id": "_design/application",
"_rev": "1-C1687D17",
"views": {
"viewname": {
"map": "function(doc) { ... }",
"reduce": "function(keys, values) { ... }"
},
"anotherview": {
"map": "function(doc) { ... }",
"reduce": "function(keys, values) { ... }"
}
}
}
The name of the design document and the name of the view are significant for querying the view. To query the view viewname, you perform an HTTP GET request to the following URI:
/database/_design/application/_view/viewname
database is the name of the database you created your design document in. Next up is the design document name, and then the view name prefixed with _view/. To query anotherview, replace viewname in that URI with anotherview. If you want to query a view in a different design document, adjust the design document name.
MapReduce is a concept that solves problems by applying a two-step process, aptly named the map phase and the reduce phase. The map phase looks at all documents in CouchDB separately one after the other and creates a map result. The map result is an ordered list of key/value pairs. Both key and value can be specified by the user writing the map function. A map function may call the built-in emit(key, value) function 0 to N times per document, creating a row in the map result per invocation.
CouchDB is smart enough to run a map function only once for every document, even on subsequent queries on a view. Only changes to documents or new documents need to be processed anew.
Map functions run in isolation for every document. They can’t modify the document, and they can’t talk to the outside world—they can’t have side effects. This is required so that CouchDB can guarantee correct results without having to recalculate a complete result when only one document gets changed.
The map result looks like this:
{"total_rows":3,"offset":0,"rows":[
{"id":"fc2636bf50556346f1ce46b4bc01fe30","key":"Lena","value":5},
{"id":"1fb2449f9b9d4e466dbfa47ebe675063","key":"Lisa","value":4},
{"id":"8ede09f6f6aeb35d948485624b28f149","key":"Sarah","value":6}
]}
It is a list of rows sorted by the value of key. The id is added automatically and refers back to the document that created this row. The value is the data you’re looking for. For example purposes, it’s the girl’s age.
The map function that produces this result is:
function(doc) {
if(doc.name && doc.age) {
emit(doc.name, doc.age);
}
}
It includes the if statement as a sanity check to ensure that we’re operating on the right fields and calls the emit function with the name and age as the key and value.
How you would do this in SQL:
SELECT field FROM table WHERE value="searchterm"
How you can do this in CouchDB?
Use case: get a result (which can be a record or set of records) associated with a key (“searchterm”).
To look something up quickly, regardless of the storage mechanism, an index is needed. An index is a data structure optimized for quick search and retrieval. CouchDB’s map result is stored in such an index, which happens to be a B+ tree.
To look up a value by “searchterm”, we need to put all values into the key of a view. All we need is a simple map function:
function(doc) {
if(doc.value) {
emit(doc.value, null);
}
}
This creates a list of documents that have a value field sorted by the data in the value field. To find all the records that match “searchterm”, we query the view and specify the search term as a query parameter:
/database/_design/application/_view/viewname?key="searchterm"
Consider the documents from the previous section, and say we’re indexing on the age field of the documents to find all the five-year-olds:
function(doc) {
if(doc.age && doc.name) {
emit(doc.age, doc.name);
}
}
Query:
/ladies/_design/ladies/_view/age?key=5
Result:
{"total_rows":3,"offset":1,"rows":[
{"id":"fc2636bf50556346f1ce46b4bc01fe30","key":5,"value":"Lena"}
]}
Easy.
Note that you have to emit a value. The view result includes the associated document ID in every row. We can use it to look up more data from the document itself. We can also use the ?include_docs=true parameter to have CouchDB fetch the documents individually for us.
How you would do this in SQL:
SELECT field FROM table WHERE value LIKE "searchterm%"
How you can do this in CouchDB?
Use case: find all documents that have a field value that starts with searchterm. For example, say you stored a MIME type (like text/html or image/jpg) for each document and now you want to find all documents that are images according to the MIME type.
The solution is very similar to the previous example: all we need is a map function that is a little more clever than the first one. But first, an example document:
{
"_id": "Hugh Laurie",
"_rev": "1-9fded7deef52ac373119d05435581edf",
"mime-type": "image/jpg",
"description": "some dude"
}
The clue lies in extracting the prefix that we want to search for from our document and putting it into our view index. We use a regular expression to match our prefix:
function(doc) {
if(doc["mime-type"]) {
// from the start (^) match everything that is not a slash ([^\/]+) until
// we find a slash (\/). Slashes needs to be escaped with a backslash (\/)
var prefix = doc["mime-type"].match(/^[^\/]+\//);
if(prefix) {
emit(prefix, null);
}
}
}
We can now query this view with our desired MIME type prefix and not only find all images, but also text, video, and all other formats:
/files/_design/finder/_view/by-mime-type?key="image/"
How you would do this in SQL:
SELECT COUNT(field) FROM table
How you can do this in CouchDB?
Use case: calculate a derived value from your data.
We haven’t explained reduce functions yet. Reduce functions are similar to aggregate functions in SQL. They compute a value over multiple documents.
To explain the mechanics of reduce functions, we’ll create one that doesn’t make a whole lot of sense. But this example is easy to understand. We’ll explore more useful reductions later.
Reduce functions operate on the output of the map function (also called the map result or intermediate result). The reduce function’s job, unsurprisingly, is to reduce the list that the map function produces.
Here’s what our summing reduce function looks like:
function(keys, values) {
var sum = 0;
for(var idx in values) {
sum = sum + values[idx];
}
return sum;
}
Here’s an alternate, more idiomatic JavaScript version:
function(keys, values) {
var sum = 0;
values.forEach(function(element) {
sum = sum + element;
});
return sum;
}
Note
Don’t miss effective builtin reduce functions like _sum and _count
This reduce function takes two arguments: a list of keys and a list of values. For our summing purposes we can ignore the keys-list and consider only the value list. We’re looping over the list and add each item to a running total that we’re returning at the end of the function.
You’ll see one difference between the map and the reduce function. The map function uses emit() to create its result, whereas the reduce function returns a value.
For example, from a list of integer values that specify the age, calculate the sum of all years of life for the news headline, “786 life years present at event.” A little contrived, but very simple and thus good for demonstration purposes. Consider the documents and the map view we used earlier in this chapter.
The reduce function to calculate the total age of all girls is:
function(keys, values) {
return sum(values);
}
Note that, instead of the two earlier versions, we use CouchDB’s predefined sum() function. It does the same thing as the other two, but it is such a common piece of code that CouchDB has it included.
The result for our reduce view now looks like this:
{"rows":[
{"key":null,"value":15}
]}
The total sum of all age fields in all our documents is 15. Just what we wanted. The key member of the result object is null, as we can’t know anymore which documents took part in the creation of the reduced result. We’ll cover more advanced reduce cases later on.
As a rule of thumb, the reduce function should reduce to a single scalar value. That is, an integer; a string; or a small, fixed-size list or object that includes an aggregated value (or values) from the values argument. It should never just return values or similar. CouchDB will give you a warning if you try to use reduce “the wrong way”:
{
"error":"reduce_overflow_error",
"message":"Reduce output must shrink more rapidly: Current output: ..."
}
How you would do this in SQL:
SELECT DISTINCT field FROM table
How you can do this in CouchDB?
Getting unique values is not as easy as adding a keyword. But a reduce view and a special query parameter give us the same result. Let’s say you want a list of tags that your users have tagged themselves with and no duplicates.
First, let’s look at the source documents. We punt on _id and _rev attributes here:
{
"name":"Chris",
"tags":["mustache", "music", "couchdb"]
}
{
"name":"Noah",
"tags":["hypertext", "philosophy", "couchdb"]
}
{
"name":"Jan",
"tags":["drums", "bike", "couchdb"]
}
Next, we need a list of all tags. A map function will do the trick:
function(doc) {
if(doc.name && doc.tags) {
doc.tags.forEach(function(tag) {
emit(tag, null);
});
}
}
The result will look like this:
{"total_rows":9,"offset":0,"rows":[
{"id":"3525ab874bc4965fa3cda7c549e92d30","key":"bike","value":null},
{"id":"3525ab874bc4965fa3cda7c549e92d30","key":"couchdb","value":null},
{"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"couchdb","value":null},
{"id":"da5ea89448a4506925823f4d985aabbd","key":"couchdb","value":null},
{"id":"3525ab874bc4965fa3cda7c549e92d30","key":"drums","value":null},
{"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"hypertext","value":null},
{"id":"da5ea89448a4506925823f4d985aabbd","key":"music","value":null},
{"id":"da5ea89448a4506925823f4d985aabbd","key":"mustache","value":null},
{"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"philosophy","value":null}
]}
As promised, these are all the tags, including duplicates. Since each document gets run through the map function in isolation, it cannot know if the same key has been emitted already. At this stage, we need to live with that. To achieve uniqueness, we need a reduce:
function(keys, values) {
return true;
}
This reduce doesn’t do anything, but it allows us to specify a special query parameter when querying the view:
/dudes/_design/dude-data/_view/tags?group=true
CouchDB replies:
{"rows":[
{"key":"bike","value":true},
{"key":"couchdb","value":true},
{"key":"drums","value":true},
{"key":"hypertext","value":true},
{"key":"music","value":true},
{"key":"mustache","value":true},
{"key":"philosophy","value":true}
]}
In this case, we can ignore the value part because it is always true, but the result includes a list of all our tags and no duplicates!
With a small change we can put the reduce to good use, too. Let’s see how many of the non-unique tags are there for each tag. To calculate the tag frequency, we just use the summing up we already learned about. In the map function, we emit a 1 instead of null:
function(doc) {
if(doc.name && doc.tags) {
doc.tags.forEach(function(tag) {
emit(tag, 1);
});
}
}
In the reduce function, we return the sum of all values:
function(keys, values) {
return sum(values);
}
Now, if we query the view with the ?group=true parameter, we get back the count for each tag:
{"rows":[
{"key":"bike","value":1},
{"key":"couchdb","value":3},
{"key":"drums","value":1},
{"key":"hypertext","value":1},
{"key":"music","value":1},
{"key":"mustache","value":1},
{"key":"philosophy","value":1}
]}
How you would do this in SQL:
UNIQUE KEY(column)
How you can do this in CouchDB?
Use case: your applications require that a certain value exists only once in a database.
This is an easy one: within a CouchDB database, each document must have a unique _id field. If you require unique values in a database, just assign them to a document’s _id field and CouchDB will enforce uniqueness for you.
There’s one caveat, though: in the distributed case, when you are running more than one CouchDB node that accepts write requests, uniqueness can be guaranteed only per node or outside of CouchDB. CouchDB will allow two identical IDs to be written to two different nodes. On replication, CouchDB will detect a conflict and flag the document accordingly.
This recipe explains how to paginate over view results. Pagination is a user interface (UI) pattern that allows the display of a large number of rows (the result set) without loading all the rows into the UI at once. A fixed-size subset, the page, is displayed along with next and previous links or buttons that can move the viewport over the result set to an adjacent page.
We assume you’re familiar with creating and querying documents and views as well as the multiple view query options.
To have some data to work with, we’ll create a list of bands, one document per band:
{ "name":"Biffy Clyro" }
{ "name":"Foo Fighters" }
{ "name":"Tool" }
{ "name":"Nirvana" }
{ "name":"Helmet" }
{ "name":"Tenacious D" }
{ "name":"Future of the Left" }
{ "name":"A Perfect Circle" }
{ "name":"Silverchair" }
{ "name":"Queens of the Stone Age" }
{ "name":"Kerub" }
We need a simple map function that gives us an alphabetical list of band names. This should be easy, but we’re adding extra smarts to filter out “The” and “A” in front of band names to put them into the right position:
function(doc) {
if(doc.name) {
var name = doc.name.replace(/^(A|The) /, "");
emit(name, null);
}
}
The views result is an alphabetical list of band names. Now say we want to display band names five at a time and have a link pointing to the next five names that make up one page, and a link for the previous five, if we’re not on the first page.
We learned how to use the startkey, limit, and skip parameters in earlier chapters. We’ll use these again here. First, let’s have a look at the full result set:
{"total_rows":11,"offset":0,"rows":[
{"id":"a0746072bba60a62b01209f467ca4fe2","key":"Biffy Clyro","value":null},
{"id":"b47d82284969f10cd1b6ea460ad62d00","key":"Foo Fighters","value":null},
{"id":"45ccde324611f86ad4932555dea7fce0","key":"Tenacious D","value":null},
{"id":"d7ab24bb3489a9010c7d1a2087a4a9e4","key":"Future of the Left","value":null},
{"id":"ad2f85ef87f5a9a65db5b3a75a03cd82","key":"Helmet","value":null},
{"id":"a2f31cfa68118a6ae9d35444fcb1a3cf","key":"Nirvana","value":null},
{"id":"67373171d0f626b811bdc34e92e77901","key":"Kerub","value":null},
{"id":"3e1b84630c384f6aef1a5c50a81e4a34","key":"Perfect Circle","value":null},
{"id":"84a371a7b8414237fad1b6aaf68cd16a","key":"Queens of the Stone Age","value":null},
{"id":"dcdaf08242a4be7da1a36e25f4f0b022","key":"Silverchair","value":null},
{"id":"fd590d4ad53771db47b0406054f02243","key":"Tool","value":null}
]}
The mechanics of paging are very simple:
Or in a pseudo-JavaScript snippet:
var result = new Result();
var page = result.getPage();
page.display();
if(result.hasPrev()) {
page.display_link('prev');
}
if(result.hasNext()) {
page.display_link('next');
}
Don’t use this method! We just show it because it might seem natural to use, and you need to know why it is a bad idea. To get the first five rows from the view result, you use the ?limit=5 query parameter:
curl -X GET http://<IP_ADDRESS>:5984/artists/_design/artists/_view/by-name?limit=5
The result:
{"total_rows":11,"offset":0,"rows":[
{"id":"a0746072bba60a62b01209f467ca4fe2","key":"Biffy Clyro","value":null},
{"id":"b47d82284969f10cd1b6ea460ad62d00","key":"Foo Fighters","value":null},
{"id":"45ccde324611f86ad4932555dea7fce0","key":"Tenacious D","value":null},
{"id":"d7ab24bb3489a9010c7d1a2087a4a9e4","key":"Future of the Left","value":null},
{"id":"ad2f85ef87f5a9a65db5b3a75a03cd82","key":"Helmet","value":null}
]}
By comparing the total_rows value to our limit value, we can determine if there are more pages to display. We also know by the offset member that we are on the first page. We can calculate the value for skip= to get the results for the next page:
var rows_per_page = 5;
var page = (offset / rows_per_page) + 1; // == 1
var skip = page * rows_per_page; // == 5 for the first page, 10 for the second ...
So we query CouchDB with:
curl -X GET 'http://<IP_ADDRESS>:5984/artists/_design/artists/_view/by-name?limit=5&skip=5'
Note we have to use ' (single quotes) to escape the & character that is special to the shell we execute curl in.
The result:
{"total_rows":11,"offset":5,"rows":[
{"id":"a2f31cfa68118a6ae9d35444fcb1a3cf","key":"Nirvana","value":null},
{"id":"67373171d0f626b811bdc34e92e77901","key":"Kerub","value":null},
{"id":"3e1b84630c384f6aef1a5c50a81e4a34","key":"Perfect Circle","value":null},
{"id":"84a371a7b8414237fad1b6aaf68cd16a","key":"Queens of the Stone Age",
"value":null},
{"id":"dcdaf08242a4be7da1a36e25f4f0b022","key":"Silverchair","value":null}
]}
Implementing the hasPrev() and hasNext() method is pretty straightforward:
function hasPrev()
{
return page > 1;
}
function hasNext()
{
var last_page = Math.floor(total_rows / rows_per_page) +
(total_rows % rows_per_page);
return page != last_page;
}
This all looks easy and straightforward, but it has one fatal flaw. Remember how view results are generated from the underlying B-tree index: CouchDB jumps to the first row (or the first row that matches startkey, if provided) and reads one row after the other from the index until there are no more rows (or limit or endkey match, if provided).
The skip argument works like this: in addition to going to the first row and starting to read, skip will skip as many rows as specified, but CouchDB will still read from the first row; it just won’t return any values for the skipped rows. If you specify skip=100, CouchDB will read 100 rows and not create output for them. This doesn’t sound too bad, but it is very bad, when you use 1000 or even 10000 as skip values. CouchDB will have to look at a lot of rows unnecessarily.
As a rule of thumb, skip should be used only with single digit values. While it’s possible that there are legitimate use cases where you specify a larger value, they are a good indicator for potential problems with your solution. Finally, for the calculations to work, you need to add a reduce function and make two calls to the view per page to get all the numbering right, and there’s still a potential for error.
The correct solution is not much harder. Instead of slicing the result set into equally sized pages, we look at 10 rows at a time and use startkey to jump to the next 10 rows. We even use skip, but only with the value 1.
Here is how it works:
The trick to finding the next page is pretty simple. Instead of requesting 10 rows for a page, you request 11 rows, but display only 10 and use the values in the 11th row as the startkey for the next page. Populating the link to the previous page is as simple as carrying the current startkey over to the next page. If there’s no previous startkey, we are on the first page. We stop displaying the link to the next page if we get rows_per_page or less rows back. This is called linked list pagination, as we go from page to page, or list item to list item, instead of jumping directly to a pre-computed page. There is one caveat, though. Can you spot it?
CouchDB view keys do not have to be unique; you can have multiple index entries read. What if you have more index entries for a key than rows that should be on a page? startkey jumps to the first row, and you’d be screwed if CouchDB didn’t have an additional parameter for you to use. All view keys with the same value are internally sorted by docid, that is, the ID of the document that created that view row. You can use the startkey_docid and endkey_docid parameters to get subsets of these rows. For pagination, we still don’t need endkey_docid, but startkey_docid is very handy. In addition to startkey and limit, you also use startkey_docid for pagination if, and only if, the extra row you fetch to find the next page has the same key as the current startkey.
It is important to note that the *_docid parameters only work in addition to the *key parameters and are only useful to further narrow down the result set of a view for a single key. They do not work on their own (the one exception being the built-in _all_docs view that already sorts by document ID).
The advantage of this approach is that all the key operations can be performed on the super-fast B-tree index behind the view. Looking up a page doesn’t include scanning through hundreds and thousands of rows unnecessarily.
One drawback of the linked list style pagination is that you can’t pre-compute the rows for a particular page from the page number and the rows per page. Jumping to a specific page doesn’t really work. Our gut reaction, if that concern is raised, is, “Not even Google is doing that!” and we tend to get away with it. Google always pretends on the first page to find 10 more pages of results. Only if you click on the second page (something very few people actually do) might Google display a reduced set of pages. If you page through the results, you get links for the previous and next 10 pages, but no more. Pre-computing the necessary startkey and startkey_docid for 20 pages is a feasible operation and a pragmatic optimization to know the rows for every page in a result set that is potentially tens of thousands of rows long, or more.
If you really do need to jump to a page over the full range of documents (we have seen applications that require that), you can still maintain an integer value index as the view index and take a hybrid approach at solving pagination.
| Author: | Paul Joseph Davis |
|---|---|
| Date: | 2010-09-26 |
| Source: | http://davispj.com/2010/09/26/new-couchdb-externals-api.html |
For a bit of background, CouchDB has had an API for managing external OS processes that are capable of handling HTTP requests for a given URL prefix. These OS processes communicate with CouchDB using JSON over stdio. They’re dead simple to write and provide CouchDB users an easy way to extend CouchDB functionality.
Even though they’re dead simple to write, there are a few issues. The implementation in CouchDB does not provide fancy pooling semantics. The current API is explicitly synchronous which prevents people from writing event driven code in an external handler. In the end, they may be simple, but their simplicity is also quite limiting.
During CouchCamp a few weeks ago I had multiple discussions with various people that wanted to see the _externals API modified in slight ways that weren’t mutually compatible. After having multiple discussions with multiple people we formed a general consensus on what a new API could look like.
So the first idea for improving the _external API was to make CouchDB act as a reverse proxy. This would allow people to write an HTTP server that was as simple or as complicated as they wanted. It will allow people to change their networking configuration more easily and also allow for external processes to be hosted on nodes other than the one running CouchDB. Bottom line, it not only allows us to have similar semantics as _externals, it provides a lot more fringe benefits as well. I’m always a fan of extra awesomeness.
After hitting on the idea of adding a reverse proxy, people quickly pointed out that it would require users to start manually managing their external processes using something like Runit or Supervisord. After some more discussions I ran into people that wanted something like _externals that didn’t handle HTTP requests. After that it was easy to see that adding a second feature that managed OS processes was the way to go.
I spent this weekend implementing both of these features. Both are at the stage of working but requiring more testing. In the case of the HTTP proxy I have no tests because I can’t decide how to test the thing. If you have ideas, I’d sure like to hear them.
 There’s no reason that I can’t have an HTTP client, proxy, and server all hosted in the same process. So that’s what I did. It turns out to be a fairly nice way of configuring matching assertions between the client and the server to test the proxy transmissions.
To configure a proxy handler, edit your local.ini and add a section like such:
[httpd_global_handlers]
_fti = {couch_httpd_proxy, handle_proxy_req, <<"http://<IP_ADDRESS>:5985">>}
This would be approximately what you’d need to do to get CouchDB-Lucene handled through this interface. The URL you use to access a query would be:
A couple things to note here. Anything in the path after the configured proxy name (“_fti” in this case) will be appended to the configured destination URL (“http://<IP_ADDRESS>:5985” in this case). The query string and any associated body will also be proxied transparently.
Also, of note is that there’s nothing that limits on what resources can be proxied. You’re free to choose any destination that the CouchDB node is capable of communicating with.
The second part of the new API gives CouchDB simple OS process management. When CouchDB boots it will start each configured OS daemon. If one of these daemons fails at some point, it will be restarted. If one of these daemons fails too often, CouchDB will stop attempting to start it.
OS daemons are one-to-one. For each daemon, CouchDB will make sure that exactly one instance of it is alive. If you have something where you want multiple processes, you need to either tell CouchDB about each one, or have a main process that forks off the required sub-processes.
To configure an OS daemon, add this to your local.ini:
[os_daemons]
my_daemon = /path/to/command -with args
As an added benefit, because stdio is now free, I implemented a simple API that OS daemons can use to read the configuration of their CouchDB host. This way you can have them store their configuration inside CouchDB’s config system if you desire. Or they can peek at things like the httpd/bind_address and httpd/port that CouchDB is using.
A request for a config section looks like this:
["get", "os_daemons"]\n
And the response:
{"my_daemon": "/path/to/command -with args"}\n
Or to get a specific key:
["get", "os_daemons", "my_daemon"]\n
And the response:
"/path/to/command -with args"\n
All requests and responses are terminated with a newline (indicated by \n).
There’s also an API for adding messages to CouchDB’s logs. Its simply:
["log", $MESG]\n
Where $MESG is any arbitrary JSON. There is no response from this command. As with the config API, the trailing \n represents a newline byte.
The OS daemons react in real time to changes to the configuration system. If you set or delete keys in the os_daemons section, the corresponding daemons will be started or killed as appropriate.
It was suggested that a good first demo would be a Node.js handler. So, I present to you a “Hello, World” Node.js handler. Also, remember that this currently relies on code in my fork on GitHub.
File node-hello-world.js:
var http = require('http');
var sys = require('sys');
// Send a log message to be included in CouchDB's
// log files.
var log = function(mesg) {
console.log(JSON.stringify(["log", mesg]));
}
// The Node.js example HTTP server
var server = http.createServer(function (req, resp) {
resp.writeHead(200, {'Content-Type': 'text/plain'});
resp.end('Hello World\n');
log(req.method + " " + req.url);
})
// We use stdin in a couple ways. First, we
// listen for data that will be the requested
// port information. We also listen for it
// to close which indicates that CouchDB has
// exited and that means its time for us to
// exit as well.
var stdin = process.openStdin();
stdin.on('data', function(d) {
server.listen(parseInt(JSON.parse(d)));
});
stdin.on('end', function () {
process.exit(0);
});
// Send the request for the port to listen on.
console.log(JSON.stringify(["get", "node_hello", "port"]));
File local.ini (Just add these to what you have):
[log]
level = info
[os_daemons]
node_hello = /path/to/node-hello-world.js
[node_hello]
port = 8000
[httpd_global_handlers]
_hello = {couch_httpd_proxy, handle_proxy_req, <<"http://<IP_ADDRESS>:8000">>}
And then start CouchDB and try:
$ curl -v http://<IP_ADDRESS>:5984/_hello
* About to connect() to <IP_ADDRESS> port 5984 (#0)
* Trying <IP_ADDRESS>... connected
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 5984 (#0)
> GET /_hello HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
> Host: <IP_ADDRESS>:5984
> Accept: */*
>
< HTTP/1.1 200
< Transfer-Encoding: chunked
< Server: CouchDB (Erlang/OTP)
< Date: Mon, 27 Sep 2010 01:13:37 GMT
< Content-Type: text/plain
< Connection: keep-alive
<
Hello World
* Connection #0 to host <IP_ADDRESS> left intact
* Closing connection #0
The corresponding CouchDB logs look like:
Apache CouchDB 1.5.0 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [<0.31.0>] Apache CouchDB has started on http://<IP_ADDRESS>:5984/
[info] [<0.105.0>] <IP_ADDRESS> - - 'GET' /_hello 200
[info] [<0.95.0>] Daemon "node-hello" :: GET /
The Query server is an external process that communicates with CouchDB by JSON protocol through stdio interface and processed all design functions calls: views, shows, lists and more.
The default query server is written in JavaScript, running via Mozilla SpiderMonkey. You can use other languages by setting a Query server key in the language property of a design document or the Content-Type header of a temporary view. Design documents that do not specify a language property are assumed to be of type javascript, as are ad hoc queries that are POSTed to _temp_view without a Content-Type header.
The Query Server is an external process that communicates with CouchDB via a JSON protocol over stdio and processes all design functions calls: views, shows, lists, filters, updates and validate_doc_update.
CouchDB communicates with the Query Server process though stdio interface by JSON messages that terminated by newline character. Messages that are sent to the Query Server are always array-typed that could be matched by the pattern [<command>, <*arguments>]\n.
Note
To simplify examples reading we omitted trailing \n character to let Sphinx highlight them well. Also, all examples contain formatted JSON values while real data transfers in compact mode without formatting spaces.
| Command: | reset |
|---|---|
| Arguments: | Query server state (optional) |
| Returns: | true |
This resets the state of the Query Server and makes it forget all previous input.
CouchDB sends:
["reset"]
The Query Server answers:
true
To set up new Query Server state the second argument is used with object data. This argument is used
CouchDB sends:
["reset", {"reduce_limit": true, "timeout": 5000}]
The Query Server answers:
true
| Command: | add_lib |
|---|---|
| Arguments: | CommonJS library object by views/lib path |
| Returns: | true |
Adds CommonJS library to Query Server state for further usage in map functions.
CouchDB sends:
[
"add_lib",
{
"utils": "exports.MAGIC = 42;"
}
]
The Query Server answers:
true
Note
This library shouldn’t have any side effects nor track its own state or you’ll have a lot of happy debugging time if something went wrong. Remember that a complete index rebuild is a heavy operation and this is the only way to fix your mistakes with shared state.
| Command: | add_fun |
|---|---|
| Arguments: | Map function source code. |
| Returns: | true |
When creating or updating a view the Query Server gets sent the view function for evaluation. The Query Server should parse, compile and evaluate the function it receives to make it callable later. If this fails, the Query Server returns an error. CouchDB might store several functions before sending in any actual documents.
CouchDB sends:
[
"add_fun",
"function(doc) { if(doc.score > 50) emit(null, {'player_name': doc.name}); }"
]
The Query Server answers:
true
| Command: | map_doc |
|---|---|
| Arguments: | Document object |
| Returns: | Array of key-value pairs per applied function |
When the view function is stored in the Query Server, CouchDB starts sending in all the documents in the database, one at a time. The Query Server calls the previously stored functions one after another with a document and stores its result. When all functions have been called, the result is returned as a JSON string.
CouchDB sends:
[
"map_doc",
{
"_id": "8877AFF9789988EE",
"_rev": "3-235256484",
"name": "John Smith",
"score": 60
}
]
If the function above is the only function stored, the Query Server answers:
[
[
[null, {"player_name": "John Smith"}]
]
]
That is, an array with the result for every function for the given document.
If a document is to be excluded from the view, the array should be empty.
CouchDB sends:
[
"map_doc",
{
"_id": "9590AEB4585637FE",
"_rev": "1-674684684",
"name": "Jane Parker",
"score": 43
}
]
The Query Server answers:
[[]]
| Command: | reduce |
|---|---|
| Arguments: |
|
| Returns: | Array with pair values: true and another array with reduced result |
If the view has a reduce function defined, CouchDB will enter into the reduce phase. The view server will receive a list of reduce functions and some map results on which it can apply them.
CouchDB sends:
[
"reduce",
[
"function(k, v) { return sum(v); }"
],
[
[[1, "699b524273605d5d3e9d4fd0ff2cb272"], 10],
[[2, "c081d0f69c13d2ce2050d684c7ba2843"], 20],
[[null, "foobar"], 3]
]
]
The Query Server answers:
[
true,
[33]
]
Note that even though the view server receives the map results in the form [[key, id-of-doc], value], the function may receive them in a different form. For example, the JavaScript Query Server applies functions on the list of keys and the list of values.
| Command: | rereduce |
|---|---|
| Arguments: | List of values. |
When building a view, CouchDB will apply the reduce step directly to the output of the map step and the rereduce step to the output of a previous reduce step.
CouchDB will send a list of values, with no keys or document ids, to the rereduce step.
CouchDB sends:
[
"rereduce",
[
"function(k, v, r) { return sum(v); }"
],
[
33,
55,
66
]
]
The Query Server answers:
[
true,
[154]
]
| Command: | ddoc |
|---|---|
| Arguments: | Array of objects.
|
| Returns: |
|
This command acts in two phases: ddoc registration and design function execution.
In the first phase CouchDB sends a full design document content to the Query Server to let it cache it by _id value for further function execution.
To do this, CouchDB sends:
[
"ddoc",
"new",
"_design/temp",
{
"_id": "_design/temp",
"_rev": "8-d7379de23a751dc2a19e5638a7bbc5cc",
"language": "javascript",
"shows": {
"request": "function(doc,req){ return {json: req}; }",
"hello": "function(doc,req){ return {body: 'Hello, ' + (doc || {})._id + '!'}; }"
}
}
]
The Query Server answers:
true
After than this design document is ready to serve next subcommands - that’s the second phase.
Note
Each ddoc subcommand is the root design document key, so they are not actually subcommands, but first elements of the JSON path that may be handled and processed.
The pattern for subcommand execution is common:
["ddoc", <design_doc_id>, [<subcommand>, <funcname>], [<argument1>, <argument2>, ...]]
| Command: | ddoc |
|---|---|
| SubCommand: | shows |
| Arguments: |
|
| Returns: | Array with two elements:
|
Executes show function.
Couchdb sends:
[
"ddoc",
"_design/temp",
[
"shows",
"doc"
],
[
null,
{
"info": {
"db_name": "test",
"doc_count": 8,
"doc_del_count": 0,
"update_seq": 105,
"purge_seq": 0,
"compact_running": false,
"disk_size": 15818856,
"data_size": 1535048,
"instance_start_time": "1359952188595857",
"disk_format_version": 6,
"committed_update_seq": 105
},
"id": null,
"uuid": "169cb4cc82427cc7322cb4463d0021bb",
"method": "GET",
"requested_path": [
"api",
"_design",
"temp",
"_show",
"request"
],
"path": [
"api",
"_design",
"temp",
"_show",
"request"
],
"raw_path": "/api/_design/temp/_show/request",
"query": {},
"headers": {
"Accept": "*/*",
"Host": "localhost:5984",
"User-Agent": "curl/7.26.0"
},
"body": "undefined",
"peer": "<IP_ADDRESS>",
"form": {},
"cookie": {},
"userCtx": {
"db": "api",
"name": null,
"roles": [
"_admin"
]
},
"secObj": {}
}
]
]
The Query Server sends:
[
"resp",
{
"body": "Hello, undefined!"
}
]
| Command: | ddoc |
|---|---|
| SubCommand: | lists |
| Arguments: | |
| Returns: | Array. See below for details. |
Executes list function.
The communication protocol for list functions is a bit complex so let’s use an example for illustration.
Let’s assume that we have view a function that emits id-rev pairs:
function(doc) {
emit(doc._id, doc._rev);
}
And we’d like to emulate _all_docs JSON response with list function. Our first version of the list functions looks like this:
function(head, req){
start({'headers': {'Content-Type': 'application/json'}});
var resp = head;
var rows = [];
while(row=getRow()){
rows.push(row);
}
resp.rows = rows;
return toJSON(resp);
}
The whole communication session during list function execution could be divided on three parts:
Initialization
The first returned object from list function is an array of next structure:
["start", <chunks>, <headers>]
Where <chunks> is an array of text chunks that will be sent to client and <headers> is an object with response HTTP headers.
This message is sent from the Query Server to CouchDB on the start() call which initialize HTTP response to the client:
[
"start",
[],
{
"headers": {
"Content-Type": "application/json"
}
}
]
After this, the list function may start to process view rows.
View Processing
Since view results can be extremely large, it is not wise to pass all its rows in a single command. Instead, CouchDB can send view rows one by one to the Query Server allowing processing view and output generation in a streaming way.
CouchDB sends a special array that carries view row data:
[
"list_row",
{
"id": "0cb42c267fe32d4b56b3500bc503e030",
"key": "0cb42c267fe32d4b56b3500bc503e030",
"value": "1-967a00dff5e02add41819138abb3284d"
}
]
If Query Server has something to return on this, it returns an array with a "chunks" item in the head and an array of data in the tail. Now, for our case it has nothing to return, so the response will be:
[
"chunks",
[]
]
When there is no more view rows to process, CouchDB sends special message, that signs about that there is no more data to send from his side:
["list_end"]
Finalization
The last stage of the communication process is the returning list tail: the last data chunk. After this, processing list function will be completed and client will receive complete response.
For our example the last message will be the next:
[
"end",
[
"{\"total_rows\":2,\"offset\":0,\"rows\":[{\"id\":\"0cb42c267fe32d4b56b3500bc503e030\",\"key\":\"0cb42c267fe32d4b56b3500bc503e030\",\"value\":\"1-967a00dff5e02add41819138abb3284d\"},{\"id\":\"431926a69504bde41851eb3c18a27b1f\",\"key\":\"431926a69504bde41851eb3c18a27b1f\",\"value\":\"1-967a00dff5e02add41819138abb3284d\"}]}"
]
]
There, we had made a big mistake: we had returned out result in a single message from the Query Server. That’s ok when there are only a few rows in the view result, but it’s not acceptable for millions documents and millions view rows
Let’s fix our list function and see the changes in communication:
function(head, req){
start({'headers': {'Content-Type': 'application/json'}});
send('{');
send('"total_rows":' + toJSON(head.total_rows) + ',');
send('"offset":' + toJSON(head.offset) + ',');
send('"rows":[');
if (row=getRow()){
send(toJSON(row));
}
while(row=getRow()){
send(',' + toJSON(row));
}
send(']');
return '}';
}
“Wait, what?” - you’d like to ask. Yes, we’d build JSON response manually by string chunks, but let’s take a look on logs:
[Wed, 24 Jul 2013 05:45:30 GMT] [debug] [<0.19191.1>] OS Process #Port<0.4444> Output :: ["start",["{","\"total_rows\":2,","\"offset\":0,","\"rows\":["],{"headers":{"Content-Type":"application/json"}}]
[Wed, 24 Jul 2013 05:45:30 GMT] [info] [<0.18963.1>] <IP_ADDRESS> - - GET /blog/_design/post/_list/index/all_docs 200
[Wed, 24 Jul 2013 05:45:30 GMT] [debug] [<0.19191.1>] OS Process #Port<0.4444> Input :: ["list_row",{"id":"0cb42c267fe32d4b56b3500bc503e030","key":"0cb42c267fe32d4b56b3500bc503e030","value":"1-967a00dff5e02add41819138abb3284d"}]
[Wed, 24 Jul 2013 05:45:30 GMT] [debug] [<0.19191.1>] OS Process #Port<0.4444> Output :: ["chunks",["{\"id\":\"0cb42c267fe32d4b56b3500bc503e030\",\"key\":\"0cb42c267fe32d4b56b3500bc503e030\",\"value\":\"1-967a00dff5e02add41819138abb3284d\"}"]]
[Wed, 24 Jul 2013 05:45:30 GMT] [debug] [<0.19191.1>] OS Process #Port<0.4444> Input :: ["list_row",{"id":"431926a69504bde41851eb3c18a27b1f","key":"431926a69504bde41851eb3c18a27b1f","value":"1-967a00dff5e02add41819138abb3284d"}]
[Wed, 24 Jul 2013 05:45:30 GMT] [debug] [<0.19191.1>] OS Process #Port<0.4444> Output :: ["chunks",[",{\"id\":\"431926a69504bde41851eb3c18a27b1f\",\"key\":\"431926a69504bde41851eb3c18a27b1f\",\"value\":\"1-967a00dff5e02add41819138abb3284d\"}"]]
[Wed, 24 Jul 2013 05:45:30 GMT] [debug] [<0.19191.1>] OS Process #Port<0.4444> Input :: ["list_end"]
[Wed, 24 Jul 2013 05:45:30 GMT] [debug] [<0.19191.1>] OS Process #Port<0.4444> Output :: ["end",["]","}"]]
Note, that now the Query Server sends response by lightweight chunks and if our communication process was extremely slow, the client will see how response data appears on their screen. Chunk by chunk, without waiting for the complete result, like he have for our previous list function.
| Command: | ddoc |
|---|---|
| SubCommand: | updates |
| Arguments: |
|
| Returns: | Array with there elements:
|
Executes update function.
CouchDB sends:
[
"ddoc",
"_design/id",
[
"updates",
"nothing"
],
[
null,
{
"info": {
"db_name": "test",
"doc_count": 5,
"doc_del_count": 0,
"update_seq": 16,
"purge_seq": 0,
"compact_running": false,
"disk_size": 8044648,
"data_size": 7979601,
"instance_start_time": "1374612186131612",
"disk_format_version": 6,
"committed_update_seq": 16
},
"id": null,
"uuid": "7b695cb34a03df0316c15ab529002e69",
"method": "POST",
"requested_path": [
"test",
"_design",
"1139",
"_update",
"nothing"
],
"path": [
"test",
"_design",
"1139",
"_update",
"nothing"
],
"raw_path": "/test/_design/1139/_update/nothing",
"query": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "identity, gzip, deflate, compress",
"Content-Length": "0",
"Host": "localhost:5984"
},
"body": "",
"peer": "<IP_ADDRESS>",
"form": {},
"cookie": {},
"userCtx": {
"db": "test",
"name": null,
"roles": [
"_admin"
]
},
"secObj": {}
}
]
]
The Query Server answers:
[
"up",
null,
{"body": "document id wasn't provided"}
]
or in case of successful update:
[
"up",
{
"_id": "7b695cb34a03df0316c15ab529002e69",
"hello": "world!"
},
{"body": "document was updated"}
]
| Command: | ddoc |
|---|---|
| SubCommand: | filters |
| Arguments: |
|
| Returns: | Array of two elements:
|
Executes filter function.
CouchDB sends:
[
"ddoc",
"_design/test",
[
"filters",
"random"
],
[
[
{
"_id": "431926a69504bde41851eb3c18a27b1f",
"_rev": "1-967a00dff5e02add41819138abb3284d",
"_revisions": {
"start": 1,
"ids": [
"967a00dff5e02add41819138abb3284d"
]
}
},
{
"_id": "0cb42c267fe32d4b56b3500bc503e030",
"_rev": "1-967a00dff5e02add41819138abb3284d",
"_revisions": {
"start": 1,
"ids": [
"967a00dff5e02add41819138abb3284d"
]
}
}
],
{
"info": {
"db_name": "test",
"doc_count": 5,
"doc_del_count": 0,
"update_seq": 19,
"purge_seq": 0,
"compact_running": false,
"disk_size": 8056936,
"data_size": 7979745,
"instance_start_time": "1374612186131612",
"disk_format_version": 6,
"committed_update_seq": 19
},
"id": null,
"uuid": "7b695cb34a03df0316c15ab529023a81",
"method": "GET",
"requested_path": [
"test",
"_changes?filter=test",
"random"
],
"path": [
"test",
"_changes"
],
"raw_path": "/test/_changes?filter=test/random",
"query": {
"filter": "test/random"
},
"headers": {
"Accept": "application/json",
"Accept-Encoding": "identity, gzip, deflate, compress",
"Content-Length": "0",
"Content-Type": "application/json; charset=utf-8",
"Host": "localhost:5984"
},
"body": "",
"peer": "<IP_ADDRESS>",
"form": {},
"cookie": {},
"userCtx": {
"db": "test",
"name": null,
"roles": [
"_admin"
]
},
"secObj": {}
}
]
]
The Query Server answers:
[
true,
[
true,
false
]
]
| Command: | ddoc |
|---|---|
| SubCommand: | views |
| Arguments: | Array of document objects |
| Returns: | Array of two elements:
|
New in version 1.2.
Executes view function in place of the filter.
Acts in the same way as filters command.
| Command: | ddoc |
|---|---|
| SubCommand: | validate_doc_update |
| Arguments: |
|
| Returns: | 1 |
Executes validation function.
CouchDB send:
[
"ddoc",
"_design/id",
["validate_doc_update"],
[
{
"_id": "docid",
"_rev": "2-e0165f450f6c89dc6b071c075dde3c4d",
"score": 10
},
{
"_id": "docid",
"_rev": "1-9f798c6ad72a406afdbf470b9eea8375",
"score": 4
},
{
"name": "Mike",
"roles": ["player"]
},
{
"admins": {},
"members": []
}
]
]
The Query Server answers:
1
Note
While the only valid response for this command is true to prevent document save the Query Server need to raise an error: forbidden or unauthorized - these errors will be turned into correct HTTP 403 and HTTP 401 responses respectively.
When something went wrong the Query Server is able to inform CouchDB about such a situation by sending special message in response of received command.
Error messages prevent further command execution and return an error description to CouchDB. All errors are logically divided into two groups:
To raise an error, the Query Server have to answer:
["error", "error_name", "reason why"]
The "error_name" helps to classify problems by their type e.g. if it’s "value_error" so probably user have entered wrong data, "not_found" notifies about missed resource and "type_error" definitely says about invalid and non expected input from user.
The "reason why" is the error message that explains why it raised and, if possible, what is needed to do to fix it.
For example, calling Update functions against non existent document could produce next error message:
["error", "not_found", "Update function requires existent document"]
The forbidden error is widely used by Validate document update functions to stop further function processing and prevent on disk store of the new document version. Since this error actually is not an error, but an assertion against user actions, CouchDB doesn’t log it at “error” level, but returns HTTP 403 Forbidden response with error information object.
To raise this error, the Query Server have to answer:
{"forbidden": "reason why"}
The unauthorized error mostly acts like forbidden one, but with the meaning of please authorize first. This small difference helps end users to understand what they can do to solve the problem. CouchDB doesn’t log it at “error” level, but returns HTTP 401 Unauthorized response with error information object.
To raise this error, the Query Server have to answer:
{"unauthorized": "reason why"}
At any time, the Query Server may send some information that will be saved in CouchDB’s log file. This is done by sending a special object with just one field, log, on a separate line:
["log", "some message"]
CouchDB responds nothing, but writes received message into log file:
[Sun, 13 Feb 2009 23:31:30 GMT] [info] [<0.72.0>] Query Server Log Message: some message
These messages are only logged at info level.
Note
While every design function has access to all JavaScript objects, the table below describes appropriate usage cases. For example, you may use emit() in List functions, but getRow() is not permitted during Map functions.
| JS Function | Reasonable to use in design doc functions |
|---|---|
| emit() | Map functions |
| getRow() | List functions |
| JSON | any |
| isArray() | any |
| log() | any |
| provides() | Show functions, List functions |
| registerType() | Show functions, List functions |
| require() | any, except Reduce and rereduce functions |
| send() | List functions |
| start() | List functions |
| sum() | any |
| toJSON() | any |
Each design function executes in a special context of predefined objects, modules and functions:
Emits a key-value pair for further processing by CouchDB after the map function is done.
| Arguments: |
|
|---|
function(doc){
emit(doc._id, doc._rev);
}
Extracts the next row from a related view result.
| Returns: | View result row |
|---|---|
| Return type: | object |
function(head, req){
send('[');
row = getRow();
if (row){
send(toJSON(row));
while(row = getRow()){
send(',');
send(toJSON(row));
}
}
return ']';
}
A helper function to check if the provided value is an Array.
| Arguments: |
|
|---|---|
| Returns: | true if obj is Array-typed, false otherwise |
| Return type: | boolean |
Log a message to the CouchDB log (at the INFO level).
| Arguments: |
|
|---|
function(doc){
log('Procesing doc ' + doc['_id']);
emit(doc['_id'], null);
}
After the map function has run, the following line can be found in CouchDB logs (e.g. at /var/log/couchdb/couch.log):
[Sat, 03 Nov 2012 17:38:02 GMT] [info] [<0.7543.0>] OS Process #Port<0.3289> Log :: Processing doc 8d300b86622d67953d102165dbe99467
Registers callable handler for specified MIME key.
| Arguments: |
|
|---|
Registers list of MIME types by associated key.
| Arguments: |
|
|---|
Predefined mappings (key-array):
Loads CommonJS module by a specified path. The path should not start with a slash.
| Arguments: |
|
|---|---|
| Returns: | Exported statements |
Sends a single string chunk in response.
| Arguments: |
|
|---|
function(head, req){
send('Hello,');
send(' ');
send('Couch');
return !
}
Initiates chunked response. As an option, a custom response object may be sent at this point. For list-functions only!
Note
list functions may set the HTTP response code and headers by calling this function. This function must be called before send(), getRow() or a return statement; otherwise, the query server will implicitly call this function with the empty object ({}).
function(head, req){
start({
"code": 302,
"headers": {
"Location": "http://couchdb.apache.org"
}
});
return "Relax!";
}
Sum arr‘s items.
| Arguments: |
|
|---|---|
| Return type: | number |
Encodes obj to JSON string. This is an alias for the JSON.stringify method.
| Arguments: |
|
|---|---|
| Returns: | JSON string |
Support for CommonJS Modules (introduced in CouchDB 0.11.0) allows you to create modular design functions without the need for duplication of functionality.
Here’s a CommonJS module that checks user permissions:
function user_context(userctx, secobj) {
var is_admin = function() {
return userctx.indexOf('_admin') != -1;
}
return {'is_admin': is_admin}
}
exports['user'] = user_context
Each module has access to additional global variables:
The CommonJS module can be added to a design document, like so:
{
"views": {
"lib": {
"security": "function user_context(userctx, secobj) { ... }"
},
"validate_doc_update": "function(newdoc, olddoc, userctx, secobj) {
user = require('lib/security').user(userctx, secobj);
return user.is_admin();
}"
},
"_id": "_design/test"
}
Modules paths are relative to the design document’s views object, but modules can only be loaded from the object referenced via lib. The lib structure can still be used for view functions as well, by simply storing view functions at e.g. views.lib.map, views.lib.reduce, etc.
Note
The Erlang query server is disabled by default. Read configuration guide about reasons why and how to enable it.
Emits key-value pairs to view indexer process.
fun({Doc}) ->
<<K,_/binary>> = proplists:get_value(<<"_rev">>, Doc, null),
V = proplists:get_value(<<"_id">>, Doc, null),
Emit(<<K>>, V)
end.
Helper to iterate over all rows in a list function.
| Arguments: |
|
|---|
fun(Head, {Req}) ->
Fun = fun({Row}, Acc) ->
Id = couch_util:get_value(<<"id">>, Row),
Send(list_to_binary(io_lib:format("Previous doc id: ~p~n", [Acc]))),
Send(list_to_binary(io_lib:format("Current doc id: ~p~n", [Id]))),
{ok, Id}
end,
FoldRows(Fun, nil),
""
end.
Retrieves the next row from a related view result.
%% FoldRows background implementation.
%% https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=blob;f=src/couchdb/couch_native_process.erl;hb=HEAD#l368
%%
foldrows(GetRow, ProcRow, Acc) ->
case GetRow() of
nil ->
{ok, Acc};
Row ->
case (catch ProcRow(Row, Acc)) of
{ok, Acc2} ->
foldrows(GetRow, ProcRow, Acc2);
{stop, Acc2} ->
{ok, Acc2}
end
end.
| Arguments: |
|
|---|
fun({Doc}) ->
<<K,_/binary>> = proplists:get_value(<<"_rev">>, Doc, null),
V = proplists:get_value(<<"_id">>, Doc, null),
Log(lists:flatten(io_lib:format("Hello from ~s doc!", [V]))),
Emit(<<K>>, V)
end.
After the map function has run, the following line can be found in CouchDB logs (e.g. at /var/log/couchdb/couch.log):
[Sun, 04 Nov 2012 11:33:58 GMT] [info] [<0.9144.2>] Hello from 8d300b86622d67953d102165dbe99467 doc!
Sends a single string Chunk in response.
fun(Head, {Req}) ->
Send("Hello,"),
Send(" "),
Send("Couch"),
"!"
end.
The function above produces the following response:
Hello, Couch!
| Arguments: |
|
|---|
Initialize List functions response. At this point, response code and headers may be defined. For example, this function redirects to the CouchDB web site:
fun(Head, {Req}) ->
Start({[{<<"code">>, 302},
{<<"headers">>, {[
{<<"Location">>, <<"http://couchdb.apache.org">>}]
}}
]}),
"Relax!"
end.
A recent of node.js and npm is required.
Clone the CouchDB repo:
$ git clone http://git-wip-us.apache.org/repos/asf/couchdb.git
$ cd couchdb
Install all dependencies:
couchdb/ $ cd src/fauxton
couchdb/src/fauxton/ $ npm install
Note
To avoid a npm global install add node_modules/.bin to your path:
export PATH=./node_modules/.bin:$PATH
Or just use the wrappers in ./bin/.
Development mode, non minified files:
./bin/grunt couchdebug
Or fully compiled install:
./bin/grunt couchdb
Using the dev server is the easiest way to use Fauxton, specially when developing for it:
grunt dev
Deploy Fauxton to your local CouchDB instance:
./bin/grunt couchapp_deploy
The Fauxton be available by http://localhost:5984/fauxton/_design/fauxton/index.html
Each bit of functionality is its own separate module or addon.
All core modules are stored under app/module and any addons that are optional are under app/addons.
We use backbone.js and Backbone.layoutmanager quite heavily, so best to get an idea how they work. Its best at this point to read through a couple of the modules and addons to get an idea of how they work.
Two good starting points are app/addon/config and app/modules/databases.
Each module must have a base.js file, this is read and compile when Fauxton is deployed.
The resource.js file is usually for your Backbone.Models and Backbone.Collections, view.js for your Backbone.Views.
The routes.js is used to register a url path for your view along with what layout, data, breadcrumbs and api point is required for the view.
Addons allow you to extend Fauxton for a specific use case. Usually, they have the following structure:
+ my_addon/
| ---+ assets [optional]
| \ ---+ less
| \ ---- my_addon.less
| ---+ templates/
| \ ---- my_addon.html - underscore template fragments
| ---- resources.js - models and collections of the addon
| ---- routes.js - URL routing for the addon
| ---- views.js - views that the model provides
We have a grunt-init template that lets you create a skeleton addon, including all the boiler plate code. Run grunt-init tasks/addon and answer the questions it asks to create an addon:
± grunt-init tasks/addon
path.existsSync is now called `fs.existsSync`.
Running "addon" task
Please answer the following:
[?] Add on Name (WickedCool) SuperAddon
[?] Location of add ons (app/addons)
[?] Do you need an assets folder?(for .less) (y/N)
[?] Do you need to make any changes to the above before continuing? (y/N)
Created addon SuperAddon in app/addons
Done, without errors.
Once the addon is created add the name to the settings.json file to get it compiled and added on the next install.
An addon can insert itself into Fauxton in two ways; via a route or via a hook.
An addon will override an existing route should one exist, but in all other ways is just a normal backbone route/view. This is how you would add a whole new feature.
Hooks let you modify/extend an existing feature. They modify a DOM element by selector for a named set of routes, for example:
var Search = new FauxtonAPI.addon();
Search.hooks = {
// Render additional content into the sidebar
"#sidebar-content": {
routes:[
"database/:database/_design/:ddoc/_search/:search",
"database/:database/_design/:ddoc/_view/:view",
"database/:database/_:handler"],
callback: searchSidebar
}
};
return Search;
adds the searchSidebar callback to #sidebar-content for three routes.
First create the addon skeleton:
± bbb addon
path.existsSync is now called `fs.existsSync`.
Running "addon" task
Please answer the following:
[?] Add on Name (WickedCool) Hello
[?] Location of add ons (app/addons)
[?] Do you need to make any changes to the above before continuing? (y/N)
Created addon Hello in app/addons
Done, without errors.
In app/addons/hello/templates/hello.html place:
<h1>Hello!</h1>
Next, we’ll defined a simple view in resources.js (for more complex addons you may want to have a views.js) that renders that template:
define([
"app",
"api"
],
function (app, FauxtonAPI) {
var Resources = {};
Resources.Hello = FauxtonAPI.View.extend({
template: "addons/hello/templates/hello"
});
return Resources;
});
Then define a route in routes.js that the addon is accessible at:
define([
"app",
"api",
"addons/hello/resources"
],
function(app, FauxtonAPI, Resources) {
var helloRoute = function () {
console.log('helloRoute callback yo');
return {
layout: "one_pane",
crumbs: [
{"name": "Hello","link": "_hello"}
],
views: {
"#dashboard-content": new Resources.Hello({})
},
apiUrl: 'hello'
};
};
Routes = {
"_hello": helloRoute
};
return Routes;
});
Then wire it all together in base.js:
define([
"app",
"api",
"addons/hello/routes"
],
function(app, FauxtonAPI, HelloRoutes) {
var Hello = new FauxtonAPI.addon();
console.log('hello from hello');
Hello.initialize = function() {
FauxtonAPI.addHeaderLink({title: "Hello", href: "#_hello"});
};
Hello.Routes = HelloRoutes;
console.log(Hello);
return Hello;
});
Once the code is in place include the add on in your settings.json so that it gets included by the require task. Your addon is included in one of three ways; a local path, a git URL or a name. Named plugins assume the plugin is in the Fauxton base directory, addons with a git URL will be cloned into the application, local paths will be copied. Addons included from a local path will be cleaned out by the clean task, others are left alone.
The components of the API URL path help determine the part of the CouchDB server that is being accessed. The result is the structure of the URL request both identifies and effectively describes the area of the database you are accessing.
As with all URLs, the individual components are separated by a forward slash.
As a general rule, URL components and JSON fields starting with the _ (underscore) character represent a special component or entity within the server or returned object. For example, the URL fragment /_all_dbs gets a list of all of the databases in a CouchDB instance.
This reference is structured according to the URL structure, as below.
The CouchDB API is the primary method of interfacing to a CouchDB instance. Requests are made using HTTP and requests are used to request information from the database, store new data, and perform views and formatting of the information stored within the documents.
Requests to the API can be categorised by the different areas of the CouchDB system that you are accessing, and the HTTP method used to send the request. Different methods imply different operations, for example retrieval of information from the database is typically handled by the GET operation, while updates are handled by either a POST or PUT request. There are some differences between the information that must be supplied for the different methods. For a guide to the basic HTTP methods and request structure, see Request Format and Responses.
For nearly all operations, the submitted data, and the returned data structure, is defined within a JavaScript Object Notation (JSON) object. Basic information on the content and data types for JSON are provided in JSON Basics.
Errors when accessing the CouchDB API are reported using standard HTTP Status Codes. A guide to the generic codes returned by CouchDB are provided in HTTP Status Codes.
When accessing specific areas of the CouchDB API, specific information and examples on the HTTP methods and request, JSON structures, and error codes are provided.
CouchDB supports the following HTTP request methods:
GET
Request the specified item. As with normal HTTP requests, the format of the URL defines what is returned. With CouchDB this can include static items, database documents, and configuration and statistical information. In most cases the information is returned in the form of a JSON document.
HEAD
The HEAD method is used to get the HTTP header of a GET request without the body of the response.
POST
Upload data. Within CouchDB POST is used to set values, including uploading documents, setting document values, and starting certain administration commands.
PUT
Used to put a specified resource. In CouchDB PUT is used to create new objects, including databases, documents, views and design documents.
DELETE
Deletes the specified resource, including documents, views, and design documents.
COPY
A special method that can be used to copy documents and objects.
If you use the an unsupported HTTP request type with a URL that does not support the specified type, a 405 error will be returned, listing the supported HTTP methods. For example:
{
"error":"method_not_allowed",
"reason":"Only GET,HEAD allowed"
}
The CouchDB design document API and the functions when returning HTML (for example as part of a show or list) enables you to include custom HTTP headers through the headers block of the return object.
Because CouchDB uses HTTP for all communication, you need to ensure that the correct HTTP headers are supplied (and processed on retrieval) so that you get the right format and encoding. Different environments and clients will be more or less strict on the effect of these HTTP headers (especially when not present). Where possible you should be as specific as possible.
Content-type
Specifies the content type of the information being supplied within the request. The specification uses MIME type specifications. For the majority of requests this will be JSON (application/json). For some settings the MIME type will be plain text. When uploading attachments it should be the corresponding MIME type for the attachment or binary (application/octet-stream).
The use of the Content-type on a request is highly recommended.
Accept
Specifies the list of accepted data types to be returned by the server (i.e. that are accepted/understandable by the client). The format should be a list of one or more MIME types, separated by colons.
For the majority of requests the definition should be for JSON data (application/json). For attachments you can either specify the MIME type explicitly, or use */* to specify that all file types are supported. If the Accept header is not supplied, then the */* MIME type is assumed (i.e. client accepts all formats).
The use of Accept in queries for CouchDB is not required, but is highly recommended as it helps to ensure that the data returned can be processed by the client.
If you specify a data type using the Accept header, CouchDB will honor the specified type in the Content-type header field returned. For example, if you explicitly request application/json in the Accept of a request, the returned HTTP headers will use the value in the returned Content-type field.
For example, when sending a request without an explicit Accept header, or when specifying */*:
GET /recipes HTTP/1.1
Host: couchdb:5984
Accept: */*
The returned headers are:
Server: CouchDB (Erlang/OTP)
Date: Thu, 13 Jan 2011 13:39:34 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 227
Cache-Control: must-revalidate
Note that the returned content type is text/plain even though the information returned by the request is in JSON format.
Explicitly specifying the Accept header:
GET /recipes HTTP/1.1
Host: couchdb:5984
Accept: application/json
The headers returned include the application/json content type:
Server: CouchDB (Erlang/OTP)
Date: Thu, 13 Jan 2013 13:40:11 GMT
Content-Type: application/json
Content-Length: 227
Cache-Control: must-revalidate
Response headers are returned by the server when sending back content and include a number of different header fields, many of which are standard HTTP response header and have no significance to CouchDB operation. The list of response headers important to CouchDB are listed below.
Content-type
Specifies the MIME type of the returned data. For most request, the returned MIME type is text/plain. All text is encoded in Unicode (UTF-8), and this is explicitly stated in the returned Content-type, as text/plain;charset=utf-8.
Cache-control
The cache control HTTP response header provides a suggestion for client caching mechanisms on how to treat the returned information. CouchDB typically returns the must-revalidate, which indicates that the information should be revalidated if possible. This is used to ensure that the dynamic nature of the content is correctly updated.
Content-length
The length (in bytes) of the returned content.
Etag
The Etag HTTP header field is used to show the revision for a document, or a view.
ETags have been assigned to a map/reduce group (the collection of views in a single design document). Any change to any of the indexes for those views would generate a new ETag for all view URLs in a single design doc, even if that specific view’s results had not changed.
Each _view URL has its own ETag which only gets updated when changes are made to the database that effect that index. If the index for that specific view does not change, that view keeps the original ETag head (therefore sending back 304 Not Modified more often).
The majority of requests and responses to CouchDB use the JavaScript Object Notation (JSON) for formatting the content and structure of the data and responses.
JSON is used because it is the simplest and easiest to use solution for working with data within a web browser, as JSON structures can be evaluated and used as JavaScript objects within the web browser environment. JSON also integrates with the server-side JavaScript used within CouchDB.
JSON supports the same basic types as supported by JavaScript, these are:
Number (either integer or floating-point).
String; this should be enclosed by double-quotes and supports Unicode characters and backslash escaping. For example:
"A String"
Boolean - a true or false value. You can use these strings directly. For example:
{ "value": true}
Array - a list of values enclosed in square brackets. For example:
["one", "two", "three"]
Object - a set of key/value pairs (i.e. an associative array, or hash). The key must be a string, but the value can be any of the supported JSON values. For example:
{
"servings" : 4,
"subtitle" : "Easy to make in advance, and then cook when ready",
"cooktime" : 60,
"title" : "Chicken Coriander"
}
In CouchDB, the JSON object is used to represent a variety of structures, including the main CouchDB document.
Parsing JSON into a JavaScript object is supported through the JSON.parse() function in JavaScript, or through various libraries that will perform the parsing of the content into a JavaScript object for you. Libraries for parsing and generating JSON are available in many languages, including Perl, Python, Ruby, Erlang and others.
Warning
Care should be taken to ensure that your JSON structures are valid, invalid structures will cause CouchDB to return an HTTP status code of 500 (server error).
Developers and users new to computer handling of numbers often encounter suprises when expecting that a number stored in JSON format does not necessarily return as the same number as compared character by character.
Any numbers defined in JSON that contain a decimal point or exponent will be passed through the Erlang VM’s idea of the “double” data type. Any numbers that are used in views will pass through the view server’s idea of a number (the common JavaScript case means even integers pass through a double due to JavaScript’s definition of a number).
Consider this document that we write to CouchDB:
{
"_id":"30b3b38cdbd9e3a587de9b8122000cff",
"number": 1.1
}
Now let’s read that document back from CouchDB:
{
"_id":"30b3b38cdbd9e3a587de9b8122000cff",
"_rev":"1-f065cee7c3fd93aa50f6c97acde93030",
"number":1.1000000000000000888
}
What happens is CouchDB is changing the textual representation of the result of decoding what it was given into some numerical format. In most cases this is an IEEE 754 double precision floating point number which is exactly what almost all other languages use as well.
What Erlang does a bit differently than other languages is that it does not attempt to pretty print the resulting output to use the shortest number of characters. For instance, this is why we have this relationship:
ejson:encode(ejson:decode(<<"1.1">>)).
<<"1.1000000000000000888">>
What can be confusing here is that internally those two formats decode into the same IEEE-754 representation. And more importantly, it will decode into a fairly close representation when passed through all major parsers that we know about.
While we’ve only been discussing cases where the textual representation changes, another important case is when an input value contains more precision than can actually represented in a double. (You could argue that this case is actually “losing” data if you don’t accept that numbers are stored in doubles).
Here’s a log for a couple of the more common JSON libraries that happen to be on the author’s machine:
Spidermonkey:
$ js -h 2>&1 | head -n 1
JavaScript-C 1.8.5 2011-03-31
$ js
js> JSON.stringify(JSON.parse("1.01234567890123456789012345678901234567890"))
"1.0123456789012346"
js> var f = JSON.stringify(JSON.parse("1.01234567890123456789012345678901234567890"))
js> JSON.stringify(JSON.parse(f))
"1.0123456789012346"
Node:
$ node -v
v0.6.15
$ node
JSON.stringify(JSON.parse("1.01234567890123456789012345678901234567890"))
'1.0123456789012346'
var f = JSON.stringify(JSON.parse("1.01234567890123456789012345678901234567890"))
undefined
JSON.stringify(JSON.parse(f))
'1.0123456789012346'
Python:
$ python
Python 2.7.2 (default, Jun 20 2012, 16:23:33)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
import json
json.dumps(json.loads("1.01234567890123456789012345678901234567890"))
'1.0123456789012346'
f = json.dumps(json.loads("1.01234567890123456789012345678901234567890"))
json.dumps(json.loads(f))
'1.0123456789012346'
Ruby:
$ irb --version
irb 0.9.5(05/04/13)
require 'JSON'
=> true
JSON.dump(JSON.load("[1.01234567890123456789012345678901234567890]"))
=> "[1.01234567890123]"
f = JSON.dump(JSON.load("[1.01234567890123456789012345678901234567890]"))
=> "[1.01234567890123]"
JSON.dump(JSON.load(f))
=> "[1.01234567890123]"
Note
A small aside on Ruby, it requires a top level object or array, so I just wrapped the value. Should be obvious it doesn’t affect the result of parsing the number though.
Ejson (CouchDB’s current parser) at CouchDB sha 168a663b:
$ ./utils/run -i
Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2]
[async-threads:4] [hipe] [kernel-poll:true]
Eshell V5.8.5 (abort with ^G)
1> ejson:encode(ejson:decode(<<"1.01234567890123456789012345678901234567890">>)).
<<"1.0123456789012346135">>
2> F = ejson:encode(ejson:decode(<<"1.01234567890123456789012345678901234567890">>)).
<<"1.0123456789012346135">>
3> ejson:encode(ejson:decode(F)).
<<"1.0123456789012346135">>
As you can see they all pretty much behave the same except for Ruby actually does appear to be losing some precision over the other libraries.
The astute observer will notice that ejson (the CouchDB JSON library) reported an extra three digits. While its tempting to think that this is due to some internal difference, its just a more specific case of the 1.1 input as described above.
The important point to realize here is that a double can only hold a finite number of values. What we’re doing here is generating a string that when passed through the “standard” floating point parsing algorithms (ie, strtod) will result in the same bit pattern in memory as we started with. Or, slightly different, the bytes in a JSON serialized number are chosen such that they refer to a single specific value that a double can represent.
The important point to understand is that we’re mapping from one infinite set onto a finite set. An easy way to see this is by reflecting on this:
1.0 == 1.00 == 1.000 = 1.(infinite zeroes)
Obviously a computer can’t hold infinite bytes so we have to decimate our infinitely sized set to a finite set that can be represented concisely.
The game that other JSON libraries are playing is merely:
“How few characters do I have to use to select this specific value for a double”
And that game has lots and lots of subtle details that are difficult to duplicate in C without a significant amount of effort (it took Python over a year to get it sorted with their fancy build systems that automatically run on a number of different architectures).
Hopefully we’ve shown that CouchDB is not doing anything “funky” by changing input. Its behaving the same as any other common JSON library does, its just not pretty printing its output.
On the other hand, if you actually are in a position where an IEEE-754 double is not a satisfactory datatype for your numbers, then the answer as has been stated is to not pass your numbers through this representation. In JSON this is accomplished by encoding them as a string or by using integer types (although integer types can still bite you if you use a platform that has a different integer representation than normal, ie, JavaScript).
Further information can be found easily, including the Floating Point Guide, and David Goldberg’s Reference.
Also, if anyone is really interested in changing this behavior, we’re all ears for contributions to jiffy (which is theoretically going to replace ejson when we get around to updating the build system). The places we’ve looked for inspiration are TCL and Python. If you know a decent implementation of this float printing algorithm give us a holler.
With the interface to CouchDB working through HTTP, error codes and statuses are reported using a combination of the HTTP status code number, and corresponding data in the body of the response data.
A list of the error codes returned by CouchDB, and generic descriptions of the related errors are provided below. The meaning of different status codes for specific request types are provided in the corresponding API call reference.
200 - OK
Request completed successfully.
201 - Created
Document created successfully.
202 - Accepted
Request has been accepted, but the corresponding operation may not have completed. This is used for background operations, such as database compaction.
304 - Not Modified
The additional content requested has not been modified. This is used with the ETag system to identify the version of information returned.
400 - Bad Request
Bad request structure. The error can indicate an error with the request URL, path or headers. Differences in the supplied MD5 hash and content also trigger this error, as this may indicate message corruption.
401 - Unauthorized
The item requested was not available using the supplied authorization, or authorization was not supplied.
403 - Forbidden
The requested item or operation is forbidden.
404 - Not Found
The requested content could not be found. The content will include further information, as a JSON object, if available. The structure will contain two keys, error and reason. For example:
{"error":"not_found","reason":"no_db_file"}
405 - Resource Not Allowed
A request was made using an invalid HTTP request type for the URL requested. For example, you have requested a PUT when a POST is required. Errors of this type can also triggered by invalid URL strings.
406 - Not Acceptable
The requested content type is not supported by the server.
409 - Conflict
Request resulted in an update conflict.
412 - Precondition Failed
The request headers from the client and the capabilities of the server do not match.
415 - Bad Content Type
The content types supported, and the content type of the information being requested or submitted indicate that the content type is not supported.
416 - Requested Range Not Satisfiable
The range specified in the request header cannot be satisfied by the server.
417 - Expectation Failed
When sending documents in bulk, the bulk load operation failed.
500 - Internal Server Error
The request was invalid, either because the supplied JSON was invalid, or invalid information was supplied as part of the request.
The CouchDB server interface provides the basic interface to a CouchDB server for obtaining CouchDB information and getting and setting configuration information.
Accessing the root of a CouchDB instance returns meta information about the instance. The response is a JSON structure containing information about the server, including a welcome message and the version of the server.
| Request Headers: | |
|---|---|
|
|
| Response Headers: | |
|
|
| Status Codes: |
|
Request:
GET / HTTP/1.1
Accept: application/json
Host: localhost:5984
Response:
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 179
Content-Type: application/json
Date: Sat, 10 Aug 2013 06:33:33 GMT
Server: CouchDB (Erlang/OTP)
{
"couchdb": "Welcome",
"uuid": "85fb71bf700c17267fef77535820e371",
"vendor": {
"name": "The Apache Software Foundation",
"version": "1.3.1"
},
"version": "1.3.1"
}
List of running tasks, including the task type, name, status and process ID. The result is a JSON array of the currently running tasks, with each task being described with a single object. Depending on operation type set of response object fields might be different.
| Request Headers: | |
|---|---|
|
|
| Response Headers: | |
|
|
| Response JSON Object: | |
|
|
| Status Codes: |
|
Request:
GET /_active_tasks HTTP/1.1
Accept: application/json
Host: localhost:5984
Response:
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 1690
Content-Type: application/json
Date: Sat, 10 Aug 2013 06:37:31 GMT
Server: CouchDB (Erlang/OTP)
[
{
"changes_done": 64438,
"database": "mailbox",
"pid": "<0.12986.1>",
"progress": 84,
"started_on":<PHONE_NUMBER>,
"total_changes": 76215,
"type": "database_compaction",
"updated_on":<PHONE_NUMBER>
},
{
"changes_done": 14443,
"database": "mailbox",
"design_document": "c9753817b3ba7c674d92361f24f59b9f",
"pid": "<0.10461.3>",
"progress": 18,
"started_on":<PHONE_NUMBER>,
"total_changes": 76215,
"type": "indexer",
"updated_on":<PHONE_NUMBER>
},
{
"changes_done": 5454,
"database": "mailbox",
"design_document": "_design/meta",
"pid": "<0.6838.4>",
"progress": 7,
"started_on":<PHONE_NUMBER>,
"total_changes": 76215,
"type": "indexer",
"updated_on":<PHONE_NUMBER>
},
{
"checkpointed_source_seq": 68585,
"continuous": false,
"doc_id": null,
"doc_write_failures": 0,
"docs_read": 4524,
"docs_written": 4524,
"missing_revisions_found": 4524,
"pid": "<0.1538.5>",
"progress": 44,
"replication_id": "9bc1727d74d49d9e157e260bb8bbd1d5",
"revisions_checked": 4524,
"source": "mailbox",
"source_seq": 154419,
"started_on":<PHONE_NUMBER>,
"target": "http://mailsrv:5984/mailbox",
"type": "replication",
"updated_on":<PHONE_NUMBER>
}
]
Returns a list of all the databases in the CouchDB instance.
| Request Headers: | |
|---|---|
|
|
| Response Headers: | |
|
|
| Status Codes: |
|
Request:
GET /_all_dbs HTTP/1.1
Accept: application/json
Host: localhost:5984
Response:
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 52
Content-Type: application/json
Date: Sat, 10 Aug 2013 06:57:48 GMT
Server: CouchDB (Erlang/OTP)
[
"_users",
"contacts",
"docs",
"invoices",
"locations"
]
New in version 1.4.
Returns a list of all database events in the CouchDB instance.
| Request Headers: | |
|---|---|
|
|
| Query Parameters: | |
|
|
| Response Headers: | |
|
|
| Response JSON Object: | |
|
|
| Status Codes: |
|
Request:
GET /_db_updates HTTP/1.1
Accept: application/json
Host: localhost:5984
Response:
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Type: application/json
Date: Sat, 10 Aug 2013 07:02:41 GMT
Server: CouchDB (Erlang/OTP)
Transfer-Encoding: chunked
{
"db_name": "mailbox",
"ok": true,
"type": "created"
}
Gets the CouchDB log, equivalent to accessing the local log file of the corresponding CouchDB instance.
| Request Headers: | |
|---|---|
|
|
| Query Parameters: | |
|
|
| Response Headers: | |
|
|
| Status Codes: |
|
Request:
GET /_log HTTP/1.1
Accept: application/json
Host: localhost:5984
Response:
[Wed, 27 Oct 2010 10:49:42 GMT] [info] [<0.23338.2>] <IP_ADDRESS> - - 'PUT' /authdb 401
[Wed, 27 Oct 2010 11:02:19 GMT] [info] [<0.23428.2>] <IP_ADDRESS> - - 'GET' /recipes/FishStew 200
[Wed, 27 Oct 2010 11:02:19 GMT] [info] [<0.23428.2>] <IP_ADDRESS> - - 'GET' /_session 200
[Wed, 27 Oct 2010 11:02:19 GMT] [info] [<0.24199.2>] <IP_ADDRESS> - - 'GET' / 200
[Wed, 27 Oct 2010 13:03:38 GMT] [info] [<0.24207.2>] <IP_ADDRESS> - - 'GET' /_log?offset=5 200
If you want to pick out specific parts of the log information you can use the bytes argument, which specifies the number of bytes to be returned, and offset, which specifies where the reading of the log should start, counted back from the end. For example, if you use the following request:
GET /_log?bytes=500&offset=2000
Reading of the log will start at 2000 bytes from the end of the log, and 500 bytes will be shown.
How bytes/offset works?
CouchDB reads specified amount of bytes from the end of log file, jumping to offset bytes towards the beginning of the file first:
Log File FilePos
----------
| | 10
| | 20
| | 30
| | 40
| | 50
| | 60
| | 70 -- Bytes = 20 --
| | 80 | Chunk
| | 90 -- Offset = 10 --
|__________| 100
Request, configure, or stop, a replication operation.
| Request Headers: | |
|---|---|
|
|
| Request JSON Object: | |
|
|
| Response Headers: | |
|
|
| Response JSON Object: | |
|
|
| Status Codes: |
|
The specification of the replication request is controlled through the JSON content of the request. The JSON should be an object with the fields defining the source, target and other options.
The Replication history is an array of objects with following structure:
| JSON Object: |
|
|---|
Request
POST /_replicate HTTP/1.1
Accept: application/json
Content-Length: 36
Content-Type: application/json
Host: localhost:5984
{
"source": "db_a",
"target": "db_b"
}
Response
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 692
Content-Type: application/json
Date: Sun, 11 Aug 2013 20:38:50 GMT
Server: CouchDB (Erlang/OTP)
{
"history": [
{
"doc_write_failures": 0,
"docs_read": 10,
"docs_written": 10,
"end_last_seq": 28,
"end_time": "Sun, 11 Aug 2013 20:38:50 GMT",
"missing_checked": 10,
"missing_found": 10,
"recorded_seq": 28,
"session_id": "142a35854a08e205c47174d91b1f9628",
"start_last_seq": 1,
"start_time": "Sun, 11 Aug 2013 20:38:50 GMT"
},
{
"doc_write_failures": 0,
"docs_read": 1,
"docs_written": 1,
"end_last_seq": 1,
"end_time": "Sat, 10 Aug 2013 15:41:54 GMT",
"missing_checked": 1,
"missing_found": 1,
"recorded_seq": 1,
"session_id": "6314f35c51de3ac408af79d6ee0c1a09",
"start_last_seq": 0,
"start_time": "Sat, 10 Aug 2013 15:41:54 GMT"
}
],
"ok": true,
"replication_id_version": 3,
"session_id": "142a35854a08e205c47174d91b1f9628",
"source_last_seq": 28
}
The aim of the replication is that at the end of the process, all active documents on the source database are also in the destination database and all documents that were deleted in the source databases are also deleted (if they exist) on the destination database.
Replication can be described as either push or pull replication:
Pull replication is where the source is the remote CouchDB instance, and the target is the local database.
Pull replication is the most useful solution to use if your source database has a permanent IP address, and your destination (local) database may have a dynamically assigned IP address (for example, through DHCP). This is particularly important if you are replicating to a mobile or other device from a central server.
Push replication is where the source is a local database, and target is a remote database.
You must use the URL specification of the CouchDB database if you want to perform replication in either of the following two situations:
For example, to request replication between a database local to the CouchDB instance to which you send the request, and a remote database you might use the following request:
POST http://couchdb:5984/_replicate
Content-Type: application/json
Accept: application/json
{
"source" : "recipes",
"target" : "http://coucdb-remote:5984/recipes",
}
In all cases, the requested databases in the source and target specification must exist. If they do not, an error will be returned within the JSON object:
{
"error" : "db_not_found"
"reason" : "could not open http://couchdb-remote:5984/ol1ka/",
}
You can create the target database (providing your user credentials allow it) by adding the create_target field to the request object:
POST http://couchdb:5984/_replicate
Content-Type: application/json
Accept: application/json
{
"create_target" : true
"source" : "recipes",
"target" : "http://couchdb-remote:5984/recipes",
}
The create_target field is not destructive. If the database already exists, the replication proceeds as normal.
You can request replication of a database so that the two databases can be synchronized. By default, the replication process occurs one time and synchronizes the two databases together. For example, you can request a single synchronization between two databases by supplying the source and target fields within the request JSON content.
POST http://couchdb:5984/_replicate
Accept: application/json
Content-Type: application/json
{
"source" : "recipes",
"target" : "recipes-snapshot",
}
In the above example, the databases recipes and recipes-snapshot will be synchronized. These databases are local to the CouchDB instance where the request was made. The response will be a JSON structure containing the success (or failure) of the synchronization process, and statistics about the process:
{
"ok" : true,
"history" : [
{
"docs_read" : 1000,
"session_id" : "52c2370f5027043d286daca4de247db0",
"recorded_seq" : 1000,
"end_last_seq" : 1000,
"doc_write_failures" : 0,
"start_time" : "Thu, 28 Oct 2010 10:24:13 GMT",
"start_last_seq" : 0,
"end_time" : "Thu, 28 Oct 2010 10:24:14 GMT",
"missing_checked" : 0,
"docs_written" : 1000,
"missing_found" : 1000
}
],
"session_id" : "52c2370f5027043d286daca4de247db0",
"source_last_seq" : 1000
}
Synchronization of a database with the previously noted methods happens only once, at the time the replicate request is made. To have the target database permanently replicated from the source, you must set the continuous field of the JSON object within the request to true.
With continuous replication changes in the source database are replicated to the target database in perpetuity until you specifically request that replication ceases.
POST http://couchdb:5984/_replicate
Accept: application/json
Content-Type: application/json
{
"continuous" : true
"source" : "recipes",
"target" : "http://couchdb-remote:5984/recipes",
}
Changes will be replicated between the two databases as long as a network connection is available between the two instances.
Note
Two keep two databases synchronized with each other, you need to set replication in both directions; that is, you must replicate from source to target, and separately from target to source.
You can cancel continuous replication by adding the cancel field to the JSON request object and setting the value to true. Note that the structure of the request must be identical to the original for the cancellation request to be honoured. For example, if you requested continuous replication, the cancellation request must also contain the continuous field.
For example, the replication request:
POST http://couchdb:5984/_replicate
Content-Type: application/json
Accept: application/json
{
"source" : "recipes",
"target" : "http://couchdb-remote:5984/recipes",
"create_target" : true,
"continuous" : true
}
Must be canceled using the request:
POST http://couchdb:5984/_replicate
Accept: application/json
Content-Type: application/json
{
"cancel" : true,
"continuous" : true
"create_target" : true,
"source" : "recipes",
"target" : "http://couchdb-remote:5984/recipes",
}
Requesting cancellation of a replication that does not exist results in a 404 error.
Restarts the CouchDB instance. You must be authenticated as a user with administration privileges for this to work.
| Request Headers: | |
|---|---|
|
|
| Response Headers: | |
|
|
| Status Codes: |
|
Request:
POST /_restart HTTP/1.1
Accept: application/json
Host: localhost:5984
Response:
HTTP/1.1 202 Accepted
Cache-Control: must-revalidate
Content-Length: 12
Content-Type: application/json
Date: Sat, 10 Aug 2013 11:33:50 GMT
Server: CouchDB (Erlang/OTP)
{
"ok": true
}
The _stats resource returns a JSON object containing the statistics for the running server. The object is structured with top-level sections collating the statistics for a range of entries, with each individual statistic being easily identified, and the content of each statistic is self-describing
| Request Headers: | |
|---|---|
|
|
| Response Headers: | |
|
|
| Status Codes: |
|
Request:
GET /_stats/couchdb/request_time HTTP/1.1
Accept: application/json
Host: localhost:5984
Response:
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 187
Content-Type: application/json
Date: Sat, 10 Aug 2013 11:41:11 GMT
Server: CouchDB (Erlang/OTP)
{
"couchdb": {
"request_time": {
"current": 21.0,
"description": "length of a request inside CouchDB without MochiWeb",
"max": 19.0,
"mean": 7.0,
"min": 1.0,
"stddev": 10.392,
"sum": 21.0
}
}
}
The fields provide the current, minimum and maximum, and a collection of statistical means and quantities. The quantity in each case is not defined, but the descriptions below provide
The statistics are divided into the following top-level sections:
Describes statistics specific to the internals of CouchDB
| Statistic ID | Description | Unit |
|---|---|---|
| auth_cache_hits | Number of authentication cache hits | number |
| auth_cache_misses | Number of authentication cache misses | number |
| database_reads | Number of times a document was read from a database | number |
| database_writes | Number of times a database was changed | number |
| open_databases | Number of open databases | number |
| open_os_files | Number of file descriptors CouchDB has open | number |
| request_time | Length of a request inside CouchDB without MochiWeb | milliseconds |
| Statistic ID | Description | Unit |
|---|---|---|
| COPY | Number of HTTP COPY requests | number |
| DELETE | Number of HTTP DELETE requests | number |
| GET | Number of HTTP GET requests | number |
| HEAD | Number of HTTP HEAD requests | number |
| POST | Number of HTTP POST requests | number |
| PUT | Number of HTTP PUT requests | number |
| Statistic ID | Description | Unit |
|---|---|---|
| 200 | Number of HTTP 200 OK responses | number |
| 201 | Number of HTTP 201 Created responses | number |
| 202 | Number of HTTP 202 Accepted responses | number |
| 301 | Number of HTTP 301 Moved Permanently responses | number |
| 304 | Number of HTTP 304 Not Modified responses | number |
| 400 | Number of HTTP 400 Bad Request responses | number |
| 401 | Number of HTTP 401 Unauthorized responses | number |
| 403 | Number of HTTP 403 Forbidden responses | number |
| 404 | Number of HTTP 404 Not Found responses | number |
| 405 | Number of HTTP 405 Method Not Allowed responses | number |
| 409 | Number of HTTP 409 Conflict responses | number |
| 412 | Number of HTTP 412 Precondition Failed responses | number |
| 500 | Number of HTTP 500 Internal Server Error responses | number |
| Statistic ID | Description | Unit |
|---|---|---|
| bulk_requests | Number of bulk requests | number |
| clients_requesting_changes | Number of clients for continuous _changes | number |
| requests | Number of HTTP requests | number |
| temporary_view_reads | Number of temporary view reads | number |
| view_reads | Number of view reads | number |
You can also access individual statistics by quoting the statistics sections and statistic ID as part of the URL path. For example, to get the request_time statistics, you can use:
GET /_stats/couchdb/request_time
This returns an entire statistics object, as with the full request, but containing only the request individual statistic. Hence, the returned structure is as follows:
{
"couchdb" : {
"request_time" : {
"stddev" : 7454.305,
"min" : 1,
"max" : 34185,
"current" : 34697.803,
"mean" : 1652.276,
"sum" : 34697.803,
"description" : "length of a request inside CouchDB without MochiWeb"
}
}
}
Accesses the built-in Futon administration interface for CouchDB.
| Response Headers: | |
|---|---|
|
|
| Status Codes: |
|
| Response Headers: | |
|---|---|
|
|
| Status Codes: |
|
Changed in version 1.5.1.
Requests one or more Universally Unique Identifiers (UUIDs) from the CouchDB instance. The response is a JSON object providing a list of UUIDs.
| Request Headers: | |
|---|---|
|
|
| Query Parameters: | |
|
|
| Response Headers: | |
|
|
| Status Codes: |
|
Request:
GET /_uuids?count=10 HTTP/1.1
Accept: application/json
Host: localhost:5984
Response:
HTTP/1.1 200 OK
Content-Length: 362
Content-Type: application/json
Date: Sat, 10 Aug 2013 11:46:25 GMT
ETag: "DGRWWQFLUDWN5MRKSLKQ425XV"
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: CouchDB (Erlang/OTP)
{
"uuids": [
"75480ca477454894678e22eec6002413",
"75480ca477454894678e22eec600250b",
"75480ca477454894678e22eec6002c41",
"75480ca477454894678e22eec6003b90",
"75480ca477454894678e22eec6003fca",
"75480ca477454894678e22eec6004bef",
"75480ca477454894678e22eec600528f",
"75480ca477454894678e22eec6005e0b",
"75480ca477454894678e22eec6006158",
"75480ca477454894678e22eec6006161"
]
}
The UUID type is determined by the UUID algorithm setting in the CouchDB configuration.
The UUID type may be changed at any time through the Configuration API. For example, the UUID type could be changed to random by sending this HTTP request:
PUT http://couchdb:5984/_config/uuids/algorithm
Content-Type: application/json
Accept: */*
"random"
You can verify the change by obtaining a list of UUIDs:
{
"uuids" : [
"031aad7b469956cf2826fcb2a9260492",
"6ec875e15e6b385120938df18ee8e496",
"cff9e881516483911aa2f0e98949092d",
"b89d37509d39dd712546f9510d4a9271",
"2e0dbf7f6c4ad716f21938a016e4e59f"
]
}
Binary content for the favicon.ico site icon.
| Response Headers: | |
|---|---|
|
|
| Status Codes: |
|
The CouchDB Authentication methods provide an interface for obtaining session and authorization data.
Note
We’re also strongly recommend you to setup SSL to improve all authentication methods security.