pw user mod couchdb -c 'CouchDB, time to relax' -s /usr/sbin/nologin -d /var/lib/couchdb
pw group add couchdb
The user is added to /etc/passwd and should look similar to the following:
shell# grep couchdb /etc/passwd
couchdb:*:1013:1013:Couchdb, time to relax:/var/lib/couchdb/:/usr/sbin/nologin
To change any of these settings, please refrain from editing /etc/passwd and instead use pw user mod ... or vipw. Make sure that the user has no shell, but instead uses /usr/sbin/nologin. The ‘*’ in the second field means that this user can not login via password authorization. For details use man 5 passwd.
By default, CouchDB reads configuration files from the following locations, in the following order:
The LOCALCONFDIR points to the directory that contains configuration files (/usr/local/etc/couchdb by default). This variable may vary from the target operation system and may be changed during building from the source code. For binary distributions, it mostly points to the installation path (e.g. C:\Program Files\CouchDB\etc\couchdb for Windows).
To see the actual configuration files chain run in shell:
couchdb -c
This will print out all actual configuration files that will form the result CouchDB configuration:
/etc/couchdb/default.ini
/etc/couchdb/default.d/geocouch.ini
/etc/couchdb/local.ini
/etc/couchdb/local.d/geocouch.ini
/etc/couchdb/local.d/vendor.ini
Settings in successive documents override the settings in earlier entries. For example, setting the httpd/bind_address parameter in local.ini would override any setting in default.ini.
Warning
The default.ini file may be overwritten during an upgrade or re-installation, so localised changes should be made to the local.ini file or files within the local.d directory.
The configuration files chain may be changed by specifying additional sources by using next command line options:
Let’s add these options and see how the configuration chain changes:
shell> couchdb -c -a /home/couchdb/custom.ini
/etc/couchdb/default.ini
/etc/couchdb/default.d/geocouch.ini
/etc/couchdb/local.ini
/etc/couchdb/local.d/geocouch.ini
/etc/couchdb/local.d/vendor.ini
/home/couchdb/custom.ini
In case when /home/couchdb/custom.ini exists it will be added to the configuration chain.
All parameter names are case-sensitive. Every parameter takes a value of one of five types: boolean, integer, string, tuple and proplist. Boolean values can be written as true or false.
Parameters with value type of tuple or proplist are following the Erlang requirement for style and naming.
The common way to set some parameters is to edit the local.ini file which is mostly located in the etc/couchdb directory relative your installation path root.
For example:
; This is a comment
[section]
param = value ; inline comments are allowed
Each configuration file line may contains section definition, parameter specification, empty (space and newline characters only) or commented line. You can setup inline commentaries for sections or parameters.
The section defines group of parameters that are belongs to some specific CouchDB subsystem. For instance, httpd section holds not only HTTP server parameters, but also others that directly interacts with it.
The parameter specification contains two parts divided by the equal sign (=): the parameter name on the left side and the parameter value on the right one. The leading and following whitespace for = is an optional to improve configuration readability.
Note
In case when you’d like to remove some parameter from the default.ini without his modification, you may specify him within the local.ini, but without any value:
[httpd_global_handlers]
_all_dbs =
This could be read as: “remove the _all_dbs parameter from the httpd_global_handlers section if it was ever set before”.
The semicolon (;) signs about commentary start: everything after this character is counted as commentary and doesn’t process by CouchDB.
After editing of configuration file CouchDB server instance should be restarted to apply these changes.
Alternatively, configuration parameters could be set via the HTTP API. This API allows to change CouchDB configuration on-fly without need for server restart:
curl -X PUT http://localhost:5984/_config/uuids/algorithm -d '"random"'
In the response the old parameter’s value returns:
"sequential"
You should be careful with changing configuration via the HTTP API since it’s easy to make CouchDB unavailable. For instance, you’d like to change the httpd/bind_address for new one:
curl -X PUT http://localhost:5984/_config/httpd/bind_address -d '"<IP_ADDRESS>"'
However, if you would made a typo or the specified IP address is not available from your network, you’ll make CouchDB unavailable for you in both cases and you will have the only way to fix the problem by edit the configuration file and restart the server. To protect yourself against such accidents you may setup the httpd/config_whitelist of configuration parameters that are allowed to edit via the HTTP API. For others you’ll need to directly edit the configuration file so you may quick fix any problems that had occurred due to misconfiguring.
Higher values may result in better read performance due to fewer read operations and/or more OS page cache hits. However, they can also increase overall response time for writes when there are many attachment write requests in parallel.
[couchdb]
attachment_stream_buffer_size = 4096
Specifies location of CouchDB database files (*.couch named). This location should be writable and readable for the user the CouchDB service runs as (couchdb by default).
[couchdb]
database_dir = /var/lib/couchdb
When this config value as false the CouchDB provides guaranty of fsync call before return 201 Created response on each document saving. Setting this config value as true may raise some overall performance with cost of losing durability - it’s strongly not recommended to do such in production:
[couchdb]
delayed_commits = false
Warning
Delayed commits are a feature of CouchDB that allows it to achieve better write performance for some workloads while sacrificing a small amount of durability. The setting causes CouchDB to wait up to a full second before committing new data after an update. If the server crashes before the header is written then any writes since the last commit are lost.
Changed in version 1.2: Added Google Snappy compression algorithm.
Method used to compress everything that is appended to database and view index files, except for attachments (see the attachments section). Available methods are:
[couchdb]
file_compression = snappy
Specifies when to make fsync calls. fsync makes sure that the contents of any file system buffers kept by the operating system are flushed to disk. There is generally no need to modify this parameter.
[couchdb]
fsync_options = [before_header, after_header, on_file_open]
This option places an upper bound on the number of databases that can be open at once. CouchDB reference counts database accesses internally and will close idle databases as needed. Sometimes it is necessary to keep more than the default open at once, such as in deployments where many databases will be replicating continuously.
[couchdb]
max_dbs_open = 100
Changed in version 1.3: This option now actually works.
Defines a maximum size for JSON documents, in bytes. This limit does not apply to attachments, since they are transferred as a stream of chunks. If you set this to a small value, you might be unable to modify configuration options, database security and other larger documents until a larger value is restored by editing the configuration file.
[couchdb]
max_document_size =<PHONE_NUMBER> ; 4 GB
If an external process, such as a query server or external process, runs for this amount of microseconds without returning any results, it will be terminated. Keeping this value smaller ensures you get expedient errors, but you may want to tweak it for your specific needs.
[couchdb]
os_process_timeout = 5000 ; 5 sec
This file contains the full URI that can be used to access this instance of CouchDB. It is used to help discover the port CouchDB is running on (if it was set to 0 (e.g. automatically assigned any free one). This file should be writable and readable for the user that runs the CouchDB service (couchdb by default).
[couchdb]
uri_file = /var/run/couchdb/couchdb.uri
Specifies location of binary drivers (icu, ejson, etc.). This location and its contents should be readable for the user that runs the CouchDB service.
[couchdb]
util_driver_dir = /usr/lib/couchdb/erlang/lib/couch-1.5.0/priv/lib
New in version 1.3.
Unique identifier for this CouchDB server instance.
[couchdb]
uuid = 0a959b9b8227188afc2ac26ccdf345a6
Specifies location of CouchDB view index files. This location should be writable and readable for the user that runs the CouchDB service (couchdb by default).
[couchdb]
view_index_dir = /var/lib/couchdb
The true value of this option enables JSONP support (it’s false by default):
[httpd]
allow_jsonp = false
List of used authentication handlers that used by CouchDB. You may extend them via third-party plugins or remove some of them if you won’t let users to use one of provided methods:
[httpd]
authentication_handlers = {couch_httpd_oauth, oauth_authentication_handler}, {couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler}
Defines the IP address by which CouchDB will be accessible:
[httpd]
bind_address = <IP_ADDRESS>
To let CouchDB listen any available IP address, just setup <IP_ADDRESS> value:
[httpd]
bind_address = <IP_ADDRESS>
For IPv6 support you need to set ::1 if you want to let CouchDB listen local address:
[httpd]
bind_address = ::1
or :: for any available:
[httpd]
bind_address = ::
Specifies default timeout value for Changes Feed in milliseconds (60000 by default):
[httpd]
changes_feed = 60000 ; 60 seconds
Sets the configuration modification whitelist. Only whitelisted values may be changed via the config API. To allow the admin to change this value over HTTP, remember to include {httpd,config_whitelist} itself. Excluding it from the list would require editing this file to update the whitelist:
[httpd]
config_whitelist = [{httpd,config_whitelist}, {log,level}, {etc,etc}]
Specifies default HTTP requests handler:
[httpd]
default_handler = {couch_httpd_db, handle_request}
Defines maximum chunk size in bytes for _log resource:
[httpd]
log_max_chunk_size = 1000000
Defined the port number to listen:
[httpd]
port = 5984
To let CouchDB handle any free port, set this option to 0:
[httpd]
port = 0
After that, CouchDB URI could be located within the URI file.
This option customizes the default function that handles requests to virtual hosts:
[httpd]
redirect_vhost_handler = {Module, Fun}
The specified function take 2 arguments: the Mochiweb request object and the target path.
Server options for the MochiWeb component of CouchDB can be added to the configuration files:
[httpd]
server_options = [{backlog, 128}, {acceptor_pool_size, 16}]
This option allow to isolate databases via subdomains:
[httpd]
secure_rewrites = true
The socket options for the listening socket in CouchDB can be specified as a list of tuples.  A list of the supported options are provided in the Erlang inet documentation.
List of global handlers that are available for virtual hosts:
[httpd]
vhost_global_handlers = _utils, _uuids, _session, _oauth, _users
The x_forwarded_host header (X-Forwarded-Host by default) is used to forward the original value of the Host header field in case, for example, if a reverse proxy is rewriting the “Host” header field to some internal host name before forward the request to CouchDB:
[httpd]
x_forwarded_host = X-Forwarded-Host
This header has higher priority above Host one, if only it exists in the request.
x_forwarded_proto header (X-Forwarder-Proto by default) is used for identifying the originating protocol of an HTTP request, since a reverse proxy may communicate with CouchDB instance using HTTP even if the request to the reverse proxy is HTTPS:
[httpd]
x_forwarded_proto = X-Forwarded-Proto
The x_forwarded_ssl header (X-Forwarded-Ssl by default) tells CouchDB that it should use the https scheme instead of the http. Actually, it’s a synonym for X-Forwarded-Proto: https header, but used by some reverse proxies:
[httpd]
x_forwarded_ssl = X-Forwarded-Ssl
Set this option to trigger basic-auth popup on unauthorized requests:
[httpd]
WWW-Authenticate = Basic realm="Welcome to the Couch!"
CouchDB supports SSL natively. All your secure connection needs can now be served without needing to setup and maintain a separate proxy server that handles SSL.
SSL setup can be tricky, but the configuration in CouchDB was designed to be as easy as possible. All you need is two files; a certificate and a private key. If you bought an official SSL certificate from a certificate authority, both should be in your possession already.
If you just want to try this out and don’t want to pay anything upfront, you can create a self-signed certificate. Everything will work the same, but clients will get a warning about an insecure certificate.
You will need the OpenSSL command line tool installed. It probably already is.
shell> mkdir /etc/couchdb/cert
shell> cd /etc/couchdb/cert
shell> openssl genrsa > privkey.pem
shell> openssl req -new -x509 -key privkey.pem -out couchdb.pem -days 1095
shell> chmod 600 privkey.pem couchdb.pem
shell> chown couchdb privkey.pem couchdb.pem
Now, you need to edit CouchDB’s configuration, either by editing your local.ini file or using the /_config API calls or the configuration screen in Futon. Here is what you need to do in local.ini, you can infer what needs doing in the other places.
At first, enable the HTTPS daemon:
[daemons]
httpsd = {couch_httpd, start_link, [https]}
Next, under [ssl] section setup newly generated certificates:
[ssl]
cert_file = /etc/couchdb/cert/couchdb.pem
key_file = /etc/couchdb/cert/privkey.pem
For more information please read certificates HOWTO.
Now start (or restart) CouchDB. You should be able to connect to it using HTTPS on port 6984:
shell> curl https://<IP_ADDRESS>:6984/
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Oh no! What happened?! Remember, clients will notify their users that your certificate is self signed. curl is the client in this case and it notifies you. Luckily you trust yourself (don’t you?) and you can specify the -k option as the message reads:
shell> curl -k https://<IP_ADDRESS>:6984/
{"couchdb":"Welcome","version":"1.5.0"}
All done.
Path to file containing PEM encoded CA certificates (trusted certificates used for verifying a peer certificate). May be omitted if you do not want to verify the peer:
[ssl]
cacert_file = /etc/ssl/certs/ca-certificates.crt
Path to a file containing the user’s certificate:
[ssl]
cert_file = /etc/couchdb/cert/couchdb.pem
Path to file containing user’s private PEM encoded key:
[ssl]
key_file = /etc/couchdb/cert/privkey.pem
String containing the user’s password. Only used if the private keyfile is password protected:
[ssl]
password = somepassword
Maximum peer certificate depth (must be set even if certificate validation is off):
[ssl]
ssl_certificate_max_depth = 1
The verification fun (optional) if not specified, the default verification fun will be used:
[ssl]
verify_fun = {Module, VerifyFun}
Set to true to validate peer certificates:
[ssl]
verify_ssl_certificates = false
New in version 1.3: added CORS support, see JIRA COUCHDB-431
CORS, or “Cross-Origin Resource Sharing”, allows a resource such as a web page running JavaScript inside a browser, to make AJAX requests (XMLHttpRequests) to a different domain, without compromising the security of either party.
A typical use case is to have a static website hosted on a CDN make requests to another resource, such as a hosted CouchDB instance. This avoids needing an intermediary proxy, using JSONP or similar workarounds to retrieve and host content.
While CouchDB’s integrated HTTP server has support for document attachments makes this less of a constraint for pure CouchDB projects, there are many cases where separating the static content from the database access is desirable, and CORS makes this very straightforward.
By supporting CORS functionality, a CouchDB instance can accept direct connections to protected databases and instances, without the browser functionality being blocked due to same-origin constraints. CORS is supported today on over 90% of recent browsers.
CORS support is provided as experimental functionality in 1.3, and as such will need to be enabled specifically in CouchDB’s configuration. While all origins are forbidden from making requests by default, support is available for simple requests, preflight requests and per-vhost configuration.
This section requires httpd/enable_cors option have true value:
[httpd]
enable_cors = true
By default, neither authentication headers nor cookies are included in requests and responses. To do so requires both setting XmlHttpRequest.withCredentials = true on the request object in the browser and enabling credentials support in CouchDB.
[cors]
credentials = true
CouchDB will respond to a credentials-enabled CORS request with an additional header, Access-Control-Allow-Credentials=true.
List of origins separated by a comma, * means accept all. You can’t set origins = * and credentials = true option at the same time:
[cors]
origins = *
Access can be restricted by protocol, host and optionally by port. Origins must follow the scheme: http://example.com:80:
[cors]
origins = http://localhost, https://localhost, http://couch.mydev.name:8080
Note that by default, no origins are accepted. You must define them explicitly.
List of accepted headers separated by a comma:
[cors]
headers = X-Couch-Id, X-Couch-Rev
List of accepted methods:
[cors]
methods = GET,POST
See also
Original JIRA implementation ticket
Standards and References:
Mozilla Developer Network Resources:
Client-side CORS support and usage:
To set the options for a vhosts, you will need to create a section with the vhost name prefixed by cors:. Example case for the vhost example.com:
[cors:example.com]
credentials = false
; List of origins separated by a comma
origins = *
; List of accepted headers separated by a comma
headers = X-CouchDB-Header
; List of accepted methods
methods = HEAD, GET
CouchDB can map requests to different locations based on the Host header, even if they arrive on the some inbound IP address.
This allows different virtual hosts on the same machine to map to different databases or design documents, etc. The most common use case is to map a virtual host to a Rewrite Handler, to provide full control over the application’s URIs.
To add a virtual host, add a CNAME pointer to the DNS for your domain name. For development and testing, it is sufficient to add an entry in the hosts file, typically /etc/hosts` on Unix-like operating systems:
# CouchDB vhost definitions, refer to local.ini for further details
<IP_ADDRESS> couchdb.local
Test that this is working:
$ ping -n 2 couchdb.local
PING couchdb.local (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from localhost (<IP_ADDRESS>): icmp_req=1 ttl=64 time=0.025 ms
64 bytes from localhost (<IP_ADDRESS>): icmp_req=2 ttl=64 time=0.051 ms
Finally, add an entry to your configuration file in the [vhosts] section:
[vhosts]
couchdb.local:5984 = /example
*.couchdb.local:5984 = /example
If your CouchDB is listening on the the default HTTP port (80), or is sitting behind a proxy, then you don’t need to specify a port number in the vhost key.
The first line will rewrite the request to display the content of the example database. This rule works only if the Host header is couchdb.local and won’t work for CNAMEs. The second rule, on the other hand, matches all CNAMEs to example db, so that both www.couchdb.local and db.couchdb.local will work.
Like in the _rewrite handler you can match some variable and use them to create the target path. Some examples:
[vhosts]
*.couchdb.local = /*
:dbname. = /:dbname
:ddocname.:dbname.example.com = /:dbname/_design/:ddocname/_rewrite
The first rule passes the wildcard as dbname. The second one does the same, but uses a variable name. And the third one allows you to use any URL with ddocname in any database with dbname.
You could also change the default function to handle request by changing the setting httpd/redirect_vhost_handler.
A default CouchDB install provides admin-level access to all connecting users. This configuration is known as Admin Party, and is not recommended for in-production usage. You can crash the party simply by creating the first admin account. CouchDB server administrators and passwords are not stored in the _users database, but in the local.ini file, which should be appropriately secured and readable only by system administrators:
[admins]
;admin = mysecretpassword
admin = -hashed-6d3c30241ba0aaa4e16c6ea99224f915687ed8cd,7f4a3e05e0cbc6f48a0035e3508eef90
architect = -pbkdf2-43ecbd256a70a3a2f7de40d2374b6c3002918834,921a12f74df0c1052b3e562a23cd227f,10000
Administrators can be added directly to the [admins] section, and when CouchDB is restarted, the passwords will be salted and encrypted. You may also use the HTTP interface to create administrator accounts; this way, you don’t need to restart CouchDB, and there’s no need to temporarily store or transmit passwords in plaintext. The HTTP _config/admins endpoint supports querying, deleting or creating new admin accounts:
GET /_config/admins HTTP/1.1
Accept: application/json
Host: localhost:5984
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 196
Content-Type: application/json
Date: Fri, 30 Nov 2012 11:37:18 GMT
Server: CouchDB (Erlang/OTP)
{
"admin": "-hashed-6d3c30241ba0aaa4e16c6ea99224f915687ed8cd,7f4a3e05e0cbc6f48a0035e3508eef90",
"architect": "-pbkdf2-43ecbd256a70a3a2f7de40d2374b6c3002918834,921a12f74df0c1052b3e562a23cd227f,10000"
}
If you already have a salted, encrypted password string (for example, from an old local.ini file, or from a different CouchDB server), then you can store the “raw” encrypted string, without having CouchDB doubly encrypt it.
PUT /_config/admins/architect?raw=true HTTP/1.1
Accept: application/json
Content-Type: application/json
Content-Length: 89
Host: localhost:5984
"-pbkdf2-43ecbd256a70a3a2f7de40d2374b6c3002918834,921a12f74df0c1052b3e562a23cd227f,10000"
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 89
Content-Type: application/json
Date: Fri, 30 Nov 2012 11:39:18 GMT
Server: CouchDB (Erlang/OTP)
"-pbkdf2-43ecbd256a70a3a2f7de40d2374b6c3002918834,921a12f74df0c1052b3e562a23cd227f,10000"
Further details are available in security, including configuring the work factor for PBKDF2, and the algorithm itself at PBKDF2 (RFC-2898).
Changed in version 1.4: PBKDF2 server-side hashed salted password support added, now as a synchronous call for the _config/admins API.
Makes cookies persistent if true.
[couch_httpd_auth]
allow_persistent_cookies = false
Number of User Context Object to cache in memory, to reduce disk lookups.
[couch_httpd_auth]
auth_cache_size = 50
Specifies the name of the system database for storing CouchDB users.
[couch_httpd_auth]
authentication_db = _users
Warning
If you change the database name, do not forget to remove or clean up the old database, since it will no longer be protected by CouchDB.
Specifies the location for redirection on successful authentication if a text/html response is accepted by the client (via an Accept header).
[couch_httpd_auth]
authentication_redirect = /_utils/session.html
New in version 1.3.
The number of iterations for password hashing by the PBKDF2 algorithm. A higher number provides better hash durability, but comes at a cost in performance for each request that requires authentication.
[couch_httpd_auth]
iterations = 10000
When this option is set to true, the couch_httpd_auth/secret option is required for Proxy Authentication.
[couch_httpd_auth]
proxy_use_secret = false
New in version 1.4.
A comma-separated list of field names in user documents (in couch_httpd_auth/authentication_db) that can be read by any user. If unset or not specified, authenticated users can only retrieve their own document.
[couch_httpd_auth]
public_fields = first_name, last_name, contacts, url
Note
Using the public_fields whitelist for user document properties requires setting the couch_httpd_auth/users_db_public option to true (the latter option has no other purpose):
[couch_httpd_auth]
users_db_public = true
When this option is set to true, no requests are allowed from anonymous users. Everyone must be authenticated.
[couch_httpd_auth]
require_valid_user = false
The secret token used for Proxy Authentication method.
[couch_httpd_auth]
secret = 92de07df7e7a3fe14808cef90a7cc0d91
Number of seconds since the last request before sessions will be expired.
[couch_httpd_auth]
timeout = 600
New in version 1.4.
Allow all users to view user documents. By default, only admins may browse all users documents, while users may browse only their own document.
[couch_httpd_auth]
users_db_public = false
The HTTP header name (X-Auth-CouchDB-Roles by default) that contains the list of a user’s roles, separated by a comma. Used for Proxy Authentication.
[couch_httpd_auth]
x_auth_roles = X-Auth-CouchDB-Roles
The HTTP header name (X-Auth-CouchDB-Token by default) containing the token used to authenticate the authorization. This token is an HMAC-SHA1 created from the couch_httpd_auth/secret and couch_httpd_auth/x_auth_username. The secret key should be the same on the client and the CouchDB node. This token is optional if the value of the couch_httpd_auth/proxy_use_secret option is not true. Used for Proxy Authentication.
[couch_httpd_auth]
x_auth_roles = X-Auth-CouchDB-Token
The HTTP header name (X-Auth-CouchDB-UserName by default) containing the username. Used for Proxy Authentication.
[couch_httpd_auth]
x_auth_username = X-Auth-CouchDB-UserName
New in version 1.2.
CouchDB is able to store OAuth credentials within user documents instead of config file by using this option:
[couch_httpd_oauth]
use_users_db = true
If set to true, OAuth token and consumer secrets will be looked up in the authentication database. These secrets are stored in a top level field named "oauth" in user documents, as below.
{
"_id": "org.couchdb.user:joe",
"type": "user",
"name": "joe",
"password_sha": "fe95df1ca59a9b567bdca5cbaf8412abd6e06121",
"salt": "4e170ffeb6f34daecfd814dfb4001a73"
"roles": ["foo", "bar"],
"oauth": {
"consumer_keys": {
"consumerKey1": "key1Secret",
"consumerKey2": "key2Secret"
},
"tokens": {
"token1": "token1Secret",
"token2": "token2Secret"
}
}
}
To let users be authenticated by OAuth Authentication (RFC 5849), three special sections must be set up in the configuration file:
[oauth_consumer_secrets]
consumer1 = sekr1t
[oauth_token_secrets]
token1 = tokensekr1t
[oauth_token_users]
token1 = couchdb_username
Specifies the copy buffer’s maximum size in bytes:
[database_compaction]
doc_buffer_size = 524288
Triggers a checkpoint after the specified amount of bytes were successfully copied to the compacted database:
[database_compaction]
checkpoint_after = 5242880
A list of rules to determine when to run automatic compaction. The daemons/compaction_daemon compacts databases and their respective view groups when all the condition parameters are satisfied. Configuration can be per-database or global, and it has the following format:
[compactions]
database_name = [ {ParamName, ParamValue}, {ParamName, ParamValue}, ... ]
_default = [ {ParamName, ParamValue}, {ParamName, ParamValue}, ... ]
For example:
[compactions]
_default = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]
db_fragmentation: If the ratio of legacy data, including metadata, to current data in the database file size is equal to or greater than this value, this condition is satisfied. The percentage is expressed as an integer percentage. This value is computed as:
(file_size - data_size) / file_size * 100
The data_size and file_size values can be obtained when querying GET /{db}.
view_fragmentation: If the ratio of legacy data, including metadata, to current data in a view index file size is equal to or greater then this value, this database compaction condition is satisfied. The percentage is expressed as an integer percentage. This value is computed as:
(file_size - data_size) / file_size * 100
The data_size and file_size values can be obtained when querying a view group’s information URI.
from and to: The period for which a database (and its view group) compaction is allowed. The value for these parameters must obey the format:
HH:MM - HH:MM (HH in [0..23], MM in [0..59])
strict_window: If a compaction is still running after the end of the allowed period, it will be canceled if this parameter is set to true. It defaults to false and is meaningful only if the period parameter is also specified.
parallel_view_compaction: If set to true, the database and its views are compacted in parallel. This is only useful on certain setups, like for example when the database and view index directories point to different disks. It defaults to false.
Before a compaction is triggered, an estimation of how much free disk space is needed is computed. This estimation corresponds to two times the data size of the database or view index. When there’s not enough free disk space to compact a particular database or view index, a warning message is logged.
Examples:
[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}]
The foo database is compacted if its fragmentation is 70% or more. Any view index of this database is compacted only if its fragmentation is 60% or more.
[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "00:00"}, {to, "04:00"}]
Similar to the preceding example but a compaction (database or view index) is only triggered if the current time is between midnight and 4 AM.
[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "00:00"}, {to, "04:00"}, {strict_window, true}]
Similar to the preceding example - a compaction (database or view index) is only triggered if the current time is between midnight and 4 AM. If at 4 AM the database or one of its views is still compacting, the compaction process will be canceled.
[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "00:00"}, {to, "04:00"}, {strict_window, true}, {parallel_view_compaction, true}]
Similar to the preceding example, but a database and its views can be compacted in parallel.
The delay, in seconds, between each check for which database and view indexes need to be compacted:
[compaction_daemon]
check_interval = 300
If a database or view index file is smaller than this value (in bytes), compaction will not happen. Very small files always have high fragmentation, so compacting them is inefficient.
[compaction_daemon]
min_file_size = 131072
CouchDB logging configuration.
Specifies the location of file for logging output:
[log]
file = /var/log/couchdb/couch.log
This path should be readable and writable for user that runs CouchDB service (couchdb by default).
Changed in version 1.3:: Added warning level.
Logging level defines how verbose and detailed logging will be:
[log]
level = info
Available levels:
New in version 1.3.
In this section you can specify log level on a per-module basis:
[log_level_by_module]
couch_httpd = debug
couch_replicator = info
couch_query_servers = error
See src/*/*.erl for available modules.
New in version 1.2.
Specifies replicator database name:
[replicator]
db = _replicator
Maximum replication retry count can be a non-negative integer or “infinity”
[replicator]
max_replication_retry_count = 10
With lower batch sizes checkpoints are done more frequently. Lower batch sizes also reduce the total amount of used RAM memory:
[replicator]
worker_batch_size = 500
More worker processes can give higher network throughput but can also imply more disk and network IO:
[replicator]
worker_processes = 4
Maximum number of HTTP connections per replication:
[replicator]
http_connections = 20
HTTP connection timeout per replication. Even for very fast/reliable networks it might need to be increased if a remote database is too busy:
[replicator]
connection_timeout = 30000
If a request fails, the replicator will retry it up to N times:
[replicator]
retries_per_request = 10
Some socket options that might boost performance in some scenarios:
See the inet Erlang module’s man page for the full list of options:
[replicator]
socket_options = [{keepalive, true}, {nodelay, false}]
Path to a file containing the user’s certificate:
[replicator]
cert_file = /full/path/to/server_cert.pem
Path to file containing user’s private PEM encoded key:
[replicator]
key_file = /full/path/to/server_key.pem
String containing the user’s password. Only used if the private keyfile is password protected:
[replicator]
password = somepassword
Set to true to validate peer certificates:
[replicator]
verify_ssl_certificates = false
File containing a list of peer trusted certificates (in the PEM format):
[replicator]
ssl_trusted_certificates_file = /etc/ssl/certs/ca-certificates.crt
Maximum peer certificate depth (must be set even if certificate validation is off):
[replicator]
ssl_certificate_max_depth = 3
Changed in version 1.2:: Added CoffeeScript query server
CouchDB delegates computation of design documents functions to external query servers. The external query server is a special OS process which communicates with CouchDB over standard input/output using a very simple line-based protocol with JSON messages.
The external query server may be defined in configuration file following next pattern:
[query_servers]
LANGUAGE = PATH ARGS
Where:
LANGUAGE: is a programming language which code this query server may execute. For instance, there are python, ruby, clojure and other query servers in wild. This value is also used for ddoc field language to determine query server that will process his functions.
Note, that you may setup multiple query servers for the same programming language, but you have to name them different (like python-dev etc.).
PATH: is a system path to the executable binary program that runs the query server.
ARGS: optionally, you may specify additional command line arguments for the executable PATH.
The default query server is written in JavaScript, running via Mozilla SpiderMonkey:
[query_servers]
javascript = /usr/bin/couchjs /usr/share/couchdb/server/main.js
coffeescript = /usr/bin/couchjs /usr/share/couchdb/server/main-coffee.js
See also
Native Erlang Query Server that allows to process Erlang ddocs and runs within CouchDB bypassing stdio communication and JSON serialization/deserialization round trip overhead.
Specifies the delay in seconds before view index changes are committed to disk. The default value is 5:
[query_server_config]
commit_freq = 5
Amount of time in seconds that the Query Server may process CouchDB command:
[query_server_config]
os_process_limit = 10
CouchDB will raise os_process_timeout error and kill the process in case the Query Server doesn’t return any result within this limit.
Controls Reduce overflow error that raises when output of reduce functions is too big:
[query_server_config]
reduce_limit = true
Normally, you don’t have to disable (by setting false value) this option since main propose of reduce functions is to reduce the input.
Warning
Due to security restrictions, the Erlang query server is disabled by default.
Unlike the JavaScript query server, the Erlang one does not runs in a sandbox mode. This means that Erlang code has full access to your OS, filesystem and network, which may lead to security issues. While Erlang functions are faster than JavaScript ones, you need to be careful about running them, especially if they were written by someone else.
CouchDB has a native Erlang query server, allowing you to write your map/reduce functions in Erlang.
First, you’ll need to edit your local.ini to include a [native_query_servers] section:
[native_query_servers]
erlang = {couch_native_process, start_link, []}
To see these changes you will also need to restart the server. To test out using Erlang views, visit the Futon admin interface, create a new database and open a temporary view. You should now be able to select erlang from the language drop-down.
Let’s try an example of map/reduce functions which count the total documents at each number of revisions (there are x many documents at version “1”, and y documents at “2”... etc). Add a few documents to the database, then enter the following functions as a temporary view:
%% Map Function
fun({Doc}) ->
<<K,_/binary>> = proplists:get_value(<<"_rev">>, Doc, null),
V = proplists:get_value(<<"_id">>, Doc, null),
Emit(<<K>>, V)
end.
%% Reduce Function
fun(Keys, Values, ReReduce) -> length(Values) end.
If all has gone well, after running the view you should see a list of the total number of documents at each revision number.
This is a simple feature that allows users to configure CouchDB so that it maintains a given OS level process alive. If the process dies for any reason, CouchDB will restart it. If the process restarts too often, then CouchDB will mark it has halted and not attempt to restart it. The default max restart rate is 3 times in the last 5 seconds. These parameters are adjustable.
Commands that are started in this manner will have access to a simple API over stdio to request configuration parameters or to add log statements to CouchDB’s logs.
To configure an OS process as a CouchDB os_daemon, create a section in your local.ini like such:
[os_daemons]
daemon_name = /path/to/command -with args
This will make CouchDB bring up the command and attempt to keep it alive. To request a configuration parameter, an os_daemon can write a simple JSON message to stdout like such:
["get", "os_daemons"]\n
which would return:
{"daemon_name": "/path/to/command -with args"}\n
Or:
["get", "os_daemons", "daemon_name"]\n
which would return:
"/path/to/command -with args"\n
There’s no restriction on what configuration variables are visible. There’s also no method for altering the configuration.
If you would like your OS daemon to be restarted in the event that the configuration changes, you can send the following messages:
["register", $(SECTION)]\n
When anything in that section changes, your OS process will be rebooted so it can pick up the new configuration settings. If you want to listen for changes on a specific key, you can send something like:
["register", $(SECTION), $(KEY)]\n
In this case, CouchDB will only restart your daemon if that exact section/key pair changes, instead of anything in that entire section.
Logging commands look like:
["log", $(JSON_MESSAGE)]\n
Where $(JSON_MESSAGE) is arbitrary JSON data. These messages are logged at the ‘info’ level. If you want to log at a different level you can pass messages like such:
["log", $(JSON_MESSAGE), {"level": $(LEVEL)}]\n
Where $(LEVEL) is one of “debug”, “info”, or “error”.
When implementing a daemon process to be managed by CouchDB you should remember to use a method like checking the parent process id or if stdin has been closed. These flags can tell you if your daemon process has been orphaned so you can exit cleanly.
There is no interactivity between CouchDB and the running process, but you can use the OS Daemons service to create new HTTP servers and responders and then use the new proxy service to redirect requests and output to the CouchDB managed service. For more information on proxying, see CouchDB As Proxy. For further background on the OS Daemon service, see CouchDB Externals API.
Specifies maximum attempts to run os_daemons before mark them halted:
[os_daemon_settings]
max_retries = 3
Delay in seconds between os_daemons restarts:
[os_daemon_settings]
retry_time = 5
CouchDB is able to spawn OS processes to notify them about recent databases updates. The notifications are in form of JSON messages sent as a line of text, terminated by CR (\n) character, to the OS processes through stdout:
[update_notification]
;unique notifier name=/full/path/to/exe -with "cmd line arg"
index_updater = ruby /usr/local/bin/index_updater.rb
The update notification messages are depend upon of event type:
Database created:
{"type":"created","db":"dbname"}
Database updated: this event raises when any document gets updated for specified database:
{"type":"updated","db":"dbname"}
Design document updated: for design document updates there is special event raised in additional to regular db update one:
{"type":"ddoc_updated","db":"dbname","id":"_design/ddoc_name"}
Database deleted:
{"type":"deleted","db":"dbname"}
Note
New line (\n) trailing character was removed from examples.
These HTTP resources are provided for CouchDB server root level.
[httpd_global_handlers]
/ = {couch_httpd_misc_handlers, handle_welcome_req, <<"Welcome">>}
The favicon handler looks for favicon.ico file within specified directory:
[httpd_global_handlers]
favicon.ico = {couch_httpd_misc_handlers, handle_favicon_req, "/usr/share/couchdb/www"}
[httpd_global_handlers]
_active_tasks = {couch_httpd_misc_handlers, handle_task_status_req}
Provides a list of all server’s databases:
[httpd_global_handlers]
_all_dbs = {couch_httpd_misc_handlers, handle_all_dbs_req}
Note
Sometimes you don’t want to disclose database names for everyone, but you also don’t like/want/able to setup any proxies in front of CouchDB. Removing this handler disables _all_dbs resource and there will be no way to get list of available databases.
The same also is true for other resource handlers.
Provides resource to work with CouchDB config remotely. Any config changes that was made via HTTP API are applied automatically on fly and doesn’t requires server instance to be restarted:
[httpd_global_handlers]
_config = {couch_httpd_misc_handlers, handle_config_req}
[httpd_global_handlers]
_log = {couch_httpd_misc_handlers, handle_log_req}
[httpd_global_handlers]
_oauth = {couch_httpd_oauth, handle_oauth_req}
Provides an API to run temporary replications:
[httpd_global_handlers]
_replicate = {couch_replicator_httpd, handle_req}
[httpd_global_handlers]
_restart = {couch_httpd_misc_handlers, handle_restart_req}
Provides a resource with information about the current user’s session:
[httpd_global_handlers]
_session = {couch_httpd_auth, handle_session_req}
[httpd_global_handlers]
_stats = {couch_httpd_stats_handlers, handle_stats_req}
The _utils handler serves Futon‘s web administration page:
[httpd_global_handlers]
_utils = {couch_httpd_misc_handlers, handle_utils_dir_req, "/usr/share/couchdb/www"}
In similar way, you may setup custom handler to let CouchDB serve any static files.
Provides a resource to get UUIDs generated by CouchDB:
[httpd_global_handlers]
_uuids = {couch_httpd_misc_handlers, handle_uuids_req}
This is useful when your client environment isn’t capable of providing truly random IDs (web browsers e.g.).
These HTTP resources are available on every CouchDB database.
[httpd_db_handlers]
_all_docs = {couch_mrview_http, handle_all_docs_req}
[httpd_db_handlers]
_changes = {couch_httpd_db, handle_changes_req}
[httpd_db_handlers]
_compact = {couch_httpd_db, handle_compact_req}
[httpd_db_handlers]
_design = {couch_httpd_db, handle_design_req}
[httpd_db_handlers]
_temp_view = {couch_mrview_http, handle_temp_view_req}
[httpd_db_handlers]
_view_cleanup = {couch_mrview_http, handle_cleanup_req}
These HTTP resources are provided for design documents.
- _compact¶
[httpd_design_handlers] _compact = {couch_mrview_http, handle_compact_req}
- _info¶
[httpd_design_handlers] _info = {couch_mrview_http, handle_info_req}
- _list¶
[httpd_design_handlers] _list = {couch_mrview_show, handle_view_list_req}
- _rewrite¶
[httpd_design_handlers] _rewrite = {couch_httpd_rewrite, handle_rewrite_req}
- _show¶
[httpd_design_handlers] _show = {couch_mrview_show, handle_doc_show_req}
- _update¶
[httpd_design_handlers] _update = {couch_mrview_show, handle_doc_update_req}
- _view¶
[httpd_design_handlers] _view = {couch_mrview_http, handle_view_req}
This daemon provides authentication caching to avoid repeated opening and closing of the _users database for each request requiring authentication:
[daemons]
auth_cache={couch_auth_cache, start_link, []}
Automatic compaction daemon:
[daemons]
compaction_daemon={couch_compaction_daemon, start_link, []}
External processes manager:
[daemons]
external_manager={couch_external_manager, start_link, []}
HTTP server daemon:
[daemons]
httpd={couch_httpd, start_link, []}
Provides SSL support. The default ssl port CouchDB listens on is 6984:
[daemons]
httpsd = {couch_httpd, start_link, [https]}
The couch_index application is responsible for managing all of the different types of indexers. This manages the process handling for keeping track of the index state as well as managing the updater and compactor handling:
[daemons]
index_server={couch_index_server, start_link, []}
OS Daemons manager:
[daemons]
os_daemons={couch_os_daemons, start_link, []}
Query servers manager:
[daemons]
query_servers={couch_query_servers, start_link, []}
Replications manager:
[daemons]
replicator_manager={couch_replicator_manager, start_link, []}
Runtime statistics aggregator:
[daemons]
stats_aggregator={couch_stats_aggregator, start, []}
Runtime statistics collector:
[daemons]
stats_collector={couch_stats_collector, start, []}
Virtual hosts manager. Provides dynamic add of vhosts without restart, wildcards support and dynamic routing via pattern matching
[daemons]
vhosts={couch_httpd_vhost, start_link, []}
Defines zlib compression level for the attachments from 1 (lowest, fastest) to 9 (highest, slowest). A value of 0 disables compression
[attachments]
compression_level = 8
Since compression is ineffective for some types of files, it is possible to let CouchDB compress only some types of attachments, specified by their MIME type:
[attachments]
compressible_types = text/*, application/javascript, application/json, application/xml
Changed in version 1.3: Added utc_id algorithm.
CouchDB provides various algorithms to generate the UUID values that are used for document _id‘s by default:
[uuids]
algorithm = sequential
Available algorithms:
random: 128 bits of random awesome. All awesome, all the time:
{
"uuids": [
"5fcbbf2cb171b1d5c3bc6df3d4affb32",
"9115e0942372a87a977f1caf30b2ac29",
"3840b51b0b81b46cab99384d5cd106e3",
"b848dbdeb422164babf2705ac18173e1",
"b7a8566af7e0fc02404bb676b47c3bf7",
"a006879afdcae324d70e925c420c860d",
"5f7716ee487cc4083545d4ca02cd45d4",
"35fdd1c8346c22ccc43cc45cd632e6d6",
"97bbdb4a1c7166682dc026e1ac97a64c",
"eb242b506a6ae330bda6969bb2677079"
]
}
sequential: Monotonically increasing ids with random increments. The first 26 hex characters are random, the last 6 increment in random amounts until an overflow occurs. On overflow, the random prefix is regenerated and the process starts over.
{
"uuids": [
"4e17c12963f4bee0e6ec90da54804894",
"4e17c12963f4bee0e6ec90da5480512f",
"4e17c12963f4bee0e6ec90da54805c25",
"4e17c12963f4bee0e6ec90da54806ba1",
"4e17c12963f4bee0e6ec90da548072b3",
"4e17c12963f4bee0e6ec90da54807609",
"4e17c12963f4bee0e6ec90da54807718",
"4e17c12963f4bee0e6ec90da54807754",
"4e17c12963f4bee0e6ec90da54807e5d",
"4e17c12963f4bee0e6ec90da54808d28"
]
}
utc_random: The time since Jan 1, 1970 UTC, in microseconds. The first 14 characters are the time in hex. The last 18 are random.
{
"uuids": [
"04dd32b3af699659b6db9486a9c58c62",
"04dd32b3af69bb1c2ac7ebfee0a50d88",
"04dd32b3af69d8591b99a8e86a76e0fb",
"04dd32b3af69f4a18a76efd89867f4f4",
"04dd32b3af6a1f7925001274bbfde952",
"04dd32b3af6a3fe8ea9b120ed906a57f",
"04dd32b3af6a5b5c518809d3d4b76654",
"04dd32b3af6a78f6ab32f1e928593c73",
"04dd32b3af6a99916c665d6bbf857475",
"04dd32b3af6ab558dd3f2c0afacb7d66"
]
}
utc_id: The time since Jan 1, 1970 UTC, in microseconds, plus the utc_id_suffix string. The first 14 characters are the time in hex. The uuids/utc_id_suffix string value is appended to these.
{
"uuids": [
"04dd32bd5eabcc@mycouch",
"04dd32bd5eabee@mycouch",
"04dd32bd5eac05@mycouch",
"04dd32bd5eac28@mycouch",
"04dd32bd5eac43@mycouch",
"04dd32bd5eac58@mycouch",
"04dd32bd5eac6e@mycouch",
"04dd32bd5eac84@mycouch",
"04dd32bd5eac98@mycouch",
"04dd32bd5eacad@mycouch"
]
}
Note
Impact of UUID choices: the choice of UUID has a significant impact on the layout of the B-tree, prior to compaction.
For example, using a sequential UUID algorithm while uploading a large batch of documents will avoid the need to rewrite many intermediate B-tree nodes. A random UUID algorithm may require rewriting intermediate nodes on a regular basis, resulting in significantly decreased throughput and wasted disk space space due to the append-only B-tree design.
It is generally recommended to set your own UUIDs, or use the sequential algorithm unless you have a specific need and take into account the likely need for compaction to re-balance the B-tree and reclaim wasted space.
New in version 1.3.
The utc_id_suffix value will be appended to UUIDs generated by the utc_id algorithm. Replicating instances should have unique utc_id_suffix values to ensure uniqueness of utc_id ids.
[uuid]
utc_id_suffix = my-awesome-suffix
New in version 1.5.1.
No more than this number of UUIDs will be sent in a single request. If more UUIDs are requested, an HTTP error response will be thrown.
[uuid]
max_count = 1000
The HTTP proxy feature makes it easy to map and redirect different content through your CouchDB URL. The proxy works by mapping a pathname and passing all content after that prefix through to the configured proxy address.
Configuration of the proxy redirect is handled through the [httpd_global_handlers] section of the CouchDB configuration file (typically local.ini). The format is:
[httpd_global_handlers]
PREFIX = {couch_httpd_proxy, handle_proxy_req, <<"DESTINATION">>}
Where:
PREFIX
Is the string that will be matched. The string can be any valid qualifier, although to ensure that existing database names are not overridden by a proxy configuration, you can use an underscore prefix.
DESTINATION
The fully-qualified URL to which the request should be sent. The destination must include the http prefix. The content is used verbatim in the original request, so you can also forward to servers on different ports and to specific paths on the target host.
The proxy process then translates requests of the form:
http://couchdb:5984/PREFIX/path
To:
DESTINATION/path
Note
Everything after PREFIX including the required forward slash will be appended to the DESTINATION.
The response is then communicated back to the original client.
For example, the following configuration:
[httpd_global_handlers]
_google = {couch_httpd_proxy, handle_proxy_req, <<"http://www.google.com">>}
Would forward all requests for http://couchdb:5984/_google to the Google website.
The service can also be used to forward to related CouchDB services, such as Lucene:
[httpd_global_handlers]
_fti = {couch_httpd_proxy, handle_proxy_req, <<"http://<IP_ADDRESS>:5985">>}
Note
The proxy service is basic. If the request is not identified by the DESTINATION, or the remainder of the PATH specification is incomplete, the original request URL is interpreted as if the PREFIX component of that URL does not exist.
For example, requesting http://couchdb:5984/_intranet/media when /media on the proxy destination does not exist, will cause the request URL to be interpreted as http://couchdb:5984/media. Care should be taken to ensure that both requested URLs and destination URLs are able to cope.
The replication is an incremental one way process involving two databases (a source and a destination).
The aim of the replication is that at the end of the process, all active documents on the source database are also in the destination database and all documents that were deleted in the source databases are also deleted (if exists) on the destination database.
The replication process only copies the last revision of a document, so all previous revisions that were only on the source database are not copied to the destination database.
One of CouchDB’s strengths is the ability to synchronize two copies of the same database. This enables users to distribute data across several nodes or datacenters, but also to move data more closely to clients.
Replication involves a source and a destination database, which can be one the same or on different CouchDB instances. The aim of the replication is that at the end of the process, all active documents on the source database are also in the destination database and all documents that were deleted in the source databases are also deleted on the destination database (if they even existed).
Replication is controlled through documents in the Replicator Database, where each document describes one replication process (see Replication Settings).
A replication is triggered by storing a replication document in the replicator database. Its status can be inspected through the active tasks API (see /_active_tasks and Replication Status). A replication can be stopped by deleting the document, or by updating it with its cancel property set to true.
During replication, CouchDB will compare the source and the destination database to determine which documents differ between the source and the destination database. It does so by following the Changes Feeds on the source and comparing the documents to the destination. Changes are submitted to the destination in batches where they can introduce conflicts. Documents that already exist on the destination in the same revision are not transferred. As the deletion of documents is represented by a new revision, a document deleted on the source will also be deleted on the target.
A replication task will finish once it reaches the end of the changes feed. If its continuous property is set to true, it will wait for new changes to appear until the task is cancelled. Replication tasks also create checkpoint documents on the destination to ensure that a restarted task can continue from where it stopped, for example after it has crashed.
When a replication task is initiated on the sending node, it is called push replication, if it is initiated by the receiving node, it is called pull replication.
One replication task will only transfer changes in one direction. To achieve master-master replication it is possible to set up two replication tasks in different directions. When a change is replication from database A to B by the first task, the second will discover that the new change on B already exists in A and will wait for further changes.
There are two ways for controlling which documents are replicated, and which are skipped. Local documents are never replicated (see Local (non-replicating) Document Methods).
Additionally, Filter functions can be used in a replication documents (see Replication Settings). The replication task will then evaluate the filter function for each document in the changes feed. The document will only be replicated if the filter returns true.
Replication can be especially useful for bringing data closer to clients. PouchDB implements the replication algorithm of CouchDB in JavaScript, making it possible to make data from a CouchDB database available in an offline browser application, and synchronize changes back to CouchDB.
The CouchDB Replication protocol is a protocol for synchronizing documents between 2 peers over HTTP/1.1.
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.
The CouchDB Replication protocol is a synchronization protocol for synchronizing documents between 2 peers over HTTP/1.1.
In theory the CouchDB protocol can be used between products that implement it. However the reference implementation, written in Erlang, is provided by the couch_replicator module available in Apache CouchDB.
The CouchDB replication protocol is using the CouchDB REST API and so is based on HTTP and the Apache CouchDB MVC Data model. The primary goal of this specification is to describe the CouchDB replication algorithm.
Note
This step can be done continuously using the feed=longpoll or feed=continuous parameters. Then the feed will continuously get the changes.
Note
The replication can be filtered by passing the filter parameter to the changes feeds with a function name. This will call a function on each changes. If this function return True, the document will be added to the feed.
A database where you PUT/POST documents to trigger replications and you DELETE to cancel ongoing replications. These documents have exactly the same content as the JSON objects we used to POST to _replicate (fields source, target, create_target, continuous, doc_ids, filter, query_params.
Replication documents can have a user defined _id. Design documents (and _local documents) added to the replicator database are ignored.
The default name of this database is _replicator. The name can be changed in the local.ini configuration, section [replicator], parameter db.
Let’s say you PUT the following document into _replicator:
{
"_id": "my_rep",
"source": "http://myserver.com:5984/foo",
"target": "bar",
"create_target": true
}
In the couch log you’ll see 2 entries like these:
[Thu, 17 Feb 2011 19:43:59 GMT] [info] [<0.291.0>] Document `my_rep` triggered replication `c0ebe9256695ff083347cbf95f93e280+create_target`
[Thu, 17 Feb 2011 19:44:37 GMT] [info] [<0.124.0>] Replication `c0ebe9256695ff083347cbf95f93e280+create_target` finished (triggered by document `my_rep`)
As soon as the replication is triggered, the document will be updated by CouchDB with 3 new fields:
{
"_id": "my_rep",
"source": "http://myserver.com:5984/foo",
"target": "bar",
"create_target": true,
"_replication_id": "c0ebe9256695ff083347cbf95f93e280",
"_replication_state": "triggered",
"_replication_state_time":<PHONE_NUMBER>
}
Special fields set by the replicator start with the prefix _replication_.
_replication_id
The ID internally assigned to the replication. This is also the ID exposed by /_active_tasks.
_replication_state
The current state of the replication.
_replication_state_time
A Unix timestamp (number of seconds since 1 Jan 1970) that tells us when the current replication state (marked in _replication_state) was set.
When the replication finishes, it will update the _replication_state field (and _replication_state_time) with the value completed, so the document will look like:
{
"_id": "my_rep",
"source": "http://myserver.com:5984/foo",
"target": "bar",
"create_target": true,
"_replication_id": "c0ebe9256695ff083347cbf95f93e280",
"_replication_state": "completed",
"_replication_state_time":<PHONE_NUMBER>
}
When an error happens during replication, the _replication_state field is set to error (and _replication_state_time gets updated of course).
When you PUT/POST a document to the _replicator database, CouchDB will attempt to start the replication up to 10 times (configurable under [replicator], parameter max_replication_retry_count). If it fails on the first attempt, it waits 5 seconds before doing a second attempt. If the second attempt fails, it waits 10 seconds before doing a third attempt. If the third attempt fails, it waits 20 seconds before doing a fourth attempt (each attempt doubles the previous wait period). When an attempt fails, the Couch log will show you something like:
[error] [<0.149.0>] Error starting replication `67c1bb92010e7abe35d7d629635f18b6+create_target` (document `my_rep_2`): {db_not_found,<<"could not open http://myserver:5986/foo/">>
Note
The _replication_state field is only set to error when all the attempts were unsuccessful.
There are only 3 possible values for the _replication_state field: triggered, completed and error. Continuous replications never get their state set to completed.
Lets suppose 2 documents are added to the _replicator database in the following order:
{
"_id": "doc_A",
"source": "http://myserver.com:5984/foo",
"target": "bar"
}
and
{
"_id": "doc_B",
"source": "http://myserver.com:5984/foo",
"target": "bar"
}
Both describe exactly the same replication (only their _ids differ). In this case document doc_A triggers the replication, getting updated by CouchDB with the fields _replication_state, _replication_state_time and _replication_id, just like it was described before. Document doc_B however, is only updated with one field, the _replication_id so it will look like this:
{
"_id": "doc_B",
"source": "http://myserver.com:5984/foo",
"target": "bar",
"_replication_id": "c0ebe9256695ff083347cbf95f93e280"
}
While document doc_A will look like this:
{
"_id": "doc_A",
"source": "http://myserver.com:5984/foo",
"target": "bar",
"_replication_id": "c0ebe9256695ff083347cbf95f93e280",
"_replication_state": "triggered",
"_replication_state_time":<PHONE_NUMBER>
}
Note that both document get exactly the same value for the _replication_id field. This way you can identify which documents refer to the same replication - you can for example define a view which maps replication IDs to document IDs.
To cancel a replication simply DELETE the document which triggered the replication. The Couch log will show you an entry like the following:
[Thu, 17 Feb 2011 20:16:29 GMT] [info] [<0.125.0>] Stopped replication `c0ebe9256695ff083347cbf95f93e280+continuous+create_target` because replication document `doc_A` was deleted
Note
You need to DELETE the document that triggered the replication. DELETE-ing another document that describes the same replication but did not trigger it, will not cancel the replication.
When CouchDB is restarted, it checks its _replicator database and restarts any replication that is described by a document that either has its _replication_state field set to triggered or it doesn’t have yet the _replication_state field set.
Note
Continuous replications always have a _replication_state field with the value triggered, therefore they’re always restarted when CouchDB is restarted.
Imagine your replicator database (default name is _replicator) has the two following documents that represent pull replications from servers A and B:
{
"_id": "rep_from_A",
"source": "http://aserver.com:5984/foo",
"target": "foo_a",
"continuous": true,
"_replication_id": "c0ebe9256695ff083347cbf95f93e280",
"_replication_state": "triggered",
"_replication_state_time":<PHONE_NUMBER>
}
{
"_id": "rep_from_B",
"source": "http://bserver.com:5984/foo",
"target": "foo_b",
"continuous": true,
"_replication_id": "231bb3cf9d48314eaa8d48a9170570d1",
"_replication_state": "triggered",
"_replication_state_time":<PHONE_NUMBER>
}
Now without stopping and restarting CouchDB, you change the name of the replicator database to another_replicator_db:
$ curl -X PUT http://localhost:5984/_config/replicator/db -d '"another_replicator_db"'
"_replicator"
As soon as this is done, both pull replications defined before, are stopped. This is explicitly mentioned in CouchDB’s log:
[Fri, 11 Mar 2011 07:44:20 GMT] [info] [<0.104.0>] Stopping all ongoing replications because the replicator database was deleted or changed
[Fri, 11 Mar 2011 07:44:20 GMT] [info] [<0.127.0>] <IP_ADDRESS> - - PUT /_config/replicator/db 200
Imagine now you add a replication document to the new replicator database named another_replicator_db:
{
"_id": "rep_from_X",
"source": "http://xserver.com:5984/foo",
"target": "foo_x",
"continuous": true
}
From now own you have a single replication going on in your system: a pull replication pulling from server X. Now you change back the replicator database to the original one _replicator:
$ curl -X PUT http://localhost:5984/_config/replicator/db -d '"_replicator"'
"another_replicator_db"
Immediately after this operation, the replication pulling from server X will be stopped and the replications defined in the _replicator database (pulling from servers A and B) will be resumed.
Changing again the replicator database to another_replicator_db will stop the pull replications pulling from servers A and B, and resume the pull replication pulling from server X.
Imagine you have in server C a replicator database with the two following pull replication documents in it:
{
"_id": "rep_from_A",
"source": "http://aserver.com:5984/foo",
"target": "foo_a",
"continuous": true,
"_replication_id": "c0ebe9256695ff083347cbf95f93e280",
"_replication_state": "triggered",
"_replication_state_time":<PHONE_NUMBER>
}
{
"_id": "rep_from_B",
"source": "http://bserver.com:5984/foo",
"target": "foo_b",
"continuous": true,
"_replication_id": "231bb3cf9d48314eaa8d48a9170570d1",
"_replication_state": "triggered",
"_replication_state_time":<PHONE_NUMBER>
}
Now you would like to have the same pull replications going on in server D, that is, you would like to have server D pull replicating from servers A and B. You have two options:
Both alternatives accomplish exactly the same goal.
Replication documents can have a custom user_ctx property. This property defines the user context under which a replication runs. For the old way of triggering replications (POSTing to /_replicate/), this property was not needed (it didn’t exist in fact) - this is because at the moment of triggering the replication it has information about the authenticated user. With the replicator database, since it’s a regular database, the information about the authenticated user is only present at the moment the replication document is written to the database - the replicator database implementation is like a _changes feed consumer (with ?include_docs=true) that reacts to what was written to the replicator database - in fact this feature could be implemented with an external script/program. This implementation detail implies that for non admin users, a user_ctx property, containing the user’s name and a subset of his/her roles, must be defined in the replication document. This is ensured by the document update validation function present in the default design document of the replicator database. This validation function also ensure that a non admin user can set a user name property in the user_ctx property that doesn’t match his/her own name (same principle applies for the roles).
For admins, the user_ctx property is optional, and if it’s missing it defaults to a user context with name null and an empty list of roles - this mean design documents will not be written to local targets. If writing design documents to local targets is desired, the a user context with the roles _admin must be set explicitly.
Also, for admins the user_ctx property can be used to trigger a replication on behalf of another user. This is the user context that will be passed to local target database document validation functions.
Note
The user_ctx property only has effect for local endpoints.
Example delegated replication document:
{
"_id": "my_rep",
"source": "http://bserver.com:5984/foo",
"target": "bar",
"continuous": true,
"user_ctx": {
"name": "joe",
"roles": ["erlanger", "researcher"]
}
}
As stated before, for admins the user_ctx property is optional, while for regular (non admin) users it’s mandatory. When the roles property of user_ctx is missing, it defaults to the empty list [ ].
Let’s take the following example to illustrate replication and conflict handling.
So on the desktop the document has Bob’s new E-mail address and his old mobile number, and on the laptop it has his old E-mail address and his new mobile number.
The question is, what happens to these conflicting updated documents?
CouchDB works with JSON documents inside databases. Replication of databases takes place over HTTP, and can be either a “pull” or a “push”, but is unidirectional. So the easiest way to perform a full sync is to do a “push” followed by a “pull” (or vice versa).
So, Alice creates v1 and sync it. She updates to v2a on one side and v2b on the other, and then replicates. What happens?
The answer is simple: both versions exist on both sides!
DESKTOP LAPTOP
+---------+
| /db/bob | INITIAL
| v1 | CREATION
+---------+
+---------+ +---------+
| /db/bob | -----------------> | /db/bob | PUSH
| v1 | | v1 |
+---------+ +---------+
+---------+ +---------+ INDEPENDENT
| /db/bob | | /db/bob | LOCAL
| v2a | | v2b | EDITS
+---------+ +---------+
+---------+ +---------+
| /db/bob | -----------------> | /db/bob | PUSH
| v2a | | v2a |
+---------+ | v2b |
+---------+
+---------+ +---------+
| /db/bob | <----------------- | /db/bob | PULL
| v2a | | v2a |
| v2b | | v2b |
+---------+ +---------+
After all, this is not a filesystem, so there’s no restriction that only one document can exist with the name /db/bob. These are just “conflicting” revisions under the same name.
Because the changes are always replicated, the data is safe. Both machines have identical copies of both documents, so failure of a hard drive on either side won’t lose any of the changes.
Another thing to notice is that peers do not have to be configured or tracked. You can do regular replications to peers, or you can do one-off, ad-hoc pushes or pulls. After the replication has taken place, there is no record kept of which peer any particular document or revision came from.
So the question now is: what happens when you try to read /db/bob? By default, CouchDB picks one arbitrary revision as the “winner”, using a deterministic algorithm so that the same choice will be made on all peers. The same happens with views: the deterministically-chosen winner is the only revision fed into your map function.
Let’s say that the winner is v2a. On the desktop, if Alice reads the document she’ll see v2a, which is what she saved there. But on the laptop, after replication, she’ll also see only v2a. It could look as if the changes she made there have been lost - but of course they have not, they have just been hidden away as a conflicting revision. But eventually she’ll need these changes merged into Bob’s business card, otherwise they will effectively have been lost.
Any sensible business-card application will, at minimum, have to present the conflicting versions to Alice and allow her to create a new version incorporating information from them all. Ideally it would merge the updates itself.
When working on a single node, CouchDB will avoid creating conflicting revisions by returning a 409 Conflict error. This is because, when you PUT a new version of a document, you must give the _rev of the previous version. If that _rev has already been superseded, the update is rejected with a 409 Conflict response.
So imagine two users on the same node are fetching Bob’s business card, updating it concurrently, and writing it back:
USER1 -----------> GET /db/bob
<----------- {"_rev":"1-aaa", ...}
USER2 -----------> GET /db/bob
<----------- {"_rev":"1-aaa", ...}
USER1 -----------> PUT /db/bob?rev=1-aaa
<----------- {"_rev":"2-bbb", ...}
USER2 -----------> PUT /db/bob?rev=1-aaa
<----------- 409 Conflict (not saved)
User2’s changes are rejected, so it’s up to the app to fetch /db/bob again, and either:
So when working in this mode, your application still has to be able to handle these conflicts and have a suitable retry strategy, but these conflicts never end up inside the database itself.
There are two different ways that conflicts can end up in the database:
The _bulk_docs API lets you submit multiple updates (and/or deletes) in a single HTTP POST. Normally, these are treated as independent updates; some in the batch may fail because the _rev is stale (just like a 409 from a PUT) whilst others are written successfully. The response from _bulk_docs lists the success/fail separately for each document in the batch.
However there is another mode of working, whereby you specify {"all_or_nothing":true} as part of the request. This is CouchDB’s nearest equivalent of a “transaction”, but it’s not the same as a database transaction which can fail and roll back. Rather, it means that all of the changes in the request will be forcibly applied to the database, even if that introduces conflicts. The only guarantee you are given is that they will either all be applied to the database, or none of them (e.g. if the power is pulled out before the update is finished writing to disk).
So this gives you a way to introduce conflicts within a single database instance. If you choose to do this instead of PUT, it means you don’t have to write any code for the possibility of getting a 409 response, because you will never get one. Rather, you have to deal with conflicts appearing later in the database, which is what you’d have to do in a multi-master application anyway.
POST /db/_bulk_docs
{
"all_or_nothing": true,
"docs": [
{"_id":"x", "_rev":"1-xxx", ...},
{"_id":"y", "_rev":"1-yyy", ...},
...
]
}
When you update a document in CouchDB, it keeps a list of the previous revisions. In the case where conflicting updates are introduced, this history branches into a tree, where the current conflicting revisions for this document form the tips (leaf nodes) of this tree:
,--> r2a
r1 --> r2b
`--> r2c
Each branch can then extend its history - for example if you read revision r2b and then PUT with ?rev=r2b then you will make a new revision along that particular branch.
,--> r2a -> r3a -> r4a
r1 --> r2b -> r3b
`--> r2c -> r3c
Here, (r4a, r3b, r3c) are the set of conflicting revisions. The way you resolve a conflict is to delete the leaf nodes along the other branches. So when you combine (r4a+r3b+r3c) into a single merged document, you would replace r4a and delete r3b and r3c.
,--> r2a -> r3a -> r4a -> r5a
r1 --> r2b -> r3b -> (r4b deleted)
`--> r2c -> r3c -> (r4c deleted)
Note that r4b and r4c still exist as leaf nodes in the history tree, but as deleted docs. You can retrieve them but they will be marked "_deleted":true.
When you compact a database, the bodies of all the non-leaf documents are discarded. However, the list of historical _revs is retained, for the benefit of later conflict resolution in case you meet any old replicas of the database at some time in future. There is “revision pruning” to stop this getting arbitrarily large.
The basic :get:`/{doc}/{docid}` operation will not show you any information about conflicts. You see only the deterministically-chosen winner, and get no indication as to whether other conflicting revisions exist or not:
{
"_id":"test",
"_rev":"2-b91bb807b4685080c6a651115ff558f5",
"hello":"bar"
}
If you do GET /db/bob?conflicts=true, and the document is in a conflict state, then you will get the winner plus a _conflicts member containing an array of the revs of the other, conflicting revision(s). You can then fetch them individually using subsequent GET /db/bob?rev=xxxx operations:
{
"_id":"test",
"_rev":"2-b91bb807b4685080c6a651115ff558f5",
"hello":"bar",
"_conflicts":[
"2-65db2a11b5172bf928e3bcf59f728970",
"2-5bc3c6319edf62d4c624277fdd0ae191"
]
}
If you do GET /db/bob?open_revs=all then you will get all the leaf nodes of the revision tree. This will give you all the current conflicts, but will also give you leaf nodes which have been deleted (i.e. parts of the conflict history which have since been resolved). You can remove these by filtering out documents with "_deleted":true:
[
{"ok":{"_id":"test","_rev":"2-5bc3c6319edf62d4c624277fdd0ae191","hello":"foo"}},
{"ok":{"_id":"test","_rev":"2-65db2a11b5172bf928e3bcf59f728970","hello":"baz"}},
{"ok":{"_id":"test","_rev":"2-b91bb807b4685080c6a651115ff558f5","hello":"bar"}}
]
The "ok" tag is an artifact of open_revs, which also lets you list explicit revisions as a JSON array, e.g. open_revs=[rev1,rev2,rev3]. In this form, it would be possible to request a revision which is now missing, because the database has been compacted.
Note
The order of revisions returned by open_revs=all is NOT related to the deterministic “winning” algorithm. In the above example, the winning revision is 2-b91b... and happens to be returned last, but in other cases it can be returned in a different position.
Once you have retrieved all the conflicting revisions, your application can then choose to display them all to the user. Or it could attempt to merge them, write back the merged version, and delete the conflicting versions - that is, to resolve the conflict permanently.
As described above, you need to update one revision and delete all the conflicting revisions explicitly. This can be done using a single POST to _bulk_docs, setting "_deleted":true on those revisions you wish to delete.
You can fetch multiple documents at once using include_docs=true on a view. However, a conflicts=true request is ignored; the “doc” part of the value never includes a _conflicts member. Hence you would need to do another query to determine for each document whether it is in a conflicting state:
$ curl 'http://<IP_ADDRESS>:5984/conflict_test/_all_docs?include_docs=true&conflicts=true'
{
"total_rows":1,
"offset":0,
"rows":[
{
"id":"test",
"key":"test",
"value":{"rev":"2-b91bb807b4685080c6a651115ff558f5"},
"doc":{
"_id":"test",
"_rev":"2-b91bb807b4685080c6a651115ff558f5",
"hello":"bar"
}
}
]
}
$ curl 'http://<IP_ADDRESS>:5984/conflict_test/test?conflicts=true'
{
"_id":"test",
"_rev":"2-b91bb807b4685080c6a651115ff558f5",
"hello":"bar",
"_conflicts":[
"2-65db2a11b5172bf928e3bcf59f728970",
"2-5bc3c6319edf62d4c624277fdd0ae191"
]
}
Views only get the winning revision of a document. However they do also get a _conflicts member if there are any conflicting revisions. This means you can write a view whose job is specifically to locate documents with conflicts. Here is a simple map function which achieves this:
function(doc) {
if (doc._conflicts) {
emit(null, [doc._rev].concat(doc._conflicts));
}
}
which gives the following output:
{
"total_rows":1,
"offset":0,
"rows":[
{
"id":"test",
"key":null,
"value":[
"2-b91bb807b4685080c6a651115ff558f5",
"2-65db2a11b5172bf928e3bcf59f728970",
"2-5bc3c6319edf62d4c624277fdd0ae191"
]
}
]
}
If you do this, you can have a separate “sweep” process which periodically scans your database, looks for documents which have conflicts, fetches the conflicting revisions, and resolves them.
Whilst this keeps the main application simple, the problem with this approach is that there will be a window between a conflict being introduced and it being resolved. From a user’s viewpoint, this may appear that the document they just saved successfully may suddenly lose their changes, only to be resurrected some time later. This may or may not be acceptable.
Also, it’s easy to forget to start the sweeper, or not to implement it properly, and this will introduce odd behaviour which will be hard to track down.
CouchDB’s “winning” revision algorithm may mean that information drops out of a view until a conflict has been resolved. Consider Bob’s business card again; suppose Alice has a view which emits mobile numbers, so that her telephony application can display the caller’s name based on caller ID. If there are conflicting documents with Bob’s old and new mobile numbers, and they happen to be resolved in favour of Bob’s old number, then the view won’t be able to recognise his new one. In this particular case, the application might have preferred to put information from both the conflicting documents into the view, but this currently isn’t possible.
Suggested algorithm to fetch a document with conflict resolution:
This could either be done on every read (in which case you could replace all calls to GET in your application with calls to a library which does the above), or as part of your sweeper code.
And here is an example of this in Ruby using the low-level RestClient:
require 'rubygems'
require 'rest_client'
require 'json'
DB="http://<IP_ADDRESS>:5984/conflict_test"
# Write multiple documents as all_or_nothing, can introduce conflicts
def writem(docs)
JSON.parse(RestClient.post("#{DB}/_bulk_docs", {
"all_or_nothing" => true,
"docs" => docs,
}.to_json))
end
# Write one document, return the rev
def write1(doc, id=nil, rev=nil)
doc['_id'] = id if id
doc['_rev'] = rev if rev
writem([doc]).first['rev']
end
# Read a document, return *all* revs
def read1(id)
retries = 0
loop do
# FIXME: escape id
res = [JSON.parse(RestClient.get("#{DB}/#{id}?conflicts=true"))]
if revs = res.first.delete('_conflicts')
begin
revs.each do |rev|
res << JSON.parse(RestClient.get("#{DB}/#{id}?rev=#{rev}"))
end
rescue
retries += 1
raise if retries >= 5
next
end
end
return res
end
end
# Create DB
RestClient.delete DB rescue nil
RestClient.put DB, {}.to_json
# Write a document
rev1 = write1({"hello"=>"xxx"},"test")
p read1("test")
# Make three conflicting versions
write1({"hello"=>"foo"},"test",rev1)
write1({"hello"=>"bar"},"test",rev1)
write1({"hello"=>"baz"},"test",rev1)
res = read1("test")
p res
# Now let's replace these three with one
res.first['hello'] = "foo+bar+baz"
res.each_with_index do |r,i|
unless i == 0
r.replace({'_id'=>r['_id'], '_rev'=>r['_rev'], '_deleted'=>true})
end
end
writem(res)
p read1("test")
An application written this way never has to deal with a PUT 409, and is automatically multi-master capable.
You can see that it’s straightforward enough when you know what you’re doing. It’s just that CouchDB doesn’t currently provide a convenient HTTP API for “fetch all conflicting revisions”, nor “PUT to supersede these N revisions”, so you need to wrap these yourself. I also don’t know of any client-side libraries which provide support for this.
Actually performing the merge is an application-specific function. It depends on the structure of your data. Sometimes it will be easy: e.g. if a document contains a list which is only ever appended to, then you can perform a union of the two list versions.
Some merge strategies look at the changes made to an object, compared to its previous version. This is how git’s merge function works.
For example, to merge Bob’s business card versions v2a and v2b, you could look at the differences between v1 and v2b, and then apply these changes to v2a as well.
With CouchDB, you can sometimes get hold of old revisions of a document. For example, if you fetch /db/bob?rev=v2b&revs_info=true you’ll get a list of the previous revision ids which ended up with revision v2b. Doing the same for v2a you can find their common ancestor revision. However if the database has been compacted, the content of that document revision will have been lost. revs_info will still show that v1 was an ancestor, but report it as “missing”:
BEFORE COMPACTION AFTER COMPACTION
,-> v2a v2a
v1
`-> v2b v2b
So if you want to work with diffs, the recommended way is to store those diffs within the new revision itself. That is: when you replace v1 with v2a, include an extra field or attachment in v2a which says which fields were changed from v1 to v2a. This unfortunately does mean additional book-keeping for your application.
The same issues arise with other replicating systems, so it can be instructive to look at these and see how they compare with CouchDB. Please feel free to add other examples.
Unison is a bi-directional file synchronisation tool. In this case, the business card would be a file, say bob.vcf.
When you run unison, changes propagate both ways. If a file has changed on one side but not the other, the new replaces the old. Unison maintains a local state file so that it knows whether a file has changed since the last successful replication.
In our example it has changed on both sides. Only one file called bob.vcf can exist within the filesystem. Unison solves the problem by simply ducking out: the user can choose to replace the remote version with the local version, or vice versa (both of which would lose data), but the default action is to leave both sides unchanged.
From Alice’s point of view, at least this is a simple solution. Whenever she’s on the desktop she’ll see the version she last edited on the desktop, and whenever she’s on the laptop she’ll see the version she last edited there.
But because no replication has actually taken place, the data is not protected. If her laptop hard drive dies, she’ll lose all her changes made on the laptop; ditto if her desktop hard drive dies.
It’s up to her to copy across one of the versions manually (under a different filename), merge the two, and then finally push the merged version to the other side.
Note also that the original file (version v1) has been lost by this point. So it’s not going to be known from inspection alone which of v2a and v2b has the most up-to-date E-mail address for Bob, and which has the most up-to-date mobile number. Alice has to remember which she entered last.
Git is a well-known distributed source control system. Like Unison, git deals with files. However, git considers the state of a whole set of files as a single object, the “tree”. Whenever you save an update, you create a “commit” which points to both the updated tree and the previous commit(s), which in turn point to the previous tree(s). You therefore have a full history of all the states of the files. This forms a branch, and a pointer is kept to the tip of the branch, from which you can work backwards to any previous state. The “pointer” is actually an SHA1 hash of the tip commit.
If you are replicating with one or more peers, a separate branch is made for each of the peers. For example, you might have:
master -- my local branch
remotes/foo/master -- branch on peer 'foo'
remotes/bar/master -- branch on peer 'bar'
In the normal way of working, replication is a “pull”, importing changes from a remote peer into the local repository. A “pull” does two things: first “fetch” the state of the peer into the remote tracking branch for that peer; and then attempt to “merge” those changes into the local branch.
Now let’s consider the business card. Alice has created a git repo containing bob.vcf, and cloned it across to the other machine. The branches look like this, where AAAAAAAA is the SHA1 of the commit:
---------- desktop ---------- ---------- laptop ----------
master: AAAAAAAA master: AAAAAAAA
remotes/laptop/master: AAAAAAAA remotes/desktop/master: AAAAAAAA
Now she makes a change on the desktop, and commits it into the desktop repo; then she makes a different change on the laptop, and commits it into the laptop repo:
---------- desktop ---------- ---------- laptop ----------
master: BBBBBBBB master: CCCCCCCC
remotes/laptop/master: AAAAAAAA remotes/desktop/master: AAAAAAAA
Now on the desktop she does git pull laptop. Firstly, the remote objects are copied across into the local repo and the remote tracking branch is updated:
---------- desktop ---------- ---------- laptop ----------
master: BBBBBBBB master: CCCCCCCC
remotes/laptop/master: CCCCCCCC remotes/desktop/master: AAAAAAAA
Note
repo still contains AAAAAAAA because commits BBBBBBBB and CCCCCCCC point to it
Then git will attempt to merge the changes in. It can do this because it knows the parent commit to CCCCCCCC is AAAAAAAA, so it takes a diff between AAAAAAAA and CCCCCCCC and tries to apply it to BBBBBBBB.
If this is successful, then you’ll get a new version with a merge commit:
---------- desktop ---------- ---------- laptop ----------
master: DDDDDDDD master: CCCCCCCC
remotes/laptop/master: CCCCCCCC remotes/desktop/master: AAAAAAAA
Then Alice has to logon to the laptop and run git pull desktop. A similar process occurs. The remote tracking branch is updated:
---------- desktop ---------- ---------- laptop ----------
master: DDDDDDDD master: CCCCCCCC
remotes/laptop/master: CCCCCCCC remotes/desktop/master: DDDDDDDD
Then a merge takes place. This is a special-case: CCCCCCCC one of the parent commits of DDDDDDDD, so the laptop can fast forward update from CCCCCCCC to DDDDDDDD directly without having to do any complex merging. This leaves the final state as:
---------- desktop ---------- ---------- laptop ----------
master: DDDDDDDD master: DDDDDDDD
remotes/laptop/master: CCCCCCCC remotes/desktop/master: DDDDDDDD
Now this is all and good, but you may wonder how this is relevant when thinking about CouchDB.
Firstly, note what happens in the case when the merge algorithm fails. The changes are still propagated from the remote repo into the local one, and are available in the remote tracking branch; so unlike Unison, you know the data is protected. It’s just that the local working copy may fail to update, or may diverge from the remote version. It’s up to you to create and commit the combined version yourself, but you are guaranteed to have all the history you might need to do this.
Note that whilst it’s possible to build new merge algorithms into Git, the standard ones are focused on line-based changes to source code. They don’t work well for XML or JSON if it’s presented without any line breaks.
The other interesting consideration is multiple peers. In this case you have multiple remote tracking branches, some of which may match your local branch, some of which may be behind you, and some of which may be ahead of you (i.e. contain changes that you haven’t yet merged):
master: AAAAAAAA
remotes/foo/master: BBBBBBBB
remotes/bar/master: CCCCCCCC
remotes/baz/master: AAAAAAAA
Note that each peer is explicitly tracked, and therefore has to be explicitly created.  This is different to CouchDB, which doesn’t keep any peer state in the database.
Another difference with git is that it maintains all history back to time zero - git compaction keeps diffs between all those versions in order to reduce size, but CouchDB discards them. If you are constantly updating a document, the size of a git repo would grow forever. It is possible (with some effort) to use “history rewriting” to make git forget commits earlier than a particular one.
| Author: | Jason Smith |
|---|---|
| Date: | 2011-01-29 |
| Source: | http://stackoverflow.com/questions/4766391/what-is-the-couchdb-replication-protocol-is-it-like-git |
Key points
If you know Git, then you know how Couch replication works. Replicating is very similar to pushing or pulling with distributed source managers like Git.
CouchDB replication does not have its own protocol. A replicator simply connects to two DBs as a client, then reads from one and writes to the other. Push replication is reading the local data and updating the remote DB; pull replication is vice versa.
Everything flows from the data model
The replication algorithm is trivial, uninteresting. A trained monkey could design it. It’s simple because the cleverness is the data model, which has these useful characteristics:
Final notes
At least one sentence in this writeup (possibly this one) is complete BS.
The compaction operation is the way to reduce disk space usage by removing unused and old data from database or view index files. This operation is a very similar to the vacuum (SQLite ex.) available for other database management systems.
During compaction of the target CouchDB creates new file with the .compact extension and transfers only actual data into. Because of this, CouchDB checks first for the available disk space - it should be twice greater than the compacted file’s data.
When all actual data successful transferred to the compacted file CouchDB removes the target replacing him with it.
Database compaction compresses the database file by removing unused file sections created during updates. Old documents revisions are replaced with small amount of metadata called tombstone which are used for conflicts resolution during replication. The number of stored revisions (and their tombstones) can be configured by using the _revs_limit URL endpoint.
Compaction is manually triggered operation per database and runs as a background task. To start it for specific database there is need to send HTTP POST /{db}/_compact sub-resource of the target database:
curl -H "Content-Type: application/json" -X POST http://localhost:5984/my_db/_compact
On success, HTTP status 202 Accepted is returned immediately:
HTTP/1.1 202 Accepted
Cache-Control: must-revalidate
Content-Length: 12
Content-Type: text/plain; charset=utf-8
Date: Wed, 19 Jun 2013 09:43:52 GMT
Server: CouchDB (Erlang/OTP)
{"ok":true}
Although the request body is not used you must still specify Content-Type header with application/json value for the request. If you don’t, you will be aware about with HTTP status 415 Unsupported Media Type response:
HTTP/1.1 415 Unsupported Media Type
Cache-Control: must-revalidate
Content-Length: 78
Content-Type: application/json
Date: Wed, 19 Jun 2013 09:43:44 GMT
Server: CouchDB (Erlang/OTP)
{"error":"bad_content_type","reason":"Content-Type must be application/json"}
When the compaction is successful started and running it is possible to get information about it via database information resource:
curl http://localhost:5984/my_db
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 246
Content-Type: application/json
Date: Wed, 19 Jun 2013 16:51:20 GMT
Server: CouchDB (Erlang/OTP)
{
"committed_update_seq": 76215,
"compact_running": true,
"data_size": 3787996,
"db_name": "my_db",
"disk_format_version": 6,
"disk_size": 17703025,
"doc_count": 5091,
"doc_del_count": 0,
"instance_start_time": "1371660751878859",
"purge_seq": 0,
"update_seq": 76215
}
Note that compaction_running field is true indicating that compaction is actually running. To track the compaction progress you may query the _active_tasks resource:
curl http://localhost:5984/my_db
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 175
Content-Type: application/json
Date: Wed, 19 Jun 2013 16:27:23 GMT
Server: CouchDB (Erlang/OTP)
[
{
"changes_done": 44461,
"database": "my_db",
"pid": "<0.218.0>",
"progress": 58,
"started_on":<PHONE_NUMBER>,
"total_changes": 76215,
"type": "database_compaction",
"updated_on":<PHONE_NUMBER>
}
]
Views are also need compaction like databases, unlike databases views are compacted by groups per design document. To start their compaction there is need to send HTTP POST /{db}/_compact/{ddoc} request:
curl -H "Content-Type: application/json" -X POST http://localhost:5984/dbname/_compact/designname
{"ok":true}
This compacts the view index from the current version of the specified design document. The HTTP response code is 202 Accepted (like compaction for databases) and a compaction background task will be created.
View indexes on disk are named after their MD5 hash of the view definition. When you change a view, old indexes remain on disk. To clean up all outdated view indexes (files named after the MD5 representation of views, that does not exist anymore) you can trigger a view cleanup:
curl -H "Content-Type: application/json" -X POST http://localhost:5984/dbname/_view_cleanup
{"ok":true}
While both database and views compactions are required be manually triggered, it is also possible to configure automatic compaction, so that compaction of databases and views is automatically triggered based on various criteria. Automatic compaction is configured in CouchDB’s configuration files.
The daemons/compaction_daemon is responsible for triggering the compaction. It is automatically started, but disabled by default. The criteria for triggering the compactions is configured in the compactions section.
With up to tens of thousands of documents you will generally find CouchDB to perform well no matter how you write your code. Once you start getting into the millions of documents you need to be a lot more careful.
The smaller your file size, the less I/O operations there will be, the more of the file can be cached by CouchDB and the operating system, the quicker it is to replicate, backup etc. Consequently you should carefully examine the data you are storing. For example it would be silly to use keys that are hundreds of characters long, but your program would be hard to maintain if you only used single character keys. Carefully consider data that is duplicated by putting it in views.
Using faster disks, striped RAID arrays and modern file systems can all speed up your CouchDB deployment. However, there is one option that can increase the responsiveness of your CouchDB server when disk performance is a bottleneck. From the Erlang documentation for the file module:
On operating systems with thread support, it is possible to let file operations be performed in threads of their own, allowing other Erlang processes to continue executing in parallel with the file operations. See the command line flag +A in erl(1).
Setting this argument to a number greater than zero can keep your CouchDB installation responsive even during periods of heavy disk utilization. The easiest way to set this option is through the ERL_FLAGS environment variable. For example, to give Erlang four threads with which to perform I/O operations add the following to (prefix)/etc/defaults/couchdb (or equivalent):
export ERL_FLAGS="+A 4"
One of the problems that administrators run into as their deployments become large are resource limits imposed by the system and by the application configuration. Raising these limits can allow your deployment to grow beyond what the default configuration will support.
The delayed commits allows to achieve better write performance for some workloads while sacrificing a small amount of durability. The setting causes CouchDB to wait up to a full second before committing new data after an update. If the server crashes before the header is written then any writes since the last commit are lost. Keep this option enabled on your own risk.
In your configuration (local.ini or similar) familiarize yourself with the couchdb/max_dbs_open:
[couchdb]
max_dbs_open = 100
This option places an upper bound on the number of databases that can be open at one time. CouchDB reference counts database accesses internally and will close idle databases when it must. Sometimes it is necessary to keep more than the default open at once, such as in deployments where many databases will be continuously replicating.
Even if you’ve increased the maximum connections CouchDB will allow, the Erlang runtime system will not allow more than 1024 connections by default. Adding the following directive to (prefix)/etc/default/couchdb (or equivalent) will increase this limit (in this case to 4096):
export ERL_MAX_PORTS=4096
CouchDB versions up to 1.1.x also create Erlang Term Storage (ETS) tables for each replication. If you are using a version of CouchDB older than 1.2 and must support many replications, also set the ERL_MAX_ETS_TABLES variable. The default is approximately 1400 tables.
Note that on Mac OS X, Erlang will not actually increase the file descriptor limit past 1024 (i.e. the system header–defined value of FD_SETSIZE). See this tip for a possible workaround and this thread for a deeper explanation.
Finally, most *nix operating systems impose various resource limits on every process. If your system is set up to use the Pluggable Authentication Modules (PAM) system, increasing this limit is straightforward. For example, creating a file named /etc/security/limits.d/100-couchdb.conf with the following contents will ensure that CouchDB can open enough file descriptors to service your increased maximum open databases and Erlang ports:
#<domain> <type> <item> <value>
couchdb hard nofile 4096
couchdb soft nofile 4096
If your system does not use PAM, a ulimit command is usually available for use in a custom script to launch CouchDB with increased resource limits. If necessary, feel free to increase this limits as long as your hardware can handle the load.
There is latency overhead making and receiving each request/response. In general you should do your requests in batches. Most APIs have some mechanism to do batches, usually by supplying lists of documents or keys in the request body. Be careful what size you pick for the batches. The larger batch requires more time your client has to spend encoding the items into JSON and more time is spent decoding that number of responses. Do some benchmarking with your own configuration and typical data to find the sweet spot. It is likely to be between one and ten thousand documents.
If you have a fast I/O system then you can also use concurrency - have multiple requests/responses at the same time. This mitigates the latency involved in assembling JSON, doing the networking and decoding JSON.
As of CouchDB 1.1.0, users often report lower write performance of documents compared to older releases. The main reason is that this release ships with the more recent version of the HTTP server library Mochiweb, which by default sets the TCP socket option SO_NODELAY to false. This means that small data sent to the TCP socket, like the reply to a document write request (or reading a very small document), will not be sent immediately to the network - TCP will buffer it for a while hoping that it will be asked to send more data through the same socket and then send all the data at once for increased performance. This TCP buffering behaviour can be disabled via httpd/socket_options:
[httpd]
socket_options = [{nodelay, true}]
When you DELETE a document the database will create a new revision which contains the _id and _rev fields as well as the _deleted flag. This revision will remain even after a database compaction so that the deletion can be replicated. Deleted documents, like non-deleted documents, can affect view build times, PUT and DELETE requests time and size of database on disk, since they increase the size of the B+Tree’s. You can see the number of deleted documents in database information. If your use case creates lots of deleted documents (for example, if you are storing short-term data like logfile entries, message queues, etc), you might want to periodically switch to a new database and delete the old one (once the entries in it have all expired).
The db file size is derived from your document and view sizes but also on a multiple of your _id sizes. Not only is the _id present in the document, but it and parts of it are duplicated in the binary tree structure CouchDB uses to navigate the file to find the document in the first place. As a real world example for one user switching from 16 byte ids to 4 byte ids made a database go from 21GB to 4GB with 10 million documents (the raw JSON text when from 2.5GB to 2GB).
Inserting with sequential (and at least sorted) ids is faster than random ids. Consequently you should consider generating ids yourself, allocating them sequentially and using an encoding scheme that consumes fewer bytes. For example, something that takes 16 hex digits to represent can be done in 4 base 62 digits (10 numerals, 26 lower case, 26 upper case).
Views with the Javascript query server are extremely slow to generate when there are a non-trivial number of documents to process. The generation process won’t even saturate a single CPU let alone your I/O. The cause is the latency involved in the CouchDB server and separate couchjs query server, dramatically indicating how important it is to take latency out of your implementation.
You can let view access be “stale” but it isn’t practical to determine when that will occur giving you a quick response and when views will be updated which will take a long time. (A 10 million document database took about 10 minutes to load into CouchDB but about 4 hours to do view generation).
View information isn’t replicated - it is rebuilt on each database so you can’t do the view generation on a separate sever.
If you’re using a very simple view function that only performs a sum or count reduction, you can call native Erlang implementations of them by simply writing _sum or _count in place of your function declaration. This will speed up things dramatically, as it cuts down on IO between CouchDB and the JavaScript query server. For example, as mentioned on the mailing list, the time for outputting an (already indexed and cached) view with about 78,000 items went down from 60 seconds to 4 seconds.
Before:
{
"_id": "_design/foo",
"views": {
"bar": {
"map": "function (doc) { emit(doc.author, 1); }",
"reduce": "function (keys, values, rereduce) { return sum(values); }"
}
}
}
After:
{
"_id": "_design/foo",
"views": {
"bar": {
"map": "function (doc) { emit(doc.author, 1); }",
"reduce": "_sum"
}
}
}
See also
CouchApps are web applications served directly from CouchDB, mostly driven by JavaScript and HTML5. If you can fit your application into those constraints, then you get CouchDB’s scalability and flexibility “for free” (and deploying your app is as simple as replicating it to the production server).
In this section we’ll show how to write design documents, using the built-in JavaScript Query Server.
But before we start to write our first function, let’s take a look at the list of common objects that will be used during our code journey - we’ll be using them extensively within each function:
Views are the primary tool used for querying and reporting on CouchDB databases.
| Arguments: |
|
|---|
Map functions accept a single document as the argument and (optionally) emit() key/value pairs that are stored in a view.
function (doc) {
if (doc.type === 'post' && doc.tags && Array.isArray(doc.tags)) {
doc.tags.forEach(function (tag) {
emit(tag.toLowerCase(), 1);
});
}
}
In this example a key/value pair is emitted for each value in the tags array of a document with a type of “post”. Note that emit() may be called many times for a single document, so the same document may be available by several different keys.
Also keep in mind that each document is sealed to prevent situation when one map function changes document state and the other one received a modified version.
For efficiency reasons, documents are passed to a group of map functions - each document is processed by group of map functions from all views of related design document. This means that if you trigger index update for one view in ddoc, all others will get updated too.
Since 1.1.0 release map function supports CommonJS modules and access to require() function.
| Arguments: |
|
|---|---|
| Returns: | Reduces values |
Reduce functions takes two required arguments of keys and values lists - the result of the related map function - and optional third one which indicates if rereduce mode is active or not. Rereduce is using for additional reduce values list, so when it is true there is no information about related keys (first argument is null).
Note, that if produced result by reduce function is longer than initial values list then a Query Server error will be raised. However, this behavior could be disabled by setting reduce_limit config option to false:
[query_server_config]
reduce_limit = false
While disabling reduce_limit might be useful for debug proposes, remember, that main task of reduce functions is to reduce mapped result, not to make it even bigger. Generally, your reduce function should converge rapidly to a single value - which could be an array or similar object.
Additionally, CouchDB has three built-in reduce functions. These are implemented in Erlang and runs inside CouchDB, so they are much faster than the equivalent JavaScript functions: _sum, _count and _stats. Their equivalents in JavaScript below:
// could be replaced by _sum
function(keys, values){
sum(values);
}
// could be replaced by _count
function(keys, values, rereduce){
if (rereduce) {
return sum(values);
} else {
return values.length;
}
}
// could be replaced by _stats
function(keys, values, rereduce){
return {
'sum': sum(values),
'min': Math.min.apply(null, values),
'max': Math.max.apply(null, values),
'count': values.length,
'sumsqr': (function(){
var sumsqr = 0;
values.forEach(function (value) {
sumsqr += value * value;
});
return sumsqr;
})(),
}
}
Note
Why don’t reduce functions support CommonJS modules?
While map functions have limited access to stored modules through require() function there is no such feature for reduce functions. The reason lies deep inside in mechanism how map and reduce functions are processed by Query Server. Let’s take a look on map functions first:
Now let’s see how reduce functions are handled:
As you may note, reduce functions been applied in single shot while map ones are applied in an iterative way per each document. This means that it’s possible for map functions to precompile CommonJS libraries and use them during the entire view processing, but for reduce functions it will be compiled again and again for each view result reduction, which will lead to performance degradation (reduce function are already does hard work to make large result smaller).
| Arguments: |
|
|---|---|
| Returns: | |
| Return type: | object or string |
Show functions are used to represent documents in various formats, commonly as HTML page with nicer formatting. They can also be used to run server-side functions without requiring a pre-existing document.
Basic example of show function could be:
function(doc, req){
if (doc) {
return "Hello from " + doc._id + "!";
} else {
return "Hello, world!";
}
}
Also, there is more simple way to return json encoded data:
function(doc, req){
return {
'json': {
'id': doc['_id'],
'rev': doc['_rev']
}
}
}
and even files (this one is CouchDB logo):
function(doc, req){
return {
'headers': {
'Content-Type' : 'image/png',
},
'base64': ''.concat(
'iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAMAAAAoLQ9TAAAAsV',
'BMVEUAAAD////////////////////////5ur3rEBn////////////////wDBL/',
'AADuBAe9EB3IEBz/7+//X1/qBQn2AgP/f3/ilpzsDxfpChDtDhXeCA76AQH/v7',
'/84eLyWV/uc3bJPEf/Dw/uw8bRWmP1h4zxSlD6YGHuQ0f6g4XyQkXvCA36MDH6',
'wMH/z8/yAwX64ODeh47BHiv/Ly/20dLQLTj98PDXWmP/Pz//39/wGyJ7Iy9JAA',
'AADHRSTlMAbw8vf08/bz+Pv19jK/W3AAAAg0lEQVR4Xp3LRQ4DQRBD0QqTm4Y5',
'zMxw/4OleiJlHeUtv2X6RbNO1Uqj9g0RMCuQO0vBIg4vMFeOpCWIWmDOw82fZx',
'vaND1c8OG4vrdOqD8YwgpDYDxRgkSm5rwu0nQVBJuMg++pLXZyr5jnc1BaH4GT',
'LvEliY253nA3pVhQqdPt0f/erJkMGMB8xucAAAAASUVORK5CYII=')
}
}
But what if you need to represent data in different formats via a single function? Functions registerType() and provides() are your the best friends in that question:
function(doc, req){
provides('json', function(){
return {'json': doc}
});
provides('html', function(){
return '<pre>' + toJSON(doc) + '</pre>'
})
provides('xml', function(){
return {
'headers': {'Content-Type': 'application/xml'},
'body' : ''.concat(
'<?xml version="1.0" encoding="utf-8"?>\n',
'<doc>',
(function(){
escape = function(s){
return s.replace(/"/g, '"')
.replace(/>/g, '>')
.replace(/</g, '<')
.replace(/&/g, '&');
};
var content = '';
for(var key in doc){
if(!doc.hasOwnProperty(key)) continue;
var value = escape(toJSON(doc[key]));
var key = escape(key);
content += ''.concat(
'<' + key + '>',
value
'</' + key + '>'
)
}
return content;
})(),
'</doc>'
)
}
})
registerType('text-json', 'text/json')
provides('text-json', function(){
return toJSON(doc);
})
}
This function may return html, json , xml or our custom text json format representation of same document object with same processing rules. Probably, the xml provider in our function needs more care to handle nested objects correctly, and keys with invalid characters, but you’ve got the idea!
See also
| Arguments: |
|
|---|---|
| Returns: | Last chunk. |
| Return type: | string |
While Show functions are used to customize document presentation, List functions are used for same purpose, but against View functions results.
The next list function formats view and represents it as a very simple HTML page:
function(head, req){
start({
'headers': {
'Content-Type': 'text/html'
}
});
send('<html><body><table>');
send('<tr><th>ID</th><th>Key</th><th>Value</th></tr>')
while(row = getRow()){
send(''.concat(
'<tr>',
'<td>' + toJSON(row.id) + '</td>',
'<td>' + toJSON(row.key) + '</td>',
'<td>' + toJSON(row.value) + '</td>',
'</tr>'
));
}
send('</table></body></html>');
}
Templates and styles could obviously be used to present data in a nicer fashion, but this is an excellent starting point. Note that you may also use registerType() and provides() functions in the same way as for Show functions!
See also
| Arguments: |
|
|---|---|
| Returns: | Two-element array: the first element is the (updated or new) document, which is committed to the database. If the first element is null no document will be committed to the database. If you are updating an existing, it should already have an _id set, and if you are creating a new document, make sure to set its _id to something, either generated based on the input or the req.uuid provided. The second element is the response that will be sent back to the caller. |
Update handlers are functions that clients can request to invoke server-side logic that will create or update a document. This feature allows a range of use cases such as providing a server-side last modified timestamp, updating individual fields in a document without first getting the latest revision, etc.
When the request to an update handler includes a document ID in the URL, the server will provide the function with the most recent version of that document. You can provide any other values needed by the update handler function via the POST/PUT entity body or query string parameters of the request.
The basic example that demonstrates all use-cases of update handlers below:
function(doc, req){
if (!doc){
if ('id' in req){
// create new document
return [{'_id': req['id']}, 'New World']
}
// change nothing in database
return [null, 'Empty World']
}
doc['world'] = 'hello';
doc['edited_by'] = req['userCtx']['name']
return [doc, 'Edited World!']
}
See also
| Arguments: |
|
|---|---|
| Returns: | Boolean value: true means that doc passes the filter rules, false that not. |
Filter functions are mostly acts like Show functions and List functions: they formats, but more correctly to say, they filters changes feed.
By default the changes feed emits all database documents changes. But if you’re waiting for some special changes, processing all documents is inefficient.
Filters are special design document functions that allows changes feed to emit only specific documents that pass filter rules.
Lets assume that our database is a mailbox and we need to to handle only new mails (documents with status new) events. Assuming that, our filter function will looks like next one:
function(doc, req){
// we need only `mail` documents
if (doc.type != 'mail'){
return false;
}
// we're interested only in `new` ones
if (doc.status != 'new'){
return false;
}
return true; // passed!
}
Filter functions must return true in fact if document passed all defined rules. Now, if you apply this function to changes feed it will emit only changes about “new mails”:
GET /somedatabase/_changes?filter=mailbox/new_mail HTTP/1.1
{"results":[
{"seq":1,"id":"df8eca9da37dade42ee4d7aa3401f1dd","changes":[{"rev":"1-c2e0085a21d34fa1cecb6dc26a4ae657"}]},
{"seq":7,"id":"df8eca9da37dade42ee4d7aa34024714","changes":[{"rev":"1-29d748a6e87b43db967fe338bcb08d74"}]},
],
"last_seq":27}
Note, that last_seq number is 27, but we’d received only two records. Seems like any other changes was about documents that hasn’t passed our filter.
Probably, we also need to filter changes feed of our mailbox not only by single status value: we’re also interested in statuses like “spam” to update spam-filter heuristic rules, “outgoing” to let mail daemon actually send mails and so on. Creating a lot of similar functions that actually does similar work isn’t good idea - so we need dynamic filter to go.
If you have noted, filter functions takes second argument as request object - it allows to create dynamic filters based on query parameters, user context and more.
The dynamic version of our filter now will be next:
function(doc, req){
// we need only `mail` documents
if (doc.type != 'mail'){
return false;
}
// we're interested only in requested status
if (doc.status != req.query.status){
return false;
}
return true; // passed!
}
and now we have pass status query parameter in request to let filter match only required documents:
GET /somedatabase/_changes?filter=mailbox/by_status&status=new HTTP/1.1
{"results":[
{"seq":1,"id":"df8eca9da37dade42ee4d7aa3401f1dd","changes":[{"rev":"1-c2e0085a21d34fa1cecb6dc26a4ae657"}]},
{"seq":7,"id":"df8eca9da37dade42ee4d7aa34024714","changes":[{"rev":"1-29d748a6e87b43db967fe338bcb08d74"}]},
],
"last_seq":27}
and we can change filter behavior with easy:
GET /somedatabase/_changes?filter=mailbox/by_status&status=spam HTTP/1.1
{"results":[
{"seq":11,"id":"8960e91220798fc9f9d29d24ed612e0d","changes":[{"rev":"3-cc6ff71af716ddc2ba114967025c0ee0"}]},
],
"last_seq":27}
Combining filters with continuous feed allows to create powerful event-driven systems.
View filters are the same as above, with one small difference: they use views map function instead to filter one to process the changes feed. Each time when a key-value pair could be emitted, a change is returned. This allows to avoid creating filter functions that are mostly does same works as views.
To use them just specify _view value for filter parameter and designdoc/viewname for view one:
GET /somedatabase/_changes?filter=_view&view=dname/viewname HTTP/1.1
Note
Since view filters uses map functions as filters, they can’t show any dynamic behavior since request object is not available.
See also
| Arguments: |
|
|---|---|
| Throws: | forbidden error to gracefully prevent document storing. |
| Throws: | unauthorized error to prevent storage and allow the user to re-auth. |
A design document may contain a function named validate_doc_update which can be used to prevent invalid or unauthorized document update requests from being stored. The function is passed the new document from the update request, the current document stored in the database, a User Context Object containing information about the user writing the document (if present), and a Security Object with lists of database security roles.
Validation functions typically examine the structure of the new document to ensure that required fields are present and to verify that the requesting user should be allowed to make changes to the document properties. For example, an application may require that a user must be authenticated in order to create a new document or that specific document fields be present when a document is updated. The validation function can abort the pending document write by throwing one of two error objects:
// user is not authorized to make the change but may re-authenticate
throw({ unauthorized: 'Error message here.' });
// change is not allowed
throw({ forbidden: 'Error message here.' });
Document validation is optional, and each design document in the database may have at most one validation function. When a write request is received for a given database, the validation function in each design document in that database is called in an unspecified order. If any of the validation functions throw an error, the write will not succeed.
Example: The _design/_auth ddoc from _users database uses a validation function to ensure that documents contain some required fields and are only modified by a user with the _admin role:
function(newDoc, oldDoc, userCtx, secObj) {
if (newDoc._deleted === true) {
// allow deletes by admins and matching users
// without checking the other fields
if ((userCtx.roles.indexOf('_admin') !== -1) ||
(userCtx.name == oldDoc.name)) {
return;
} else {
throw({forbidden: 'Only admins may delete other user docs.'});
}
}
if ((oldDoc && oldDoc.type !== 'user') || newDoc.type !== 'user') {
throw({forbidden : 'doc.type must be user'});
} // we only allow user docs for now
if (!newDoc.name) {
throw({forbidden: 'doc.name is required'});
}
if (!newDoc.roles) {
throw({forbidden: 'doc.roles must exist'});
}
if (!isArray(newDoc.roles)) {
throw({forbidden: 'doc.roles must be an array'});
}
if (newDoc._id !== ('org.couchdb.user:' + newDoc.name)) {
throw({
forbidden: 'Doc ID must be of the form org.couchdb.user:name'
});
}
if (oldDoc) { // validate all updates
if (oldDoc.name !== newDoc.name) {
throw({forbidden: 'Usernames can not be changed.'});
}
}
if (newDoc.password_sha && !newDoc.salt) {
throw({
forbidden: 'Users with password_sha must have a salt.' +
'See /_utils/script/couch.js for example code.'
});
}
var is_server_or_database_admin = function(userCtx, secObj) {
// see if the user is a server admin
if(userCtx.roles.indexOf('_admin') !== -1) {
return true; // a server admin
}
// see if the user a database admin specified by name
if(secObj && secObj.admins && secObj.admins.names) {
if(secObj.admins.names.indexOf(userCtx.name) !== -1) {
return true; // database admin
}
}
// see if the user a database admin specified by role
if(secObj && secObj.admins && secObj.admins.roles) {
var db_roles = secObj.admins.roles;
for(var idx = 0; idx < userCtx.roles.length; idx++) {
var user_role = userCtx.roles[idx];
if(db_roles.indexOf(user_role) !== -1) {
return true; // role matches!
}
}
}
return false; // default to no admin
}
if (!is_server_or_database_admin(userCtx, secObj)) {
if (oldDoc) { // validate non-admin updates
if (userCtx.name !== newDoc.name) {
throw({
forbidden: 'You may only update your own user document.'
});
}
// validate role updates
var oldRoles = oldDoc.roles.sort();
var newRoles = newDoc.roles.sort();
if (oldRoles.length !== newRoles.length) {
throw({forbidden: 'Only _admin may edit roles'});
}
for (var i = 0; i < oldRoles.length; i++) {
if (oldRoles[i] !== newRoles[i]) {
throw({forbidden: 'Only _admin may edit roles'});
}
}
} else if (newDoc.roles.length > 0) {
throw({forbidden: 'Only _admin may set roles'});
}
}
// no system roles in users db
for (var i = 0; i < newDoc.roles.length; i++) {
if (newDoc.roles[i][0] === '_') {
throw({
forbidden:
'No system roles (starting with underscore) in users db.'
});
}
}
// no system names as names
if (newDoc.name[0] === '_') {
throw({forbidden: 'Username may not start with underscore.'});
}
var badUserNameChars = [':'];
for (var i = 0; i < badUserNameChars.length; i++) {
if (newDoc.name.indexOf(badUserNameChars[i]) >= 0) {
throw({forbidden: 'Character `' + badUserNameChars[i] +
'` is not allowed in usernames.'});
}
}
}
Note
The return statement used only for function, it has no impact on the validation process.
See also
Views are the primary tool used for querying and reporting on CouchDB documents. There you’ll learn how they works and how to use them to build effective applications with CouchDB
Views are useful for many purposes:
Let’s go through the different use cases. First is extracting data that you might need for a special purpose in a specific order. For a front page, we want a list of blog post titles sorted by date. We’ll work with a set of example documents as we walk through how views work:
{
"_id":"biking",
"_rev":"AE19EBC7654",
"title":"Biking",
"body":"My biggest hobby is mountainbiking. The other day...",
"date":"2009/01/30 18:04:11"
}
{
"_id":"bought-a-cat",
"_rev":"4A3BBEE711",
"title":"Bought a Cat",
"body":"I went to the the pet store earlier and brought home a little kitty...",
"date":"2009/02/17 21:13:39"
}
{
"_id":"hello-world",
"_rev":"43FBA4E7AB",
"title":"Hello World",
"body":"Well hello and welcome to my new blog...",
"date":"2009/01/15 15:52:20"
}
Three will do for the example. Note that the documents are sorted by “_id”, which is how they are stored in the database. Now we define a view. Bear with us without an explanation while we show you some code:
function(doc) {
if(doc.date && doc.title) {
emit(doc.date, doc.title);
}
}
This is a map function, and it is written in JavaScript. If you are not familiar with JavaScript but have used C or any other C-like language such as Java, PHP, or C#, this should look familiar. It is a simple function definition.
You provide CouchDB with view functions as strings stored inside the views field of a design document.  Instead, when you query your view, CouchDB takes the source code and runs it for you on every document in the database your view was defined in. You query your view to retrieve the view result.
All map functions have a single parameter doc. This is a single document in the database. Our map function checks whether our document has a date and a title attribute — luckily, all of our documents have them — and then calls the built-in emit() function with these two attributes as arguments.
The emit() function always takes two arguments: the first is key, and the second is value. The emit(key, value) function creates an entry in our view result. One more thing: the emit() function can be called multiple times in the map function to create multiple entries in the view results from a single document, but we are not doing that yet.
CouchDB takes whatever you pass into the emit() function and puts it into a list (see Table 1, “View results” below). Each row in that list includes the key and value. More importantly, the list is sorted by key (by doc.date in our case). The most important feature of a view result is that it is sorted by key. We will come back to that over and over again to do neat things. Stay tuned.
Table 1. View results:
| Key | Value |
|---|---|
| “2009/01/15 15:52:20” | “Hello World” |
| “2009/01/30 18:04:11” | “Biking” |
| “2009/02/17 21:13:39” | “Bought a Cat” |
When you query your view, CouchDB takes the source code and runs it for you on every document in the database. If you have a lot of documents, that takes quite a bit of time and you might wonder if it is not horribly inefficient to do this. Yes, it would be, but CouchDB is designed to avoid any extra costs: it only runs through all documents once, when you first query your view. If a document is changed, the map function is only run once, to recompute the keys and values for that single document.
The view result is stored in a B-tree, just like the structure that is responsible for holding your documents. View B-trees are stored in their own file, so that for high-performance CouchDB usage, you can keep views on their own disk. The B-tree provides very fast lookups of rows by key, as well as efficient streaming of rows in a key range. In our example, a single view can answer all questions that involve time: “Give me all the blog posts from last week” or “last month” or “this year.” Pretty neat.
When we query our view, we get back a list of all documents sorted by date. Each row also includes the post title so we can construct links to posts. Table 1 is just a graphical representation of the view result. The actual result is JSON-encoded and contains a little more metadata:
{
"total_rows": 3,
"offset": 0,
"rows": [
{
"key": "2009/01/15 15:52:20",
"id": "hello-world",
"value": "Hello World"
},
{
"key": "2009/01/30 18:04:11",
"id": "biking",
"value": "Biking"
},
{
"key": "2009/02/17 21:13:39",
"id": "bought-a-cat",
"value": "Bought a Cat"
}
]
}
Now, the actual result is not as nicely formatted and doesn’t include any superfluous whitespace or newlines, but this is better for you (and us!) to read and understand. Where does that “id” member in the result rows come from? That wasn’t there before. That’s because we omitted it earlier to avoid confusion. CouchDB automatically includes the document ID of the document that created the entry in the view result. We’ll use this as well when constructing links to the blog post pages.
Let’s move on to the second use case for views: “building efficient indexes to find documents by any value or structure that resides in them.” We already explained the efficient indexing, but we skipped a few details. This is a good time to finish this discussion as we are looking at map functions that are a little more complex.