Getting changes to documents in the database

Changes in a distributed database

IBM Cloudant databases are distributed. They have shard and fault-tolerant characteristics. These characteristics mean that the responses that are provided by the _changes request might be different from the behavior you expect.

In particular, if you ask for a list of changes _since a sequence identifier, you get the requested information in response. But you might also get changes that were made before the change indicated by the sequence identifier. The reason these extra changes are included, along with the implications for applications, is explained in the Replication guide.

Any application that uses the _changes request must be able to process a list of changes correctly as shown in the following list:

• A different order for the changes that are listed in the response, when compared with an earlier request for the same information.
• Changes that occur before the change specified by the sequence identifier.

The feed argument

The feed argument changes how IBM Cloudant sends the response. By default, _changes reports all changes, then the connection closes. This behavior is the same as using the feed=normal argument.

If you set feed=longpoll, requests sent to the server stay open until changes are reported. This option helps when monitoring changes continuously.

If you set feed=continuous, new changes are reported as they occur. This option means that the database connection stays open for a while. The response may end at any time and clients should reconnect if they wish to continue receiving changes.

Each line in the continuous response is either empty or a JSON object that represents a single change. The option ensures that the following guidelines are met:

• The format of the report entries reflects the continuous nature of the changes.
• Validity of the JSON output is maintained.

See the following example (abbreviated) responses from a continuous changes feed:

{
	"seq": "1-g1A...qyw",
	"id": "2documentation22d01513-c30f-417b-8c27-56b3c0de12ac",
	"changes": [
		{
			"rev": "1-967a00dff5e02add41819138abb3284d"
		}
	]
},
{
	"seq": "2-g1A...ssQ",
	"id": "1documentation22d01513-c30f-417b-8c27-56b3c0de12ac",
	"changes": [
		{
			"rev": "1-967a00dff5e02add41819138abb3284d"
		}
	]
},
{
	"seq": "3-g1A...qyy",
	"id": "1documentation22d01513-c30f-417b-8c27-56b3c0de12ac",
	"changes": [
		{
			"rev": "2-eec205a9d413992850a6e32678485900"
		}
	],
	"deleted": true
},
{
	"seq": "4-g1A...qyz",
	"id": "2documentation22d01513-c30f-417b-8c27-56b3c0de12ac",
	"changes": [
		{
			"rev": "2-eec205a9d413992850a6e32678485900"
		}
	],
	"deleted": true
}

The filter argument

The filter argument designates a pre-defined filter function to apply to the changes feed. Additionally, several built-in filters are available:

_design

The _design filter accepts only changes to design documents.

_doc_ids

This filter accepts only changes for documents whose ID is specified in the doc_ids parameter.

_selector

Returns changes for documents that match the selector request body parameter. The selector syntax is the same as the syntax that is used for _find. If you want to use a selector filter, you must use the POST changes feed (as you can't supply a document body with a GET request). Use the _selector method of filtering instead of the _view filtering method because it's faster and easier to use.

For more information, see the API documentation.

_view

Enables use of an existing map function as the filter.

The since argument

Use the since argument to get a list of changes that occurred after a specified sequence identifier. If the since identifier is 0 (the default), or omitted, the request returns all changes. If the since identifier is now, the request asks for changes that are made after the current time.

The distributed nature of IBM Cloudant can affect the results that you get in a response. For example, if you request a list of changes twice, by using the same since sequence identifier both times, the order of changes in the resulting list might not be the same.

You might also see some results that appear to be from before the since parameter. The reason is that you might be getting results from a different replica of a shard (a shard replica).

Shard replicas automatically and continuously replicate to each other and eventually have the same data. However, at any point in time, a shard replica might differ from another shard replica because the replication between them isn't yet complete.

When you request a list of changes, normally the same replicas are used to respond. But if the node that holds the shard replica isn't available, the system substitutes a corresponding shard replica that is held on another node. To help ensure that you see all the applicable changes, the most recent checkpoint between the replicas is used. Using the checkpoint is effectively "rolling back" the list of changes to the most recent point in time when the shard replicas were confirmed to agree with each other. This "rolling back" means you might see changes listed that took place "before" the since sequence identifier you supplied.

Your application must be able to handle a change that is reported more than one time if you make a _changes request several times.

For more information about the behavior of the _changes response, see the replication guide.

Responses from the _changes request

The response from a _changes request is a JSON object that contains a list of the changes that were made to documents within the database. The following table describes the meaning of the individual fields:

See the following example (abbreviated) response to a _changes request:

{
	"results": [
		{
			"seq": "1-g1A...sIg",
			"id": "foo",
			"changes": [
				{
					"rev": "1-967...84d"
				}
			]
		}
	],
	"last_seq": "1-g1A...sIg",
	"pending": 0
}

Important notes about _changes

• The results that are returned by _changes are partially ordered. In other words, the order might not be preserved for multiple calls. You might decide to get a current list by using _changes and including the last_seq value. The resulting list provides the starting point for subsequent _changes lists that use the since query argument.
• Although shard copies of the same range contain the same data, their _changes history is often unique. This difference is a result of how writes were applied to the shard. For example, they might be applied in a different order. To be sure that all changes are reported for your specified sequence, it might be necessary to go further back into the shard's history to find a suitable starting point. The changes are then reported from that starting point. This "rolling back" might give the appearance of duplicate updates, or updates that are apparently before the specified since value.
• _changes reported by a shard are always presented in order. But the ordering between all the contributing shards might appear to be different. For more information, see A Changes Feed Example.
• Sequence values are unique for a shard, but might vary between shards. This variation means that, if you have sequence values from different shards, you can't assume that the same sequence value refers to the same document within the different shards.

Using POST to get changes

Instead of GET, you can also use POST to query the changes feed. The only difference, if you're using POST and you're using either of the docs_ids or selector filters, is that it's possible to include the "doc_ids" : [...] or "selector": {...} parts in the request body. All other parameters are expected to be in the query string, the same as using GET.

See the following example that uses HTTP to POST to the _changes endpoint:

POST /$DATABASE/_changes?filter=_selector HTTP/1.1
Host: $ACCOUNT.cloudant.com
Content-Type: application/json

See the following example to POST to the _changes endpoint:

curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X POST "$SERVICE_URL/orders/_changes" -H "Content-Type: application/json"'

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.ChangesResult;
import com.ibm.cloud.cloudant.v1.model.PostChangesOptions;

Cloudant service = Cloudant.newInstance();

PostChangesOptions changesOptions = new PostChangesOptions.Builder()
    .db("orders")
    .build();

ChangesResult response =
    service.postChanges(changesOptions).execute()
        .getResult();

System.out.println(response);

import { CloudantV1 } from '@ibm-cloud/cloudant';

const service = CloudantV1.newInstance({});

service.postChanges({
  db: 'orders'
}).then(response => {
  console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()
response = service.post_changes(
  db='orders'
).get_result()

print(response)

postChangesOptions := service.NewPostChangesOptions(
  "orders",
)

changesResult, response, err := service.PostChanges(postChangesOptions)
if err != nil {
  panic(err)
}

b, _ := json.MarshalIndent(changesResult, "", "  ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
   "encoding/json"
   "fmt"
   "github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's Authentication section for examples.

When you POST to the _changes endpoint, you see an example similar to the following JSON object:

{"results":[
{"seq":"1-g1AAAA...","id":"0007741142412418284","changes":[{"rev":"1-9d0c2676941ec3a3b3cc2f08fe9a51e0"}]},
{"seq":"2-g1AAAA...","id":"_design/applianceId","changes":[{"rev":"1-b1f67a8b672c1324680d6d7dc1e1fd3c"}]},
...
],
"last_seq":"18-g1AAAA...","pending":0}