Document APIs
This section starts with a short introduction to OpenSearch’s data replication model, followed by a detailed description of the following CRUD APIs:
|
Note
|
All CRUD APIs are single-index APIs. The index parameter accepts a single
index name, or an alias which points to a single index.
|
Reading and Writing documents
Introduction
Each index in OpenSearch is divided into shards and each shard can have multiple copies. These copies are known as a replication group and must be kept in sync when documents are added or removed. If we fail to do so, reading from one copy will result in very different results than reading from another. The process of keeping the shard copies in sync and serving reads from them is what we call the data replication model.
OpenSearch’s data replication model is based on the primary-backup model and is described very well in the PacificA paper of Microsoft Research. That model is based on having a single copy from the replication group that acts as the primary shard. The other copies are called replica shards. The primary serves as the main entry point for all indexing operations. It is in charge of validating them and making sure they are correct. Once an index operation has been accepted by the primary, the primary is also responsible for replicating the operation to the other copies.
This purpose of this section is to give a high level overview of the OpenSearch replication model and discuss the implications it has for various interactions between write and read operations.
Basic write model
Every indexing operation in OpenSearch is first resolved to a replication group using routing, typically based on the document ID. Once the replication group has been determined, the operation is forwarded internally to the current primary shard of the group. This stage of indexing is referred to as the coordinating stage.
The next stage of indexing is the primary stage, performed on the primary shard. The primary shard is responsible for validating the operation and forwarding it to the other replicas. Since replicas can be offline, the primary is not required to replicate to all replicas. Instead, OpenSearch maintains a list of shard copies that should receive the operation. This list is called the in-sync copies and is maintained by the master node. As the name implies, these are the set of "good" shard copies that are guaranteed to have processed all of the index and delete operations that have been acknowledged to the user. The primary is responsible for maintaining this invariant and thus has to replicate all operations to each copy in this set.
The primary shard follows this basic flow:
-
Validate incoming operation and reject it if structurally invalid (Example: have an object field where a number is expected)
-
Execute the operation locally i.e. indexing or deleting the relevant document. This will also validate the content of fields and reject if needed (Example: a keyword value is too long for indexing in Lucene).
-
Forward the operation to each replica in the current in-sync copies set. If there are multiple replicas, this is done in parallel.
-
Once all replicas have successfully performed the operation and responded to the primary, the primary acknowledges the successful completion of the request to the client.
Each in-sync replica copy performs the indexing operation locally so that it has a copy. This stage of indexing is the replica stage.
These indexing stages (coordinating, primary, and replica) are sequential. To enable internal retries, the lifetime of each stage encompasses the lifetime of each subsequent stage. For example, the coordinating stage is not complete until each primary stage, which may be spread out across different primary shards, has completed. Each primary stage will not complete until the in-sync replicas have finished indexing the docs locally and responded to the replica requests.
Failure handling
Many things can go wrong during indexing — disks can get corrupted, nodes can be disconnected from each other, or some configuration mistake could cause an operation to fail on a replica despite it being successful on the primary. These are infrequent but the primary has to respond to them.
In the case that the primary itself fails, the node hosting the primary will send a message to the master about it. The indexing operation will wait (up to 1 minute, by default) for the master to promote one of the replicas to be a new primary. The operation will then be forwarded to the new primary for processing. Note that the master also monitors the health of the nodes and may decide to proactively demote a primary. This typically happens when the node holding the primary is isolated from the cluster by a networking issue. See here for more details.
Once the operation has been successfully performed on the primary, the primary has to deal with potential failures when executing it on the replica shards. This may be caused by an actual failure on the replica or due to a network issue preventing the operation from reaching the replica (or preventing the replica from responding). All of these share the same end result: a replica which is part of the in-sync replica set misses an operation that is about to be acknowledged. In order to avoid violating the invariant, the primary sends a message to the master requesting that the problematic shard be removed from the in-sync replica set. Only once removal of the shard has been acknowledged by the master does the primary acknowledge the operation. Note that the master will also instruct another node to start building a new shard copy in order to restore the system to a healthy state.
While forwarding an operation to the replicas, the primary will use the replicas to validate that it is still the active primary. If the primary has been isolated due to a network partition (or a long GC) it may continue to process incoming indexing operations before realising that it has been demoted. Operations that come from a stale primary will be rejected by the replicas. When the primary receives a response from the replica rejecting its request because it is no longer the primary then it will reach out to the master and will learn that it has been replaced. The operation is then routed to the new primary.
Basic read model
Reads in OpenSearch can be very lightweight lookups by ID or a heavy search request with complex aggregations that take non-trivial CPU power. One of the beauties of the primary-backup model is that it keeps all shard copies identical (with the exception of in-flight operations). As such, a single in-sync copy is sufficient to serve read requests.
When a read request is received by a node, that node is responsible for forwarding it to the nodes that hold the relevant shards, collating the responses, and responding to the client. We call that node the coordinating node for that request. The basic flow is as follows:
-
Resolve the read requests to the relevant shards. Note that since most searches will be sent to one or more indices, they typically need to read from multiple shards, each representing a different subset of the data.
-
Select an active copy of each relevant shard, from the shard replication group. This can be either the primary or a replica. By default, OpenSearch uses adaptive replica selection to select the shard copies.
-
Send shard level read requests to the selected copies.
-
Combine the results and respond. Note that in the case of get by ID look up, only one shard is relevant and this step can be skipped.
Shard failures
When a shard fails to respond to a read request, the coordinating node sends the request to another shard copy in the same replication group. Repeated failures can result in no available shard copies.
To ensure fast responses, the following APIs will respond with partial results if one or more shards fail:
Responses containing partial results still provide a 200 OK
HTTP status code.
Shard failures are indicated by the timed_out
and _shards
fields of
the response header.
A few simple implications
Each of these basic flows determines how OpenSearch behaves as a system for both reads and writes. Furthermore, since read and write requests can be executed concurrently, these two basic flows interact with each other. This has a few inherent implications:
- Efficient reads
-
Under normal operation each read operation is performed once for each relevant replication group. Only under failure conditions do multiple copies of the same shard execute the same search.
- Read unacknowledged
-
Since the primary first indexes locally and then replicates the request, it is possible for a concurrent read to already see the change before it has been acknowledged.
- Two copies by default
-
This model can be fault tolerant while maintaining only two copies of the data. This is in contrast to quorum-based system where the minimum number of copies for fault tolerance is 3.
Failures
Under failures, the following is possible:
- A single shard can slow down indexing
-
Because the primary waits for all replicas in the in-sync copies set during each operation, a single slow shard can slow down the entire replication group. This is the price we pay for the read efficiency mentioned above. Of course a single slow shard will also slow down unlucky searches that have been routed to it.
- Dirty reads
-
An isolated primary can expose writes that will not be acknowledged. This is caused by the fact that an isolated primary will only realize that it is isolated once it sends requests to its replicas or when reaching out to the master. At that point the operation is already indexed into the primary and can be read by a concurrent read. OpenSearch mitigates this risk by pinging the master every second (by default) and rejecting indexing operations if no master is known.
Index API
Adds a JSON document to the specified index and makes it searchable. If the target is an index and the document already exists, the request updates the document and increments its version.
Request
PUT /<target>/_doc/<_id>
POST /<target>/_doc/
PUT /<target>/_create/<_id>
POST /<target>/_create/<_id>
Path parameters
<target>
-
(Required, string) Name of the index to target.
If the target doesn’t exist, this request creates the index.
You can check for existing targets using the resolve index API.
<_id>
-
(Optional, string) Unique identifier for the document.
This parameter is required for the following request formats:
-
PUT /<target>/_doc/<_id>
-
PUT /<target>/_create/<_id>
-
POST /<target>/_create/<_id>
To automatically generate a document ID, use the
POST /<target>/_doc/
request format and omit this parameter. -
Query parameters
if_seq_no
-
(Optional, integer) Only perform the operation if the document has this sequence number. See Optimistic concurrency control.
if_primary_term
-
(Optional, integer) Only perform the operation if the document has this primary term. See Optimistic concurrency control.
op_type
-
(Optional, enum) Set to
create
to only index the document if it does not already exist (put if absent). If a document with the specified_id
already exists, the indexing operation will fail. Same as using the<index>/_create
endpoint. Valid values:index
,create
. If document id is specified, it defaults toindex
. Otherwise, it defaults tocreate
. pipeline
-
(Optional, string) ID of the pipeline to use to preprocess incoming documents.
refresh
-
(Optional, enum) If
true
, OpenSearch refreshes the affected shards to make this operation visible to search, ifwait_for
then wait for a refresh to make this operation visible to search, iffalse
do nothing with refreshes. Valid values:true
,false
,wait_for
. Default:false
. routing
-
(Optional, string) Target the specified primary shard.
timeout
-
(Optional, time units) Period the request waits for the following operations:
Defaults to
1m
(one minute). This guarantees OpenSearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur. version
-
(Optional, integer) Explicit version number for concurrency control. The specified version must match the current version of the document for the request to succeed.
version_type
-
(Optional, enum) Specific version type:
internal
,external
,external_gte
. wait_for_active_shards
-
(Optional, string) The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). Default: 1, the primary shard.See Active shards.
require_alias
-
(Optional, Boolean) If
true
, the destination must be an index alias. Defaults tofalse
.
Request body
<field>
-
(Required, string) Request body contains the JSON source for the document data.
Response body
_shards
-
Provides information about the replication process of the index operation.
_shards.total
-
Indicates how many shard copies (primary and replica shards) the index operation should be executed on.
_shards.successful
-
Indicates the number of shard copies the index operation succeeded on. When the index operation is successful,
successful
is at least 1.NoteReplica shards might not all be started when an indexing operation returns successfully—by default, only the primary is required. Set wait_for_active_shards
to change this default behavior. See Active shards. _shards.failed
-
An array that contains replication-related errors in the case an index operation failed on a replica shard. 0 indicates there were no failures.
_index
-
The name of the index the document was added to.
_type
-
The document type. OpenSearch indices now support a single document type,
_doc
. _id
-
The unique identifier for the added document.
_version
-
The document version. Incremented each time the document is updated.
_seq_no
-
The sequence number assigned to the document for the indexing operation. Sequence numbers are used to ensure an older version of a document doesn’t overwrite a newer version. See Optimistic concurrency control.
_primary_term
-
The primary term assigned to the document for the indexing operation. See Optimistic concurrency control.
result
-
The result of the indexing operation,
created
orupdated
.
Description
You can index a new JSON document with the _doc
or _create
resource. Using
_create
guarantees that the document is only indexed if it does not already
exist. To update an existing document, you must use the _doc
resource.
Automatically create indices
If the target doesn’t exist, the operation automatically creates the index and applies any matching index templates.
If no mapping exists, the index operation creates a dynamic mapping. By default, new fields and objects are automatically added to the mapping if needed. For more information about field mapping, see mapping and the put mapping API.
Automatic index creation is controlled by the action.auto_create_index
setting. This setting defaults to true
, which allows any index to be created
automatically. You can modify this setting to explicitly allow or block
automatic creation of indices that match specified patterns, or set it to
false
to disable automatic index creation entirely. Specify a
comma-separated list of patterns you want to allow, or prefix each pattern with
+
or -
to indicate whether it should be allowed or blocked. When a list is
specified, the default behaviour is to disallow.
PUT _cluster/settings
{
"persistent": {
"action.auto_create_index": "my-index-000001,index10,-index1*,+ind*" (1)
}
}
PUT _cluster/settings
{
"persistent": {
"action.auto_create_index": "false" (2)
}
}
PUT _cluster/settings
{
"persistent": {
"action.auto_create_index": "true" (3)
}
}
-
Allow auto-creation of indices called
my-index-000001
orindex10
, block the creation of indices that match the patternindex1*
, and allow creation of any other indices that match theind*
pattern. Patterns are matched in the order specified. -
Disable automatic index creation entirely.
-
Allow automatic creation of any index. This is the default.
Put if absent
You can force a create operation by using the create
resource or
setting the op_type
parameter to _create. In this case,
the index operation fails if a document with the specified ID
already exists in the index.
Create document IDs automatically
When using the POST /<target>/_doc/
request format, the op_type
is
automatically set to create
and the index operation generates a unique ID for
the document.
POST my-index-000001/_doc/
{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1<PHONE_NUMBER>",
"user": {
"id": "bilbo"
}
}
The API returns the following result:
{
"_shards": {
"total": 2,
"failed": 0,
"successful": 2
},
"_index": "my-index-000001",
"_type": "_doc",
"_id": "W0tpsmIBdwcYyG50zbta",
"_version": 1,
"_seq_no": 0,
"_primary_term": 1,
"result": "created"
}
Optimistic concurrency control
Index operations can be made conditional and only be performed if the last
modification to the document was assigned the sequence number and primary
term specified by the if_seq_no
and if_primary_term
parameters. If a
mismatch is detected, the operation will result in a VersionConflictException
and a status code of 409. See Optimistic concurrency control for more details.
Routing
By default, shard placement — or routing
— is controlled by using a
hash of the document’s id value. For more explicit control, the value
fed into the hash function used by the router can be directly specified
on a per-operation basis using the routing
parameter. For example:
POST my-index-000001/_doc?routing=bilbo
{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1<PHONE_NUMBER>",
"user": {
"id": "bilbo"
}
}
In this example, the document is routed to a shard based on
the routing
parameter provided: "bilbo".
When setting up explicit mapping, you can also use the _routing
field
to direct the index operation to extract the routing value from the
document itself. This does come at the (very minimal) cost of an
additional document parsing pass. If the _routing
mapping is defined
and set to be required
, the index operation will fail if no routing
value is provided or extracted.
Distributed
The index operation is directed to the primary shard based on its route (see the Routing section above) and performed on the actual node containing this shard. After the primary shard completes the operation, if needed, the update is distributed to applicable replicas.
Active shards
To improve the resiliency of writes to the system, indexing operations
can be configured to wait for a certain number of active shard copies
before proceeding with the operation. If the requisite number of active
shard copies are not available, then the write operation must wait and
retry, until either the requisite shard copies have started or a timeout
occurs. By default, write operations only wait for the primary shards
to be active before proceeding (i.e. wait_for_active_shards=1
).
This default can be overridden in the index settings dynamically
by setting index.write.wait_for_active_shards
. To alter this behavior
per operation, the wait_for_active_shards
request parameter can be used.
Valid values are all
or any positive integer up to the total number
of configured copies per shard in the index (which is number_of_replicas+1
).
Specifying a negative value or a number greater than the number of
shard copies will throw an error.
For example, suppose we have a cluster of three nodes, A
, B
, and C
and
we create an index index
with the number of replicas set to 3 (resulting in
4 shard copies, one more copy than there are nodes). If we
attempt an indexing operation, by default the operation will only ensure
the primary copy of each shard is available before proceeding. This means
that even if B
and C
went down, and A
hosted the primary shard copies,
the indexing operation would still proceed with only one copy of the data.
If wait_for_active_shards
is set on the request to 3
(and all 3 nodes
are up), then the indexing operation will require 3 active shard copies
before proceeding, a requirement which should be met because there are 3
active nodes in the cluster, each one holding a copy of the shard. However,
if we set wait_for_active_shards
to all
(or to 4
, which is the same),
the indexing operation will not proceed as we do not have all 4 copies of
each shard active in the index. The operation will timeout
unless a new node is brought up in the cluster to host the fourth copy of
the shard.
It is important to note that this setting greatly reduces the chances of
the write operation not writing to the requisite number of shard copies,
but it does not completely eliminate the possibility, because this check
occurs before the write operation commences. Once the write operation
is underway, it is still possible for replication to fail on any number of
shard copies but still succeed on the primary. The _shards
section of the
write operation’s response reveals the number of shard copies on which
replication succeeded/failed.
{
"_shards": {
"total": 2,
"failed": 0,
"successful": 2
}
}
Refresh
Control when the changes made by this request are visible to search. See refresh.
Noop updates
When updating a document using the index API a new version of the document is
always created even if the document hasn’t changed. If this isn’t acceptable
use the _update
API with detect_noop
set to true. This option isn’t
available on the index API because the index API doesn’t fetch the old source
and isn’t able to compare it against the new source.
There isn’t a hard and fast rule about when noop updates aren’t acceptable. It’s a combination of lots of factors like how frequently your data source sends updates that are actually noops and how many queries per second OpenSearch runs on the shard receiving the updates.
Timeout
The primary shard assigned to perform the index operation might not be
available when the index operation is executed. Some reasons for this
might be that the primary shard is currently recovering from a gateway
or undergoing relocation. By default, the index operation will wait on
the primary shard to become available for up to 1 minute before failing
and responding with an error. The timeout
parameter can be used to
explicitly specify how long it waits. Here is an example of setting it
to 5 minutes:
PUT my-index-000001/_doc/1?timeout=5m
{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1<PHONE_NUMBER>",
"user": {
"id": "bilbo"
}
}
Versioning
Each indexed document is given a version number. By default,
internal versioning is used that starts at 1 and increments
with each update, deletes included. Optionally, the version number can be
set to an external value (for example, if maintained in a
database). To enable this functionality, version_type
should be set to
external
. The value provided must be a numeric, long value greater than or equal to 0,
and less than around 9.2e+18.
When using the external version type, the system checks to see if the version number passed to the index request is greater than the version of the currently stored document. If true, the document will be indexed and the new version number used. If the value provided is less than or equal to the stored document’s version number, a version conflict will occur and the index operation will fail. For example:
PUT my-index-000001/_doc/1?version=2&version_type=external
{
"user": {
"id": "elkbee"
}
}
|
Note
|
Versioning is completely real time, and is not affected by the near real time aspects of search operations. If no version is provided, then the operation is executed without any version checks. |
In the previous example, the operation will succeed since the supplied version of 2 is higher than the current document version of 1. If the document was already updated and its version was set to 2 or higher, the indexing command will fail and result in a conflict (409 http status code).
A nice side effect is that there is no need to maintain strict ordering of async indexing operations executed as a result of changes to a source database, as long as version numbers from the source database are used. Even the simple case of updating the OpenSearch index using data from a database is simplified if external versioning is used, as only the latest version will be used if the index operations arrive out of order for whatever reason.
Version types
In addition to the external
version type, OpenSearch
also supports other types for specific use cases:
internal
-
Only index the document if the given version is identical to the version of the stored document.
external
orexternal_gt
-
Only index the document if the given version is strictly higher than the version of the stored document or if there is no existing document. The given version will be used as the new version and will be stored with the new document. The supplied version must be a non-negative long number.
external_gte
-
Only index the document if the given version is equal or higher than the version of the stored document. If there is no existing document the operation will succeed as well. The given version will be used as the new version and will be stored with the new document. The supplied version must be a non-negative long number.
|
Note
|
The external_gte version type is meant for special use cases and
should be used with care. If used incorrectly, it can result in loss of data.
There is another option, force , which is deprecated because it can cause
primary and replica shards to diverge.
|
Examples
Insert a JSON document into the my-index-000001
index with an _id
of 1:
PUT my-index-000001/_doc/1
{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1<PHONE_NUMBER>",
"user": {
"id": "bilbo"
}
}
The API returns the following result:
{
"_shards": {
"total": 2,
"failed": 0,
"successful": 2
},
"_index": "my-index-000001",
"_type": "_doc",
"_id": "1",
"_version": 1,
"_seq_no": 0,
"_primary_term": 1,
"result": "created"
}
Use the _create
resource to index a document into the my-index-000001
index if
no document with that ID exists:
PUT my-index-000001/_create/1
{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1<PHONE_NUMBER>",
"user": {
"id": "bilbo"
}
}
Set the op_type
parameter to create to index a document into the my-index-000001
index if no document with that ID exists:
PUT my-index-000001/_doc/1?op_type=create
{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1<PHONE_NUMBER>",
"user": {
"id": "bilbo"
}
}
Get API
Retrieves the specified JSON document from an index.
GET my-index-000001/_doc/0
Request
GET <index>/_doc/<_id>
HEAD <index>/_doc/<_id>
GET <index>/_source/<_id>
HEAD <index>/_source/<_id>
Description
You use GET to retrieve a document and its source or stored fields from a
particular index. Use HEAD to verify that a document exists. You can
use the _source
resource retrieve just the document source or verify
that it exists.
Realtime
By default, the get API is realtime, and is not affected by the refresh
rate of the index (when data will become visible for search). In case where
stored fields are requested (see stored_fields
parameter) and the document
has been updated but is not yet refreshed, the get API will have to parse
and analyze the source to extract the stored fields. In order to disable
realtime GET, the realtime
parameter can be set to false
.
Source filtering
By default, the get operation returns the contents of the _source
field unless
you have used the stored_fields
parameter or if the _source
field is disabled.
You can turn off _source
retrieval by using the _source
parameter:
GET my-index-000001/_doc/0?_source=false
If you only need one or two fields from the _source
, use the _source_includes
or _source_excludes
parameters to include or filter out particular fields.
This can be especially helpful with large documents where partial retrieval can
save on network overhead. Both parameters take a comma separated list
of fields or wildcard expressions. Example:
GET my-index-000001/_doc/0?_source_includes=*.id&_source_excludes=entities
If you only want to specify includes, you can use a shorter notation:
GET my-index-000001/_doc/0?_source=*.id
Routing
If routing is used during indexing, the routing value also needs to be specified to retrieve a document. For example:
GET my-index-000001/_doc/2?routing=user1
This request gets the document with id 2
, but it is routed based on the
user. The document is not fetched if the correct routing is not specified.
Preference
Controls a preference
of which shard replicas to execute the get
request on. By default, the operation is randomized between the shard
replicas.
The preference
can be set to:
_local
-
The operation will prefer to be executed on a local allocated shard if possible.
- Custom (string) value
-
A custom value will be used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session id, or the user name.
Refresh
The refresh
parameter can be set to true
in order to refresh the
relevant shard before the get operation and make it searchable. Setting
it to true
should be done after careful thought and verification that
this does not cause a heavy load on the system (and slows down
indexing).
Distributed
The get operation gets hashed into a specific shard id. It then gets redirected to one of the replicas within that shard id and returns the result. The replicas are the primary shard and its replicas within that shard id group. This means that the more replicas we have, the better GET scaling we will have.
Versioning support
You can use the version
parameter to retrieve the document only if
its current version is equal to the specified one.
Internally, OpenSearch has marked the old document as deleted and added an entirely new document. The old version of the document doesn’t disappear immediately, although you won’t be able to access it. OpenSearch cleans up deleted documents in the background as you continue to index more data.
Path parameters
<index>
-
(Required, string) Name of the index that contains the document.
<_id>
-
(Required, string) Unique identifier of the document.
Query parameters
preference
-
(Optional, string) Specifies the node or shard the operation should be performed on. Random by default.
realtime
-
(Optional, Boolean) If
true
, the request is real-time as opposed to near-real-time. Defaults totrue
. See Realtime. refresh
-
(Optional, Boolean) If
true
, OpenSearch refreshes the affected shards to make this operation visible to search. Iffalse
, do nothing with refreshes. Defaults tofalse
. routing
-
(Optional, string) Target the specified primary shard.
stored_fields
-
(Optional, Boolean) If
true
, retrieves the document fields stored in the index rather than the document_source
. Defaults tofalse
. _source
-
(Optional, string) True or false to return the
_source
field or not, or a list of fields to return. _source_excludes
-
(Optional, string) A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. _source_includes
-
(Optional, string) A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. version
-
(Optional, integer) Explicit version number for concurrency control. The specified version must match the current version of the document for the request to succeed.
version_type
-
(Optional, enum) Specific version type:
internal
,external
,external_gte
.
Response body
_index
-
The name of the index the document belongs to.
_type
-
The document type. OpenSearch indices now support a single document type,
_doc
. _id
-
The unique identifier for the document.
_version
-
The document version. Incremented each time the document is updated.
_seq_no
-
The sequence number assigned to the document for the indexing operation. Sequence numbers are used to ensure an older version of a document doesn’t overwrite a newer version. See Optimistic concurrency control.
_primary_term
-
The primary term assigned to the document for the indexing operation. See Optimistic concurrency control.
found
-
Indicates whether the document exists:
true
orfalse
. _routing
-
The explicit routing, if set.
- '_source'
-
If
found
istrue
, contains the document data formatted in JSON. Excluded if the_source
parameter is set tofalse
or thestored_fields
parameter is set totrue
. - '_fields'
-
If the
stored_fields
parameter is set totrue
andfound
istrue
, contains the document fields stored in the index.
Examples
Retrieve the JSON document with the _id
0 from the my-index-000001
index:
GET my-index-000001/_doc/0
The API returns the following result:
{
"_index": "my-index-000001",
"_type": "_doc",
"_id": "0",
"_version": 1,
"_seq_no": 0,
"_primary_term": 1,
"found": true,
"_source": {
"@timestamp": "2099-11-15T14:12:12",
"http": {
"request": {
"method": "get"
},
"response": {
"status_code": 200,
"bytes": 1070000
},
"version": "1.1"
},
"source": {
"ip": "<IP_ADDRESS>"
},
"message": "GET /search HTTP/1.1<PHONE_NUMBER>",
"user": {
"id": "bilbo"
}
}
}
Check to see if a document with the _id
0 exists:
HEAD my-index-000001/_doc/0
OpenSearch returns a status code of 200 - OK
if the document exists, or
404 - Not Found
if it doesn’t.
Get the source field only
Use the <index>/_source/<id>
resource to get
just the _source
field of a document. For example:
GET my-index-000001/_source/1
You can use the source filtering parameters to control which parts of the
_source
are returned:
GET my-index-000001/_source/1/?_source_includes=*.id&_source_excludes=entities
You can use HEAD with the _source
endpoint to efficiently
test whether or not the document _source exists. A document’s source is not
available if it is disabled in the mapping.
HEAD my-index-000001/_source/1
Get stored fields
Use the stored_fields
parameter to specify the set of stored fields you want
to retrieve. Any requested fields that are not stored are ignored.
Consider for instance the following mapping:
PUT my-index-000001
{
"mappings": {
"properties": {
"counter": {
"type": "integer",
"store": false
},
"tags": {
"type": "keyword",
"store": true
}
}
}
}
Now we can add a document:
PUT my-index-000001/_doc/1
{
"counter": 1,
"tags": [ "production" ]
}
And then try to retrieve it:
GET my-index-000001/_doc/1?stored_fields=tags,counter
The API returns the following result:
{
"_index": "my-index-000001",
"_type": "_doc",
"_id": "1",
"_version": 1,
"_seq_no" : 22,
"_primary_term" : 1,
"found": true,
"fields": {
"tags": [
"production"
]
}
}
Field values fetched from the document itself are always returned as an array.
Since the counter
field is not stored, the get request ignores it.
You can also retrieve metadata fields like the _routing
field:
PUT my-index-000001/_doc/2?routing=user1
{
"counter" : 1,
"tags" : ["env2"]
}
GET my-index-000001/_doc/2?routing=user1&stored_fields=tags,counter
The API returns the following result:
{
"_index": "my-index-000001",
"_type": "_doc",
"_id": "2",
"_version": 1,
"_seq_no" : 13,
"_primary_term" : 1,
"_routing": "user1",
"found": true,
"fields": {
"tags": [
"env2"
]
}
}
Only leaf fields can be retrieved with the stored_field
option. Object fields
can’t be returned—if specified, the request fails.
Delete API
Removes a JSON document from the specified index.
Request
DELETE /<index>/_doc/<_id>
Description
You use DELETE to remove a document from an index. You must specify the index name and document ID.
Optimistic concurrency control
Delete operations can be made conditional and only be performed if the last
modification to the document was assigned the sequence number and primary
term specified by the if_seq_no
and if_primary_term
parameters. If a
mismatch is detected, the operation will result in a VersionConflictException
and a status code of 409. See Optimistic concurrency control for more details.
Versioning
Each document indexed is versioned. When deleting a document, the version
can
be specified to make sure the relevant document we are trying to delete is
actually being deleted and it has not changed in the meantime. Every write
operation executed on a document, deletes included, causes its version to be
incremented. The version number of a deleted document remains available for a
short time after deletion to allow for control of concurrent operations. The
length of time for which a deleted document’s version remains available is
determined by the index.gc_deletes
index setting and defaults to 60 seconds.
Routing
If routing is used during indexing, the routing value also needs to be specified to delete a document.
If the _routing
mapping is set to required
and no routing value is
specified, the delete API throws a RoutingMissingException
and rejects
the request.
For example:
DELETE /my-index-000001/_doc/1?routing=shard-1
This request deletes the document with id 1
, but it is routed based on the
user. The document is not deleted if the correct routing is not specified.
Automatic index creation
If an external versioning variant is used, the delete operation automatically creates the specified index if it does not exist. For information about manually creating indices, see create index API.
Distributed
The delete operation gets hashed into a specific shard id. It then gets redirected into the primary shard within that id group, and replicated (if needed) to shard replicas within that id group.
Wait for active shards
When making delete requests, you can set the wait_for_active_shards
parameter to require a minimum number of shard copies to be active
before starting to process the delete request. See
here for further details and a usage
example.
Refresh
Control when the changes made by this request are visible to search. See
?refresh
.
Timeout
The primary shard assigned to perform the delete operation might not be
available when the delete operation is executed. Some reasons for this
might be that the primary shard is currently recovering from a store
or undergoing relocation. By default, the delete operation will wait on
the primary shard to become available for up to 1 minute before failing
and responding with an error. The timeout
parameter can be used to
explicitly specify how long it waits. Here is an example of setting it
to 5 minutes:
DELETE /my-index-000001/_doc/1?timeout=5m
Path parameters
<index>
-
(Required, string) Name of the target index.
<_id>
-
(Required, string) Unique identifier for the document.
Query parameters
if_seq_no
-
(Optional, integer) Only perform the operation if the document has this sequence number. See Optimistic concurrency control.
if_primary_term
-
(Optional, integer) Only perform the operation if the document has this primary term. See Optimistic concurrency control.
pipeline
-
(Optional, string) ID of the pipeline to use to preprocess incoming documents.
refresh
-
(Optional, enum) If
true
, OpenSearch refreshes the affected shards to make this operation visible to search, ifwait_for
then wait for a refresh to make this operation visible to search, iffalse
do nothing with refreshes. Valid values:true
,false
,wait_for
. Default:false
. routing
-
(Optional, string) Target the specified primary shard.
timeout
-
(Optional, time units) Period to wait for active shards. Defaults to
1m
(one minute). version
-
(Optional, integer) Explicit version number for concurrency control. The specified version must match the current version of the document for the request to succeed.
version_type
-
(Optional, enum) Specific version type:
internal
,external
,external_gte
. wait_for_active_shards
-
(Optional, string) The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). Default: 1, the primary shard.See Active shards.
Examples
Delete the JSON document 1
from the my-index-000001
index:
DELETE /my-index-000001/_doc/1
The API returns the following result:
{
"_shards": {
"total": 2,
"failed": 0,
"successful": 2
},
"_index": "my-index-000001",
"_type": "_doc",
"_id": "1",
"_version": 2,
"_primary_term": 1,
"_seq_no": 5,
"result": "deleted"
}
Delete by query API
Deletes documents that match the specified query.
POST /my-index-000001/_delete_by_query
{
"query": {
"match": {
"user.id": "elkbee"
}
}
}
Request
POST /<target>/_delete_by_query
Description
You can specify the query criteria in the request URI or the request body using the same syntax as the Search API.
When you submit a delete by query request, OpenSearch gets a snapshot of the index
when it begins processing the request and deletes matching documents using
internal
versioning. If a document changes between the time that the
snapshot is taken and the delete operation is processed, it results in a version
conflict and the delete operation fails.
|
Note
|
Documents with a version equal to 0 cannot be deleted using delete by
query because internal versioning does not support 0 as a valid
version number.
|
While processing a delete by query request, OpenSearch performs multiple search requests sequentially to find all of the matching documents to delete. A bulk delete request is performed for each batch of matching documents. If a search or bulk request is rejected, the requests are retried up to 10 times, with exponential back off. If the maximum retry limit is reached, processing halts and all failed requests are returned in the response. Any delete requests that completed successfully still stick, they are not rolled back.
You can opt to count version conflicts instead of halting and returning by
setting conflicts
to proceed
.
Refreshing shards
Specifying the refresh
parameter refreshes all shards involved in the delete
by query once the request completes. This is different than the delete API’s
refresh
parameter, which causes just the shard that received the delete
request to be refreshed. Unlike the delete API, it does not support
wait_for
.
Running delete by query asynchronously
If the request contains wait_for_completion=false
, OpenSearch
performs some preflight checks, launches the request, and returns a
task
you can use to cancel or get the status of the task. OpenSearch creates a
record of this task as a document at .tasks/task/${taskId}
. When you are
done with a task, you should delete the task document so OpenSearch can reclaim the
space.
Waiting for active shards
wait_for_active_shards
controls how many copies of a shard must be active
before proceeding with the request. See Active shards
for details. timeout
controls how long each write request waits for unavailable
shards to become available. Both work exactly the way they work in the
Bulk API. Delete by query uses scrolled searches, so you can also
specify the scroll
parameter to control how long it keeps the search context
alive, for example ?scroll=10m
. The default is 5 minutes.
Throttling delete requests
To control the rate at which delete by query issues batches of delete operations,
you can set requests_per_second
to any positive decimal number. This pads each
batch with a wait time to throttle the rate. Set requests_per_second
to -1
to disable throttling.
Throttling uses a wait time between batches so that the internal scroll requests
can be given a timeout that takes the request padding into account. The padding
time is the difference between the batch size divided by the
requests_per_second
and the time spent writing. By default the batch size is
1000
, so if requests_per_second
is set to 500
:
target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
Since the batch is issued as a single _bulk
request, large batch sizes
cause OpenSearch to create many requests and wait before starting the next set.
This is "bursty" instead of "smooth".
Slicing
Delete by query supports sliced scroll to parallelize the delete process. This can improve efficiency and provide a convenient way to break the request down into smaller parts.
Setting slices
to auto
chooses a reasonable number for most indices.
If you’re slicing manually or otherwise tuning automatic slicing, keep in mind
that:
-
Query performance is most efficient when the number of
slices
is equal to the number of shards in the index. If that number is large (for example, 500), choose a lower number as too manyslices
hurts performance. Settingslices
higher than the number of shards generally does not improve efficiency and adds overhead. -
Delete performance scales linearly across available resources with the number of slices.
Whether query or delete performance dominates the runtime depends on the documents being reindexed and cluster resources.
Path parameters
<target>
-
(Optional, string) Comma-separated list of indices and index aliases to search. Wildcard (
*
) expressions are supported.To search all indices in a cluster, omit this parameter or use
_all
or*
.
Query parameters
allow_no_indices
-
(Optional, Boolean) If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
.Defaults to
true
. analyzer
-
(Optional, string) Analyzer to use for the query string.
analyze_wildcard
-
(Optional, Boolean) If
true
, wildcard and prefix queries are analyzed. Defaults tofalse
. conflicts
-
(Optional, string) What to do if delete by query hits version conflicts:
abort
orproceed
. Defaults toabort
. default_operator
-
(Optional, string) The default operator for query string query: AND or OR. Defaults to
OR
. df
-
(Optional, string) Field to use as default where no field prefix is given in the query string.
expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
Defaults to
open
. from
-
(Optional, integer) Starting document offset. Defaults to
0
. ignore_unavailable
-
(Optional, Boolean) If
true
, missing or closed indices are not included in the response. Defaults tofalse
. lenient
-
(Optional, Boolean) If
true
, format-based query failures (such as providing text to a numeric field) will be ignored. Defaults tofalse
. max_docs
-
(Optional, integer) Maximum number of documents to process. Defaults to all documents.
preference
-
(Optional, string) Specifies the node or shard the operation should be performed on. Random by default.
q
-
(Optional, string) Query in the Lucene query string syntax.
request_cache
-
(Optional, Boolean) If
true
, the request cache is used for this request. Defaults to the index-level setting. refresh
-
(Optional, enum) If
true
, OpenSearch refreshes the affected shards to make this operation visible to search, ifwait_for
then wait for a refresh to make this operation visible to search, iffalse
do nothing with refreshes. Valid values:true
,false
,wait_for
. Default:false
. requests_per_second
-
(Optional, integer) The throttle for this request in sub-requests per second. Defaults to
-1
(no throttle). routing
-
(Optional, string) Target the specified primary shard.
scroll
-
(Optional, time value) Period to retain the search context for scrolling. See Scroll search results.
scroll_size
-
(Optional, integer) Size of the scroll request that powers the operation. Defaults to 1000.
search_type
-
(Optional, string) The type of the search operation. Available options:
-
query_then_fetch
-
dfs_query_then_fetch
-
search_timeout
-
(Optional, time units) Explicit timeout for each search request. Defaults to no timeout.
slices
-
(Optional, integer) The number of slices this task should be divided into. Defaults to 1 meaning the task isn’t sliced into subtasks.
sort
-
(Optional, string) A comma-separated list of <field>:<direction> pairs.
_source
-
(Optional, string) True or false to return the
_source
field or not, or a list of fields to return. _source_excludes
-
(Optional, string) A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. _source_includes
-
(Optional, string) A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. stats
-
(Optional, string) Specific
tag
of the request for logging and statistical purposes. terminate_after
-
(Optional, integer) The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early.
timeout
-
(Optional, time units) Period each deletion request waits for active shards. Defaults to
1m
(one minute). version
-
(Optional, Boolean) If
true
, returns the document version as part of a hit. wait_for_active_shards
-
(Optional, string) The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). Default: 1, the primary shard.See Active shards.
Request body
query
-
(Optional, query object) Specifies the documents to delete using the Query DSL.
Response body
The JSON response looks like this:
{
"took" : 147,
"timed_out": false,
"total": 119,
"deleted": 119,
"batches": 1,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1.0,
"throttled_until_millis": 0,
"failures" : [ ]
}
took
-
The number of milliseconds from start to end of the whole operation.
timed_out
-
This flag is set to
true
if any of the requests executed during the delete by query execution has timed out. total
-
The number of documents that were successfully processed.
deleted
-
The number of documents that were successfully deleted.
batches
-
The number of scroll responses pulled back by the delete by query.
version_conflicts
-
The number of version conflicts that the delete by query hit.
noops
-
This field is always equal to zero for delete by query. It only exists so that delete by query, update by query, and reindex APIs return responses with the same structure.
retries
-
The number of retries attempted by delete by query.
bulk
is the number of bulk actions retried, andsearch
is the number of search actions retried. throttled_millis
-
Number of milliseconds the request slept to conform to
requests_per_second
. requests_per_second
-
The number of requests per second effectively executed during the delete by query.
throttled_until_millis
-
This field should always be equal to zero in a
_delete_by_query
response. It only has meaning when using the Task API, where it indicates the next time (in milliseconds since epoch) a throttled request will be executed again in order to conform torequests_per_second
. failures
-
Array of failures if there were any unrecoverable errors during the process. If this is non-empty then the request aborted because of those failures. Delete by query is implemented using batches, and any failure causes the entire process to abort but all failures in the current batch are collected into the array. You can use the
conflicts
option to prevent reindex from aborting on version conflicts.
Examples
Delete all documents from the my-index-000001
index:
POST my-index-000001/_delete_by_query?conflicts=proceed
{
"query": {
"match_all": {}
}
}
Delete documents from multiple indices:
POST /my-index-000001,my-index-000002/_delete_by_query
{
"query": {
"match_all": {}
}
}
Limit the delete by query operation to shards that a particular routing value:
POST my-index-000001/_delete_by_query?routing=1
{
"query": {
"range" : {
"age" : {
"gte" : 10
}
}
}
}
By default _delete_by_query
uses scroll batches of 1000. You can change the
batch size with the scroll_size
URL parameter:
POST my-index-000001/_delete_by_query?scroll_size=5000
{
"query": {
"term": {
"user.id": "bilbo"
}
}
}
Slice manually
Slice a delete by query manually by providing a slice id and total number of slices:
POST my-index-000001/_delete_by_query
{
"slice": {
"id": 0,
"max": 2
},
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}
POST my-index-000001/_delete_by_query
{
"slice": {
"id": 1,
"max": 2
},
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}
Which you can verify works with:
GET _refresh
POST my-index-000001/_search?size=0&filter_path=hits.total
{
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}
Which results in a sensible total
like this one:
{
"hits": {
"total" : {
"value": 0,
"relation": "eq"
}
}
}
Use automatic slicing
You can also let delete-by-query automatically parallelize using
sliced scroll to slice on _id
. Use slices
to specify
the number of slices to use:
POST my-index-000001/_delete_by_query?refresh&slices=5
{
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}
Which you also can verify works with:
POST my-index-000001/_search?size=0&filter_path=hits.total
{
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}
Which results in a sensible total
like this one:
{
"hits": {
"total" : {
"value": 0,
"relation": "eq"
}
}
}
Setting slices
to auto
will let OpenSearch choose the number of slices
to use. This setting will use one slice per shard, up to a certain limit. If
there are multiple source indices, it will choose the number of slices based
on the index with the smallest number of shards.
Adding slices
to _delete_by_query
just automates the manual process used in
the section above, creating sub-requests which means it has some quirks:
-
You can see these requests in the Tasks APIs. These sub-requests are "child" tasks of the task for the request with
slices
. -
Fetching the status of the task for the request with
slices
only contains the status of completed slices. -
These sub-requests are individually addressable for things like cancellation and rethrottling.
-
Rethrottling the request with
slices
will rethrottle the unfinished sub-request proportionally. -
Canceling the request with
slices
will cancel each sub-request. -
Due to the nature of
slices
each sub-request won’t get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution. -
Parameters like
requests_per_second
andmax_docs
on a request with slices` are distributed proportionally to each sub-request. Combine that with the point above about distribution being uneven and you should conclude that usingmax_docs
withslices
might not result in exactlymax_docs
documents being deleted. -
Each sub-request gets a slightly different snapshot of the source index though these are all taken at approximately the same time.
Change throttling for a request
The value of requests_per_second
can be changed on a running delete by query
using the _rethrottle
API. Rethrottling that speeds up the
query takes effect immediately but rethrotting that slows down the query
takes effect after completing the current batch to prevent scroll
timeouts.
POST _delete_by_query/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
Use the tasks API to get the task ID. Set requests_per_second
to any positive decimal value or -1
to disable throttling.
Get the status of a delete by query operation
Use the tasks API to get the status of a delete by query operation:
GET _tasks?detailed=true&actions=*/delete/byquery
The response looks like:
{
"nodes" : {
"r1A2WoRbTwKZ516z6NEs5A" : {
"name" : "r1A2WoR",
"transport_address" : "<IP_ADDRESS>:9300",
"host" : "<IP_ADDRESS>",
"ip" : "<IP_ADDRESS>:9300",
"attributes" : {
"testattr" : "test",
"portsfile" : "true"
},
"tasks" : {
"r1A2WoRbTwKZ516z6NEs5A:36619" : {
"node" : "r1A2WoRbTwKZ516z6NEs5A",
"id" : 36619,
"type" : "transport",
"action" : "indices:data/write/delete/byquery",
"status" : { (1)
"total" : 6154,
"updated" : 0,
"created" : 0,
"deleted" : 3500,
"batches" : 36,
"version_conflicts" : 0,
"noops" : 0,
"retries": 0,
"throttled_millis": 0
},
"description" : ""
}
}
}
}
}
-
This object contains the actual status. It is just like the response JSON with the important addition of the
total
field.total
is the total number of operations that the reindex expects to perform. You can estimate the progress by adding theupdated
,created
, anddeleted
fields. The request will finish when their sum is equal to thetotal
field.
With the task id you can look up the task directly:
GET /_tasks/r1A2WoRbTwKZ516z6NEs5A:36619
The advantage of this API is that it integrates with wait_for_completion=false
to transparently return the status of completed tasks. If the task is completed
and wait_for_completion=false
was set on it then it’ll come back with
results
or an error
field. The cost of this feature is the document that
wait_for_completion=false
creates at .tasks/task/${taskId}
. It is up to
you to delete that document.
Cancel a delete by query operation
Any delete by query can be canceled using the task cancel API:
POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel
The task ID can be found using the tasks API.
Cancellation should happen quickly but might take a few seconds. The task status API above will continue to list the delete by query task until this task checks that it has been cancelled and terminates itself.
Update API
Updates a document using the specified script.
Request
POST /<index>/_update/<_id>
Description
Enables you to script document updates. The script can update, delete, or skip
modifying the document. The update API also supports passing a partial document,
which is merged into the existing document. To fully replace an existing
document, use the index
API.
This operation:
-
Gets the document (collocated with the shard) from the index.
-
Runs the specified script.
-
Indexes the result.
The document must still be reindexed, but using update
removes some network
roundtrips and reduces chances of version conflicts between the GET and the
index operation.
The _source
field must be enabled to use update
. In addition to _source
,
you can access the following variables through the ctx
map: _index
,
_type
, _id
, _version
, _routing
, and _now
(the current timestamp).
Path parameters
<index>
-
(Required, string) Name of the target index. By default, the index is created automatically if it doesn’t exist. For more information, see Automatically create indices.
<_id>
-
(Required, string) Unique identifier for the document to be updated.
Query parameters
if_seq_no
-
(Optional, integer) Only perform the operation if the document has this sequence number. See Optimistic concurrency control.
if_primary_term
-
(Optional, integer) Only perform the operation if the document has this primary term. See Optimistic concurrency control.
lang
-
(Optional, string) The script language. Default:
painless
. require_alias
-
(Optional, Boolean) If
true
, the destination must be an index alias. Defaults tofalse
. refresh
-
(Optional, enum) If
true
, OpenSearch refreshes the affected shards to make this operation visible to search, ifwait_for
then wait for a refresh to make this operation visible to search, iffalse
do nothing with refreshes. Valid values:true
,false
,wait_for
. Default:false
. retry_on_conflict
-
(Optional, integer) Specify how many times should the operation be retried when a conflict occurs. Default: 0.
routing
-
(Optional, string) Target the specified primary shard.
_source
-
(Optional, list) Set to
false
to disable source retrieval (default:true
). You can also specify a comma-separated list of the fields you want to retrieve. _source_excludes
-
(Optional, list) Specify the source fields you want to exclude.
_source_includes
-
(Optional, list) Specify the source fields you want to retrieve.
timeout
-
(Optional, time units) Period to wait for the following operations:
Defaults to
1m
(one minute). This guarantees OpenSearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur. wait_for_active_shards
-
(Optional, string) The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). Default: 1, the primary shard.See Active shards.
Examples
First, let’s index a simple doc:
PUT test/_doc/1
{
"counter" : 1,
"tags" : ["red"]
}
To increment the counter, you can submit an update request with the following script:
POST test/_update/1
{
"script" : {
"source": "ctx._source.counter += params.count",
"lang": "painless",
"params" : {
"count" : 4
}
}
}
Similarly, you could use and update script to add a tag to the list of tags (this is just a list, so the tag is added even it exists):
POST test/_update/1
{
"script": {
"source": "ctx._source.tags.add(params.tag)",
"lang": "painless",
"params": {
"tag": "blue"
}
}
}
You could also remove a tag from the list of tags. The Painless
function to remove
a tag takes the array index of the element
you want to remove. To avoid a possible runtime error, you first need to
make sure the tag exists. If the list contains duplicates of the tag, this
script just removes one occurrence.
POST test/_update/1
{
"script": {
"source": "if (ctx._source.tags.contains(params.tag)) { ctx._source.tags.remove(ctx._source.tags.indexOf(params.tag)) }",
"lang": "painless",
"params": {
"tag": "blue"
}
}
}
You can also add and remove fields from a document. For example, this script
adds the field new_field
:
POST test/_update/1
{
"script" : "ctx._source.new_field = 'value_of_new_field'"
}
Conversely, this script removes the field new_field
:
POST test/_update/1
{
"script" : "ctx._source.remove('new_field')"
}
Instead of updating the document, you can also change the operation that is
executed from within the script. For example, this request deletes the doc if
the tags
field contains green
, otherwise it does nothing (noop
):
POST test/_update/1
{
"script": {
"source": "if (ctx._source.tags.contains(params.tag)) { ctx.op = 'delete' } else { ctx.op = 'none' }",
"lang": "painless",
"params": {
"tag": "green"
}
}
}
Update part of a document
The following partial update adds a new field to the existing document:
POST test/_update/1
{
"doc": {
"name": "new_name"
}
}
If both doc
and script
are specified, then doc
is ignored. If you
specify a scripted update, include the fields you want to update in the script.
Detect noop updates
By default updates that don’t change anything detect that they don’t change
anything and return "result": "noop"
:
POST test/_update/1
{
"doc": {
"name": "new_name"
}
}
If the value of name
is already new_name
, the update
request is ignored and the result
element in the response returns noop
:
{
"_shards": {
"total": 0,
"successful": 0,
"failed": 0
},
"_index": "test",
"_type": "_doc",
"_id": "1",
"_version": 7,
"_primary_term": 1,
"_seq_no": 6,
"result": "noop"
}
You can disable this behavior by setting "detect_noop": false
:
POST test/_update/1
{
"doc": {
"name": "new_name"
},
"detect_noop": false
}
Upsert
If the document does not already exist, the contents of the upsert
element
are inserted as a new document. If the document exists, the
script
is executed:
POST test/_update/1
{
"script": {
"source": "ctx._source.counter += params.count",
"lang": "painless",
"params": {
"count": 4
}
},
"upsert": {
"counter": 1
}
}
Scripted upsert
To run the script whether or not the document exists, set scripted_upsert
to
true
:
POST sessions/_update/dh3sgudg8gsrgl
{
"scripted_upsert": true,
"script": {
"id": "my_web_session_summariser",
"params": {
"pageViewEvent": {
"url": "foo.com/bar",
"response": 404,
"time": "2014-01-01 12:32"
}
}
},
"upsert": {}
}
Doc as upsert
Instead of sending a partial doc
plus an upsert
doc, you can set
doc_as_upsert
to true
to use the contents of doc
as the upsert
value:
POST test/_update/1
{
"doc": {
"name": "new_name"
},
"doc_as_upsert": true
}
|
Note
|
Using ingest pipelines with |
Update By Query API
Updates documents that match the specified query. If no query is specified, performs an update on every document in the index without modifying the source, which is useful for picking up mapping changes.
POST my-index-000001/_update_by_query?conflicts=proceed
Request
POST /<target>/_update_by_query
Description
You can specify the query criteria in the request URI or the request body using the same syntax as the Search API.
When you submit an update by query request, OpenSearch gets a snapshot of the index
when it begins processing the request and updates matching documents using
internal
versioning.
When the versions match, the document is updated and the version number is incremented.
If a document changes between the time that the snapshot is taken and
the update operation is processed, it results in a version conflict and the operation fails.
You can opt to count version conflicts instead of halting and returning by
setting conflicts
to proceed
.
|
Note
|
Documents with a version equal to 0 cannot be updated using update by
query because internal versioning does not support 0 as a valid
version number.
|
While processing an update by query request, OpenSearch performs multiple search requests sequentially to find all of the matching documents. A bulk update request is performed for each batch of matching documents. Any query or update failures cause the update by query request to fail and the failures are shown in the response. Any update requests that completed successfully still stick, they are not rolled back.
Refreshing shards
Specifying the refresh
parameter refreshes all shards once the request completes.
This is different than the update API’s refresh
parameter, which causes just the shard
that received the request to be refreshed. Unlike the update API, it does not support
wait_for
.
Running update by query asynchronously
If the request contains wait_for_completion=false
, OpenSearch
performs some preflight checks, launches the request, and returns a
task
you can use to cancel or get the status of the task.
OpenSearch creates a record of this task as a document at .tasks/task/${taskId}
.
When you are done with a task, you should delete the task document so
OpenSearch can reclaim the space.
Waiting for active shards
wait_for_active_shards
controls how many copies of a shard must be active
before proceeding with the request. See Active shards
for details. timeout
controls how long each write request waits for unavailable
shards to become available. Both work exactly the way they work in the
Bulk API. Update by query uses scrolled searches, so you can also
specify the scroll
parameter to control how long it keeps the search context
alive, for example ?scroll=10m
. The default is 5 minutes.
Throttling update requests
To control the rate at which update by query issues batches of update operations,
you can set requests_per_second
to any positive decimal number. This pads each
batch with a wait time to throttle the rate. Set requests_per_second
to -1
to disable throttling.
Throttling uses a wait time between batches so that the internal scroll requests
can be given a timeout that takes the request padding into account. The padding
time is the difference between the batch size divided by the
requests_per_second
and the time spent writing. By default the batch size is
1000
, so if requests_per_second
is set to 500
:
target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
Since the batch is issued as a single _bulk
request, large batch sizes
cause OpenSearch to create many requests and wait before starting the next set.
This is "bursty" instead of "smooth".
Slicing
Update by query supports sliced scroll to parallelize the update process. This can improve efficiency and provide a convenient way to break the request down into smaller parts.
Setting slices
to auto
chooses a reasonable number for most indices.
If you’re slicing manually or otherwise tuning automatic slicing, keep in mind
that:
-
Query performance is most efficient when the number of
slices
is equal to the number of shards in the index. If that number is large (for example, 500), choose a lower number as too manyslices
hurts performance. Settingslices
higher than the number of shards generally does not improve efficiency and adds overhead. -
Update performance scales linearly across available resources with the number of slices.
Whether query or update performance dominates the runtime depends on the documents being reindexed and cluster resources.
Path parameters
<target>
-
(Optional, string) Comma-separated list of indices and index aliases to search. Wildcard (
*
) expressions are supported.To search all indices in a cluster, omit this parameter or use
_all
or*
.
Query parameters
allow_no_indices
-
(Optional, Boolean) If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
.Defaults to
true
. analyzer
-
(Optional, string) Analyzer to use for the query string.
analyze_wildcard
-
(Optional, Boolean) If
true
, wildcard and prefix queries are analyzed. Defaults tofalse
. conflicts
-
(Optional, string) What to do if update by query hits version conflicts:
abort
orproceed
. Defaults toabort
. default_operator
-
(Optional, string) The default operator for query string query: AND or OR. Defaults to
OR
. df
-
(Optional, string) Field to use as default where no field prefix is given in the query string.
expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
Defaults to
open
. from
-
(Optional, integer) Starting document offset. Defaults to
0
. ignore_unavailable
-
(Optional, Boolean) If
true
, missing or closed indices are not included in the response. Defaults tofalse
. lenient
-
(Optional, Boolean) If
true
, format-based query failures (such as providing text to a numeric field) will be ignored. Defaults tofalse
. max_docs
-
(Optional, integer) Maximum number of documents to process. Defaults to all documents.
pipeline
-
(Optional, string) ID of the pipeline to use to preprocess incoming documents.
preference
-
(Optional, string) Specifies the node or shard the operation should be performed on. Random by default.
q
-
(Optional, string) Query in the Lucene query string syntax.
request_cache
-
(Optional, Boolean) If
true
, the request cache is used for this request. Defaults to the index-level setting. refresh
-
(Optional, Boolean) If
true
, OpenSearch refreshes affected shards to make the operation visible to search. Defaults tofalse
. requests_per_second
-
(Optional, integer) The throttle for this request in sub-requests per second. Defaults to
-1
(no throttle). routing
-
(Optional, string) Target the specified primary shard.
scroll
-
(Optional, time value) Period to retain the search context for scrolling. See Scroll search results.
scroll_size
-
(Optional, integer) Size of the scroll request that powers the operation. Defaults to 1000.
search_type
-
(Optional, string) The type of the search operation. Available options:
-
query_then_fetch
-
dfs_query_then_fetch
-
search_timeout
-
(Optional, time units) Explicit timeout for each search request. Defaults to no timeout.
slices
-
(Optional, integer) The number of slices this task should be divided into. Defaults to 1 meaning the task isn’t sliced into subtasks.
sort
-
(Optional, string) A comma-separated list of <field>:<direction> pairs.
_source
-
(Optional, string) True or false to return the
_source
field or not, or a list of fields to return. _source_excludes
-
(Optional, string) A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. _source_includes
-
(Optional, string) A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. stats
-
(Optional, string) Specific
tag
of the request for logging and statistical purposes. terminate_after
-
(Optional, integer) The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early.
timeout
-
(Optional, time units) Period each update request waits for the following operations:
-
Dynamic mapping updates
Defaults to
1m
(one minute). This guarantees OpenSearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur. -
version
-
(Optional, Boolean) If
true
, returns the document version as part of a hit. wait_for_active_shards
-
(Optional, string) The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). Default: 1, the primary shard.See Active shards.
Request body
query
-
(Optional, query object) Specifies the documents to update using the Query DSL.
Response body
took
-
The number of milliseconds from start to end of the whole operation.
timed_out
-
This flag is set to
true
if any of the requests executed during the update by query execution has timed out. total
-
The number of documents that were successfully processed.
updated
-
The number of documents that were successfully updated.
deleted
-
The number of documents that were successfully deleted.
batches
-
The number of scroll responses pulled back by the update by query.
version_conflicts
-
The number of version conflicts that the update by query hit.
noops
-
The number of documents that were ignored because the script used for the update by query returned a
noop
value forctx.op
. retries
-
The number of retries attempted by update by query.
bulk
is the number of bulk actions retried, andsearch
is the number of search actions retried. throttled_millis
-
Number of milliseconds the request slept to conform to
requests_per_second
. requests_per_second
-
The number of requests per second effectively executed during the update by query.
throttled_until_millis
-
This field should always be equal to zero in an
_update_by_query
response. It only has meaning when using the Task API, where it indicates the next time (in milliseconds since epoch) a throttled request will be executed again in order to conform torequests_per_second
. failures
-
Array of failures if there were any unrecoverable errors during the process. If this is non-empty then the request aborted because of those failures. Update by query is implemented using batches. Any failure causes the entire process to abort, but all failures in the current batch are collected into the array. You can use the
conflicts
option to prevent reindex from aborting on version conflicts.
Examples
The simplest usage of _update_by_query
just performs an update on every
document in the index without changing the source. This is useful to
pick up a new property or some other online
mapping change.
To update selected documents, specify a query in the request body:
POST my-index-000001/_update_by_query?conflicts=proceed
{
"query": { (1)
"term": {
"user.id": "bilbo"
}
}
}
-
The query must be passed as a value to the
query
key, in the same way as the Search API. You can also use theq
parameter in the same way as the search API.
Update documents in multiple indices:
POST my-index-000001,my-index-000002/_update_by_query
Limit the update by query operation to shards that a particular routing value:
POST my-index-000001/_update_by_query?routing=1
By default update by query uses scroll batches of 1000.
You can change the batch size with the scroll_size
parameter:
POST my-index-000001/_update_by_query?scroll_size=100
Update the document source
Update by query supports scripts to update the document source.
For example, the following request increments the count
field for all
documents with a user.id
of bilbo
in my-index-000001
:
POST my-index-000001/_update_by_query
{
"script": {
"source": "ctx._source.count++",
"lang": "painless"
},
"query": {
"term": {
"user.id": "bilbo"
}
}
}
Note that conflicts=proceed
is not specified in this example. In this case, a
version conflict should halt the process so you can handle the failure.
As with the Update API, you can set ctx.op
to change the
operation that is performed:
noop
|
Set |
delete
|
Set |
Update by query only supports update
, noop
, and delete
.
Setting ctx.op
to anything else is an error. Setting any other field in ctx
is an error.
This API only enables you to modify the source of matching documents, you cannot move them.
Update documents using an ingest pipeline
Update by query can use the Ingest node feature by specifying a pipeline
:
PUT _ingest/pipeline/set-foo
{
"description" : "sets foo",
"processors" : [ {
"set" : {
"field": "foo",
"value": "bar"
}
} ]
}
POST my-index-000001/_update_by_query?pipeline=set-foo
Get the status of update by query operations
You can fetch the status of all running update by query requests with the Task API:
GET _tasks?detailed=true&actions=*byquery
The responses looks like:
{
"nodes" : {
"r1A2WoRbTwKZ516z6NEs5A" : {
"name" : "r1A2WoR",
"transport_address" : "<IP_ADDRESS>:9300",
"host" : "<IP_ADDRESS>",
"ip" : "<IP_ADDRESS>:9300",
"attributes" : {
"testattr" : "test",
"portsfile" : "true"
},
"tasks" : {
"r1A2WoRbTwKZ516z6NEs5A:36619" : {
"node" : "r1A2WoRbTwKZ516z6NEs5A",
"id" : 36619,
"type" : "transport",
"action" : "indices:data/write/update/byquery",
"status" : { (1)
"total" : 6154,
"updated" : 3500,
"created" : 0,
"deleted" : 0,
"batches" : 4,
"version_conflicts" : 0,
"noops" : 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0
},
"description" : ""
}
}
}
}
}
-
This object contains the actual status. It is just like the response JSON with the important addition of the
total
field.total
is the total number of operations that the reindex expects to perform. You can estimate the progress by adding theupdated
,created
, anddeleted
fields. The request will finish when their sum is equal to thetotal
field.
With the task id you can look up the task directly. The following example
retrieves information about task r1A2WoRbTwKZ516z6NEs5A:36619
:
GET /_tasks/r1A2WoRbTwKZ516z6NEs5A:36619
The advantage of this API is that it integrates with wait_for_completion=false
to transparently return the status of completed tasks. If the task is completed
and wait_for_completion=false
was set on it, then it’ll come back with a
results
or an error
field. The cost of this feature is the document that
wait_for_completion=false
creates at .tasks/task/${taskId}
. It is up to
you to delete that document.
Cancel an update by query operation
Any update by query can be cancelled using the Task Cancel API:
POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel
The task ID can be found using the tasks API.
Cancellation should happen quickly but might take a few seconds. The task status API above will continue to list the update by query task until this task checks that it has been cancelled and terminates itself.
Change throttling for a request
The value of requests_per_second
can be changed on a running update by query
using the _rethrottle
API:
POST _update_by_query/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
The task ID can be found using the tasks API.
Just like when setting it on the _update_by_query
API, requests_per_second
can be either -1
to disable throttling or any decimal number
like 1.7
or 12
to throttle to that level. Rethrottling that speeds up the
query takes effect immediately, but rethrotting that slows down the query will
take effect after completing the current batch. This prevents scroll
timeouts.
Slice manually
Slice an update by query manually by providing a slice id and total number of slices to each request:
POST my-index-000001/_update_by_query
{
"slice": {
"id": 0,
"max": 2
},
"script": {
"source": "ctx._source['extra'] = 'test'"
}
}
POST my-index-000001/_update_by_query
{
"slice": {
"id": 1,
"max": 2
},
"script": {
"source": "ctx._source['extra'] = 'test'"
}
}
Which you can verify works with:
GET _refresh
POST my-index-000001/_search?size=0&q=extra:test&filter_path=hits.total
Which results in a sensible total
like this one:
{
"hits": {
"total": {
"value": 120,
"relation": "eq"
}
}
}
Use automatic slicing
You can also let update by query automatically parallelize using
Sliced scroll to slice on _id
. Use slices
to specify the number of
slices to use:
POST my-index-000001/_update_by_query?refresh&slices=5
{
"script": {
"source": "ctx._source['extra'] = 'test'"
}
}
Which you also can verify works with:
POST my-index-000001/_search?size=0&q=extra:test&filter_path=hits.total
Which results in a sensible total
like this one:
{
"hits": {
"total": {
"value": 120,
"relation": "eq"
}
}
}
Setting slices
to auto
will let OpenSearch choose the number of slices
to use. This setting will use one slice per shard, up to a certain limit. If
there are multiple source indices, it will choose the number of slices based
on the index with the smallest number of shards.
Adding slices
to _update_by_query
just automates the manual process used in
the section above, creating sub-requests which means it has some quirks:
-
You can see these requests in the Tasks APIs. These sub-requests are "child" tasks of the task for the request with
slices
. -
Fetching the status of the task for the request with
slices
only contains the status of completed slices. -
These sub-requests are individually addressable for things like cancellation and rethrottling.
-
Rethrottling the request with
slices
will rethrottle the unfinished sub-request proportionally. -
Canceling the request with
slices
will cancel each sub-request. -
Due to the nature of
slices
each sub-request won’t get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution. -
Parameters like
requests_per_second
andmax_docs
on a request withslices
are distributed proportionally to each sub-request. Combine that with the point above about distribution being uneven and you should conclude that usingmax_docs
withslices
might not result in exactlymax_docs
documents being updated. -
Each sub-request gets a slightly different snapshot of the source index though these are all taken at approximately the same time.
Pick up a new property
Say you created an index without dynamic mapping, filled it with data, and then added a mapping value to pick up more fields from the data:
PUT test
{
"mappings": {
"dynamic": false, (1)
"properties": {
"text": {"type": "text"}
}
}
}
POST test/_doc?refresh
{
"text": "words words",
"flag": "bar"
}
POST test/_doc?refresh
{
"text": "words words",
"flag": "foo"
}
PUT test/_mapping (2)
{
"properties": {
"text": {"type": "text"},
"flag": {"type": "text", "analyzer": "keyword"}
}
}
-
This means that new fields won’t be indexed, just stored in
_source
. -
This updates the mapping to add the new
flag
field. To pick up the new field you have to reindex all documents with it.
Searching for the data won’t find anything:
POST test/_search?filter_path=hits.total
{
"query": {
"match": {
"flag": "foo"
}
}
}
{
"hits" : {
"total": {
"value": 0,
"relation": "eq"
}
}
}
But you can issue an _update_by_query
request to pick up the new mapping:
POST test/_update_by_query?refresh&conflicts=proceed
POST test/_search?filter_path=hits.total
{
"query": {
"match": {
"flag": "foo"
}
}
}
{
"hits" : {
"total": {
"value": 1,
"relation": "eq"
}
}
}
You can do the exact same thing when adding a field to a multifield.
Multi get (mget) API
Retrieves multiple JSON documents by ID.
GET /_mget
{
"docs": [
{
"_index": "my-index-000001",
"_id": "1"
},
{
"_index": "my-index-000001",
"_id": "2"
}
]
}
Request
GET /_mget
GET /<index>/_mget
Description
You use mget
to retrieve multiple documents from one or more indices.
If you specify an index in the request URI, you only need to specify the document IDs in the request body.
Security
Partial responses
To ensure fast responses, the multi get API responds with partial results if one or more shards fail. See Shard failures for more information.
Path parameters
<index>
-
(Optional, string) Name of the index to retrieve documents from when
ids
are specified, or when a document in thedocs
array does not specify an index.
Query parameters
preference
-
(Optional, string) Specifies the node or shard the operation should be performed on. Random by default.
realtime
-
(Optional, Boolean) If
true
, the request is real-time as opposed to near-real-time. Defaults totrue
. See Realtime. refresh
-
(Optional, enum) If
true
, OpenSearch refreshes the affected shards to make this operation visible to search, ifwait_for
then wait for a refresh to make this operation visible to search, iffalse
do nothing with refreshes. Valid values:true
,false
,wait_for
. Default:false
. routing
-
(Optional, string) Target the specified primary shard.
stored_fields
-
(Optional, Boolean) If
true
, retrieves the document fields stored in the index rather than the document_source
. Defaults tofalse
. _source
-
(Optional, string) True or false to return the
_source
field or not, or a list of fields to return. _source_excludes
-
(Optional, string) A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. _source_includes
-
(Optional, string) A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored.
Request body
docs
-
(Optional, array) The documents you want to retrieve. Required if no index is specified in the request URI. You can specify the following attributes for each document:
_id
-
(Required, string) The unique document ID.
_index
-
(Optional, string) The index that contains the document. Required if no index is specified in the request URI.
_routing
-
(Optional, string) The key for the primary shard the document resides on. Required if routing is used during indexing.
_source
-
(Optional, Boolean) If
false
, excludes all_source
fields. Defaults totrue
.source_include
-
(Optional, array) The fields to extract and return from the
_source
field. source_exclude
-
(Optional, array) The fields to exclude from the returned
_source
field.
_stored_fields
-
(Optional, array) The stored fields you want to retrieve.
ids
-
(Optional, array) The IDs of the documents you want to retrieve. Allowed when the index is specified in the request URI.
Response body
The response includes a docs
array that contains the documents in the order specified in the request.
The structure of the returned documents is similar to that returned by the get API.
If there is a failure getting a particular document, the error is included in place of the document.
Examples
Get documents by ID
If you specify an index in the request URI, only the document IDs are required in the request body:
GET /my-index-000001/_mget
{
"docs": [
{
"_type": "_doc",
"_id": "1"
},
{
"_type": "_doc",
"_id": "2"
}
]
}
And type:
GET /test/_doc/_mget
{
"docs": [
{
"_id": "1"
},
{
"_id": "2"
}
]
}
You can use the ids
element to simplify the request:
GET /my-index-000001/_mget
{
"ids" : ["1", "2"]
}
Filter source fields
By default, the _source
field is returned for every document (if stored).
Use the _source
and _source_include
or source_exclude
attributes to
filter what fields are returned for a particular document.
You can include the _source
, _source_includes
, and _source_excludes
query parameters in the
request URI to specify the defaults to use when there are no per-document instructions.
For example, the following request sets _source
to false for document 1 to exclude the
source entirely, retrieves field3
and field4
from document 2, and retrieves the user
field
from document 3 but filters out the user.location
field.
GET /_mget
{
"docs": [
{
"_index": "test",
"_type": "_doc",
"_id": "1",
"_source": false
},
{
"_index": "test",
"_type": "_doc",
"_id": "2",
"_source": [ "field3", "field4" ]
},
{
"_index": "test",
"_type": "_doc",
"_id": "3",
"_source": {
"include": [ "user" ],
"exclude": [ "user.location" ]
}
}
]
}
Get stored fields
Use the stored_fields
attribute to specify the set of stored fields you want
to retrieve. Any requested fields that are not stored are ignored.
You can include the stored_fields
query parameter in the request URI to specify the defaults
to use when there are no per-document instructions.
For example, the following request retrieves field1
and field2
from document 1, and
field3
and `field4`from document 2:
GET /_mget
{
"docs": [
{
"_index": "test",
"_type": "_doc",
"_id": "1",
"stored_fields": [ "field1", "field2" ]
},
{
"_index": "test",
"_type": "_doc",
"_id": "2",
"stored_fields": [ "field3", "field4" ]
}
]
}
The following request retrieves field1
and field2
from all documents by default.
These default fields are returned for document 1, but
overridden to return field3
and field4
for document 2.
GET /test/_doc/_mget?stored_fields=field1,field2
{
"docs": [
{
"_id": "1"
},
{
"_id": "2",
"stored_fields": [ "field3", "field4" ]
}
]
}
Specify document routing
If routing is used during indexing, you need to specify the routing value to retrieve documents.
For example, the following request fetches test/_doc/2
from the shard corresponding to routing key key1
,
and fetches test/_doc/1
from the shard corresponding to routing key key2
.
GET /_mget?routing=key1
{
"docs": [
{
"_index": "test",
"_type": "_doc",
"_id": "1",
"routing": "key2"
},
{
"_index": "test",
"_type": "_doc",
"_id": "2"
}
]
}
Bulk API
Performs multiple indexing or delete operations in a single API call. This reduces overhead and can greatly increase indexing speed.
POST _bulk
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }
Request
POST /_bulk
POST /<target>/_bulk
-
To use the
create
action, you must have thecreate_doc
,create
,index
, orwrite
index privilege. -
To use the
index
action, you must have thecreate
,index
, orwrite
index privilege. -
To use the
delete
action, you must have thedelete
orwrite
index privilege. -
To use the
update
action, you must have theindex
orwrite
index privilege.
Description
Provides a way to perform multiple index
, create
, delete
, and update
actions in a single request.
The actions are specified in the request body using a newline delimited JSON (NDJSON) structure:
action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n
The index
and create
actions expect a source on the next line,
and have the same semantics as the op_type
parameter in the standard index API:
create
fails if a document with the same ID already exists in the target,
index
adds or replaces a document as necessary.
update
expects that the partial doc, upsert,
and script and its options are specified on the next line.
delete
does not expect a source on the next line and
has the same semantics as the standard delete API.
|
Note
|
The final line of data must end with a newline character |
Because this format uses literal `\n’s as delimiters, make sure that the JSON actions and sources are not pretty printed.
If you provide a <target>
in the request path,
it is used for any actions that don’t explicitly specify an _index
argument.
A note on the format: The idea here is to make processing of this as
fast as possible. As some of the actions are redirected to other
shards on other nodes, only action_meta_data
is parsed on the
receiving node side.
Client libraries using this protocol should try and strive to do something similar on the client side, and reduce buffering as much as possible.
There is no "correct" number of actions to perform in a single bulk request. Experiment with different settings to find the optimal size for your particular workload.
When using the HTTP API, make sure that the client does not send HTTP chunks, as this will slow things down.
Submitting bulk requests with cURL
If you’re providing text file input to curl
, you must use the
--data-binary
flag instead of plain -d
. The latter doesn’t preserve
newlines. Example:
$ cat requests
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
$ curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk --data-binary "@requests"; echo
{"took":7, "errors": false, "items":[{"index":{"_index":"test","_type":"_doc","_id":"1","_version":1,"result":"created","forced_refresh":false}}]}
Optimistic concurrency control
Each index
and delete
action within a bulk API call may include the
if_seq_no
and if_primary_term
parameters in their respective action
and meta data lines. The if_seq_no
and if_primary_term
parameters control
how operations are executed, based on the last modification to existing
documents. See Optimistic concurrency control for more details.
Versioning
Each bulk item can include the version value using the
version
field. It automatically follows the behavior of the
index / delete operation based on the _version
mapping. It also
support the version_type
(see versioning).
Routing
Each bulk item can include the routing value using the
routing
field. It automatically follows the behavior of the
index / delete operation based on the _routing
mapping.
Wait for active shards
When making bulk calls, you can set the wait_for_active_shards
parameter to require a minimum number of shard copies to be active
before starting to process the bulk request. See
here for further details and a usage
example.
Refresh
Control when the changes made by this request are visible to search. See refresh.
|
Note
|
Only the shards that receive the bulk request will be affected by
refresh . Imagine a _bulk?refresh=wait_for request with three
documents in it that happen to be routed to different shards in an index
with five shards. The request will only wait for those three shards to
refresh. The other two shards that make up the index do not
participate in the _bulk request at all.
|
Security
Path parameters
<target>
-
(Optional, string) Name of the index or index alias to perform bulk actions on.
Query parameters
pipeline
-
(Optional, string) ID of the pipeline to use to preprocess incoming documents.
refresh
-
(Optional, enum) If
true
, OpenSearch refreshes the affected shards to make this operation visible to search, ifwait_for
then wait for a refresh to make this operation visible to search, iffalse
do nothing with refreshes. Valid values:true
,false
,wait_for
. Default:false
. require_alias
-
(Optional, Boolean) If
true
, the request’s actions must target an index alias. Defaults tofalse
. routing
-
(Optional, string) Target the specified primary shard.
_source
-
(Optional, string) True or false to return the
_source
field or not, or a list of fields to return. _source_excludes
-
(Optional, string) A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. _source_includes
-
(Optional, string) A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. timeout
-
(Optional, time units) Period each action waits for the following operations:
Defaults to
1m
(one minute). This guarantees OpenSearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur. wait_for_active_shards
-
(Optional, string) The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). Default: 1, the primary shard.See Active shards.
Request body
The request body contains a newline-delimited list of create
, delete
, index
,
and update
actions and their associated source data.
create
-
(Optional, string) Indexes the specified document if it does not already exist. The following line must contain the source data to be indexed.
_index
-
(Optional, string) Name of the index or index alias to perform the action on. This parameter is required if a
<target>
is not specified in the request path. _id
-
(Optional, string) The document ID. If no ID is specified, a document ID is automatically generated.
require_alias
-
(Optional, Boolean) If
true
, the action must target an index alias. Defaults tofalse
.
delete
-
(Optional, string) Removes the specified document from the index.
_index
-
(Optional, string) Name of the index or index alias to perform the action on. This parameter is required if a
<target>
is not specified in the request path. _id
-
(Required, string) The document ID.
require_alias
-
(Optional, Boolean) If
true
, the action must target an index alias. Defaults tofalse
.
index
-
(Optional, string) Indexes the specified document. If the document exists, replaces the document and increments the version. The following line must contain the source data to be indexed.
_index
-
(Optional, string) Name of the index or index alias to perform the action on. This parameter is required if a
<target>
is not specified in the request path. _id
-
(Optional, string) The document ID. If no ID is specified, a document ID is automatically generated.
require_alias
-
(Optional, Boolean) If
true
, the action must target an index alias. Defaults tofalse
.
update
-
(Optional, string) Performs a partial document update. The following line must contain the partial document and update options.
_index
-
(Optional, string) Name of the index or index alias to perform the action on. This parameter is required if a
<target>
is not specified in the request path. _id
-
(Optional, string) The document ID. If no ID is specified, a document ID is automatically generated.
require_alias
-
(Optional, Boolean) If
true
, the action must target an index alias. Defaults tofalse
.
doc
-
(Optional, object) The partial document to index. Required for
update
operations. <fields>
-
(Optional, object) The document source to index. Required for
create
andindex
operations.
Response body
The bulk API’s response contains the individual results of each operation in the request, returned in the order submitted. The success or failure of an individual operation does not affect other operations in the request.
took
-
(integer) How long, in milliseconds, it took to process the bulk request.
errors
-
(Boolean) If
true
, one or more of the operations in the bulk request did not complete successfully. items
-
(array of objects) Contains the result of each operation in the bulk request, in the order they were submitted.
Properties of
items
objects- <action>
-
(object) The parameter name is an action associated with the operation. Possible values are
create
,delete
,index
, andupdate
.The parameter value is an object that contains information for the associated operation.
Properties of
<action>
_index
-
(string) Name of the index associated with the operation.
_type
-
(string) The document type associated with the operation. OpenSearch indices support a single document type of
_doc
. _id
-
(integer) The document ID associated with the operation.
_version
-
(integer) The document version associated with the operation. The document version is incremented each time the document is updated.
This parameter is only returned for successful actions.
result
-
(string) Result of the operation. Successful values are
created
,deleted
, andupdated
.This parameter is only returned for successful operations.
_shards
-
(object) Contains shard information for the operation.
This parameter is only returned for successful operations.
Properties of
_shards
total
-
(integer) Number of shards the operation attempted to execute on.
successful
-
(integer) Number of shards the operation succeeded on.
failed
-
(integer) Number of shards the operation attempted to execute on but failed.
_seq_no
-
(integer) The sequence number assigned to the document for the operation. Sequence numbers are used to ensure an older version of a document doesn’t overwrite a newer version. See Optimistic concurrency control.
This parameter is only returned for successful operations.
_primary_term
-
(integer) The primary term assigned to the document for the operation. See Optimistic concurrency control.
This parameter is only returned for successful operations.
status
-
(integer) HTTP status code returned for the operation.
error
-
(object) Contains additional information about the failed operation.
The parameter is only returned for failed operations.
Properties of
error
type
-
(string) Error type for the operation.
reason
-
(string) Reason for the failed operation.
index_uuid
-
(string) The universally unique identifier (UUID) of the index associated with the failed operation.
shard
-
(string) ID of the shard associated with the failed operation.
index
-
(string) Name of the index associated with the failed operation.
Examples
POST _bulk
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }
The API returns the following result:
{
"took": 30,
"errors": false,
"items": [
{
"index": {
"_index": "test",
"_type": "_doc",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 0,
"_primary_term": 1
}
},
{
"delete": {
"_index": "test",
"_type": "_doc",
"_id": "2",
"_version": 1,
"result": "not_found",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 404,
"_seq_no" : 1,
"_primary_term" : 2
}
},
{
"create": {
"_index": "test",
"_type": "_doc",
"_id": "3",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 2,
"_primary_term" : 3
}
},
{
"update": {
"_index": "test",
"_type": "_doc",
"_id": "1",
"_version": 2,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 200,
"_seq_no" : 3,
"_primary_term" : 4
}
}
]
}
Bulk update example
When using the update
action, retry_on_conflict
can be used as a field in
the action itself (not in the extra payload line), to specify how many
times an update should be retried in the case of a version conflict.
The update
action payload supports the following options: doc
(partial document), upsert
, doc_as_upsert
, script
, params
(for
script), lang
(for script), and _source
. See update documentation for details on
the options. Example with update actions:
POST _bulk
{ "update" : {"_id" : "1", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"} }
{ "update" : { "_id" : "0", "_index" : "index1", "retry_on_conflict" : 3} }
{ "script" : { "source": "ctx._source.counter += params.param1", "lang" : "painless", "params" : {"param1" : 1}}, "upsert" : {"counter" : 1}}
{ "update" : {"_id" : "2", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"}, "doc_as_upsert" : true }
{ "update" : {"_id" : "3", "_index" : "index1", "_source" : true} }
{ "doc" : {"field" : "value"} }
{ "update" : {"_id" : "4", "_index" : "index1"} }
{ "doc" : {"field" : "value"}, "_source": true}
Example with failed actions
The following bulk API request includes operations that update non-existent documents.
POST /_bulk
{ "update": {"_id": "5", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "update": {"_id": "6", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "create": {"_id": "7", "_index": "index1"} }
{ "my_field": "foo" }
Because these operations cannot complete successfully, the API returns a
response with an errors
flag of true
.
The response also includes an error
object for any failed operations. The
error
object contains additional information about the failure, such as the
error type and reason.
{
"took": 486,
"errors": true,
"items": [
{
"update": {
"_index": "index1",
"_type" : "_doc",
"_id": "5",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[_doc][5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"_index": "index1",
"_type" : "_doc",
"_id": "6",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[_doc][6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"create": {
"_index": "index1",
"_type" : "_doc",
"_id": "7",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1,
"status": 201
}
}
]
}
To return only information about failed operations, use the
filter_path
query parameter with an
argument of items.*.error
.
POST /_bulk?filter_path=items.*.error
{ "update": {"_id": "5", "_index": "index1"} }
{ "doc": {"my_field": "baz"} }
{ "update": {"_id": "6", "_index": "index1"} }
{ "doc": {"my_field": "baz"} }
{ "update": {"_id": "7", "_index": "index1"} }
{ "doc": {"my_field": "baz"} }
The API returns the following result.
{
"items": [
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[_doc][5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[_doc][6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
}
]
}
Reindex API
Copies documents from a source to a destination.
The source and destination can be any pre-existing index or index alias. However, the source and destination must be different.
|
Important
|
Reindex requires The destination should be configured as wanted before calling Mappings, shard counts, replicas, and so on must be configured ahead of time. |
POST _reindex
{
"source": {
"index": "my-index-000001"
},
"dest": {
"index": "my-new-index-000001"
}
}
Request
POST /_reindex
Prerequisites
-
If reindexing from a remote cluster, you must explicitly allow the remote host in the
reindex.remote.whitelist
setting ofopensearch.yml
. See Reindex from remote.
Description
Extracts the document source from the source index and indexes the documents into the destination index. You can copy all documents to the destination index, or reindex a subset of the documents.
Just like _update_by_query
, _reindex
gets a
snapshot of the source but its destination must be different so
version conflicts are unlikely. The dest
element can be configured like the
index API to control optimistic concurrency control. Omitting
version_type
or setting it to internal
causes OpenSearch
to blindly dump documents into the destination, overwriting any that happen to have
the same ID.
Setting version_type
to external
causes OpenSearch to preserve the
version
from the source, create any documents that are missing, and update
any documents that have an older version in the destination than they do
in the source.
Setting op_type
to create
causes _reindex
to only create missing
documents in the destination. All existing documents will cause a version
conflict.
By default, version conflicts abort the _reindex
process.
To continue reindexing if there are conflicts, set the "conflicts"
request body parameter to proceed
.
In this case, the response includes a count of the version conflicts that were encountered.
Note that the handling of other error types is unaffected by the "conflicts"
parameter.
Running reindex asynchronously
If the request contains wait_for_completion=false
, OpenSearch
performs some preflight checks, launches the request, and returns a
task
you can use to cancel or get the status of the task.
OpenSearch creates a record of this task as a document at .tasks/_doc/${taskId}
.
When you are done with a task, you should delete the task document so
OpenSearch can reclaim the space.
Reindex from multiple sources
If you have many sources to reindex it is generally better to reindex them one at a time rather than using a glob pattern to pick up multiple sources. That way you can resume the process if there are any errors by removing the partially completed source and starting over. It also makes parallelizing the process fairly simple: split the list of sources to reindex and run each list in parallel.
One-off bash scripts seem to work nicely for this:
for index in i1 i2 i3 i4 i5; do
curl -HContent-Type:application/json -XPOST localhost:9200/_reindex?pretty -d'{
"source": {
"index": "'$index'"
},
"dest": {
"index": "'$index'-reindexed"
}
}'
done
Throttling
Set requests_per_second
to any positive decimal number (1.4
, 6
,
1000
, etc.) to throttle the rate at which _reindex
issues batches of index
operations. Requests are throttled by padding each batch with a wait time.
To disable throttling, set requests_per_second
to -1
.
The throttling is done by waiting between batches so that the scroll
that _reindex
uses internally can be given a timeout that takes into account the padding.
The padding time is the difference between the batch size divided by the
requests_per_second
and the time spent writing. By default the batch size is
1000
, so if requests_per_second
is set to 500
:
target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
Since the batch is issued as a single _bulk
request, large batch sizes
cause OpenSearch to create many requests and then wait for a while before
starting the next set. This is "bursty" instead of "smooth".
Rethrottling
The value of requests_per_second
can be changed on a running reindex using
the _rethrottle
API:
POST _reindex/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
The task ID can be found using the tasks API.
Just like when setting it on the Reindex API, requests_per_second
can be either -1
to disable throttling or any decimal number
like 1.7
or 12
to throttle to that level. Rethrottling that speeds up the
query takes effect immediately, but rethrottling that slows down the query will
take effect after completing the current batch. This prevents scroll
timeouts.
Slicing
Reindex supports Sliced scroll to parallelize the reindexing process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.
|
Note
|
Reindexing from remote clusters does not support manual or automatic slicing. |
Manual slicing
Slice a reindex request manually by providing a slice id and total number of slices to each request:
POST _reindex
{
"source": {
"index": "my-index-000001",
"slice": {
"id": 0,
"max": 2
}
},
"dest": {
"index": "my-new-index-000001"
}
}
POST _reindex
{
"source": {
"index": "my-index-000001",
"slice": {
"id": 1,
"max": 2
}
},
"dest": {
"index": "my-new-index-000001"
}
}
You can verify this works by:
GET _refresh
POST my-new-index-000001/_search?size=0&filter_path=hits.total
which results in a sensible total
like this one:
{
"hits": {
"total" : {
"value": 120,
"relation": "eq"
}
}
}
Automatic slicing
You can also let _reindex
automatically parallelize using Sliced scroll to
slice on _id
. Use slices
to specify the number of slices to use:
POST _reindex?slices=5&refresh
{
"source": {
"index": "my-index-000001"
},
"dest": {
"index": "my-new-index-000001"
}
}
You can also this verify works by:
POST my-new-index-000001/_search?size=0&filter_path=hits.total
which results in a sensible total
like this one:
{
"hits": {
"total" : {
"value": 120,
"relation": "eq"
}
}
}
Setting slices
to auto
will let OpenSearch choose the number of slices to
use. This setting will use one slice per shard, up to a certain limit. If there
are multiple sources, it will choose the number of
slices based on the index with the smallest number of shards.
Adding slices
to _reindex
just automates the manual process used in the
section above, creating sub-requests which means it has some quirks:
-
You can see these requests in the Tasks APIs. These sub-requests are "child" tasks of the task for the request with
slices
. -
Fetching the status of the task for the request with
slices
only contains the status of completed slices. -
These sub-requests are individually addressable for things like cancelation and rethrottling.
-
Rethrottling the request with
slices
will rethrottle the unfinished sub-request proportionally. -
Canceling the request with
slices
will cancel each sub-request. -
Due to the nature of
slices
each sub-request won’t get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution. -
Parameters like
requests_per_second
andmax_docs
on a request withslices
are distributed proportionally to each sub-request. Combine that with the point above about distribution being uneven and you should conclude that usingmax_docs
withslices
might not result in exactlymax_docs
documents being reindexed. -
Each sub-request gets a slightly different snapshot of the source, though these are all taken at approximately the same time.
Picking the number of slices
If slicing automatically, setting slices
to auto
will choose a reasonable
number for most indices. If slicing manually or otherwise tuning
automatic slicing, use these guidelines.
Query performance is most efficient when the number of slices
is equal to the
number of shards in the index. If that number is large (e.g. 500),
choose a lower number as too many slices
will hurt performance. Setting
slices
higher than the number of shards generally does not improve efficiency
and adds overhead.
Indexing performance scales linearly across available resources with the number of slices.
Whether query or indexing performance dominates the runtime depends on the documents being reindexed and cluster resources.
Reindex routing
By default if _reindex
sees a document with routing then the routing is
preserved unless it’s changed by the script. You can set routing
on the
dest
request to change this:
keep
-
Sets the routing on the bulk request sent for each match to the routing on the match. This is the default value.
discard
-
Sets the routing on the bulk request sent for each match to
null
. =<some text>
-
Sets the routing on the bulk request sent for each match to all text after the
=
.
For example, you can use the following request to copy all documents from
the source
with the company name cat
into the dest
with
routing set to cat
.
POST _reindex
{
"source": {
"index": "source",
"query": {
"match": {
"company": "cat"
}
}
},
"dest": {
"index": "dest",
"routing": "=cat"
}
}
By default _reindex
uses scroll batches of 1000. You can change the
batch size with the size
field in the source
element:
POST _reindex
{
"source": {
"index": "source",
"size": 100
},
"dest": {
"index": "dest",
"routing": "=cat"
}
}
Reindex with an ingest pipeline
Reindex can also use the Ingest node feature by specifying a
pipeline
like this:
POST _reindex
{
"source": {
"index": "source"
},
"dest": {
"index": "dest",
"pipeline": "some_ingest_pipeline"
}
}
Query parameters
refresh
-
(Optional, enum) If
true
, OpenSearch refreshes the affected shards to make this operation visible to search, ifwait_for
then wait for a refresh to make this operation visible to search, iffalse
do nothing with refreshes. Valid values:true
,false
,wait_for
. Default:false
. timeout
-
(Optional, time units) Period each indexing waits for the following operations:
Defaults to
1m
(one minute). This guarantees OpenSearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur. wait_for_active_shards
-
(Optional, string) The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). Default: 1, the primary shard.See Active shards.
wait_for_completion
-
(Optional, Boolean) If
true
, the request blocks until the operation is complete. Defaults totrue
. requests_per_second
-
(Optional, integer) The throttle for this request in sub-requests per second. Defaults to
-1
(no throttle). require_alias
-
(Optional, Boolean) If
true
, the destination must be an index alias. Defaults tofalse
. scroll
-
(Optional, time units) Specifies how long a consistent view of the index should be maintained for scrolled search.
slices
-
(Optional, integer) The number of slices this task should be divided into. Defaults to 1 meaning the task isn’t sliced into subtasks.
max_docs
-
(Optional, integer) Maximum number of documents to process. Defaults to all documents.
Request body
conflicts
-
(Optional, enum) Set to
proceed
to continue reindexing even if there are conflicts. Defaults toabort
. source
-
index
-
(Required, string) The name of the index or index alias you are copying from. Also accepts a comma-separated list to reindex from multiple sources.
max_docs
-
(Optional, integer) The maximum number of documents to reindex.
query
-
(Optional, query object) Specifies the documents to reindex using the Query DSL.
remote
-
host
-
(Optional, string) The URL for the remote instance of OpenSearch that you want to index from. Required when indexing from remote.
username
-
(Optional, string) The username to use for authentication with the remote host.
password
-
(Optional, string) The password to use for authentication with the remote host.
socket_timeout
-
(Optional, time units) The remote socket read timeout. Defaults to 30 seconds.
connect_timeout
-
(Optional, time units) The remote connection timeout. Defaults to 30 seconds.
size
-
{Optional, integer) The number of documents to index per batch. Use when indexing from remote to ensure that the batches fit within the on-heap buffer, which defaults to a maximum size of 100 MB.
slice
-
id
-
(Optional, integer) Slice ID for manual slicing.
max
-
(Optional, integer) Total number of slices.
sort
-
(Optional, list) A comma-separated list of
<field>:<direction>
pairs to sort by before indexing. Use in conjunction withmax_docs
to control what documents are reindexed.Sort in reindex is deprecated and is not guaranteed to index documents in order. If used in combination with
max_docs
, consider using a query filter instead. _source
-
(Optional, string) If
true
reindexes all source fields. Set to a list to reindex select fields. Defaults totrue
.
dest
-
index
-
(Required, string) The name of the index or index alias you are copying to.
version_type
-
(Optional, enum) The versioning to use for the indexing operation. Valid values:
internal
,external
,external_gt
,external_gte
. See Version types for more information. op_type
-
(Optional, enum) Set to create to only index documents that do not already exist (put if absent). Valid values:
index
,create
. Defaults toindex
. type
-
(Optional, string) Types are deprecated. Document type for reindexed documents. Defaults to
_doc
.WarningTypes in source indices are always ignored, also when not specifying a destination
type
. If explicitly specifying destinationtype
, the specified type must match the type in the destination index or be either unspecified or the special value_doc
.
script
-
source
-
(Optional, string) The script to run to update the document source or metadata when reindexing.
lang
-
(Optional, enum) The script language:
painless
,expression
,mustache
,java
. For more information, see Scripting.
Response body
took
-
(integer) The total milliseconds the entire operation took.
timed_out
-
{Boolean) This flag is set to
true
if any of the requests executed during the reindex timed out. total
-
(integer) The number of documents that were successfully processed.
updated
-
(integer) The number of documents that were successfully updated, i.e. a document with same ID already existed prior to reindex updating it.
created
-
(integer) The number of documents that were successfully created.
deleted
-
(integer) The number of documents that were successfully deleted.
batches
-
(integer) The number of scroll responses pulled back by the reindex.
noops
-
(integer) The number of documents that were ignored because the script used for the reindex returned a
noop
value forctx.op
. version_conflicts
-
(integer) The number of version conflicts that reindex hits.
retries
-
(integer) The number of retries attempted by reindex.
bulk
is the number of bulk actions retried andsearch
is the number of search actions retried. throttled_millis
-
(integer) Number of milliseconds the request slept to conform to
requests_per_second
. requests_per_second
-
(integer) The number of requests per second effectively executed during the reindex.
throttled_until_millis
-
(integer) This field should always be equal to zero in a
_reindex
response. It only has meaning when using the Task API, where it indicates the next time (in milliseconds since epoch) a throttled request will be executed again in order to conform torequests_per_second
. failures
-
(array) Array of failures if there were any unrecoverable errors during the process. If this is non-empty then the request aborted because of those failures. Reindex is implemented using batches and any failure causes the entire process to abort but all failures in the current batch are collected into the array. You can use the
conflicts
option to prevent reindex from aborting on version conflicts.
Examples
Reindex select documents with a query
You can limit the documents by adding a query to the source
.
For example, the following request only copies documents with a user.id
of bilbo
into my-new-index-000001
:
POST _reindex
{
"source": {
"index": "my-index-000001",
"query": {
"term": {
"user.id": "bilbo"
}
}
},
"dest": {
"index": "my-new-index-000001"
}
}
Reindex select documents with max_docs
You can limit the number of processed documents by setting max_docs
.
For example, this request copies a single document from my-index-000001
to
my-new-index-000001
:
POST _reindex
{
"max_docs": 1,
"source": {
"index": "my-index-000001"
},
"dest": {
"index": "my-new-index-000001"
}
}
Reindex from multiple sources
The index
attribute in source
can be a list, allowing you to copy from lots
of sources in one request. This will copy documents from the
my-index-000001
and my-index-000002
indices:
POST _reindex
{
"source": {
"index": ["my-index-000001", "my-index-000002"]
},
"dest": {
"index": "my-new-index-000002"
}
}
|
Note
|
The Reindex API makes no effort to handle ID collisions so the last document written will "win" but the order isn’t usually predictable so it is not a good idea to rely on this behavior. Instead, make sure that IDs are unique using a script. |
Reindex select fields with a source filter
You can use source filtering to reindex a subset of the fields in the original documents.
For example, the following request only reindexes the user.id
and _doc
fields of each document:
POST _reindex
{
"source": {
"index": "my-index-000001",
"_source": ["user.id", "_doc"]
},
"dest": {
"index": "my-new-index-000001"
}
}
Reindex to change the name of a field
_reindex
can be used to build a copy of an index with renamed fields. Say you
create an index containing documents that look like this:
POST my-index-000001/_doc/1?refresh
{
"text": "words words",
"flag": "foo"
}
but you don’t like the name flag
and want to replace it with tag
.
_reindex
can create the other index for you:
POST _reindex
{
"source": {
"index": "my-index-000001"
},
"dest": {
"index": "my-new-index-000001"
},
"script": {
"source": "ctx._source.tag = ctx._source.remove(\"flag\")"
}
}
Now you can get the new document:
GET my-new-index-000001/_doc/1
which will return:
{
"found": true,
"_id": "1",
"_index": "my-new-index-000001",
"_type": "_doc",
"_version": 1,
"_seq_no": 44,
"_primary_term": 1,
"_source": {
"text": "words words",
"tag": "foo"
}
}
Reindex daily indices
You can use _reindex
in combination with Painless to reindex
daily indices to apply a new template to the existing documents.
Assuming you have indices that contain documents like:
PUT metricbeat-2016.05.30/_doc/1?refresh
{"system.cpu.idle.pct": 0.908}
PUT metricbeat-2016.05.31/_doc/1?refresh
{"system.cpu.idle.pct": 0.105}
The new template for the metricbeat-*
indices is already loaded into OpenSearch,
but it applies only to the newly created indices. Painless can be used to reindex
the existing documents and apply the new template.
The script below extracts the date from the index name and creates a new index
with -1
appended. All data from metricbeat-2016.05.31
will be reindexed
into metricbeat-2016.05.31-1
.
POST _reindex
{
"source": {
"index": "metricbeat-*"
},
"dest": {
"index": "metricbeat"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'metricbeat-' + (ctx._index.substring('metricbeat-'.length(), ctx._index.length())) + '-1'"
}
}
All documents from the previous metricbeat indices can now be found in the *-1
indices.
GET metricbeat-2016.05.30-1/_doc/1
GET metricbeat-2016.05.31-1/_doc/1
The previous method can also be used in conjunction with changing a field name to load only the existing data into the new index and rename any fields if needed.
Extract a random subset of the source
_reindex
can be used to extract a random subset of the source for testing:
POST _reindex
{
"max_docs": 10,
"source": {
"index": "my-index-000001",
"query": {
"function_score" : {
"random_score" : {},
"min_score" : 0.9 (1)
}
}
},
"dest": {
"index": "my-new-index-000001"
}
}
-
You may need to adjust the
min_score
depending on the relative amount of data extracted from source.