Search APIs
Search APIs are used to search and aggregate data stored in OpenSearch indices. For an overview and related tutorials, see Search your data.
Most search APIs support multi-target syntax, with the exception of the explain API.
Core search
Search testing
Search templates
Search API
Returns search hits that match the query defined in the request.
 You can provide search queries using the q
query string parameter or request body.
Path parameters
<target>
-
(Optional, string) Comma-separated list of indices and index aliases to search. Wildcard (
*
) expressions are supported.To search all indices in a cluster, omit this parameter or use
_all
or*
.
Query parameters
|
Important
|
Several options for this API can be specified using a query parameter or a request body parameter. If both parameters are specified, only the query parameter is used. |
allow_no_indices
-
(Optional, Boolean) If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
.Defaults to
true
.
allow_partial_search_results
-
(Optional, Boolean) If
true
, returns partial results if there are request timeouts or shard failures. Iffalse
, returns an error with no partial results. Defaults totrue
.To override the default for this field, set the
search.default_allow_partial_results
cluster setting tofalse
. batched_reduce_size
-
(Optional, integer) The number of shard results that should be reduced at once on the coordinating node. This value should be used as a protection mechanism to reduce the memory overhead per search request if the potential number of shards in the request can be large. Defaults to
512
.
ccs_minimize_roundtrips
-
(Optional, Boolean) If
true
, network round-trips between the coordinating node and the remote clusters are minimized when executing cross-cluster search (CCS) requests. See How cross-cluster search handles network delays. Defaults totrue
. docvalue_fields
-
(Optional, string) A comma-separated list of fields to return as the docvalue representation of a field for each hit.
expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
Defaults to
open
. explain
-
(Optional, Boolean) If
true
, returns detailed information about score computation as part of a hit. Defaults tofalse
. from
-
(Optional, integer) Starting document offset. Defaults to
0
.By default, you cannot page through more than 10,000 documents using the
from
andsize
parameters. This limit is set using theindex.max_result_window
index setting.Deep paging or requesting many results at once can result in slow searches. Results are sorted before being returned. Because search requests usually span multiple shards, each shard must generate its own sorted results. These separate results must then be combined and sorted to ensure that the overall order is correct.
As an alternative to deep paging, we recommend using scroll or the
search_after
parameter. ignore_throttled
-
(Optional, Boolean) If
true
, concrete, expanded or aliased indices will be ignored when frozen. Defaults totrue
. ignore_unavailable
-
(Optional, Boolean) If
true
, missing or closed indices are not included in the response. Defaults tofalse
. max_concurrent_shard_requests
-
(Optional, integer) Defines the number of concurrent shard requests per node this search executes concurrently. This value should be used to limit the impact of the search on the cluster in order to limit the number of concurrent shard requests. Defaults to
5
. pre_filter_shard_size
-
(Optional, integer) Defines a threshold that enforces a pre-filter roundtrip to prefilter search shards based on query rewriting if the number of shards the search request expands to exceeds the threshold. This filter roundtrip can limit the number of shards significantly if for instance a shard can not match any documents based on its rewrite method ie. if date filters are mandatory to match but the shard bounds and the query are disjoint. When unspecified, the pre-filter phase is executed if any of these conditions is met:
-
The request targets more than
128
shards. -
The request targets one or more read-only index.
-
The primary sort of the query targets an indexed field.
-
preference
-
(Optional, string) Nodes and shards used for the search. By default, OpenSearch selects from eligible nodes and shards using adaptive replica selection, accounting for allocation awareness.
Valid values for
preference
_only_local
-
Run the search only on shards on the local node.
_local
-
If possible, run the search on shards on the local node. If not, select shards using the default method.
_only_nodes:<node-id>,<node-id>
-
Run the search on only the specified nodes IDs. If suitable shards exist on more than one selected nodes, use shards on those nodes using the default method. If none of the specified nodes are available, select shards from any available node using the default method.
_prefer_nodes:<node-id>,<node-id>
-
If possible, run the search on the specified nodes IDs. If not, select shards using the default method.
_shards:<shard>,<shard>
-
Run the search only on the specified shards. This value can be combined with other
preference
values, but this value must come first. For example:_shards:2,3|_local
- <custom-string>
-
Any string that does not start with
_
. If the cluster state and selected shards do not change, searches using the same<custom-string>
value are routed to the same shards in the same order.
q
-
(Optional, string) Query in the Lucene query string syntax.
You can use the
q
parameter to run a query parameter search. Query parameter searches do not support the full OpenSearch Query DSL but are handy for testing.ImportantThe q
parameter overrides thequery
parameter in the request body. If both parameters are specified, documents matching thequery
request body parameter are not returned. request_cache
-
(Optional, Boolean) If
true
, the caching of search results is enabled for requests wheresize
is0
. See Shard request cache settings. Defaults to index level settings. rest_total_hits_as_int
-
(Optional, Boolean) Indicates whether hits.total should be rendered as an integer or an object in the rest search response. Defaults to
false
. routing
-
(Optional, string) Target the specified primary shard.
scroll
-
(Optional, time value) Period to retain the search context for scrolling. See Scroll search results.
By default, this value cannot exceed
1d
(24 hours). You can change this limit using thesearch.max_keep_alive
cluster-level setting.
search_type
-
(Optional, string) How distributed term frequencies are calculated for relevance scoring.
Valid values for
search_type
query_then_fetch
-
(Default) Distributed term frequencies are calculated locally for each shard running the search. We recommend this option for faster searches with potentially less accurate scoring.
dfs_query_then_fetch
-
Distributed term frequencies are calculated globally, using information gathered from all shards running the search. While this option increases the accuracy of scoring, it adds a round-trip to each shard, which can result in slower searches.
seq_no_primary_term
-
(Optional, Boolean) If
true
, returns sequence number and primary term of the last modification of each hit. See Optimistic concurrency control. size
-
(Optional, integer) Defines the number of hits to return. Defaults to
10
.By default, you cannot page through more than 10,000 documents using the
from
andsize
parameters. This limit is set using theindex.max_result_window
index setting.Deep paging or requesting many results at once can result in slow searches. Results are sorted before being returned. Because search requests usually span multiple shards, each shard must generate its own sorted results. These separate results must then be combined and sorted to ensure that the overall order is correct.
As an alternative to deep paging, we recommend using scroll or the
search_after
parameter.If the
scroll
parameter is specified, this value cannot be0
. sort
-
(Optional, string) A comma-separated list of <field>:<direction> pairs.
_source
-
(Optional) Indicates which source fields are returned for matching documents. These fields are returned in the
hits._source
property of the search response. Defaults totrue
.Valid values for
_source
true
-
(Boolean) The entire document source is returned.
false
-
(Boolean) The document source is not returned.
<string>
-
(string) Comma-separated list of source fields to return. Wildcard (
*
) patterns are supported.
_source_excludes
-
(Optional, string) A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. _source_includes
-
(Optional, string) A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. stats
-
(Optional, string) Specific
tag
of the request for logging and statistical purposes. stored_fields
-
(Optional, string) A comma-separated list of stored fields to return as part of a hit. If no fields are specified, no stored fields are included in the response.
If this field is specified, the
_source
parameter defaults tofalse
. You can pass_source: true
to return both source fields and stored fields in the search response. suggest_field
-
(Optional, string) Specifies which field to use for suggestions.
suggest_text
-
(Optional, string) The source text for which the suggestions should be returned.
terminate_after
-
(Optional, integer) The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early.
Defaults to
0
, which does not terminate query execution early. timeout
-
(Optional, time units) Specifies the period of time to wait for a response. If no response is received before the timeout expires, the request fails and returns an error. Defaults to no timeout.
track_scores
-
(Optional, Boolean) If
true
, calculate and return document scores, even if the scores are not used for sorting. Defaults tofalse
. track_total_hits
-
(Optional, integer or Boolean) Number of hits matching the query to count accurately. Defaults to
10000
.If
true
, the exact number of hits is returned at the cost of some performance. Iffalse
, the response does not include the total number of hits matching the query. typed_keys
-
(Optional, Boolean) If
true
, aggregation and suggester names are be prefixed by their respective types in the response. Defaults totrue
. version
-
(Optional, Boolean) If
true
, returns document version as part of a hit. Defaults tofalse
.
Request body
docvalue_fields
-
(Optional, array of strings and objects) Array of wildcard (
*
) patterns. The request returns doc values for field names matching these patterns in thehits.fields
property of the response.You can specify items in the array as a string or object. See Doc value fields.
Properties of
docvalue_fields
objectsfield
-
(Required, string) Wildcard pattern. The request returns doc values for field names matching this pattern.
format
-
(Optional, string) Format in which the doc values are returned.
For date fields, you can specify a date date
format
. For numeric fields fields, you can specify a DecimalFormat pattern.For other field data types, this parameter is not supported.
fields
-
(Optional, array of strings and objects) Array of wildcard (
*
) patterns. The request returns values for field names matching these patterns in thehits.fields
property of the response.You can specify items in the array as a string or object. See Fields for more details.
Properties of
fields
objectsfield
-
(Required, string) Wildcard pattern. The request returns values for field names matching this pattern.
format
-
(Optional, string) Format in which the values are returned.
The date fields
date
anddate_nanos
accept a date format. Spatial fields accept eithergeojson
for GeoJSON (the default) orwkt
for Well Known Text.For other field data types, this parameter is not supported.
explain
-
(Optional, Boolean) If
true
, returns detailed information about score computation as part of a hit. Defaults tofalse
. from
-
(Optional, integer) Starting document offset. Defaults to
0
.By default, you cannot page through more than 10,000 documents using the
from
andsize
parameters. This limit is set using theindex.max_result_window
index setting.Deep paging or requesting many results at once can result in slow searches. Results are sorted before being returned. Because search requests usually span multiple shards, each shard must generate its own sorted results. These separate results must then be combined and sorted to ensure that the overall order is correct.
As an alternative to deep paging, we recommend using scroll or the
search_after
parameter. indices_boost
-
(Optional, array of objects) Boosts the
_score
of documents from specified indices.Properties of
indices_boost
objects<index>: <boost-value>
-
(Required, float)
<index>
is the name of the index or index alias. Wildcard (*
) expressions are supported.<boost-value>
is the factor by which scores are multiplied.A boost value greater than
1.0
increases the score. A boost value between0
and1.0
decreases the score.
min_score
-
(Optional, float) Minimum
_score
for matching documents. Documents with a lower_score
are not included in the search results.
query
-
(Optional, query object) Defines the search definition using the Query DSL.
seq_no_primary_term
-
(Optional, Boolean) If
true
, returns sequence number and primary term of the last modification of each hit. See Optimistic concurrency control. size
-
(Optional, integer) The number of hits to return. Defaults to
10
.By default, you cannot page through more than 10,000 documents using the
from
andsize
parameters. This limit is set using theindex.max_result_window
index setting.Deep paging or requesting many results at once can result in slow searches. Results are sorted before being returned. Because search requests usually span multiple shards, each shard must generate its own sorted results. These separate results must then be combined and sorted to ensure that the overall order is correct.
As an alternative to deep paging, we recommend using scroll or the
search_after
parameter.If the
scroll
parameter is specified, this value cannot be0
. _source
-
(Optional) Indicates which source fields are returned for matching documents. These fields are returned in the
hits._source
property of the search response. Defaults totrue
.Valid values for
_source
true
-
(Boolean) The entire document source is returned.
false
-
(Boolean) The document source is not returned.
<wildcard_pattern>
-
(string or array of strings) Wildcard (
*
) pattern or array of patterns containing source fields to return. <object>
-
(object) Object containing a list of source fields to include or exclude.
Properties for
<object>
excludes
-
(string or array of strings) Wildcard (
*
) pattern or array of patterns containing source fields to exclude from the response.You can also use this property to exclude fields from the subset specified in
includes
property. includes
-
(string or array of strings) Wildcard (
*
) pattern or array of patterns containing source fields to return.If this property is specified, only these source fields are returned. You can exclude fields from this subset using the
excludes
property.
stats
-
(Optional, array of strings) Stats groups to associate with the search. Each group maintains a statistics aggregation for its associated searches. You can retrieve these stats using the indices stats API.
terminate_after
-
(Optional, integer) The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early.
Defaults to
0
, which does not terminate query execution early. timeout
-
(Optional, time units) Specifies the period of time to wait for a response. If no response is received before the timeout expires, the request fails and returns an error. Defaults to no timeout.
version
-
(Optional, Boolean) If
true
, returns document version as part of a hit. Defaults tofalse
.
Response body
_scroll_id
-
(string) Identifier for the search and its search context.
You can use this scroll ID with the scroll API to retrieve the next batch of search results for the request. See Scroll search results.
This parameter is only returned if the
scroll
query parameter is specified in the request. took
-
(integer) Milliseconds it took OpenSearch to execute the request.
This value is calculated by measuring the time elapsed between receipt of a request on the coordinating node and the time at which the coordinating node is ready to send the response.
Took time includes:
-
Communication time between the coordinating node and data nodes
-
Time the request spends in the
search
thread pool, queued for execution -
Actual execution time
Took time does not include:
-
Time needed to send the request to OpenSearch
-
Time needed to serialize the JSON response
-
Time needed to send the response to a client
-
timed_out
-
(Boolean) If
true
, the request timed out before completion; returned results may be partial or empty. _shards
-
(object) Contains a count of shards used for the request.
Properties of
_shards
total
-
(integer) Total number of shards that require querying, including unallocated shards.
successful
-
(integer) Number of shards that executed the request successfully.
skipped
-
(integer) Number of shards that skipped the request because a lightweight check helped realize that no documents could possibly match on this shard. This typically happens when a search request includes a range filter and the shard only has values that fall outside of that range.
failed
-
(integer) Number of shards that failed to execute the request. Note that shards that are not allocated will be considered neither successful nor failed. Having
failed+successful
less thantotal
is thus an indication that some of the shards were not allocated.
hits
-
(object) Contains returned documents and metadata.
Properties of
hits
total
-
(object) Metadata about the number of returned documents.
Properties of
total
value
-
(integer) Total number of returned documents.
relation
-
(string) Indicates whether the number of returned documents in the
value
parameter is accurate or a lower bound.Values of
relation
:eq
-
Accurate
gte
-
Lower bound, including returned documents
max_score
-
(float) Highest returned document
_score
.This value is
null
for requests that do not sort by_score
.
hits
-
(array of objects) Array of returned document objects.
Properties of
hits
objects_index
-
(string) Name of the index containing the returned document.
_type
-
(string) Mapping type of the returned document. Types are deprecated.
_id
-
(string) Unique identifier for the returned document. This ID is only unique within the returned index.
_score
-
(float) Positive 32-bit floating point number used to determine the relevance of the returned document.
_source
-
(object) Original JSON body passed for the document at index time.
You can use the
_source
parameter to exclude this property from the response or specify which source fields to return. fields
-
(object) Contains field values for the documents. These fields must be specified in the request using one or more of the following request parameters:
This property is returned only if one or more of these parameters are set.
Properties of
fields
<field>
-
(array) Key is the field name. Value is the value for the field.
Examples
GET /my-index-000001/_search
{
"query": {
"term": {
"user.id": "bilbo"
}
}
}
The API returns the following response:
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1.3862942,
"hits": [
{
"_index": "my-index-000001",
"_type" : "_doc",
"_id": "0",
"_score": 1.3862942,
"_source": {
"@timestamp": "2099-11-15T14:12:12",
"http": {
"request": {
"method": "get"
},
"response": {
"status_code": 200,
"bytes": 1070000
},
"version": "1.1"
},
"source": {
"ip": "<IP_ADDRESS>"
},
"message": "GET /search HTTP/1.1<PHONE_NUMBER>",
"user": {
"id": "bilbo"
}
}
}
]
}
}
Scroll API
Retrieves the next batch of results for a scrolling search.
GET /_search/scroll
{
"scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ=="
}
Request
Deprecated.
GET /_search/scroll/<scroll_id>
GET /_search/scroll
Deprecated.
POST /_search/scroll/<scroll_id>
POST /_search/scroll
Description
You can use the scroll API to retrieve large sets of results from a single scrolling search request.
The scroll API requires a scroll ID. To get a scroll ID, submit a
search API request that includes an argument for the
scroll
query parameter. The scroll
parameter indicates how long OpenSearch should retain the
search context for the request.
The search response returns a scroll ID in the _scroll_id
response body
parameter. You can then use the scroll ID with the scroll API to retrieve the
next batch of results for the request.
You can also use the scroll API to specify a new scroll
parameter that extends
or shortens the retention period for the search context.
|
Important
|
Results from a scrolling search reflect the state of the index at the time of the initial search request. Subsequent indexing or document changes only affect later search and scroll requests. |
Path parameters
<scroll_id>
-
(Optional, string) Deprecated. Scroll ID of the search.
ImportantScroll IDs can be long. We recommend only specifying scroll IDs using the scroll_id
request body parameter.
Query parameters
scroll
-
(Optional, time value) Period to retain the search context for scrolling. See Scroll search results.
This value overrides the duration set by the original search API request’s
scroll
parameter.By default, this value cannot exceed
1d
(24 hours). You can change this limit using thesearch.max_keep_alive
cluster-level setting.ImportantYou can also specify this value using the scroll
request body parameter. If both parameters are specified, only the query parameter is used. scroll_id
-
(Optional, string) Deprecated. Scroll ID for the search.
ImportantScroll IDs can be long. We recommend only specifying scroll IDs using the scroll_id
request body parameter. rest_total_hits_as_int
-
(Optional, Boolean) If
true
, the API response’shit.total
property is returned as an integer. Iffalse
, the API response’shit.total
property is returned as an object. Defaults tofalse
.
Request body
scroll
-
(Optional, time value) Period to retain the search context for scrolling. See Scroll search results.
This value overrides the duration set by the original search API request’s
scroll
parameter.By default, this value cannot exceed
1d
(24 hours). You can change this limit using thesearch.max_keep_alive
cluster-level setting.ImportantYou can also specify this value using the scroll
query parameter. If both parameters are specified, only the query parameter is used.
scroll_id
-
(Required, string) Scroll ID for the search.
Response body
The scroll API returns the same response body as the search API. See the search API’s response body parameters.
Clear scroll API
Clears the search context and results for a scrolling search.
DELETE /_search/scroll
{
"scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ=="
}
Request
DELETE /_search/scroll/<scroll_id>
DELETE /_search/scroll
Path parameters
<scroll_id>
-
(Optional, string) Deprecated. Comma-separated list of scroll IDs to clear. To clear all scroll IDs, use
_all
.ImportantScroll IDs can be long. We recommend only specifying scroll IDs using the scroll_id
request body parameter.
Query parameters
scroll_id
-
(Optional, string) Deprecated. Comma-separated list of scroll IDs to clear. To clear all scroll IDs, use
_all
.ImportantScroll IDs can be long. We recommend only specifying scroll IDs using the scroll_id
request body parameter.
Request body
scroll_id
-
(Required, string or array of strings) Scroll IDs to clear. To clear all scroll IDs, use
_all
.
Response body
succeeded
-
(Boolean) If
true
, the request succeeded. This does not indicate whether any scrolling search requests were cleared. num_freed
-
(integer) Number of scrolling search requests cleared.
Search Template
Allows you to use the mustache language to pre render search requests.
GET _search/template
{
"source" : {
"query": { "match" : { "{{my_field}}" : "{{my_value}}" } },
"size" : "{{my_size}}"
},
"params" : {
"my_field" : "message",
"my_value" : "foo",
"my_size" : 5
}
}
Request
GET _search/template
Description
The /_search/template
endpoint allows you to use the mustache language to pre-
render search requests, before they are executed and fill existing templates
with template parameters.
For more information on how Mustache templating and what kind of templating you can do with it check out the online documentation of the mustache project.
|
Note
|
The mustache language is implemented in OpenSearch as a sandboxed scripting language, hence it obeys settings that may be used to enable or disable scripts per type and context as described in the scripting docs. |
Path parameters
<index>
-
(Optional, string) Comma-separated list or wildcard expression of index names used to limit the request.
Query parameters
allow_no_indices
-
(Optional, Boolean) If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
.Defaults to
true
. ccs_minimize_roundtrips
-
(Optional, Boolean) If
true
, network round-trips are minimized for cross-cluster search requests. Defaults totrue
. expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
explain
-
(Optional, Boolean) If
true
, the response includes additional details about score computation as part of a hit. Defaults tofalse
. ignore_throttled
-
(Optional, Boolean) If
true
, specified concrete, expanded or aliased indices are not included in the response when throttled. Defaults totrue
. ignore_unavailable
-
(Optional, Boolean) If
true
, missing or closed indices are not included in the response. Defaults tofalse
. preference
-
(Optional, string) Specifies the node or shard the operation should be performed on. Random by default.
profile
-
(Optional, Boolean) If
true
, the query execution is profiled. Defaults tofalse
. rest_total_hits_as_int
-
(Optional, Boolean) If
true
,hits.total
are rendered as an integer in the response. Defaults tofalse
. routing
-
(Optional, string) Target the specified primary shard.
scroll
-
(Optional, time units) Specifies how long a consistent view of the index should be maintained for scrolled search.
search_type
-
(Optional, string) The type of the search operation. Available options:
-
query_then_fetch
-
dfs_query_then_fetch
-
typed_keys
-
(Optional, Boolean) If
true
, aggregation and suggester names are prefixed by their respective types in the response. Defaults tofalse
.
Request body
The API request body must contain the search definition template and its parameters.
Response codes
Store a search template
You can store a search template using the stored scripts API.
POST _scripts/<templateid>
{
"script": {
"lang": "mustache",
"source": {
"query": {
"match": {
"title": "{{query_string}}"
}
}
}
}
}
The template can be retrieved by calling
GET _scripts/<templateid>
The API returns the following result:
{
"script" : {
"lang" : "mustache",
"source" : """{"query":{"match":{"title":"{{query_string}}"}}}""",
"options": {
"content_type" : "application/json; charset=UTF-8"
}
},
"_id": "<templateid>",
"found": true
}
This template can be deleted by calling
DELETE _scripts/<templateid>
Using a stored search template
To use a stored template at search time send the following request:
GET _search/template
{
"id": "<templateid>", (1)
"params": {
"query_string": "search for these words"
}
}
-
Name of the stored template script.
Validating a search template
A template can be rendered in a response with given parameters by using the following request:
GET _render/template
{
"source": "{ \"query\": { \"terms\": {{#toJson}}statuses{{/toJson}} }}",
"params": {
"statuses" : {
"status": [ "pending", "published" ]
}
}
}
The API returns the rendered template:
{
"template_output": {
"query": {
"terms": {
"status": [ (1)
"pending",
"published"
]
}
}
}
}
-
status
array has been populated with values from theparams
object.
Stored templates can also be rendered by calling the following request:
GET _render/template/<template_name>
{
"params": {
"..."
}
}
Using the explain parameter
You can use the explain
parameter when running a template:
GET _search/template
{
"id": "my_template",
"params": {
"status": [ "pending", "published" ]
},
"explain": true
}
Profiling
You can use the profile
parameter when running a template:
GET _search/template
{
"id": "my_template",
"params": {
"status": [ "pending", "published" ]
},
"profile": true
}
Filling in a query string with a single value
GET _search/template
{
"source": {
"query": {
"term": {
"message": "{{query_string}}"
}
}
},
"params": {
"query_string": "search for these words"
}
}
Converting parameters to JSON
The {{#toJson}}parameter{{/toJson}}
function can be used to convert parameters
like maps and array to their JSON representation:
GET _search/template
{
"source": "{ \"query\": { \"terms\": {{#toJson}}statuses{{/toJson}} }}",
"params": {
"statuses" : {
"status": [ "pending", "published" ]
}
}
}
which is rendered as:
{
"query": {
"terms": {
"status": [
"pending",
"published"
]
}
}
}
A more complex example substitutes an array of JSON objects:
GET _search/template
{
"source": "{\"query\":{\"bool\":{\"must\": {{#toJson}}clauses{{/toJson}} }}}",
"params": {
"clauses": [
{ "term": { "user" : "foo" } },
{ "term": { "user" : "bar" } }
]
}
}
which is rendered as:
{
"query": {
"bool": {
"must": [
{
"term": {
"user": "foo"
}
},
{
"term": {
"user": "bar"
}
}
]
}
}
}
Concatenating array of values
The {{#join}}array{{/join}}
function can be used to concatenate the
values of an array as a comma delimited string:
GET _search/template
{
"source": {
"query": {
"match": {
"emails": "{{#join}}emails{{/join}}"
}
}
},
"params": {
"emails": [<EMAIL_ADDRESS><EMAIL_ADDRESS>]
}
}
which is rendered as:
{
"query" : {
"match" : {
"emails" : "username<EMAIL_ADDRESS>}
}
}
The function also accepts a custom delimiter:
GET _search/template
{
"source": {
"query": {
"range": {
"born": {
"gte" : "{{date.min}}",
"lte" : "{{date.max}}",
"format": "{{#join delimiter='||'}}date.formats{{/join delimiter='||'}}"
}
}
}
},
"params": {
"date": {
"min": "2016",
"max": "31/12/2017",
"formats": ["dd/MM/yyyy", "yyyy"]
}
}
}
which is rendered as:
{
"query": {
"range": {
"born": {
"gte": "2016",
"lte": "31/12/2017",
"format": "dd/MM/yyyy||yyyy"
}
}
}
}
Default values
A default value is written as {{var}}{{^var}}default{{/var}}
for instance:
{
"source": {
"query": {
"range": {
"line_no": {
"gte": "{{start}}",
"lte": "{{end}}{{^end}}20{{/end}}"
}
}
}
},
"params": { ... }
}
When params
is { "start": 10, "end": 15 }
this query would be rendered as:
{
"range": {
"line_no": {
"gte": "10",
"lte": "15"
}
}
}
But when params
is { "start": 10 }
this query would use the default value
for end
:
{
"range": {
"line_no": {
"gte": "10",
"lte": "20"
}
}
}
Conditional clauses
Conditional clauses cannot be expressed using the JSON form of the template.
Instead, the template must be passed as a string. For instance, let’s say
we wanted to run a match
query on the line
field, and optionally wanted
to filter by line numbers, where start
and end
are optional.
The params
would look like:
{
"params": {
"text": "words to search for",
"line_no": { (1)
"start": 10,
"end": 20
}
}
}
-
The
line_no
,start
, andend
parameters are optional.
When written as a query, the template would include invalid JSON, such as
section markers like {{#line_no}}
:
{
"query": {
"bool": {
"must": {
"match": {
"line": "{{text}}" (1)
}
},
"filter": {
{{#line_no}} (2)
"range": {
"line_no": {
{{#start}} (3)
"gte": "{{start}}" (4)
{{#end}},{{/end}} (5)
{{/start}}
{{#end}} (6)
"lte": "{{end}}" (7)
{{/end}}
}
}
{{/line_no}}
}
}
}
}
-
Fill in the value of param
text
-
Include the
range
filter only ifline_no
is specified -
Include the
gte
clause only ifline_no.start
is specified -
Fill in the value of param
line_no.start
-
Add a comma after the
gte
clause only ifline_no.start
ANDline_no.end
are specified -
Include the
lte
clause only ifline_no.end
is specified -
Fill in the value of param
line_no.end
Because search templates cannot include invalid JSON, you can pass the same query as a string instead:
"source": "{\"query\":{\"bool\":{\"must\":{\"match\":{\"line\":\"{{text}}\"}},\"filter\":{{{#line_no}}\"range\":{\"line_no\":{{{#start}}\"gte\":\"{{start}}\"{{#end}},{{/end}}{{/start}}{{#end}}\"lte\":\"{{end}}\"{{/end}}}}{{/line_no}}}}}}"
Encoding URLs
The {{#url}}value{{/url}}
function can be used to encode a string value
in a HTML encoding form as defined in by the
HTML specification.
As an example, it is useful to encode a URL:
GET _render/template
{
"source": {
"query": {
"term": {
"http_access_log": "{{#url}}{{host}}/{{page}}{{/url}}"
}
}
},
"params": {
"host": "https://www.opensearch.org/",
"page": "learn"
}
}
The previous query will be rendered as:
{
"template_output": {
"query": {
"term": {
"http_access_log": "https%3A%2F%2Fwww.opensearch.org%2F%2Flearn"
}
}
}
}
Multi Search Template
Allows to execute several search template requests.
Request
GET _msearch/template
Description
Allows to execute several search template requests within the same API using the
_msearch/template
endpoint.
The format of the request is similar to the Multi Search API format:
header\n
body\n
header\n
body\n
The header part supports the same index
, search_type
, preference
, and
routing
options as the Multi Search API.
The body includes a search template body request and supports inline, stored and file templates.
Examples
$ cat requests
{"index": "test"}
{"source": {"query": {"match": {"user" : "{{username}}" }}}, "params": {"username": "john"}} (1)
{"source": {"query": {"{{query_type}}": {"name": "{{name}}" }}}, "params": {"query_type": "match_phrase_prefix", "name": "Smith"}}
{"index": "_all"}
{"id": "template_1", "params": {"query_string": "search for these words" }} (2)
$ curl -H "Content-Type: application/x-ndjson" -XGET localhost:9200/_msearch/template --data-binary "@requests"; echo
-
Inline search template request
-
Search template request based on a stored template
The response returns a responses
array, which includes the search template
response for each search template request matching its order in the original
multi search template request. If there was a complete failure for that specific
search template request, an object with error
message will be returned in
place of the actual search response.
Search Shards API
Returns the indices and shards that a search request would be executed against.
GET /my-index-000001/_search_shards
Request
GET /<index>/_search_shards
Description
The search shards api returns the indices and shards that a search request would
be executed against. This can give useful feedback for working out issues or
planning optimizations with routing and shard preferences. When filtered aliases
are used, the filter is returned as part of the indices
section.
Path parameters
<index>
-
(Optional, string) Comma-separated list or wildcard expression of index names used to limit the request.
Query parameters
allow_no_indices
-
(Optional, Boolean) If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
.Defaults to
true
. expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
Defaults to
open
. ignore_unavailable
-
(Optional, Boolean) If
true
, missing or closed indices are not included in the response. Defaults tofalse
. local
-
(Optional, Boolean) If
true
, the request retrieves information from the local node only. Defaults tofalse
, which means information is retrieved from the master node. preference
-
(Optional, string) Specifies the node or shard the operation should be performed on. Random by default.
routing
-
(Optional, string) Target the specified primary shard.
Examples
GET /my-index-000001/_search_shards
The API returns the following result:
{
"nodes": ...,
"indices" : {
"my-index-000001": { }
},
"shards": [
[
{
"index": "my-index-000001",
"node": "JklnKbD7Tyqi9TP3_Q_tBg",
"primary": true,
"shard": 0,
"state": "STARTED",
"allocation_id": {"id":"0TvkCyF7TAmM1wHP4a42-A"},
"relocating_node": null
}
],
[
{
"index": "my-index-000001",
"node": "JklnKbD7Tyqi9TP3_Q_tBg",
"primary": true,
"shard": 1,
"state": "STARTED",
"allocation_id": {"id":"fMju3hd1QHWmWrIgFnI4Ww"},
"relocating_node": null
}
],
[
{
"index": "my-index-000001",
"node": "JklnKbD7Tyqi9TP3_Q_tBg",
"primary": true,
"shard": 2,
"state": "STARTED",
"allocation_id": {"id":"Nwl0wbMBTHCWjEEbGYGapg"},
"relocating_node": null
}
],
[
{
"index": "my-index-000001",
"node": "JklnKbD7Tyqi9TP3_Q_tBg",
"primary": true,
"shard": 3,
"state": "STARTED",
"allocation_id": {"id":"bU_KLGJISbW0RejwnwDPKw"},
"relocating_node": null
}
],
[
{
"index": "my-index-000001",
"node": "JklnKbD7Tyqi9TP3_Q_tBg",
"primary": true,
"shard": 4,
"state": "STARTED",
"allocation_id": {"id":"DMs7_giNSwmdqVukF7UydA"},
"relocating_node": null
}
]
]
}
Specifying the same request, this time with a routing value:
GET /my-index-000001/_search_shards?routing=foo,bar
The API returns the following result:
{
"nodes": ...,
"indices" : {
"my-index-000001": { }
},
"shards": [
[
{
"index": "my-index-000001",
"node": "JklnKbD7Tyqi9TP3_Q_tBg",
"primary": true,
"shard": 2,
"state": "STARTED",
"allocation_id": {"id":"fMju3hd1QHWmWrIgFnI4Ww"},
"relocating_node": null
}
],
[
{
"index": "my-index-000001",
"node": "JklnKbD7Tyqi9TP3_Q_tBg",
"primary": true,
"shard": 3,
"state": "STARTED",
"allocation_id": {"id":"0TvkCyF7TAmM1wHP4a42-A"},
"relocating_node": null
}
]
]
}
Because of the specified routing values, the search is only executed against two of the shards.
Suggesters
Suggests similar looking terms based on a provided text by using a suggester. Parts of the suggest feature are still under development.
POST my-index-000001/_search
{
"query" : {
"match": {
"message": "tring out OpenSearch"
}
},
"suggest" : {
"my-suggestion" : {
"text" : "tring out OpenSearch",
"term" : {
"field" : "message"
}
}
}
}
Request
The suggest feature suggests similar looking terms based on a provided text by
using a suggester. The suggest request part is defined alongside the query part
in a _search
request. If the query part is left out, only suggestions are
returned.
|
Note
|
_suggest endpoint has been deprecated in favour of using suggest via
_search endpoint. In 5.0, the _search endpoint has been optimized for
suggest only search requests.
|
Examples
Several suggestions can be specified per request. Each suggestion is identified
with an arbitrary name. In the example below two suggestions are requested. Both
my-suggest-1
and my-suggest-2
suggestions use the term
suggester, but have
a different text
.
POST _search
{
"suggest": {
"my-suggest-1" : {
"text" : "tring out OpenSearch",
"term" : {
"field" : "message"
}
},
"my-suggest-2" : {
"text" : "kmichy",
"term" : {
"field" : "user.id"
}
}
}
}
The below suggest response example includes the suggestion response for
my-suggest-1
and my-suggest-2
. Each suggestion part contains
entries. Each entry is effectively a token from the suggest text and
contains the suggestion entry text, the original start offset and length
in the suggest text and if found an arbitrary number of options.
{
"_shards": ...
"hits": ...
"took": 2,
"timed_out": false,
"suggest": {
"my-suggest-1": [ {
"text": "tring",
"offset": 0,
"length": 5,
"options": [ {"text": "trying", "score": 0.8, "freq": 1 } ]
}, {
"text": "out",
"offset": 6,
"length": 3,
"options": []
}, {
"text": "opensearch",
"offset": 10,
"length": 10,
"options": []
} ],
"my-suggest-2": ...
}
}
Each options array contains an option object that includes the suggested text, its document frequency and score compared to the suggest entry text. The meaning of the score depends on the used suggester. The term suggester’s score is based on the edit distance.
Global suggest text
To avoid repetition of the suggest text, it is possible to define a
global text. In the example below the suggest text is defined globally
and applies to the my-suggest-1
and my-suggest-2
suggestions.
POST _search
{
"suggest": {
"text" : "tring out OpenSearch",
"my-suggest-1" : {
"term" : {
"field" : "message"
}
},
"my-suggest-2" : {
"term" : {
"field" : "user"
}
}
}
}
The suggest text can in the above example also be specified as suggestion specific option. The suggest text specified on suggestion level override the suggest text on the global level.
Term suggester
|
Note
|
In order to understand the format of suggestions, please read the Suggesters page first. |
The term
suggester suggests terms based on edit distance. The provided
suggest text is analyzed before terms are suggested. The suggested terms
are provided per analyzed suggest text token. The term
suggester
doesn’t take the query into account that is part of request.
Common suggest options:
text
|
The suggest text. The suggest text is a required option that needs to be set globally or per suggestion. |
field
|
The field to fetch the candidate suggestions from. This is a required option that either needs to be set globally or per suggestion. |
analyzer
|
The analyzer to analyse the suggest text with. Defaults to the search analyzer of the suggest field. |
size
|
The maximum corrections to be returned per suggest text token. |
sort
|
Defines how suggestions should be sorted per suggest text term. Two possible values:
|
suggest_mode
|
The suggest mode controls what suggestions are included or controls for what suggest text terms, suggestions should be suggested. Three possible values can be specified:
|
Other term suggest options:
max_edits
|
The maximum edit distance candidate suggestions can have in order to be considered as a suggestion. Can only be a value between 1 and 2. Any other value results in a bad request error being thrown. Defaults to 2. |
prefix_length
|
The number of minimal prefix characters that must match in order be a candidate for suggestions. Defaults to 1. Increasing this number improves spellcheck performance. Usually misspellings don’t occur in the beginning of terms. (Old name "prefix_len" is deprecated) |
min_word_length
|
The minimum length a suggest text term must have in order to be included. Defaults to 4. (Old name "min_word_len" is deprecated) |
shard_size
|
Sets the maximum number of suggestions to be retrieved
from each individual shard. During the reduce phase only the top N
suggestions are returned based on the |
max_inspections
|
A factor that is used to multiply with the
|
min_doc_freq
|
The minimal threshold in number of documents a suggestion should appear in. This can be specified as an absolute number or as a relative percentage of number of documents. This can improve quality by only suggesting high frequency terms. Defaults to 0f and is not enabled. If a value higher than 1 is specified, then the number cannot be fractional. The shard level document frequencies are used for this option. |
max_term_freq
|
The maximum threshold in number of documents in which a suggest text token can exist in order to be included. Can be a relative percentage number (e.g., 0.4) or an absolute number to represent document frequencies. If a value higher than 1 is specified, then fractional can not be specified. Defaults to 0.01f. This can be used to exclude high frequency terms — which are usually spelled correctly — from being spellchecked. This also improves the spellcheck performance. The shard level document frequencies are used for this option. |
string_distance
|
Which string distance implementation to use for comparing how similar suggested terms are. Five possible values can be specified:
|
Phrase Suggester
|
Note
|
In order to understand the format of suggestions, please read the Suggesters page first. |
The term
suggester provides a very convenient API to access word
alternatives on a per token basis within a certain string distance. The API
allows accessing each token in the stream individually while
suggest-selection is left to the API consumer. Yet, often pre-selected
suggestions are required in order to present to the end-user. The
phrase
suggester adds additional logic on top of the term
suggester
to select entire corrected phrases instead of individual tokens weighted
based on ngram-language
models. In practice this suggester will be
able to make better decisions about which tokens to pick based on
co-occurrence and frequencies.
API Example
In general the phrase
suggester requires special mapping up front to work.
The phrase
suggester examples on this page need the following mapping to
work. The reverse
analyzer is used only in the last example.
PUT test
{
"settings": {
"index": {
"number_of_shards": 1,
"analysis": {
"analyzer": {
"trigram": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase","shingle"]
},
"reverse": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase","reverse"]
}
},
"filter": {
"shingle": {
"type": "shingle",
"min_shingle_size": 2,
"max_shingle_size": 3
}
}
}
}
},
"mappings": {
"properties": {
"title": {
"type": "text",
"fields": {
"trigram": {
"type": "text",
"analyzer": "trigram"
},
"reverse": {
"type": "text",
"analyzer": "reverse"
}
}
}
}
}
}
POST test/_doc?refresh=true
{"title": "noble warriors"}
POST test/_doc?refresh=true
{"title": "nobel prize"}
Once you have the analyzers and mappings set up you can use the phrase
suggester in the same spot you’d use the term
suggester:
POST test/_search
{
"suggest": {
"text": "noble prize",
"simple_phrase": {
"phrase": {
"field": "title.trigram",
"size": 1,
"gram_size": 3,
"direct_generator": [ {
"field": "title.trigram",
"suggest_mode": "always"
} ],
"highlight": {
"pre_tag": "<em>",
"post_tag": "</em>"
}
}
}
}
}
The response contains suggestions scored by the most likely spelling correction first. In this case we received the expected correction "nobel prize".
{
"_shards": ...
"hits": ...
"timed_out": false,
"took": 3,
"suggest": {
"simple_phrase" : [
{
"text" : "noble prize",
"offset" : 0,
"length" : 11,
"options" : [ {
"text" : "nobel prize",
"highlighted": "<em>nobel</em> prize",
"score" : 0.48614594
}]
}
]
}
}
Basic Phrase suggest API parameters
field
|
The name of the field used to do n-gram lookups for the language model, the suggester will use this field to gain statistics to score corrections. This field is mandatory. |
gram_size
|
Sets max size of the n-grams (shingles) in the |
real_word_error_likelihood
|
The likelihood of a term being
misspelled even if the term exists in the dictionary. The default is
|
confidence
|
The confidence level defines a factor applied to the
input phrases score which is used as a threshold for other suggest
candidates. Only candidates that score higher than the threshold will be
included in the result. For instance a confidence level of |
max_errors
|
The maximum percentage of the terms
considered to be misspellings in order to form a correction. This method
accepts a float value in the range |
separator
|
The separator that is used to separate terms in the bigram field. If not set the whitespace character is used as a separator. |
size
|
The number of candidates that are generated for each
individual query term. Low numbers like |
analyzer
|
Sets the analyzer to analyze to suggest text with.
Defaults to the search analyzer of the suggest field passed via |
shard_size
|
Sets the maximum number of suggested terms to be
retrieved from each individual shard. During the reduce phase, only the
top N suggestions are returned based on the |
text
|
Sets the text / query to provide suggestions for. |
highlight
|
Sets up suggestion highlighting. If not provided then
no |
collate
|
Checks each suggestion against the specified |
POST test/_search
{
"suggest": {
"text" : "noble prize",
"simple_phrase" : {
"phrase" : {
"field" : "title.trigram",
"size" : 1,
"direct_generator" : [ {
"field" : "title.trigram",
"suggest_mode" : "always",
"min_word_length" : 1
} ],
"collate": {
"query": { (1)
"source" : {
"match": {
"{{field_name}}" : "{{suggestion}}" (2)
}
}
},
"params": {"field_name" : "title"}, (3)
"prune": true (4)
}
}
}
}
}
-
This query will be run once for every suggestion.
-
The
{{suggestion}}
variable will be replaced by the text of each suggestion. -
An additional
field_name
variable has been specified inparams
and is used by thematch
query. -
All suggestions will be returned with an extra
collate_match
option indicating whether the generated phrase matched any document.
Smoothing Models
The phrase
suggester supports multiple smoothing models to balance
weight between infrequent grams (grams (shingles) are not existing in
the index) and frequent grams (appear at least once in the index). The
smoothing model can be selected by setting the smoothing
parameter
to one of the following options. Each smoothing model supports specific
properties that can be configured.
stupid_backoff
|
A simple backoff model that backs off to lower
order n-gram models if the higher order count is |
laplace
|
A smoothing model that uses an additive smoothing where a
constant (typically |
linear_interpolation
|
A smoothing model that takes the weighted
mean of the unigrams, bigrams, and trigrams based on user supplied
weights (lambdas). Linear Interpolation doesn’t have any default values.
All parameters ( |
POST test/_search
{
"suggest": {
"text" : "obel prize",
"simple_phrase" : {
"phrase" : {
"field" : "title.trigram",
"size" : 1,
"smoothing" : {
"laplace" : {
"alpha" : 0.7
}
}
}
}
}
}
Candidate Generators
The phrase
suggester uses candidate generators to produce a list of
possible terms per term in the given text. A single candidate generator
is similar to a term
suggester called for each individual term in the
text. The output of the generators is subsequently scored in combination
with the candidates from the other terms for suggestion candidates.
Currently only one type of candidate generator is supported, the
direct_generator
. The Phrase suggest API accepts a list of generators
under the key direct_generator
; each of the generators in the list is
called per term in the original text.
Direct Generators
The direct generators support the following parameters:
field
|
The field to fetch the candidate suggestions from. This is a required option that either needs to be set globally or per suggestion. |
size
|
The maximum corrections to be returned per suggest text token. |
suggest_mode
|
The suggest mode controls what suggestions are included on the suggestions
generated on each shard. All values other than
|
max_edits
|
The maximum edit distance candidate suggestions can have in order to be considered as a suggestion. Can only be a value between 1 and 2. Any other value results in a bad request error being thrown. Defaults to 2. |
prefix_length
|
The number of minimal prefix characters that must match in order be a candidate suggestions. Defaults to 1. Increasing this number improves spellcheck performance. Usually misspellings don’t occur in the beginning of terms. (Old name "prefix_len" is deprecated) |
min_word_length
|
The minimum length a suggest text term must have in order to be included. Defaults to 4. (Old name "min_word_len" is deprecated) |
max_inspections
|
A factor that is used to multiply with the
|
min_doc_freq
|
The minimal threshold in number of documents a suggestion should appear in. This can be specified as an absolute number or as a relative percentage of number of documents. This can improve quality by only suggesting high frequency terms. Defaults to 0f and is not enabled. If a value higher than 1 is specified, then the number cannot be fractional. The shard level document frequencies are used for this option. |
max_term_freq
|
The maximum threshold in number of documents in which a suggest text token can exist in order to be included. Can be a relative percentage number (e.g., 0.4) or an absolute number to represent document frequencies. If a value higher than 1 is specified, then fractional can not be specified. Defaults to 0.01f. This can be used to exclude high frequency terms — which are usually spelled correctly — from being spellchecked. This also improves the spellcheck performance. The shard level document frequencies are used for this option. |
pre_filter
|
A filter (analyzer) that is applied to each of the tokens passed to this candidate generator. This filter is applied to the original token before candidates are generated. |
post_filter
|
A filter (analyzer) that is applied to each of the generated tokens before they are passed to the actual phrase scorer. |
The following example shows a phrase
suggest call with two generators:
the first one is using a field containing ordinary indexed terms, and the
second one uses a field that uses terms indexed with a reverse
filter
(tokens are index in reverse order). This is used to overcome the limitation
of the direct generators to require a constant prefix to provide
high-performance suggestions. The pre_filter
and post_filter
options
accept ordinary analyzer names.
POST test/_search
{
"suggest": {
"text" : "obel prize",
"simple_phrase" : {
"phrase" : {
"field" : "title.trigram",
"size" : 1,
"direct_generator" : [ {
"field" : "title.trigram",
"suggest_mode" : "always"
}, {
"field" : "title.reverse",
"suggest_mode" : "always",
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}
}
pre_filter
and post_filter
can also be used to inject synonyms after
candidates are generated. For instance for the query captain usq
we
might generate a candidate usa
for the term usq
, which is a synonym for
america
. This allows us to present captain america
to the user if this
phrase scores high enough.
Completion Suggester
|
Note
|
In order to understand the format of suggestions, please
read the Suggesters page first. For more flexible
search-as-you-type searches that do not use suggesters, see the
search_as_you_type field type.
|
The completion
suggester provides auto-complete/search-as-you-type
functionality. This is a navigational feature to guide users to
relevant results as they are typing, improving search precision.
It is not meant for spell correction or did-you-mean functionality
like the term
or phrase
suggesters.
Ideally, auto-complete functionality should be as fast as a user
types to provide instant feedback relevant to what a user has already
typed in. Hence, completion
suggester is optimized for speed.
The suggester uses data structures that enable fast lookups,
but are costly to build and are stored in-memory.
Mapping
To use this feature, specify a special mapping for this field, which indexes the field values for fast completions.
PUT music
{
"mappings": {
"properties": {
"suggest": {
"type": "completion"
},
"title": {
"type": "keyword"
}
}
}
}
Mapping supports the following parameters:
analyzer
|
The index analyzer to use, defaults to |
search_analyzer
|
The search analyzer to use, defaults to value of |
preserve_separators
|
Preserves the separators, defaults to |
preserve_position_increments
|
Enables position increments, defaults to |
max_input_length
|
Limits the length of a single input, defaults to |
Indexing
You index suggestions like any other field. A suggestion is made of an
input
and an optional weight
attribute. An input
is the expected
text to be matched by a suggestion query and the weight
determines how
the suggestions will be scored. Indexing a suggestion is as follows:
PUT music/_doc/1?refresh
{
"suggest" : {
"input": [ "Nevermind", "Nirvana" ],
"weight" : 34
}
}
The following parameters are supported:
input
|
The input to store, this can be an array of strings or just a string. This field is mandatory.
|
||
weight
|
A positive integer or a string containing a positive integer, which defines a weight and allows you to rank your suggestions. This field is optional. |
You can index multiple suggestions for a document as follows:
PUT music/_doc/1?refresh
{
"suggest": [
{
"input": "Nevermind",
"weight": 10
},
{
"input": "Nirvana",
"weight": 3
}
]
}
You can use the following shorthand form. Note that you can not specify a weight with suggestion(s) in the shorthand form.
PUT music/_doc/1?refresh
{
"suggest" : [ "Nevermind", "Nirvana" ]
}
Querying
Suggesting works as usual, except that you have to specify the suggest
type as completion
. Suggestions are near real-time, which means
new suggestions can be made visible by refresh and
documents once deleted are never shown. This request:
POST music/_search?pretty
{
"suggest": {
"song-suggest": {
"prefix": "nir", (1)
"completion": { (2)
"field": "suggest" (3)
}
}
}
}
-
Prefix used to search for suggestions
-
Type of suggestions
-
Name of the field to search for suggestions in
returns this response:
{
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits": ...
"took": 2,
"timed_out": false,
"suggest": {
"song-suggest" : [ {
"text" : "nir",
"offset" : 0,
"length" : 3,
"options" : [ {
"text" : "Nirvana",
"_index": "music",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"suggest": ["Nevermind", "Nirvana"]
}
} ]
} ]
}
}
|
Important
|
_source metadata field must be enabled, which is the default
behavior, to enable returning _source with suggestions.
|
The configured weight for a suggestion is returned as _score
. The
text
field uses the input
of your indexed suggestion. Suggestions
return the full document _source
by default. The size of the _source
can impact performance due to disk fetch and network transport overhead.
To save some network overhead, filter out unnecessary fields from the _source
using source filtering to minimize
_source
size. Note that the _suggest endpoint doesn’t support source
filtering but using suggest on the _search
endpoint does:
POST music/_search
{
"_source": "suggest", (1)
"suggest": {
"song-suggest": {
"prefix": "nir",
"completion": {
"field": "suggest", (2)
"size": 5 (3)
}
}
}
}
-
Filter the source to return only the
suggest
field -
Name of the field to search for suggestions in
-
Number of suggestions to return
Which should look like:
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
},
"suggest": {
"song-suggest": [ {
"text": "nir",
"offset": 0,
"length": 3,
"options": [ {
"text": "Nirvana",
"_index": "music",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"suggest": [ "Nevermind", "Nirvana" ]
}
} ]
} ]
}
}
The basic completion suggester query supports the following parameters:
field
|
The name of the field on which to run the query (required). |
size
|
The number of suggestions to return (defaults to |
skip_duplicates
|
Whether duplicate suggestions should be filtered out (defaults to |
|
Note
|
The completion suggester considers all documents in the index. See Context Suggester for an explanation of how to query a subset of documents instead. |
|
Note
|
In case of completion queries spanning more than one shard, the suggest is executed in two phases, where the last phase fetches the relevant documents from shards, implying executing completion requests against a single shard is more performant due to the document fetch overhead when the suggest spans multiple shards. To get best performance for completions, it is recommended to index completions into a single shard index. In case of high heap usage due to shard size, it is still recommended to break index into multiple shards instead of optimizing for completion performance. |
Skip duplicate suggestions
Queries can return duplicate suggestions coming from different documents.
It is possible to modify this behavior by setting skip_duplicates
to true.
When set, this option filters out documents with duplicate suggestions from the result.
POST music/_search?pretty
{
"suggest": {
"song-suggest": {
"prefix": "nor",
"completion": {
"field": "suggest",
"skip_duplicates": true
}
}
}
}
|
Warning
|
When set to true, this option can slow down search because more suggestions need to be visited to find the top N. |
Fuzzy queries
The completion suggester also supports fuzzy queries — this means you can have a typo in your search and still get results back.
POST music/_search?pretty
{
"suggest": {
"song-suggest": {
"prefix": "nor",
"completion": {
"field": "suggest",
"fuzzy": {
"fuzziness": 2
}
}
}
}
}
Suggestions that share the longest prefix to the query prefix
will
be scored higher.
The fuzzy query can take specific fuzzy parameters. The following parameters are supported:
fuzziness
|
The fuzziness factor, defaults to |
transpositions
|
if set to |
min_length
|
Minimum length of the input before fuzzy
suggestions are returned, defaults |
prefix_length
|
Minimum length of the input, which is not
checked for fuzzy alternatives, defaults to |
unicode_aware
|
If |
|
Note
|
If you want to stick with the default values, but
still use fuzzy, you can either use fuzzy: {}
or fuzzy: true .
|
Regex queries
The completion suggester also supports regex queries meaning you can express a prefix as a regular expression
POST music/_search?pretty
{
"suggest": {
"song-suggest": {
"regex": "n[ever|i]r",
"completion": {
"field": "suggest"
}
}
}
}
The regex query can take specific regex parameters. The following parameters are supported:
flags
|
Possible flags are |
max_determinized_states
|
Regular expressions are dangerous because it’s easy to accidentally
create an innocuous looking one that requires an exponential number of
internal determinized automaton states (and corresponding RAM and CPU)
for Lucene to execute. Lucene prevents these using the
|
Context Suggester
The completion suggester considers all documents in the index, but it is often desirable to serve suggestions filtered and/or boosted by some criteria. For example, you want to suggest song titles filtered by certain artists or you want to boost song titles based on their genre.
To achieve suggestion filtering and/or boosting, you can add context mappings while
configuring a completion field. You can define multiple context mappings for a
completion field.
Every context mapping has a unique name and a type. There are two types: category
and geo
. Context mappings are configured under the contexts
parameter in
the field mapping.
|
Note
|
It is mandatory to provide a context when indexing and querying a context enabled completion field. |
The following defines types, each with two context mappings for a completion field:
PUT place
{
"mappings": {
"properties": {
"suggest": {
"type": "completion",
"contexts": [
{ (1)
"name": "place_type",
"type": "category"
},
{ (2)
"name": "location",
"type": "geo",
"precision": 4
}
]
}
}
}
}
PUT place_path_category
{
"mappings": {
"properties": {
"suggest": {
"type": "completion",
"contexts": [
{ (3)
"name": "place_type",
"type": "category",
"path": "cat"
},
{ (4)
"name": "location",
"type": "geo",
"precision": 4,
"path": "loc"
}
]
},
"loc": {
"type": "geo_point"
}
}
}
}
-
Defines a
category
context named 'place_type' where the categories must be sent with the suggestions. -
Defines a
geo
context named 'location' where the categories must be sent with the suggestions. -
Defines a
category
context named 'place_type' where the categories are read from thecat
field. -
Defines a
geo
context named 'location' where the categories are read from theloc
field.
|
Note
|
Adding context mappings increases the index size for completion field. The completion index is entirely heap resident, you can monitor the completion field index size using Index stats API. |
Category Context
The category
context allows you to associate one or more categories with suggestions at index
time. At query time, suggestions can be filtered and boosted by their associated categories.
The mappings are set up like the place_type
fields above. If path
is defined
then the categories are read from that path in the document, otherwise they must
be sent in the suggest field like this:
PUT place/_doc/1
{
"suggest": {
"input": [ "timmy's", "starbucks", "dunkin donuts" ],
"contexts": {
"place_type": [ "cafe", "food" ] (1)
}
}
}
-
These suggestions will be associated with 'cafe' and 'food' category.
If the mapping had a path
then the following index request would be enough to
add the categories:
PUT place_path_category/_doc/1
{
"suggest": ["timmy's", "starbucks", "dunkin donuts"],
"cat": ["cafe", "food"] (1)
}
-
These suggestions will be associated with 'cafe' and 'food' category.
|
Note
|
If context mapping references another field and the categories are explicitly indexed, the suggestions are indexed with both set of categories. |
Category Query
Suggestions can be filtered by one or more categories. The following filters suggestions by multiple categories:
POST place/_search?pretty
{
"suggest": {
"place_suggestion": {
"prefix": "tim",
"completion": {
"field": "suggest",
"size": 10,
"contexts": {
"place_type": [ "cafe", "restaurants" ]
}
}
}
}
}
|
Note
|
If multiple categories or category contexts are set on the query they are merged as a disjunction. This means that suggestions match if they contain at least one of the provided context values. |
Suggestions with certain categories can be boosted higher than others. The following filters suggestions by categories and additionally boosts suggestions associated with some categories:
POST place/_search?pretty
{
"suggest": {
"place_suggestion": {
"prefix": "tim",
"completion": {
"field": "suggest",
"size": 10,
"contexts": {
"place_type": [ (1)
{ "context": "cafe" },
{ "context": "restaurants", "boost": 2 }
]
}
}
}
}
}
-
The context query filter suggestions associated with categories 'cafe' and 'restaurants' and boosts the suggestions associated with 'restaurants' by a factor of
2
In addition to accepting category values, a context query can be composed of
multiple category context clauses. The following parameters are supported for a
category
context clause:
context
|
The value of the category to filter/boost on. This is mandatory. |
boost
|
The factor by which the score of the suggestion
should be boosted, the score is computed by
multiplying the boost with the suggestion weight,
defaults to |
prefix
|
Whether the category value should be treated as a
prefix or not. For example, if set to |
|
Note
|
If a suggestion entry matches multiple contexts the final score is computed as the maximum score produced by any matching contexts. |
Geo location Context
A geo
context allows you to associate one or more geo points or geohashes with suggestions
at index time. At query time, suggestions can be filtered and boosted if they are within
a certain distance of a specified geo location.
Internally, geo points are encoded as geohashes with the specified precision.
Geo Mapping
In addition to the path
setting, geo
context mapping accepts the following settings:
precision
|
This defines the precision of the geohash to be indexed and can be specified
as a distance value ( |
|
Note
|
The index time precision setting sets the maximum geohash precision that
can be used at query time.
|
Indexing geo contexts
geo
contexts can be explicitly set with suggestions or be indexed from a geo point field in the
document via the path
parameter, similar to category
contexts. Associating multiple geo location context
with a suggestion, will index the suggestion for every geo location. The following indexes a suggestion
with two geo location contexts:
PUT place/_doc/1
{
"suggest": {
"input": "timmy's",
"contexts": {
"location": [
{
"lat": 43.6624803,
"lon": -79.3863353
},
{
"lat": 43.6624718,
"lon": -79.3873227
}
]
}
}
}
Geo location Query
Suggestions can be filtered and boosted with respect to how close they are to one or more geo points. The following filters suggestions that fall within the area represented by the encoded geohash of a geo point:
POST place/_search
{
"suggest": {
"place_suggestion": {
"prefix": "tim",
"completion": {
"field": "suggest",
"size": 10,
"contexts": {
"location": {
"lat": 43.662,
"lon": -79.380
}
}
}
}
}
}
|
Note
|
When a location with a lower precision at query time is specified, all suggestions that fall within the area will be considered. |
|
Note
|
If multiple categories or category contexts are set on the query they are merged as a disjunction. This means that suggestions match if they contain at least one of the provided context values. |
Suggestions that are within an area represented by a geohash can also be boosted higher than others, as shown by the following:
POST place/_search?pretty
{
"suggest": {
"place_suggestion": {
"prefix": "tim",
"completion": {
"field": "suggest",
"size": 10,
"contexts": {
"location": [ (1)
{
"lat": 43.6624803,
"lon": -79.3863353,
"precision": 2
},
{
"context": {
"lat": 43.6624803,
"lon": -79.3863353
},
"boost": 2
}
]
}
}
}
}
}
-
The context query filters for suggestions that fall under the geo location represented by a geohash of '(43.662, -79.380)' with a precision of '2' and boosts suggestions that fall under the geohash representation of '(43.6624803, -79.3863353)' with a default precision of '6' by a factor of
2
|
Note
|
If a suggestion entry matches multiple contexts the final score is computed as the maximum score produced by any matching contexts. |
In addition to accepting context values, a context query can be composed of
multiple context clauses. The following parameters are supported for a
geo
context clause:
context
|
A geo point object or a geo hash string to filter or boost the suggestion by. This is mandatory. |
boost
|
The factor by which the score of the suggestion
should be boosted, the score is computed by
multiplying the boost with the suggestion weight,
defaults to |
precision
|
The precision of the geohash to encode the query geo point.
This can be specified as a distance value ( |
neighbours
|
Accepts an array of precision values at which
neighbouring geohashes should be taken into account.
precision value can be a distance value ( |
Returning the type of the suggester
Sometimes you need to know the exact type of a suggester in order to parse its results. The typed_keys
parameter
can be used to change the suggester’s name in the response so that it will be prefixed by its type.
Considering the following example with two suggesters term
and phrase
:
POST _search?typed_keys
{
"suggest": {
"text" : "some test mssage",
"my-first-suggester" : {
"term" : {
"field" : "message"
}
},
"my-second-suggester" : {
"phrase" : {
"field" : "message"
}
}
}
}
In the response, the suggester names will be changed to respectively term#my-first-suggester
and
phrase#my-second-suggester
, reflecting the types of each suggestion:
{
"suggest": {
"term#my-first-suggester": [ (1)
{
"text": "some",
"offset": 0,
"length": 4,
"options": []
},
{
"text": "test",
"offset": 5,
"length": 4,
"options": []
},
{
"text": "mssage",
"offset": 10,
"length": 6,
"options": [
{
"text": "message",
"score": 0.8333333,
"freq": 4
}
]
}
],
"phrase#my-second-suggester": [ (2)
{
"text": "some test mssage",
"offset": 0,
"length": 16,
"options": [
{
"text": "some test message",
"score": 0.030227963
}
]
}
]
},
...
}
-
The name
my-first-suggester
now contains theterm
prefix. -
The name
my-second-suggester
now contains thephrase
prefix.
Multi search API
Executes several searches with a single API request.
GET my-index-000001/_msearch
{ }
{"query" : {"match" : { "message": "this is a test"}}}
{"index": "my-index-000002"}
{"query" : {"match_all" : {}}}
Request
GET /<target>/_msearch
Description
The multi search API executes several searches from a single API request. The format of the request is similar to the bulk API format and makes use of the newline delimited JSON (NDJSON) format.
The structure is as follows:
header\n
body\n
header\n
body\n
This structure is specifically optimized to reduce parsing if a specific search ends up redirected to another node.
|
Important
|
The final line of data must end with a newline character |
Path parameters
<target>
-
(Optional, string) Comma-separated list of indices and index aliases to search.
This list acts as a fallback if a search in the request body does not specify an
index
target.Wildcard (
) expressions are supported. To search all indices in a cluster, omit this parameter or use
_all
or.
Query parameters
allow_no_indices
-
(Optional, Boolean) If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
. ccs_minimize_roundtrips
-
(Optional, Boolean) If
true
, network roundtrips between the coordinating node and remote clusters are minimized for cross-cluster search requests. Defaults totrue
. See How cross-cluster search handles network delays. expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
Defaults to
open
. ignore_throttled
-
(Optional, Boolean) If
true
, concrete, expanded or aliased indices are ignored when frozen. Defaults totrue
. ignore_unavailable
-
(Optional, Boolean) If
true
, missing or closed indices are not included in the response. Defaults tofalse
. max_concurrent_searches
-
(Optional, integer) Maximum number of concurrent searches the multi search API can execute. Defaults to max(1, (# of <<data-node,data nodes>> * min(<<search-threadpool,search thread pool size>>, 10))).
max_concurrent_shard_requests
-
(Optional, integer) Maximum number of concurrent shard requests that each sub-search request executes per node. Defaults to
5
.You can use this parameter to prevent a request from overloading a cluster. For example, a default request hits all indices in a cluster. This could cause shard request rejections if the number of shards per node is high.
In certain scenarios, parallelism isn’t achieved through concurrent requests. In those cases, a low value in this parameter could result in poor performance. For example, in an environment where a very low number of concurrent search requests are expected, a higher value in this parameter may improve performance.
pre_filter_shard_size
-
(Optional, integer) Defines a threshold that enforces a pre-filter roundtrip to prefilter search shards based on query rewriting if the number of shards the search request expands to exceeds the threshold. This filter roundtrip can limit the number of shards significantly if for instance a shard can not match any documents based on its rewrite method i.e., if date filters are mandatory to match but the shard bounds and the query are disjoint. When unspecified, the pre-filter phase is executed if any of these conditions is met:
-
The request targets more than
128
shards. -
The request targets one or more read-only index.
-
The primary sort of the query targets an indexed field.
-
rest_total_hits_as_int
-
(Optional, Boolean) If
true
,hits.total
are returned as an integer in the response. Defaults tofalse
, which returns an object. routing
-
(Optional, string) Custom routing value used to route search operations to a specific shard.
search_type
-
(Optional, string) Indicates whether global term and document frequencies should be used when scoring returned documents.
Options are:
query_then_fetch
-
(default) Documents are scored using local term and document frequencies for the shard. This is usually faster but less accurate.
dfs_query_then_fetch
-
Documents are scored using global term and document frequencies across all shards. This is usually slower but more accurate.
typed_keys
-
(Optional, Boolean) Specifies whether aggregation and suggester names should be prefixed by their respective types in the response.
Request body
The request body contains a newline-delimited list of search <header>
and
search <body>
objects.
<header>
-
(Required, object) Contains parameters used to limit or change the subsequent search body request.
This object is required for each search body but can be empty (
{}
) or a blank line.allow_no_indices
-
(Optional, Boolean) If
true
, the request does not return an error if a wildcard expression or_all
value retrieves only missing or closed indices.This parameter also applies to index aliases that point to a missing or closed index.
expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
Defaults to
open
.ignore_unavailable
-
(Optional, Boolean) If
true
, documents from missing or closed indices are not included in the response. Defaults tofalse
. index
-
(Optional, string or array of strings) Indices and index aliases to search. Wildcard (
*
) expressions are supported. You can specify multiple targets as an array.If this parameter is not specified, the
<target>
request path parameter is used as a fallback. preference
-
(Optional, string) Node or shard used to perform the search. Random by default.
request_cache
-
(Optional, Boolean) If
true
, the request cache can be used for this search. Defaults to index-level settings. See Shard request cache settings. routing
-
(Optional, string) Custom routing value used to route search operations to a specific shard.
search_type
-
(Optional, string) Indicates whether global term and document frequencies should be used when scoring returned documents.
Options are:
query_then_fetch
-
(default) Documents are scored using local term and document frequencies for the shard. This is usually faster but less accurate.
dfs_query_then_fetch
-
Documents are scored using global term and document frequencies across all shards. This is usually slower but more accurate.
<body>
-
(Optional, object) Contains parameters for a search request:
aggregations
-
(Optional, aggregation object) Aggregations you wish to run during the search. See Aggregations.
query
-
(Optional, query DSL object) Query you wish to run during the search. Hits matching this query are returned in the response.
from
-
(Optional, integer) Starting offset for returned hits. Defaults to
0
. size
-
(Optional, integer) Number of hits to return. Defaults to
10
.
Response body
responses
-
(array) Includes the search response and status code for each search request matching its order in the original multi search request. If there was a complete failure for a specific search request, an object with
error
message and corresponding status code will be returned in place of the actual search response.
Examples
The header part includes which indices, and index aliases to
search. The header also indicates the search_type
,
preference
, and routing
. The body includes the typical search body request
(including the query
, aggregations
, from
, size
, and so on).
$ cat requests
{"index" : "test"}
{"query" : {"match_all" : {}}, "from" : 0, "size" : 10}
{"index" : "test", "search_type" : "dfs_query_then_fetch"}
{"query" : {"match_all" : {}}}
{}
{"query" : {"match_all" : {}}}
{"query" : {"match_all" : {}}}
{"search_type" : "dfs_query_then_fetch"}
{"query" : {"match_all" : {}}}
$ curl -H "Content-Type: application/x-ndjson" -XGET localhost:9200/_msearch --data-binary "@requests"; echo
Note, the above includes an example of an empty header (can also be just without any content) which is supported as well.
The endpoint also allows you to search against indices and index
aliases in the request path. In this case, it will be used as the default target
unless explicitly specified in the header’s index
parameter. For example:
GET my-index-000001/_msearch
{}
{"query" : {"match_all" : {}}, "from" : 0, "size" : 10}
{}
{"query" : {"match_all" : {}}}
{"index" : "my-index-000002"}
{"query" : {"match_all" : {}}}
The above will execute the search against the my-index-000001
index for all the
requests that don’t define an index
target in the request body. The last
search will be executed against the my-index-000002
index.
The search_type
can be set in a similar manner to globally apply to
all search requests.
Security
Template support
Much like described in Search Template for the _search resource, _msearch also provides support for templates. Submit them like follows for inline templates:
GET _msearch/template
{"index" : "my-index-000001"}
{ "source" : "{ \"query\": { \"match\": { \"message\" : \"{{keywords}}\" } } } }", "params": { "query_type": "match", "keywords": "some message" } }
{"index" : "my-index-000001"}
{ "source" : "{ \"query\": { \"match_{{template}}\": {} } }", "params": { "template": "all" } }
You can also create search templates:
POST /_scripts/my_template_1
{
"script": {
"lang": "mustache",
"source": {
"query": {
"match": {
"message": "{{query_string}}"
}
}
}
}
}
POST /_scripts/my_template_2
{
"script": {
"lang": "mustache",
"source": {
"query": {
"term": {
"{{field}}": "{{value}}"
}
}
}
}
}
You can use search templates in a _msearch:
GET _msearch/template
{"index" : "main"}
{ "id": "my_template_1", "params": { "query_string": "some message" } }
{"index" : "main"}
{ "id": "my_template_2", "params": { "field": "user", "value": "test" } }
Partial responses
To ensure fast responses, the multi search API will respond with partial results if one or more shards fail. See Shard failures for more information.
Search Cancellation
Multi searches can be cancelled using standard task cancellation mechanism and are also automatically cancelled when the http connection used to perform the request is closed by the client. It is fundamental that the http client sending requests closes connections whenever requests time out or are aborted. Cancelling an msearch request will also cancel all of the corresponding sub search requests.
Count API
Gets the number of matches for a search query.
GET /my-index-000001/_count?q=user:bilbo
|
Note
|
The query being sent in the body must be nested in a query key, same as
the search API works.
|
Request
GET /<target>/_count
Description
The count API allows you to execute a query and get the number of matches for that query. The query can either be provided using a simple query string as a parameter, or using the Query DSL defined within the request body.
The count API supports multi-target syntax. You can run a single count API search across multiple indices.
The operation is broadcast across all shards. For each shard id group, a replica is chosen and executed against it. This means that replicas increase the scalability of count.
Path parameters
<target>
-
(Optional, string) Comma-separated list of indices, and index aliases to search. Wildcard (
*
) expressions are supported.To search all indices in a cluster, omit this parameter or use
_all
or*
.
Query parameters
allow_no_indices
-
(Optional, Boolean) If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
.Defaults to
true
. analyzer
-
(Optional, string) Analyzer to use for the query string.
analyze_wildcard
-
(Optional, Boolean) If
true
, wildcard and prefix queries are analyzed. Defaults tofalse
. default_operator
-
(Optional, string) The default operator for query string query: AND or OR. Defaults to
OR
. df
-
(Optional, string) Field to use as default where no field prefix is given in the query string.
expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
Defaults to
open
. ignore_throttled
-
(Optional, Boolean) If
true
, concrete, expanded or aliased indices are ignored when frozen. Defaults totrue
. ignore_unavailable
-
(Optional, Boolean) If
true
, missing or closed indices are not included in the response. Defaults tofalse
. lenient
-
(Optional, Boolean) If
true
, format-based query failures (such as providing text to a numeric field) will be ignored. Defaults tofalse
. min_score
-
(Optional, float) Sets the minimum
_score
value that documents must have to be included in the result. preference
-
(Optional, string) Specifies the node or shard the operation should be performed on. Random by default.
q
-
(Optional, string) Query in the Lucene query string syntax.
routing
-
(Optional, string) Target the specified primary shard.
terminate_after
-
(Optional, integer) The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early.
Request body
query
-
(Optional, query object) Defines the search definition using the Query DSL.
Examples
PUT /my-index-000001/_doc/1?refresh
{
"user.id": "bilbo"
}
GET /my-index-000001/_count?q=user:bilbo
GET /my-index-000001/_count
{
"query" : {
"term" : { "user.id" : "bilbo" }
}
}
Both examples above do the same: count the number of documents in
my-index-000001
with a user.id
of bilbo
. The API returns the following response:
{
"count": 1,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
}
}
The query is optional, and when not provided, it will use match_all
to
count all the docs.
Validate API
Validates a potentially expensive query without executing it.
GET my-index-000001/_validate/query?q=user.id:bilbo
Request
GET /<target>/_validate/<query>
Description
The validate API allows you to validate a potentially expensive query without executing it. The query can be sent either as a path parameter or in the request body.
Path parameters
<target>
-
(Optional, string) Comma-separated list of indices, and index aliases to search. Wildcard (
*
) expressions are supported.To search all indices in a cluster, omit this parameter or use
_all
or*
. query
-
(Optional, query object) Defines the search definition using the Query DSL.
Query parameters
all_shards
-
(Optional, Boolean) If
true
, the validation is executed on all shards instead of one random shard per index. Defaults tofalse
. allow_no_indices
-
(Optional, Boolean) If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
.Defaults to
false
. analyzer
-
(Optional, string) Analyzer to use for the query string.
analyze_wildcard
-
(Optional, Boolean) If
true
, wildcard and prefix queries are analyzed. Defaults tofalse
. default_operator
-
(Optional, string) The default operator for query string query: AND or OR. Defaults to
OR
. df
-
(Optional, string) Field to use as default where no field prefix is given in the query string.
expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
explain
-
(Optional, Boolean) If
true
, the response returns detailed information if an error has occurred. Defaults tofalse
. ignore_unavailable
-
(Optional, Boolean) If
true
, missing or closed indices are not included in the response. Defaults tofalse
. lenient
-
(Optional, Boolean) If
true
, format-based query failures (such as providing text to a numeric field) will be ignored. Defaults tofalse
. rewrite
-
(Optional, Boolean) If
true
, returns a more detailed explanation showing the actual Lucene query that will be executed. Defaults tofalse
. q
-
(Optional, string) Query in the Lucene query string syntax.
Examples
PUT my-index-000001/_bulk?refresh
{"index":{"_id":1}}
{"user" : { "id": "bilbo" }, "@timestamp" : "2099-11-15T14:12:12", "message" : "Trying out OpenSearch."}
{"index":{"_id":2}}
{"user" : { "id": "bilba" }, "@timestamp" : "2099-11-15T14:12:13", "message" : "Similar to Bilbo."}
When sent a valid query:
GET my-index-000001/_validate/query?q=user.id:bilbo
The response contains valid:true
:
{"_shards":{"total":1,"successful":1,"failed":0},"valid":true}
The query may also be sent in the request body:
GET my-index-000001/_validate/query
{
"query" : {
"bool" : {
"must" : {
"query_string" : {
"query" : "*:*"
}
},
"filter" : {
"term" : { "user.id" : "bilbo" }
}
}
}
}
|
Note
|
The query being sent in the body must be nested in a query key, same as
the search api works
|
If the query is invalid, valid
will be false
. Here the query is invalid
because OpenSearch knows the post_date
field should be a date due to dynamic
mapping, and 'foo' does not correctly parse into a date:
GET my-index-000001/_validate/query
{
"query": {
"query_string": {
"query": "@timestamp:foo",
"lenient": false
}
}
}
{"_shards":{"total":1,"successful":1,"failed":0},"valid":false}
The explain parameter
An explain
parameter can be specified to get more detailed information about
why a query failed:
GET my-index-000001/_validate/query?explain=true
{
"query": {
"query_string": {
"query": "@timestamp:foo",
"lenient": false
}
}
}
The API returns the following response:
{
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"valid": false,
"explanations": [{
"index": "my-index-000001",
"valid": false,
"error": "[my-index-000001/1Nf0oygTT6ypNZs1q3mf0w] QueryShardException[failed to create query...failed to parse date field [foo]"
}]
}
The rewrite parameter
When the query is valid, the explanation defaults to the string representation
of that query. With rewrite
set to true
, the explanation is more detailed
showing the actual Lucene query that will be executed.
GET my-index-000001/_validate/query?rewrite=true
{
"query": {
"more_like_this": {
"like": {
"_id": "2"
},
"boost_terms": 1
}
}
}
The API returns the following response:
{
"valid": true,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"explanations": [
{
"index": "my-index-000001",
"valid": true,
"explanation": "((user:terminator^3.71334 plot:future^2.763601 plot:human^2.8415773 plot:sarah^3.4193945 plot:kyle^3.8244398 plot:cyborg^3.9177752 plot:connor^4.040236 plot:reese^4.7133346 ... )~6) -ConstantScore(_id:2)) #(ConstantScore(_type:_doc))^0.0"
}
]
}
Rewrite and all_shards parameters
By default, the request is executed on a single shard only, which is randomly
selected. The detailed explanation of the query may depend on which shard is
being hit, and therefore may vary from one request to another. So, in case of
query rewrite the all_shards
parameter should be used to get response from
all available shards.
GET my-index-000001/_validate/query?rewrite=true&all_shards=true
{
"query": {
"match": {
"user.id": {
"query": "bilbo",
"fuzziness": "auto"
}
}
}
}
The API returns the following response:
{
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"valid": true,
"explanations": [{
"index": "my-index-000001",
"shard": 0,
"valid": true,
"explanation": "(user.id:bilba)^0.8 user.id:bilbo"
}]
}
Explain API
Returns information about why a specific document matches (or doesn’t match) a query.
GET /my-index-000001/_explain/0
{
"query" : {
"match" : { "message" : "opensearch" }
}
}
Request
GET /<index>/_explain/<id>
POST /<index>/_explain/<id>
Description
The explain API computes a score explanation for a query and a specific document. This can give useful feedback whether a document matches or didn’t match a specific query.
Path parameters
<id>
-
(Required, integer) Defines the document ID.
<index>
-
(Required, string) Index names used to limit the request.
Only a single index name can be provided to this parameter.
Query parameters
analyzer
-
(Optional, string) Analyzer to use for the query string.
analyze_wildcard
-
(Optional, Boolean) If
true
, wildcard and prefix queries are analyzed. Defaults tofalse
. default_operator
-
(Optional, string) The default operator for query string query: AND or OR. Defaults to
OR
. df
-
(Optional, string) Field to use as default where no field prefix is given in the query string.
lenient
-
(Optional, Boolean) If
true
, format-based query failures (such as providing text to a numeric field) will be ignored. Defaults tofalse
. preference
-
(Optional, string) Specifies the node or shard the operation should be performed on. Random by default.
q
-
(Optional, string) Query in the Lucene query string syntax.
stored_fields
-
(Optional, string) A comma-separated list of stored fields to return in the response.
routing
-
(Optional, string) Custom routing value used to route operations to a specific shard.
_source
-
(Optional, string) True or false to return the
_source
field or not, or a list of fields to return. _source_excludes
-
(Optional, string) A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored. _source_includes
-
(Optional, string) A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter.If the
_source
parameter isfalse
, this parameter is ignored.
Request body
query
-
(Optional, query object) Defines the search definition using the Query DSL.
Examples
GET /my-index-000001/_explain/0
{
"query" : {
"match" : { "message" : "opensearch" }
}
}
The API returns the following response:
{
"_index":"my-index-000001",
"_type":"_doc",
"_id":"0",
"matched":true,
"explanation":{
"value":1.6943598,
"description":"weight(message:opensearch in 0) [PerFieldSimilarity], result of:",
"details":[
{
"value":1.6943598,
"description":"score(freq=1.0), computed as boost * idf * tf from:",
"details":[
{
"value":2.2,
"description":"boost",
"details":[]
},
{
"value":1.3862944,
"description":"idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
"details":[
{
"value":1,
"description":"n, number of documents containing term",
"details":[]
},
{
"value":5,
"description":"N, total number of documents with field",
"details":[]
}
]
},
{
"value":0.5555556,
"description":"tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
"details":[
{
"value":1.0,
"description":"freq, occurrences of term within document",
"details":[]
},
{
"value":1.2,
"description":"k1, term saturation parameter",
"details":[]
},
{
"value":0.75,
"description":"b, length normalization parameter",
"details":[]
},
{
"value":3.0,
"description":"dl, length of field",
"details":[]
},
{
"value":5.4,
"description":"avgdl, average length of field",
"details":[]
}
]
}
]
}
]
}
}
There is also a simpler way of specifying the query via the q
parameter. The
specified q
parameter value is then parsed as if the query_string
query was
used. Example usage of the q
parameter in the
explain API:
GET /my-index-000001/_explain/0?q=message:search
The API returns the same result as the previous request.
Profile API
|
Warning
|
The Profile API is a debugging tool and adds significant overhead to search execution. |
Provides detailed timing information about the execution of individual components in a search request.
Description
The Profile API gives the user insight into how search requests are executed at a low level so that the user can understand why certain requests are slow, and take steps to improve them. Note that the Profile API, amongst other things, doesn’t measure network latency, time spent in the search fetch phase, time spent while the requests spends in queues or while merging shard responses on the coordinating node.
The output from the Profile API is very verbose, especially for complicated requests executed across many shards. Pretty-printing the response is recommended to help understand the output.
Examples
Any _search
request can be profiled by adding a top-level profile
parameter:
GET /my-index-000001/_search
{
"profile": true,(1)
"query" : {
"match" : { "message" : "GET /search" }
}
}
-
Setting the top-level
profile
parameter totrue
will enable profiling for the search.
The API returns the following result:
{
"took": 25,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 5,
"relation": "eq"
},
"max_score": 0.17402273,
"hits": [...] (1)
},
"profile": {
"shards": [
{
"id": "[2aE02wS1R8q_QFnYu6vDVQ][my-index-000001][0]",
"searches": [
{
"query": [
{
"type": "BooleanQuery",
"description": "message:get
message:search", "time_in_nanos" : 11972972, "breakdown" :
{
"set_min_competitive_score_count": 0,
"match_count": 5,
"shallow_advance_count": 0,
"set_min_competitive_score": 0,
"next_doc": 39022,
"match": 4456,
"next_doc_count": 5,
"score_count": 5,
"compute_max_score_count": 0,
"compute_max_score": 0,
"advance": 84525,
"advance_count": 1,
"score": 37779,
"build_scorer_count": 2,
"create_weight": 4694895,
"shallow_advance": 0,
"create_weight_count": 1,
"build_scorer": 7112295
},
"children": [
{
"type": "TermQuery",
"description": "message:get",
"time_in_nanos": 3801935,
"breakdown": {
"set_min_competitive_score_count": 0,
"match_count": 0,
"shallow_advance_count": 3,
"set_min_competitive_score": 0,
"next_doc": 0,
"match": 0,
"next_doc_count": 0,
"score_count": 5,
"compute_max_score_count": 3,
"compute_max_score": 32487,
"advance": 5749,
"advance_count": 6,
"score": 16219,
"build_scorer_count": 3,
"create_weight": 2382719,
"shallow_advance": 9754,
"create_weight_count": 1,
"build_scorer": 1355007
}
},
{
"type": "TermQuery",
"description": "message:search",
"time_in_nanos": 205654,
"breakdown": {
"set_min_competitive_score_count": 0,
"match_count": 0,
"shallow_advance_count": 3,
"set_min_competitive_score": 0,
"next_doc": 0,
"match": 0,
"next_doc_count": 0,
"score_count": 5,
"compute_max_score_count": 3,
"compute_max_score": 6678,
"advance": 12733,
"advance_count": 6,
"score": 6627,
"build_scorer_count": 3,
"create_weight": 130951,
"shallow_advance": 2512,
"create_weight_count": 1,
"build_scorer": 46153
}
}
]
}
],
"rewrite_time": 451233,
"collector": [
{
"name": "SimpleTopScoreDocCollector",
"reason": "search_top_hits",
"time_in_nanos": 775274
}
]
}
],
"aggregations": []
}
]
}
}
-
Search results are returned, but were omitted here for brevity.
Even for a simple query, the response is relatively complicated. Let’s break it down piece-by-piece before moving to more complex examples.
The overall structure of the profile response is as follows:
{
"profile": {
"shards": [
{
"id": "[2aE02wS1R8q_QFnYu6vDVQ][my-index-000001][0]", (1)
"searches": [
{
"query": [...], (2)
"rewrite_time": 51443, (3)
"collector": [...] (4)
}
],
"aggregations": [...] (5)
}
]
}
}
-
A profile is returned for each shard that participated in the response, and is identified by a unique ID.
-
Each profile contains a section which holds details about the query execution.
-
Each profile has a single time representing the cumulative rewrite time.
-
Each profile also contains a section about the Lucene Collectors which run the search.
-
Each profile contains a section which holds the details about the aggregation execution.
Because a search request may be executed against one or more shards in an index,
and a search may cover one or more indices, the top level element in the profile
response is an array of shard
objects. Each shard object lists its id
which
uniquely identifies the shard. The ID’s format is
[nodeID][indexName][shardID]
.
The profile itself may consist of one or more "searches", where a search is a
query executed against the underlying Lucene index. Most search requests
submitted by the user will only execute a single search
against the Lucene
index. But occasionally multiple searches will be executed, such as including a
global aggregation (which needs to execute a secondary "match_all" query for the
global context).
Inside each search
object there will be two arrays of profiled information:
a query
array and a collector
array. Alongside the search
object is an
aggregations
object that contains the profile information for the
aggregations. In the future, more sections may be added, such as suggest
,
highlight
, etc.
There will also be a rewrite
metric showing the total time spent rewriting the
query (in nanoseconds).
|
Note
|
As with other statistics apis, the Profile API supports human readable outputs. This can be turned on by adding
?human=true to the query string. In this case, the output contains the additional time field containing rounded,
human readable timing information (e.g. "time": "391,9ms" , "time": "123.3micros" ).
|
Profiling Queries
|
Note
|
The details provided by the Profile API directly expose Lucene class names and concepts, which means that complete interpretation of the results require fairly advanced knowledge of Lucene. This page attempts to give a crash-course in how Lucene executes queries so that you can use the Profile API to successfully diagnose and debug queries, but it is only an overview. For complete understanding, please refer to Lucene’s documentation and, in places, the code. With that said, a complete understanding is often not required to fix a slow query. It is usually
sufficient to see that a particular component of a query is slow, and not necessarily understand why
the |
query
Section
The query
section contains detailed timing of the query tree executed by
Lucene on a particular shard. The overall structure of this query tree will
resemble your original OpenSearch query, but may be slightly (or sometimes
very) different. It will also use similar but not always identical naming.
Using our previous match
query example, let’s analyze the query
section:
"query": [
{
"type": "BooleanQuery",
"description": "message:get message:search",
"time_in_nanos": "11972972",
"breakdown": {...}, (1)
"children": [
{
"type": "TermQuery",
"description": "message:get",
"time_in_nanos": "3801935",
"breakdown": {...}
},
{
"type": "TermQuery",
"description": "message:search",
"time_in_nanos": "205654",
"breakdown": {...}
}
]
}
]
-
The breakdown timings are omitted for simplicity.
Based on the profile structure, we can see that our match
query was rewritten
by Lucene into a BooleanQuery with two clauses (both holding a TermQuery). The
type
field displays the Lucene class name, and often aligns with the
equivalent name in OpenSearch. The description
field displays the Lucene
explanation text for the query, and is made available to help differentiating
between parts of your query (e.g. both message:get
and message:search
are
TermQuery’s and would appear identical otherwise.
The time_in_nanos
field shows that this query took ~11.9ms for the entire
BooleanQuery to execute. The recorded time is inclusive of all children.
The breakdown
field will give detailed stats about how the time was spent,
we’ll look at that in a moment. Finally, the children
array lists any
sub-queries that may be present. Because we searched for two values ("get
search"), our BooleanQuery holds two children TermQueries. They have identical
information (type, time, breakdown, etc). Children are allowed to have their
own children.
Timing Breakdown
The breakdown
component lists detailed timing statistics about low-level
Lucene execution:
"breakdown": {
"set_min_competitive_score_count": 0,
"match_count": 5,
"shallow_advance_count": 0,
"set_min_competitive_score": 0,
"next_doc": 39022,
"match": 4456,
"next_doc_count": 5,
"score_count": 5,
"compute_max_score_count": 0,
"compute_max_score": 0,
"advance": 84525,
"advance_count": 1,
"score": 37779,
"build_scorer_count": 2,
"create_weight": 4694895,
"shallow_advance": 0,
"create_weight_count": 1,
"build_scorer": 7112295
}
Timings are listed in wall-clock nanoseconds and are not normalized at all. All
caveats about the overall time_in_nanos
apply here. The intention of the
breakdown is to give you a feel for A) what machinery in Lucene is actually
eating time, and B) the magnitude of differences in times between the various
components. Like the overall time, the breakdown is inclusive of all children
times.
The meaning of the stats are as follows:
All parameters:
create_weight
|
A Query in Lucene must be capable of reuse across multiple IndexSearchers (think of it as the engine that
executes a search against a specific Lucene Index). This puts Lucene in a tricky spot, since many queries
need to accumulate temporary state/statistics associated with the index it is being used against, but the
Query contract mandates that it must be immutable.
|
build_scorer
|
This parameter shows how long it takes to build a Scorer for the query. A Scorer is the mechanism that
iterates over matching documents and generates a score per-document (e.g. how well does "foo" match the document?).
Note, this records the time required to generate the Scorer object, not actually score the documents. Some
queries have faster or slower initialization of the Scorer, depending on optimizations, complexity, etc.
|
next_doc
|
The Lucene method |
advance
|
|
match
|
Some queries, such as phrase queries, match documents using a "two-phase" process. First, the document is
"approximately" matched, and if it matches approximately, it is checked a second time with a more rigorous
(and expensive) process. The second phase verification is what the |
score
|
This records the time taken to score a particular document via its Scorer |
*_count
|
Records the number of invocations of the particular method. For example, |
collectors
Section
The Collectors portion of the response shows high-level execution details. Lucene works by defining a "Collector" which is responsible for coordinating the traversal, scoring, and collection of matching documents. Collectors are also how a single query can record aggregation results, execute unscoped "global" queries, execute post-query filters, etc.
Looking at the previous example:
"collector": [
{
"name": "SimpleTopScoreDocCollector",
"reason": "search_top_hits",
"time_in_nanos": 775274
}
]
We see a single collector named SimpleTopScoreDocCollector
wrapped into
CancellableCollector
. SimpleTopScoreDocCollector
is the default "scoring and
sorting" Collector
used by OpenSearch. The reason
field attempts to give a plain
English description of the class name. The time_in_nanos
is similar to the
time in the Query tree: a wall-clock time inclusive of all children. Similarly,
children
lists all sub-collectors. The CancellableCollector
that wraps
SimpleTopScoreDocCollector
is used by OpenSearch to detect if the current search was
cancelled and stop collecting documents as soon as it occurs.
It should be noted that Collector times are independent from the Query times. They are calculated, combined, and normalized independently! Due to the nature of Lucene’s execution, it is impossible to "merge" the times from the Collectors into the Query section, so they are displayed in separate portions.
For reference, the various collector reasons are:
search_sorted
|
A collector that scores and sorts documents. This is the most common collector and will be seen in most simple searches |
search_count
|
A collector that only counts the number of documents that match the query, but does not fetch the source.
This is seen when |
search_terminate_after_count
|
A collector that terminates search execution after |
search_min_score
|
A collector that only returns matching documents that have a score greater than |
search_multi
|
A collector that wraps several other collectors. This is seen when combinations of search, aggregations, global aggs, and post_filters are combined in a single search. |
search_timeout
|
A collector that halts execution after a specified period of time. This is seen when a |
aggregation
|
A collector that OpenSearch uses to run aggregations against the query scope. A single |
global_aggregation
|
A collector that executes an aggregation against the global query scope, rather than the specified query. Because the global scope is necessarily different from the executed query, it must execute its own match_all query (which you will see added to the Query section) to collect your entire dataset |
rewrite
Section
All queries in Lucene undergo a "rewriting" process. A query (and its sub-queries) may be rewritten one or more times, and the process continues until the query stops changing. This process allows Lucene to perform optimizations, such as removing redundant clauses, replacing one query for a more efficient execution path, etc. For example a Boolean → Boolean → TermQuery can be rewritten to a TermQuery, because all the Booleans are unnecessary in this case.
The rewriting process is complex and difficult to display, since queries can change drastically. Rather than showing the intermediate results, the total rewrite time is simply displayed as a value (in nanoseconds). This value is cumulative and contains the total time for all queries being rewritten.
A more complex example
To demonstrate a slightly more complex query and the associated results, we can profile the following query:
GET /my-index-000001/_search
{
"profile": true,
"query": {
"term": {
"user.id": {
"value": "elkbee"
}
}
},
"aggs": {
"my_scoped_agg": {
"terms": {
"field": "http.response.status_code"
}
},
"my_global_agg": {
"global": {},
"aggs": {
"my_level_agg": {
"terms": {
"field": "http.response.status_code"
}
}
}
}
},
"post_filter": {
"match": {
"message": "search"
}
}
}
This example has:
-
A query
-
A scoped aggregation
-
A global aggregation
-
A post_filter
The API returns the following result:
{
...
"profile": {
"shards": [
{
"id": "[P6-vulHtQRWuD4YnubWb7A][my-index-000001][0]",
"searches": [
{
"query": [
{
"type": "TermQuery",
"description": "message:search",
"time_in_nanos": 141618,
"breakdown": {
"set_min_competitive_score_count": 0,
"match_count": 0,
"shallow_advance_count": 0,
"set_min_competitive_score": 0,
"next_doc": 0,
"match": 0,
"next_doc_count": 0,
"score_count": 0,
"compute_max_score_count": 0,
"compute_max_score": 0,
"advance": 3942,
"advance_count": 4,
"score": 0,
"build_scorer_count": 2,
"create_weight": 38380,
"shallow_advance": 0,
"create_weight_count": 1,
"build_scorer": 99296
}
},
{
"type": "TermQuery",
"description": "user.id:elkbee",
"time_in_nanos": 163081,
"breakdown": {
"set_min_competitive_score_count": 0,
"match_count": 0,
"shallow_advance_count": 0,
"set_min_competitive_score": 0,
"next_doc": 2447,
"match": 0,
"next_doc_count": 4,
"score_count": 4,
"compute_max_score_count": 0,
"compute_max_score": 0,
"advance": 3552,
"advance_count": 1,
"score": 5027,
"build_scorer_count": 2,
"create_weight": 107840,
"shallow_advance": 0,
"create_weight_count": 1,
"build_scorer": 44215
}
}
],
"rewrite_time": 4769,
"collector": [
{
"name": "MultiCollector",
"reason": "search_multi",
"time_in_nanos": 1945072,
"children": [
{
"name": "FilteredCollector",
"reason": "search_post_filter",
"time_in_nanos": 500850,
"children": [
{
"name": "SimpleTopScoreDocCollector",
"reason": "search_top_hits",
"time_in_nanos": 22577
}
]
},
{
"name": "MultiBucketCollector: [[my_scoped_agg, my_global_agg]]",
"reason": "aggregation",
"time_in_nanos": 867617
}
]
}
]
}
],
"aggregations": [...] (1)
}
]
}
}
-
The
"aggregations"
portion has been omitted because it will be covered in the next section.
As you can see, the output is significantly more verbose than before. All the major portions of the query are represented:
-
The first
TermQuery
(user.id:elkbee) represents the mainterm
query. -
The second
TermQuery
(message:search) represents thepost_filter
query.
The Collector tree is fairly straightforward, showing how a single CancellableCollector wraps a MultiCollector which also wraps a FilteredCollector to execute the post_filter (and in turn wraps the normal scoring SimpleCollector), a BucketCollector to run all scoped aggregations.
Understanding MultiTermQuery output
A special note needs to be made about the MultiTermQuery
class of queries.
This includes wildcards, regex, and fuzzy queries. These queries emit very
verbose responses, and are not overly structured.
Essentially, these queries rewrite themselves on a per-segment basis. If you
imagine the wildcard query b*
, it technically can match any token that begins
with the letter "b". It would be impossible to enumerate all possible
combinations, so Lucene rewrites the query in context of the segment being
evaluated, e.g., one segment may contain the tokens [bar, baz]
, so the query
rewrites to a BooleanQuery combination of "bar" and "baz". Another segment may
only have the token [bakery]
, so the query rewrites to a single TermQuery for
"bakery".
Due to this dynamic, per-segment rewriting, the clean tree structure becomes distorted and no longer follows a clean "lineage" showing how one query rewrites into the next. At present time, all we can do is apologize, and suggest you collapse the details for that query’s children if it is too confusing. Luckily, all the timing statistics are correct, just not the physical layout in the response, so it is sufficient to just analyze the top-level MultiTermQuery and ignore its children if you find the details too tricky to interpret.
Hopefully this will be fixed in future iterations, but it is a tricky problem to solve and still in-progress. :)
Profiling Aggregations
aggregations
Section
The aggregations
section contains detailed timing of the aggregation tree
executed by a particular shard. The overall structure of this aggregation tree
will resemble your original OpenSearch request. Let’s execute the previous query again
and look at the aggregation profile this time:
GET /my-index-000001/_search
{
"profile": true,
"query": {
"term": {
"user.id": {
"value": "elkbee"
}
}
},
"aggs": {
"my_scoped_agg": {
"terms": {
"field": "http.response.status_code"
}
},
"my_global_agg": {
"global": {},
"aggs": {
"my_level_agg": {
"terms": {
"field": "http.response.status_code"
}
}
}
}
},
"post_filter": {
"match": {
"message": "search"
}
}
}
This yields the following aggregation profile output:
{
"profile": {
"shards": [
{
"aggregations": [
{
"type": "NumericTermsAggregator",
"description": "my_scoped_agg",
"time_in_nanos": 79294,
"breakdown": {
"reduce": 0,
"build_aggregation": 30885,
"build_aggregation_count": 1,
"initialize": 2623,
"initialize_count": 1,
"reduce_count": 0,
"collect": 45786,
"collect_count": 4,
"build_leaf_collector": 18211,
"build_leaf_collector_count": 1,
"post_collection": 929,
"post_collection_count": 1
},
"debug": {
"total_buckets": 1,
"result_strategy": "long_terms"
}
},
{
"type": "GlobalAggregator",
"description": "my_global_agg",
"time_in_nanos": 104325,
"breakdown": {
"reduce": 0,
"build_aggregation": 22470,
"build_aggregation_count": 1,
"initialize": 12454,
"initialize_count": 1,
"reduce_count": 0,
"collect": 69401,
"collect_count": 4,
"build_leaf_collector": 8150,
"build_leaf_collector_count": 1,
"post_collection": 1584,
"post_collection_count": 1
},
"children": [
{
"type": "NumericTermsAggregator",
"description": "my_level_agg",
"time_in_nanos": 76876,
"breakdown": {
"reduce": 0,
"build_aggregation": 13824,
"build_aggregation_count": 1,
"initialize": 1441,
"initialize_count": 1,
"reduce_count": 0,
"collect": 61611,
"collect_count": 4,
"build_leaf_collector": 5564,
"build_leaf_collector_count": 1,
"post_collection": 471,
"post_collection_count": 1
},
"debug": {
"total_buckets": 1,
"result_strategy": "long_terms"
}
}
]
}
]
}
]
}
}
From the profile structure we can see that the my_scoped_agg
is internally
being run as a NumericTermsAggregator
(because the field it is aggregating,
http.response.status_code
, is a numeric field). At the same level, we see a GlobalAggregator
which comes from my_global_agg
. That aggregation then has a child
NumericTermsAggregator
which comes from the second term’s aggregation on http.response.status_code
.
The time_in_nanos
field shows the time executed by each aggregation, and is
inclusive of all children. While the overall time is useful, the breakdown
field will give detailed stats about how the time was spent.
Some aggregations may return expert debug
information that describe features
of the underlying execution of the aggregation that are 'useful for folks that
hack on aggregations but that we don’t expect to be otherwise useful. They can
vary wildly between versions, aggregations, and aggregation execution
strategies.
Timing Breakdown
The breakdown
component lists detailed statistics about low-level execution:
"breakdown": {
"reduce": 0,
"build_aggregation": 30885,
"build_aggregation_count": 1,
"initialize": 2623,
"initialize_count": 1,
"reduce_count": 0,
"collect": 45786,
"collect_count": 4,
"build_leaf_collector": 18211,
"build_leaf_collector_count": 1,
"post_collection": 929,
"post_collection_count": 1
}
Each property in the breakdown
component corresponds to an internal method for
the aggregation. For example, the build_leaf_collector
property measures
nanoseconds spent running the aggregation’s getLeafCollector()
method.
Properties ending in _count
record the number of invocations of the particular
method. For example, "collect_count": 2
means the aggregation called the
collect()
on two different documents. The reduce
property is reserved for
future use and always returns 0
.
Timings are listed in wall-clock nanoseconds and are not normalized at all. All
caveats about the overall time
apply here. The intention of the breakdown is
to give you a feel for A) what machinery in OpenSearch is actually eating time, and B)
the magnitude of differences in times between the various components. Like the
overall time, the breakdown is inclusive of all children times.
Profiling Considerations
Like any profiler, the Profile API introduces a non-negligible overhead to
search execution. The act of instrumenting low-level method calls such as
collect
, advance
, and next_doc
can be fairly expensive, since these
methods are called in tight loops. Therefore, profiling should not be enabled
in production settings by default, and should not be compared against
non-profiled query times. Profiling is just a diagnostic tool.
There are also cases where special Lucene optimizations are disabled, since they are not amenable to profiling. This could cause some queries to report larger relative times than their non-profiled counterparts, but in general should not have a drastic effect compared to other components in the profiled query.
Limitations
-
Profiling currently does not measure the search fetch phase nor the network overhead.
-
Profiling also does not account for time spent in the queue, merging shard responses on the coordinating node, or additional work such as building global ordinals (an internal data structure used to speed up search).
-
Profiling statistics are currently not available for suggestions, highlighting,
dfs_query_then_fetch
. -
Profiling of the reduce phase of aggregation is currently not available.
-
The Profiler is still highly experimental. The Profiler is instrumenting parts of Lucene that were never designed to be exposed in this manner, and so all results should be viewed as a best effort to provide detailed diagnostics. We hope to improve this over time. If you find obviously wrong numbers, strange query structures, or other bugs, please report them!
Field capabilities API
Allows you to retrieve the capabilities of fields among multiple indices.
GET /_field_caps?fields=rating
Request
GET /_field_caps?fields=<fields>
POST /_field_caps?fields=<fields>
GET /<target>/_field_caps?fields=<fields>
POST /<target>/_field_caps?fields=<fields>
Description
The field capabilities API returns the information about the capabilities of fields among multiple indices.
Path parameters
<target>
-
(Optional, string) Comma-separated list of indices and index aliases used to limit the request. Wildcard expressions (
*
) are supported.To target all indices in a cluster, omit this parameter or use
_all
or*
.
Query parameters
fields
-
(Required, string) Comma-separated list of fields to retrieve capabilities for. Wildcard (
*
) expressions are supported. allow_no_indices
-
(Optional, Boolean) If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
.Defaults to
true
. expand_wildcards
-
(Optional, string) Controls what kind of indices that wildcard expressions can expand to. Multiple values are accepted when separated by a comma, as in
open,hidden
. Valid values are:all
-
Expand to open and closed indices, including hidden indices.
open
-
Expand only to open indices.
closed
-
Expand only to closed indices.
hidden
-
Expansion of wildcards will include hidden indices. Must be combined with
open
,closed
, or both. none
-
Wildcard expressions are not accepted.
Defaults to
open
. ignore_unavailable
-
(Optional, Boolean) If
true
, missing or closed indices are not included in the response. Defaults tofalse
. include_unmapped
-
(Optional, Boolean) If
true
, unmapped fields are included in the response. Defaults tofalse
.
Request body
index_filter
-
(Optional, query object Allows to filter indices if the provided query rewrites to
match_none
on every shard.
Response body
The types used in the response describe families of field types.
Normally a type family is the same as the field type declared in the mapping,
but to simplify matters certain field types that behave identically are
described using a type family. For example, keyword
, constant_keyword
and wildcard
field types are all described as the keyword
type family.
searchable
-
Whether this field is indexed for search on all indices.
aggregatable
-
Whether this field can be aggregated on all indices.
indices
-
The list of indices where this field has the same type family, or null if all indices have the same type family for the field.
non_searchable_indices
-
The list of indices where this field is not searchable, or null if all indices have the same definition for the field.
non_aggregatable_indices
-
The list of indices where this field is not aggregatable, or null if all indices have the same definition for the field.
meta
-
Merged metadata across all indices as a map of string keys to arrays of values. A value length of 1 indicates that all indices had the same value for this key, while a length of 2 or more indicates that not all indices had the same value for this key.
Examples
The request can be restricted to specific indices:
GET my-index-000001/_field_caps?fields=rating
The next example API call requests information about the rating
and the
title
fields:
GET _field_caps?fields=rating,title
The API returns the following response:
{
"indices": [ "index1", "index2", "index3", "index4", "index5" ],
"fields": {
"rating": { (1)
"long": {
"searchable": true,
"aggregatable": false,
"indices": [ "index1", "index2" ],
"non_aggregatable_indices": [ "index1" ] (2)
},
"keyword": {
"searchable": false,
"aggregatable": true,
"indices": [ "index3", "index4" ],
"non_searchable_indices": [ "index4" ] (3)
}
},
"title": { (4)
"text": {
"searchable": true,
"aggregatable": false
}
}
}
}
-
The field
rating
is defined as a long inindex1
andindex2
and as akeyword
inindex3
andindex4
. -
The field
rating
is not aggregatable inindex1
. -
The field
rating
is not searchable inindex4
. -
The field
title
is defined astext
in all indices.
By default unmapped fields are ignored. You can include them in the response by
adding a parameter called include_unmapped
in the request:
GET _field_caps?fields=rating,title&include_unmapped
In which case the response will contain an entry for each field that is present in some indices but not all:
{
"indices": [ "index1", "index2", "index3" ],
"fields": {
"rating": {
"long": {
"searchable": true,
"aggregatable": false,
"indices": [ "index1", "index2" ],
"non_aggregatable_indices": [ "index1" ]
},
"keyword": {
"searchable": false,
"aggregatable": true,
"indices": [ "index3", "index4" ],
"non_searchable_indices": [ "index4" ]
},
"unmapped": { (1)
"indices": [ "index5" ],
"searchable": false,
"aggregatable": false
}
},
"title": {
"text": {
"indices": [ "index1", "index2", "index3", "index4" ],
"searchable": true,
"aggregatable": false
},
"unmapped": { (2)
"indices": [ "index5" ],
"searchable": false,
"aggregatable": false
}
}
}
}
-
The
rating
field is unmapped` inindex5
. -
The
title
field is unmapped` inindex5
.