Configuring OpenSearch
OpenSearch ships with good defaults and requires very little configuration. Most settings can be changed on a running cluster using the Cluster update settings API API.
The configuration files should contain settings which are node-specific (such
as node.name
and paths), or settings which a node requires in order to be
able to join a cluster, such as cluster.name
and network.host
.
Config files location
OpenSearch has three configuration files:
-
opensearch.yml
for configuring OpenSearch -
jvm.options
for configuring OpenSearch JVM settings -
log4j2.properties
for configuring OpenSearch logging
These files are located in the config directory, whose default location depends
on whether or not the installation is from an archive distribution (tar.gz
or
zip
) or a package distribution (Debian or RPM packages).
For the archive distributions, the config directory location defaults to
$OPENSEARCH_HOME/config
. The location of the config directory can be changed via the
OPENSEARCH_PATH_CONF
environment variable as follows:
OPENSEARCH_PATH_CONF=/path/to/my/config ./bin/opensearch
Alternatively, you can export
the OPENSEARCH_PATH_CONF
environment variable via the
command line or via your shell profile.
For the package distributions, the config directory location defaults to
/etc/opensearch
. The location of the config directory can also be changed
via the OPENSEARCH_PATH_CONF
environment variable, but note that setting this in your
shell is not sufficient. Instead, this variable is sourced from
/etc/default/opensearch
(for the Debian package) and
/etc/sysconfig/opensearch
(for the RPM package). You will need to edit the
OPENSEARCH_PATH_CONF=/etc/opensearch
entry in one of these files accordingly to
change the config directory location.
Config file format
The configuration format is YAML. Here is an example of changing the path of the data and logs directories:
path:
data: /var/lib/opensearch
logs: /var/log/opensearch
Settings can also be flattened as follows:
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
In YAML, you can format non-scalar values as sequences:
discovery.seed_hosts:
- <IP_ADDRESS>:9300
- <IP_ADDRESS>
- seeds.mydomain.com
Though less common, you can also format non-scalar values as arrays:
discovery.seed_hosts: ["<IP_ADDRESS>:9300", "<IP_ADDRESS>", "seeds.mydomain.com"]
Environment variable substitution
Environment variables referenced with the ${…}
notation within the
configuration file will be replaced with the value of the environment
variable. For example:
node.name: ${HOSTNAME}
network.host: ${OPENSEARCH_NETWORK_HOST}
Values for environment variables must be simple strings. Use a comma-separated string to provide values that OpenSearch will parse as a list. For example, OpenSearch will split the following string into a list of values for the ${HOSTNAME}
environment variable:
export HOSTNAME=“host1,host2"
Cluster and node setting types
Cluster and node settings can be categorized based on how they are configured:
- Dynamic
-
You can configure and update dynamic settings on a running cluster using the cluster update settings API. You can also configure dynamic settings locally on an unstarted or shut down node using
opensearch.yml
.Updates made using the cluster update settings API can be persistent, which apply across cluster restarts, or transient, which reset after a cluster restart. You can also reset transient or persistent settings by assigning them a
null
value using the API.If you configure the same setting using multiple methods, OpenSearch applies the settings in following order of precedence:
-
Transient setting
-
Persistent setting
-
opensearch.yml
setting -
Default setting value
For example, you can apply a transient setting to override a persistent setting or
opensearch.yml
setting. However, a change to anopensearch.yml
setting will not override a defined transient or persistent setting.TipIt’s best to set dynamic, cluster-wide settings with the cluster update settings API and use opensearch.yml
only for local configurations. Using the cluster update settings API ensures the setting is the same on all nodes. If you accidentally configure different settings inopensearch.yml
on different nodes, it can be difficult to notice discrepancies. -
- Static
-
Static settings can only be configured on an unstarted or shut down node using
opensearch.yml
.Static settings must be set on every relevant node in the cluster.
Setting JVM options
You should rarely need to change Java Virtual Machine (JVM) options. If you do,
the most likely change is setting the heap size. The remainder of
this document explains in detail how to set JVM options. You can set options
either with jvm.options
files or with the OPENSEARCH_JAVA_OPTS
environment variable.
The preferred method of setting or overriding JVM options is via JVM options
files. When installing from the tar or zip distributions, the root jvm.options
configuration file is config/jvm.options
and custom JVM options files can be
added to config/jvm.options.d/
. When installing from the Debian or RPM
packages, the root jvm.options
configuration file is
/etc/opensearch/jvm.options
and custom JVM options files can be added to
/etc/opensearch/jvm.options.d/
. When using the Docker
distribution of OpenSearch you can bind mount custom JVM options files into
/usr/share/opensearch/config/jvm.options.d/
. You should never need to
modify the root jvm.options
file instead preferring to use custom JVM options
files. The processing ordering of custom JVM options is lexicographic.
JVM options files must have the suffix '.options' and contain a line-delimited list of JVM arguments following a special syntax:
-
lines consisting of whitespace only are ignored
-
lines beginning with
#
are treated as comments and are ignored# this is a comment
-
lines beginning with a
-
are treated as a JVM option that applies independent of the version of the JVM-Xmx2g
-
lines beginning with a number followed by a
:
followed by a-
are treated as a JVM option that applies only if the version of the JVM matches the number8:-Xmx2g
-
lines beginning with a number followed by a
-
followed by a:
are treated as a JVM option that applies only if the version of the JVM is greater than or equal to the number8-:-Xmx2g
-
lines beginning with a number followed by a
-
followed by a number followed by a:
are treated as a JVM option that applies only if the version of the JVM falls in the range of the two numbers8-9:-Xmx2g
-
all other lines are rejected
An alternative mechanism for setting Java Virtual Machine options is via the
OPENSEARCH_JAVA_OPTS
environment variable. For instance:
export OPENSEARCH_JAVA_OPTS="$OPENSEARCH_JAVA_OPTS -Djava.io.tmpdir=/path/to/temp/dir"
./bin/opensearch
When using the RPM or Debian packages, OPENSEARCH_JAVA_OPTS
can be specified in the
system configuration file.
The JVM has a built-in mechanism for observing the JAVA_TOOL_OPTIONS
environment variable.  The primary reason for this is that on some OS (e.g., Ubuntu)
there are agents installed by default via this environment variable that we do
not want interfering with OpenSearch.
Additionally, some other Java programs support the JAVA_OPTS
environment
variable. This is not a mechanism built into the JVM but instead a convention
in the ecosystem. However, we do not support this environment variable, instead
supporting setting JVM options via the jvm.options
file or the environment
variable OPENSEARCH_JAVA_OPTS
as above.
Secure settings
Some settings are sensitive, and relying on filesystem permissions to protect
their values is not sufficient. For this use case, OpenSearch provides a
keystore and the opensearch-keystore
tool to
manage the settings in the keystore.
|
Important
|
Only some settings are designed to be read from the keystore. However, the keystore has no validation to block unsupported settings. Adding unsupported settings to the keystore causes OpenSearch to fail to start. To see whether a setting is supported in the keystore, look for a "Secure" qualifier the setting reference. |
All the modifications to the keystore take effect only after restarting OpenSearch.
These settings, just like the regular ones in the opensearch.yml
config file,
need to be specified on each node in the cluster. Currently, all secure settings
are node-specific settings that must have the same value on every node.
Reloadable secure settings
Just like the settings values in opensearch.yml
, changes to the keystore
contents are not automatically applied to the running OpenSearch node. Re-reading
settings requires a node restart. However, certain secure settings are marked as
reloadable. Such settings can be re-read and applied on a running node.
The values of all secure settings, reloadable or not, must be identical
across all cluster nodes. After making the desired secure settings changes,
using the bin/opensearch-keystore add
command, call:
POST _nodes/reload_secure_settings
{
"secure_settings_password": "s3cr3t" (1)
}
-
The password that the OpenSearch keystore is encrypted with.
This API decrypts and re-reads the entire keystore, on every cluster node, but only the reloadable secure settings are applied. Changes to other settings do not go into effect until the next restart. Once the call returns, the reload has been completed, meaning that all internal data structures dependent on these settings have been changed. Everything should look as if the settings had the new value from the start.
When changing multiple reloadable secure settings, modify all of them on each
cluster node, then issue a reload_secure_settings
call instead of reloading after each modification.
There are reloadable secure settings for:
Circuit breaker settings
OpenSearch contains multiple circuit breakers used to prevent operations from causing an OutOfMemoryError. Each breaker specifies a limit for how much memory it can use. Additionally, there is a parent-level breaker that specifies the total amount of memory that can be used across all breakers.
Except where noted otherwise, these settings can be dynamically updated on a live cluster with the cluster-update-settings API.
Parent circuit breaker
The parent-level breaker can be configured with the following settings:
indices.breaker.total.use_real_memory
-
(Static) Determines whether the parent breaker should take real memory usage into account (
true
) or only consider the amount that is reserved by child circuit breakers (false
). Defaults totrue
.
indices.breaker.total.limit
-
(Dynamic) Starting limit for overall parent breaker. Defaults to 70% of JVM heap if
indices.breaker.total.use_real_memory
isfalse
. Ifindices.breaker.total.use_real_memory
istrue
, defaults to 95% of the JVM heap.
Field data circuit breaker
The field data circuit breaker estimates the heap memory required to load a field into the field data cache. If loading the field would cause the cache to exceed a predefined memory limit, the circuit breaker stops the operation and returns an error.
indices.breaker.fielddata.limit
-
(Dynamic) Limit for fielddata breaker. Defaults to 40% of JVM heap.
indices.breaker.fielddata.overhead
-
(Dynamic) A constant that all field data estimations are multiplied with to determine a final estimation. Defaults to
1.03
.
Request circuit breaker
The request circuit breaker allows OpenSearch to prevent per-request data structures (for example, memory used for calculating aggregations during a request) from exceeding a certain amount of memory.
indices.breaker.request.limit
-
(Dynamic) Limit for request breaker, defaults to 60% of JVM heap.
indices.breaker.request.overhead
-
(Dynamic) A constant that all request estimations are multiplied with to determine a final estimation. Defaults to
1
.
In flight requests circuit breaker
The in flight requests circuit breaker allows OpenSearch to limit the memory usage of all currently active incoming requests on transport or HTTP level from exceeding a certain amount of memory on a node. The memory usage is based on the content length of the request itself. This circuit breaker also considers that memory is not only needed for representing the raw request but also as a structured object which is reflected by default overhead.
network.breaker.inflight_requests.limit
-
(Dynamic) Limit for in flight requests breaker, defaults to 100% of JVM heap. This means that it is bound by the limit configured for the parent circuit breaker.
network.breaker.inflight_requests.overhead
-
(Dynamic) A constant that all in flight requests estimations are multiplied with to determine a final estimation. Defaults to 2.
Accounting requests circuit breaker
The accounting circuit breaker allows OpenSearch to limit the memory usage of things held in memory that are not released when a request is completed. This includes things like the Lucene segment memory.
indices.breaker.accounting.limit
-
(Dynamic) Limit for accounting breaker, defaults to 100% of JVM heap. This means that it is bound by the limit configured for the parent circuit breaker.
indices.breaker.accounting.overhead
-
(Dynamic) A constant that all accounting estimations are multiplied with to determine a final estimation. Defaults to 1
Script compilation circuit breaker
Slightly different than the previous memory-based circuit breaker, the script compilation circuit breaker limits the number of inline script compilations within a period of time.
See the "prefer-parameters" section of the scripting documentation for more information.
script.context.$CONTEXT.max_compilations_rate
-
(Dynamic) Limit for the number of unique dynamic scripts within a certain interval that are allowed to be compiled for a given context. Defaults to
75/5m
, meaning 75 every 5 minutes.
Cluster-level shard allocation and routing settings
Shard allocation is the process of allocating shards to nodes. This can happen during initial recovery, replica allocation, rebalancing, or when nodes are added or removed.
One of the main roles of the master is to decide which shards to allocate to which nodes, and when to move shards between nodes in order to rebalance the cluster.
There are a number of settings available to control the shard allocation process:
-
Cluster-level shard allocation settings control allocation and rebalancing operations.
-
Disk-based shard allocation settings explains how OpenSearch takes available disk space into account, and the related settings.
-
Shard allocation awareness and Forced awareness control how shards can be distributed across different racks or availability zones.
-
Cluster-level shard allocation filtering allows certain nodes or groups of nodes excluded from allocation so that they can be decommissioned.
Besides these, there are a few other miscellaneous cluster-level settings.
Cluster-level shard allocation settings
You can use the following settings to control shard allocation and recovery:
cluster.routing.allocation.enable
-
(Dynamic) Enable or disable allocation for specific kinds of shards:
-
all
- (default) Allows shard allocation for all kinds of shards. -
primaries
- Allows shard allocation only for primary shards. -
new_primaries
- Allows shard allocation only for primary shards for new indices. -
none
- No shard allocations of any kind are allowed for any indices.
This setting does not affect the recovery of local primary shards when restarting a node. A restarted node that has a copy of an unassigned primary shard will recover that primary immediately, assuming that its allocation id matches one of the active allocation ids in the cluster state.
-
cluster.routing.allocation.node_concurrent_incoming_recoveries
-
(Dynamic) How many concurrent incoming shard recoveries are allowed to happen on a node. Incoming recoveries are the recoveries where the target shard (most likely the replica unless a shard is relocating) is allocated on the node. Defaults to
2
. cluster.routing.allocation.node_concurrent_outgoing_recoveries
-
(Dynamic) How many concurrent outgoing shard recoveries are allowed to happen on a node. Outgoing recoveries are the recoveries where the source shard (most likely the primary unless a shard is relocating) is allocated on the node. Defaults to
2
. cluster.routing.allocation.node_concurrent_recoveries
-
(Dynamic) A shortcut to set both
cluster.routing.allocation.node_concurrent_incoming_recoveries
andcluster.routing.allocation.node_concurrent_outgoing_recoveries
. cluster.routing.allocation.node_initial_primaries_recoveries
-
(Dynamic) While the recovery of replicas happens over the network, the recovery of an unassigned primary after node restart uses data from the local disk. These should be fast so more initial primary recoveries can happen in parallel on the same node. Defaults to
4
. cluster.routing.allocation.same_shard.host
-
(Dynamic) Allows to perform a check to prevent allocation of multiple instances of the same shard on a single host, based on host name and host address. Defaults to
false
, meaning that no check is performed by default. This setting only applies if multiple nodes are started on the same machine.
Shard rebalancing settings
A cluster is balanced when it has an equal number of shards on each node without having a concentration of shards from any index on any node. OpenSearch runs an automatic process called rebalancing which moves shards between the nodes in your cluster to improve its balance. Rebalancing obeys all other shard allocation rules such as allocation filtering and forced awareness which may prevent it from completely balancing the cluster. In that case, rebalancing strives to acheve the most balanced cluster possible within the rules you have configured.
You can use the following settings to control the rebalancing of shards across the cluster:
cluster.routing.rebalance.enable
-
(Dynamic) Enable or disable rebalancing for specific kinds of shards:
-
all
- (default) Allows shard balancing for all kinds of shards. -
primaries
- Allows shard balancing only for primary shards. -
replicas
- Allows shard balancing only for replica shards. -
none
- No shard balancing of any kind are allowed for any indices.
-
cluster.routing.allocation.allow_rebalance
-
(Dynamic) Specify when shard rebalancing is allowed:
-
always
- Always allow rebalancing. -
indices_primaries_active
- Only when all primaries in the cluster are allocated. -
indices_all_active
- (default) Only when all shards (primaries and replicas) in the cluster are allocated.
-
cluster.routing.allocation.cluster_concurrent_rebalance
-
(Dynamic) Allow to control how many concurrent shard rebalances are allowed cluster wide. Defaults to
2
. Note that this setting only controls the number of concurrent shard relocations due to imbalances in the cluster. This setting does not limit shard relocations due to allocation filtering or forced awareness.
Shard balancing heuristics settings
Rebalancing works by computing a weight for each node based on its allocation of shards, and then moving shards between nodes to reduce the weight of the heavier nodes and increase the weight of the lighter ones. The cluster is balanced when there is no possible shard movement that can bring the weight of any node closer to the weight of any other node by more than a configurable threshold. The following settings allow you to control the details of these calculations.
cluster.routing.allocation.balance.shard
-
(Dynamic) Defines the weight factor for the total number of shards allocated on a node (float). Defaults to
0.45f
. Raising this raises the tendency to equalize the number of shards across all nodes in the cluster. cluster.routing.allocation.balance.index
-
(Dynamic) Defines the weight factor for the number of shards per index allocated on a specific node (float). Defaults to
0.55f
. Raising this raises the tendency to equalize the number of shards per index across all nodes in the cluster. cluster.routing.allocation.balance.threshold
-
(Dynamic) Minimal optimization value of operations that should be performed (non negative float). Defaults to
1.0f
. Raising this will cause the cluster to be less aggressive about optimizing the shard balance.
|
Note
|
Regardless of the result of the balancing algorithm, rebalancing might not be allowed due to forced awareness or allocation filtering. |
Disk-based shard allocation settings
The disk-based shard allocator ensures that all nodes have enough disk space without performing more shard movements than necessary. It allocates shards based on a pair of thresholds known as the low watermark and the high watermark. Its primary goal is to ensure that no node exceeds the high watermark, or at least that any such overage is only temporary. If a node exceeds the high watermark then OpenSearch will solve this by moving some of its shards onto other nodes in the cluster.
|
Note
|
It is normal for nodes to temporarily exceed the high watermark from time to time. |
The allocator also tries to keep nodes clear of the high watermark by forbidding the allocation of more shards to a node that exceeds the low watermark. Importantly, if all of your nodes have exceeded the low watermark then no new shards can be allocated and OpenSearch will not be able to move any shards between nodes in order to keep the disk usage below the high watermark. You must ensure that your cluster has enough disk space in total and that there are always some nodes below the low watermark.
Shard movements triggered by the disk-based shard allocator must also satisfy all other shard allocation rules such as allocation filtering and forced awareness. If these rules are too strict then they can also prevent the shard movements needed to keep the nodes' disk usage under control.
If a node is filling up its disk faster than OpenSearch can move shards elsewhere then there is a risk that the disk will completely fill up. To prevent this, as a last resort, once the disk usage reaches the flood-stage watermark OpenSearch will block writes to indices with a shard on the affected node. It will also continue to move shards onto the other nodes in the cluster. When disk usage on the affected node drops below the high watermark, OpenSearch automatically removes the write block.
|
Tip
|
It is normal for the nodes in your cluster to be using very different amounts of disk space. The balance of the cluster depends only on the number of shards on each node and the indices to which those shards belong. It considers neither the sizes of these shards nor the available disk space on each node, for the following reasons:
|
You can use the following settings to control disk-based allocation:
cluster.routing.allocation.disk.threshold_enabled
-
(Dynamic) Defaults to
true
. Set tofalse
to disable the disk allocation decider.
cluster.routing.allocation.disk.watermark.low
-
(Dynamic) Controls the low watermark for disk usage. It defaults to
85%
, meaning that OpenSearch will not allocate shards to nodes that have more than 85% disk used. It can also be set to an absolute byte value (like500mb
) to prevent OpenSearch from allocating shards if less than the specified amount of space is available. This setting has no effect on the primary shards of newly-created indices but will prevent their replicas from being allocated.
cluster.routing.allocation.disk.watermark.high
-
(Dynamic) Controls the high watermark. It defaults to
90%
, meaning that OpenSearch will attempt to relocate shards away from a node whose disk usage is above 90%. It can also be set to an absolute byte value (similarly to the low watermark) to relocate shards away from a node if it has less than the specified amount of free space. This setting affects the allocation of all shards, whether previously allocated or not. cluster.routing.allocation.disk.watermark.enable_for_single_data_node
-
(Static) For a single data node, the default is to disregard disk watermarks when making an allocation decision. This is deprecated behavior and will be changed in 8.0. This setting can be set to
true
to enable the disk watermarks for a single data node cluster (will become default in 8.0).
cluster.routing.allocation.disk.watermark.flood_stage
-
(Dynamic) Controls the flood stage watermark, which defaults to 95%. OpenSearch enforces a read-only index block (
index.blocks.read_only_allow_delete
) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark.NoteYou cannot mix the usage of percentage values and byte values within these settings. Either all values are set to percentage values, or all are set to byte values. This enforcement is so that OpenSearch can validate that the settings are internally consistent, ensuring that the low disk threshold is less than the high disk threshold, and the high disk threshold is less than the flood stage threshold. An example of resetting the read-only index block on the
my-index-000001
index:PUT /my-index-000001/_settings { "index.blocks.read_only_allow_delete": null }
cluster.info.update.interval
-
(Dynamic) How often OpenSearch should check on disk usage for each node in the cluster. Defaults to
30s
. cluster.routing.allocation.disk.include_relocations
-
Defaults to true, which means that OpenSearch will take into account shards that are currently being relocated to the target node when computing a node’s disk usage. Taking relocating shards' sizes into account may, however, mean that the disk usage for a node is incorrectly estimated on the high side, since the relocation could be 90% complete and a recently retrieved disk usage would include the total size of the relocating shard as well as the space already used by the running relocation.
|
Note
|
Percentage values refer to used disk space, while byte values refer to free disk space. This can be confusing, since it flips the meaning of high and low. For example, it makes sense to set the low watermark to 10gb and the high watermark to 5gb, but not the other way around. |
An example of updating the low watermark to at least 100 gigabytes free, a high watermark of at least 50 gigabytes free, and a flood stage watermark of 10 gigabytes free, and updating the information about the cluster every minute:
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "100gb",
"cluster.routing.allocation.disk.watermark.high": "50gb",
"cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
"cluster.info.update.interval": "1m"
}
}
Shard allocation awareness
You can use custom node attributes as awareness attributes to enable OpenSearch to take your physical hardware configuration into account when allocating shards. If OpenSearch knows which nodes are on the same physical server, in the same rack, or in the same zone, it can distribute the primary shard and its replica shards to minimise the risk of losing all shard copies in the event of a failure.
When shard allocation awareness is enabled with the
dynamic
cluster.routing.allocation.awareness.attributes
setting, shards are only
allocated to nodes that have values set for the specified awareness attributes.
If you use multiple awareness attributes, OpenSearch considers each attribute
separately when allocating shards.
By default OpenSearch uses adaptive replica selection
to route search or GET requests. However, with the presence of allocation awareness
attributes OpenSearch will prefer using shards in the same location (with the same
awareness attribute values) to process these requests. This behavior can be
disabled by specifying export OPENSEARCH_JAVA_OPTS="$OPENSEARCH_JAVA_OPTS -Des.search.ignore_awareness_attributes=true"
system property on every node that is part of the cluster.
|
Note
|
The number of attribute values determines how many shard copies are allocated in each location. If the number of nodes in each location is unbalanced and there are a lot of replicas, replica shards might be left unassigned. |
Enabling shard allocation awareness
To enable shard allocation awareness:
-
Specify the location of each node with a custom node attribute. For example, if you want OpenSearch to distribute shards across different racks, you might set an awareness attribute called
rack_id
in each node’sopensearch.yml
config file.node.attr.rack_id: rack_one
You can also set custom attributes when you start a node:
`./bin/opensearch -Enode.attr.rack_id=rack_one`
-
Tell OpenSearch to take one or more awareness attributes into account when allocating shards by setting
cluster.routing.allocation.awareness.attributes
in every master-eligible node’sopensearch.yml
config file.cluster.routing.allocation.awareness.attributes: rack_id (1)
-
Specify multiple attributes as a comma-separated list.
You can also use the cluster-update-settings API to set or update a cluster’s awareness attributes.
-
With this example configuration, if you start two nodes with
node.attr.rack_id
set to rack_one
and create an index with 5 primary
shards and 1 replica of each primary, all primaries and replicas are
allocated across the two nodes.
If you add two nodes with node.attr.rack_id
set to rack_two
,
OpenSearch moves shards to the new nodes, ensuring (if possible)
that no two copies of the same shard are in the same rack.
If rack_two
fails and takes down both its nodes, by default OpenSearch
allocates the lost shard copies to nodes in rack_one
. To prevent multiple
copies of a particular shard from being allocated in the same location, you can
enable forced awareness.
Forced awareness
By default, if one location fails, OpenSearch assigns all of the missing replica shards to the remaining locations. While you might have sufficient resources across all locations to host your primary and replica shards, a single location might be unable to host ALL of the shards.
To prevent a single location from being overloaded in the event of a failure,
you can set cluster.routing.allocation.awareness.force
so no replicas are
allocated until nodes are available in another location.
For example, if you have an awareness attribute called zone
and configure nodes
in zone1
and zone2
, you can use forced awareness to prevent OpenSearch
from allocating replicas if only one zone is available:
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2 (1)
-
Specify all possible values for the awareness attribute.
With this example configuration, if you start two nodes with node.attr.zone
set
to zone1
and create an index with 5 shards and 1 replica, OpenSearch creates
the index and allocates the 5 primary shards but no replicas. Replicas are
only allocated once nodes with node.attr.zone
set to zone2
are available.
Cluster-level shard allocation filtering
You can use cluster-level shard allocation filters to control where OpenSearch allocates shards from any index. These cluster wide filters are applied in conjunction with per-index allocation filtering and allocation awareness.
Shard allocation filters can be based on custom node attributes or the built-in
_name
, _host_ip
, _publish_ip
, _ip
, _host
and _id
attributes.
The cluster.routing.allocation
settings are dynamic, enabling live indices to
be moved from one set of nodes to another. Shards are only relocated if it is
possible to do so without breaking another routing constraint, such as never
allocating a primary and replica shard on the same node.
The most common use case for cluster-level shard allocation filtering is when you want to decommission a node. To move shards off of a node prior to shutting it down, you could create a filter that excludes the node by its IP address:
PUT _cluster/settings
{
"transient" : {
"cluster.routing.allocation.exclude._ip" : "<IP_ADDRESS>"
}
}
Cluster routing settings
cluster.routing.allocation.include.{attribute}
-
(Dynamic) Allocate shards to a node whose
{attribute}
has at least one of the comma-separated values. cluster.routing.allocation.require.{attribute}
-
(Dynamic) Only allocate shards to a node whose
{attribute}
has all of the comma-separated values. cluster.routing.allocation.exclude.{attribute}
-
(Dynamic) Do not allocate shards to a node whose
{attribute}
has any of the comma-separated values.
The cluster allocation settings support the following built-in attributes:
_name
|
Match nodes by node name |
_host_ip
|
Match nodes by host IP address (IP associated with hostname) |
_publish_ip
|
Match nodes by publish IP address |
_ip
|
Match either |
_host
|
Match nodes by hostname |
_id
|
Match nodes by node id |
You can use wildcards when specifying attribute values, for example:
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.exclude._ip": "192.168.2.*"
}
}
Miscellaneous cluster settings
Metadata
An entire cluster may be set to read-only with the following setting:
cluster.blocks.read_only
-
(Dynamic) Make the whole cluster read only (indices do not accept write operations), metadata is not allowed to be modified (create or delete indices).
cluster.blocks.read_only_allow_delete
-
(Dynamic) Identical to
cluster.blocks.read_only
but allows to delete indices to free up resources.
|
Warning
|
Don’t rely on this setting to prevent changes to your cluster. Any user with access to the cluster-update-settings API can make the cluster read-write again. |
Cluster shard limit
There is a soft limit on the number of shards in a cluster, based on the number of nodes in the cluster. This is intended to prevent operations which may unintentionally destabilize the cluster.
|
Important
|
This limit is intended as a safety net, not a sizing recommendation. The exact number of shards your cluster can safely support depends on your hardware configuration and workload, but should remain well below this limit in almost all cases, as the default limit is set quite high. |
If an operation, such as creating a new index, restoring a snapshot of an index, or opening a closed index would lead to the number of shards in the cluster going over this limit, the operation will fail with an error indicating the shard limit.
If the cluster is already over the limit, due to changes in node membership or setting changes, all operations that create or open indices will fail until either the limit is increased as described below, or some indices are closed or deleted to bring the number of shards below the limit.
The cluster shard limit defaults to 1,000 shards per data node. Both primary and replica shards of all open indices count toward the limit, including unassigned shards. For example, an open index with 5 primary shards and 2 replicas counts as 15 shards. Closed indices do not contribute to the shard count.
You can dynamically adjust the cluster shard limit with the following setting:
cluster.max_shards_per_node
-
(Dynamic) Limits the total number of primary and replica shards for the cluster. OpenSearch calculates the limit as follows:
cluster.max_shards_per_node * number of data nodes
Shards for closed indices do not count toward this limit. Defaults to
1000
. A cluster with no data nodes is unlimited.OpenSearch rejects any request that creates more shards than this limit allows. For example, a cluster with a
cluster.max_shards_per_node
setting of100
and three data nodes has a shard limit of 300. If the cluster already contains 296 shards, OpenSearch rejects any request that adds five or more shards to the cluster.NoteThis setting does not limit shards for individual nodes. To limit the number of shards for each node, use the cluster.routing.allocation.total_shards_per_node
setting.
User-defined cluster metadata
User-defined metadata can be stored and retrieved using the Cluster Settings API.
This can be used to store arbitrary, infrequently-changing data about the cluster
without the need to create an index to store it. This data may be stored using
any key prefixed with cluster.metadata.
. For example, to store the email
address of the administrator of a cluster under the key cluster.metadata.administrator
,
issue this request:
PUT /_cluster/settings
{
"persistent": {
"cluster.metadata.administrator"<EMAIL_ADDRESS>}
}
|
Important
|
User-defined cluster metadata is not intended to store sensitive or confidential information. Any information stored in user-defined cluster metadata will be viewable by anyone with access to the Cluster Get Settings API, and is recorded in the OpenSearch logs. |
Index tombstones
The cluster state maintains index tombstones to explicitly denote indices that have been deleted. The number of tombstones maintained in the cluster state is controlled by the following setting:
cluster.indices.tombstones.size
-
(Static) Index tombstones prevent nodes that are not part of the cluster when a delete occurs from joining the cluster and reimporting the index as though the delete was never issued. To keep the cluster state from growing huge we only keep the last
cluster.indices.tombstones.size
deletes, which defaults to 500. You can increase it if you expect nodes to be absent from the cluster and miss more than 500 deletes. We think that is rare, thus the default. Tombstones don’t take up much space, but we also think that a number like 50,000 is probably too big.
If OpenSearch encounters index data that is absent from the current cluster
state, those indices are considered to be dangling. For example,
this can happen if you delete more than
cluster.indices.tombstones.size
indices while an OpenSearch node is offline.
You can use the Dangling indices API to manage
this situation.
Logger
The settings which control logging can be updated dynamically with the
logger.
prefix. For instance, to increase the logging level of the
indices.recovery
module to DEBUG
, issue this request:
PUT /_cluster/settings
{
"transient": {
"logger.org.opensearch.indices.recovery": "DEBUG"
}
}
Persistent tasks allocation
Plugins can create a kind of tasks called persistent tasks. Those tasks are usually long-lived tasks and are stored in the cluster state, allowing the tasks to be revived after a full cluster restart.
Every time a persistent task is created, the master node takes care of assigning the task to a node of the cluster, and the assigned node will then pick up the task and execute it locally. The process of assigning persistent tasks to nodes is controlled by the following settings:
cluster.persistent_tasks.allocation.enable
-
(Dynamic) Enable or disable allocation for persistent tasks:
-
all
- (default) Allows persistent tasks to be assigned to nodes -
none
- No allocations are allowed for any type of persistent task
This setting does not affect the persistent tasks that are already being executed. Only newly created persistent tasks, or tasks that must be reassigned (after a node left the cluster, for example), are impacted by this setting.
-
cluster.persistent_tasks.allocation.recheck_interval
-
(Dynamic) The master node will automatically check whether persistent tasks need to be assigned when the cluster state changes significantly. However, there may be other factors, such as memory usage, that affect whether persistent tasks can be assigned to nodes but do not cause the cluster state to change. This setting controls how often assignment checks are performed to react to these factors. The default is 30 seconds. The minimum permitted value is 10 seconds.
Discovery and cluster formation settings
Discovery and cluster formation are affected by the following settings:
discovery.seed_hosts
-
(Static) Provides a list of the addresses of the master-eligible nodes in the cluster. May also be a single string containing the addresses separated by commas. Each address has the format
host:port
orhost
. Thehost
is either a host name to be resolved by DNS, an IPv4 address, or an IPv6 address. IPv6 addresses must be enclosed in square brackets. If a host name resolves via DNS to multiple addresses, OpenSearch uses all of them. DNS lookups are subject to JVM DNS caching. If theport
is not given then it is determined by checking the following settings in order:-
transport.profiles.default.port
-
transport.port
If neither of these is set then the default port is
9300
. The default value fordiscovery.seed_hosts
is["<IP_ADDRESS>", "[::1]"]
. Seediscovery.seed_hosts
. -
discovery.seed_providers
-
(Static) Specifies which types of seed hosts provider to use to obtain the addresses of the seed nodes used to start the discovery process. By default, it is the settings-based seed hosts provider which obtains the seed node addresses from the
discovery.seed_hosts
setting. discovery.type
-
(Static) Specifies whether OpenSearch should form a multiple-node cluster. By default, OpenSearch discovers other nodes when forming a cluster and allows other nodes to join the cluster later. If
discovery.type
is set tosingle-node
, OpenSearch forms a single-node cluster and suppresses the timeout set bycluster.publish.timeout
. For more information about when you might use this setting, see Single-node discovery. cluster.initial_master_nodes
-
(Static) Sets the initial set of master-eligible nodes in a brand-new cluster. By default this list is empty, meaning that this node expects to join a cluster that has already been bootstrapped. See
cluster.initial_master_nodes
.
Expert settings
Discovery and cluster formation are also affected by the following expert-level settings, although it is not recommended to change any of these from their default values.
|
Warning
|
If you adjust these settings then your cluster may not form correctly or may become unstable or intolerant of certain failures. |
discovery.cluster_formation_warning_timeout
-
(Static) Sets how long a node will try to form a cluster before logging a warning that the cluster did not form. Defaults to
10s
. If a cluster has not formed afterdiscovery.cluster_formation_warning_timeout
has elapsed then the node will log a warning message that starts with the phrasemaster not discovered
which describes the current state of the discovery process. discovery.find_peers_interval
-
(Static) Sets how long a node will wait before attempting another discovery round. Defaults to
1s
. discovery.probe.connect_timeout
-
(Static) Sets how long to wait when attempting to connect to each address. Defaults to
3s
. discovery.probe.handshake_timeout
-
(Static) Sets how long to wait when attempting to identify the remote node via a handshake. Defaults to
1s
. discovery.request_peers_timeout
-
(Static) Sets how long a node will wait after asking its peers again before considering the request to have failed. Defaults to
3s
. discovery.seed_resolver.max_concurrent_resolvers
-
(Static) Specifies how many concurrent DNS lookups to perform when resolving the addresses of seed nodes. Defaults to
10
. discovery.seed_resolver.timeout
-
(Static) Specifies how long to wait for each DNS lookup performed when resolving the addresses of seed nodes. Defaults to
5s
. cluster.auto_shrink_voting_configuration
-
(Dynamic) Controls whether the voting configuration sheds departed nodes automatically, as long as it still contains at least 3 nodes. The default value is
true
. If set tofalse
, the voting configuration never shrinks automatically and you must remove departed nodes manually with the voting configuration exclusions API. cluster.election.back_off_time
-
(Static) Sets the amount to increase the upper bound on the wait before an election on each election failure. Note that this is linear backoff. This defaults to
100ms
. Changing this setting from the default may cause your cluster to fail to elect a master node. cluster.election.duration
-
(Static) Sets how long each election is allowed to take before a node considers it to have failed and schedules a retry. This defaults to
500ms
. Changing this setting from the default may cause your cluster to fail to elect a master node. cluster.election.initial_timeout
-
(Static) Sets the upper bound on how long a node will wait initially, or after the elected master fails, before attempting its first election. This defaults to
100ms
. Changing this setting from the default may cause your cluster to fail to elect a master node. cluster.election.max_timeout
-
(Static) Sets the maximum upper bound on how long a node will wait before attempting an first election, so that an network partition that lasts for a long time does not result in excessively sparse elections. This defaults to
10s
. Changing this setting from the default may cause your cluster to fail to elect a master node. cluster.fault_detection.follower_check.interval
-
(Static) Sets how long the elected master waits between follower checks to each other node in the cluster. Defaults to
1s
. Changing this setting from the default may cause your cluster to become unstable. cluster.fault_detection.follower_check.timeout
-
(Static) Sets how long the elected master waits for a response to a follower check before considering it to have failed. Defaults to
10s
. Changing this setting from the default may cause your cluster to become unstable. cluster.fault_detection.follower_check.retry_count
-
(Static) Sets how many consecutive follower check failures must occur to each node before the elected master considers that node to be faulty and removes it from the cluster. Defaults to
3
. Changing this setting from the default may cause your cluster to become unstable. cluster.fault_detection.leader_check.interval
-
(Static) Sets how long each node waits between checks of the elected master. Defaults to
1s
. Changing this setting from the default may cause your cluster to become unstable. cluster.fault_detection.leader_check.timeout
-
(Static) Sets how long each node waits for a response to a leader check from the elected master before considering it to have failed. Defaults to
10s
. Changing this setting from the default may cause your cluster to become unstable. cluster.fault_detection.leader_check.retry_count
-
(Static) Sets how many consecutive leader check failures must occur before a node considers the elected master to be faulty and attempts to find or elect a new master. Defaults to
3
. Changing this setting from the default may cause your cluster to become unstable. cluster.follower_lag.timeout
-
(Static) Sets how long the master node waits to receive acknowledgements for cluster state updates from lagging nodes. The default value is
90s
. If a node does not successfully apply the cluster state update within this period of time, it is considered to have failed and is removed from the cluster. See Publishing the cluster state. cluster.max_voting_config_exclusions
-
(Dynamic) Sets a limit on the number of voting configuration exclusions at any one time. The default value is
10
. See Adding and removing nodes. cluster.publish.info_timeout
-
(Static) Sets how long the master node waits for each cluster state update to be completely published to all nodes before logging a message indicating that some nodes are responding slowly. The default value is
10s
. cluster.publish.timeout
-
(Static) Sets how long the master node waits for each cluster state update to be completely published to all nodes, unless
discovery.type
is set tosingle-node
. The default value is30s
. See Publishing the cluster state.
cluster.no_master_block
-
(Dynamic) Specifies which operations are rejected when there is no active master in a cluster. This setting has three valid values:
all
-
All operations on the node (both read and write operations) are rejected. This also applies for API cluster state read or write operations, like the get index settings, put mapping and cluster state API.
write
-
(default) Write operations are rejected. Read operations succeed, based on the last known cluster configuration. This situation may result in partial reads of stale data as this node may be isolated from the rest of the cluster.
metadata_write
-
Only metadata write operations (e.g. mapping updates, routing table changes) are rejected but regular indexing operations continue to work. Read and write operations succeed, based on the last known cluster configuration. This situation may result in partial reads of stale data as this node may be isolated from the rest of the cluster.
Note-
The
cluster.no_master_block
setting doesn’t apply to nodes-based APIs (for example, cluster stats, node info, and node stats APIs). Requests to these APIs are not be blocked and can run on any available node. -
For the cluster to be fully operational, it must have an active master.
monitor.fs.health.enabled
-
(Dynamic) If
true
, the node runs periodic filesystem health checks. Defaults totrue
. monitor.fs.health.refresh_interval
-
(Static) Interval between successive filesystem health checks. Defaults to
2m
. monitor.fs.health.slow_path_logging_threshold
-
(Dynamic) If a filesystem health checks takes longer than this threshold then OpenSearch logs a warning. Defaults to
5s
.
Field data cache settings
The field data cache contains field data and global ordinals, which are both used to support aggregations on certain field types. Since these are on-heap data structures, it is important to monitor the cache’s use.
Cache size
The entries in the cache are expensive to build, so the default behavior is to keep the cache loaded in memory. The default cache size is unlimited, causing the cache to grow until it reaches the limit set by the field data circuit breaker. This behavior can be configured.
If the cache size limit is set, the cache will begin clearing the least-recently-updated entries in the cache. This setting can automatically avoid the circuit breaker limit, at the cost of rebuilding the cache as needed.
If the circuit breaker limit is reached, further requests that increase the cache size will be prevented. In this case you should manually clear the cache.
indices.fielddata.cache.size
-
(Static) The max size of the field data cache, eg
38%
of node heap space, or an absolute value, eg12GB
. Defaults to unbounded. If you choose to set it, it should be smaller than Field data circuit breaker limit.
Monitoring field data
You can monitor memory usage for field data as well as the field data circuit breaker using the nodes stats API or the cat fielddata API.
HTTP
The HTTP layer exposes OpenSearch’s REST APIs over HTTP. Clients send HTTP requests to a node in the cluster which either handles it locally or else passes it on to other nodes for further processing using the Transport layer.
When possible, consider using HTTP keep alive when connecting for better performance and try to get your favorite client not to do HTTP chunking.
HTTP settings
The following settings can be configured for HTTP. These settings also use the common network settings.
http.port
-
(Static) A bind port range. Defaults to
9200-9300
. http.publish_port
-
(Static) The port that HTTP clients should use when communicating with this node. Useful when a cluster node is behind a proxy or firewall and the
http.port
is not directly addressable from the outside. Defaults to the actual port assigned viahttp.port
. http.bind_host
-
(Static) The host address to bind the HTTP service to. Defaults to
http.host
(if set) ornetwork.bind_host
. http.publish_host
-
(Static) The host address to publish for HTTP clients to connect to. Defaults to
http.host
(if set) ornetwork.publish_host
. http.host
-
(Static) Used to set the
http.bind_host
and thehttp.publish_host
. http.max_content_length
-
(Static) Maximum size of an HTTP request body. Defaults to
100mb
. http.max_initial_line_length
-
(Static) Maximum size of an HTTP URL. Defaults to
4kb
. http.max_header_size
-
(Static) Maximum size of allowed headers. Defaults to
8kb
.
http.compression
-
(Static) Support for compression when possible (with Accept-Encoding). If HTTPS is enabled, defaults to
false
. Otherwise, defaults totrue
.Disabling compression for HTTPS mitigates potential security risks, such as a BREACH attack. To compress HTTPS traffic, you must explicitly set
http.compression
totrue
. http.compression_level
-
(Static) Defines the compression level to use for HTTP responses. Valid values are in the range of 1 (minimum compression) and 9 (maximum compression). Defaults to
3
.
http.cors.enabled
-
(Static) Enable or disable cross-origin resource sharing, which determines whether a browser on another origin can execute requests against OpenSearch. Set to
true
to enable OpenSearch to process pre-flight CORS requests. OpenSearch will respond to those requests with theAccess-Control-Allow-Origin
header if theOrigin
sent in the request is permitted by thehttp.cors.allow-origin
list. Set tofalse
(the default) to make OpenSearch ignore theOrigin
request header, effectively disabling CORS requests because OpenSearch will never respond with theAccess-Control-Allow-Origin
response header.NoteIf the client does not send a pre-flight request with an Origin
header or it does not check the response headers from the server to validate theAccess-Control-Allow-Origin
response header, then cross-origin security is compromised. If CORS is not enabled on OpenSearch, the only way for the client to know is to send a pre-flight request and realize the required response headers are missing.
http.cors.allow-origin
-
(Static) Which origins to allow. If you prepend and append a forward slash (
/
) to the value, this will be treated as a regular expression, allowing you to support HTTP and HTTPs. For example, using/https?:\/\/localhost(:[0-9]+)?/
would return the request header appropriately in both cases. Defaults to no origins allowed.ImportantA wildcard ( ) is a valid value but is considered a security risk, as your OpenSearch instance is open to cross origin requests from *anywhere.
http.cors.max-age
-
(Static) Browsers send a "preflight" OPTIONS-request to determine CORS settings.
max-age
defines how long the result should be cached for. Defaults to1728000
(20 days).
http.cors.allow-methods
-
(Static) Which methods to allow. Defaults to
OPTIONS, HEAD, GET, POST, PUT, DELETE
.
http.cors.allow-headers
-
(Static) Which headers to allow. Defaults to
X-Requested-With, Content-Type, Content-Length
.
http.cors.allow-credentials
-
(Static) Whether the
Access-Control-Allow-Credentials
header should be returned. Defaults tofalse
.NoteThis header is only returned when the setting is set to true
.
http.detailed_errors.enabled
-
(Static) If
true
, enables the output of detailed error messages and stack traces in the response output. Defaults totrue
.If
false
, use theerror_trace
parameter to enable stack traces and return detailed error messages. Otherwise, only a simple message will be returned. http.pipelining.max_events
-
(Static) The maximum number of events to be queued up in memory before an HTTP connection is closed, defaults to
10000
. http.max_warning_header_count
-
(Static) The maximum number of warning headers in client HTTP responses. Defaults to
unbounded
. http.max_warning_header_size
-
(Static) The maximum total size of warning headers in client HTTP responses. Defaults to
unbounded
. http.tcp.no_delay
-
(Static) Enable or disable the TCP no delay setting. Defaults to
network.tcp.no_delay
. http.tcp.keep_alive
-
(Static) Configures the
SO_KEEPALIVE
option for this socket, which determines whether it sends TCP keepalive probes. Defaults tonetwork.tcp.keep_alive
. http.tcp.keep_idle
-
(Static) Configures the
TCP_KEEPIDLE
option for this socket, which determines the time in seconds that a connection must be idle before starting to send TCP keepalive probes. Defaults tonetwork.tcp.keep_idle
, which uses the system default. This value cannot exceed300
seconds. Only applicable on Linux and macOS, and requires Java 11 or newer. http.tcp.keep_interval
-
(Static) Configures the
TCP_KEEPINTVL
option for this socket, which determines the time in seconds between sending TCP keepalive probes. Defaults tonetwork.tcp.keep_interval
, which uses the system default. This value cannot exceed300
seconds. Only applicable on Linux and macOS, and requires Java 11 or newer. http.tcp.keep_count
-
(Static) Configures the
TCP_KEEPCNT
option for this socket, which determines the number of unacknowledged TCP keepalive probes that may be sent on a connection before it is dropped. Defaults tonetwork.tcp.keep_count
, which uses the system default. Only applicable on Linux and macOS, and requires Java 11 or newer. http.tcp.reuse_address
-
(Static) Should an address be reused or not. Defaults to
network.tcp.reuse_address
. http.tcp.send_buffer_size
-
(Static) The size of the TCP send buffer (specified with size units). Defaults to
network.tcp.send_buffer_size
. http.tcp.receive_buffer_size
-
(Static) The size of the TCP receive buffer (specified with size units). Defaults to
network.tcp.receive_buffer_size
.
REST request tracer
The HTTP layer has a dedicated tracer logger which, when activated, logs incoming requests. The log can be dynamically activated
by setting the level of the org.opensearch.http.HttpTracer
logger to TRACE
:
PUT _cluster/settings
{
"transient" : {
"logger.org.opensearch.http.HttpTracer" : "TRACE"
}
}
You can also control which uris will be traced, using a set of include and exclude wildcard patterns. By default every request will be traced.
PUT _cluster/settings
{
"transient" : {
"http.tracer.include" : "*",
"http.tracer.exclude" : ""
}
}
Index management settings
You can use the following cluster settings to enable or disable index management features.
action.auto_create_index
-
(Dynamic) Automatically create an index if it doesn’t already exist and apply any configured index templates. Defaults to
true
.
action.destructive_requires_name
-
(Dynamic) When set to
true
, you must specify the index name to delete an index. It is not possible to delete all indices with_all
or use wildcards.
cluster.indices.close.enable
-
(Dynamic) Enables opening of closed indices in OpenSearch. You might enable this setting temporarily to change the analyzer configuration for an existing index. We strongly recommend leaving this set to
false
(the default) otherwise.ImportantClosed indices are a data loss risk because they are not included when you make cluster configuration changes, such as scaling to a different capacity, failover, and many other operations. Additionally, closed indices can lead to inaccurate disk space counts.
reindex.remote.whitelist
-
(Static) Specifies the hosts that can be reindexed from remotely. Expects a YAML array of
host:port
strings. Consists of a comma-delimited list ofhost:port
entries. Defaults to["*.io:*", "*.com:*"]
.
Index recovery settings
Peer recovery syncs data from a primary shard to a new or existing shard copy.
Peer recovery automatically occurs when OpenSearch:
-
Recreates a shard lost during node failure
-
Relocates a shard to another node due to a cluster rebalance or changes to the shard allocation settings
You can view a list of in-progress and completed recoveries using the cat recovery API.
Recovery settings
indices.recovery.max_bytes_per_sec
-
(Dynamic) Limits total inbound and outbound recovery traffic for each node. Applies to both peer recoveries as well as snapshot recoveries (i.e., restores from a snapshot). Defaults to
40mb
.This limit applies to each node separately. If multiple nodes in a cluster perform recoveries at the same time, the cluster’s total recovery traffic may exceed this limit.
If this limit is too high, ongoing recoveries may consume an excess of bandwidth and other resources, which can destabilize the cluster.
This is a dynamic setting, which means you can set it in each node’s
opensearch.yml
config file and you can update it dynamically using the cluster update settings API. If you set it dynamically then the same limit applies on every node in the cluster. If you do not set it dynamically then you can set a different limit on each node, which is useful if some of your nodes have better bandwidth than others.
Expert peer recovery settings
You can use the following expert setting to manage resources for peer recoveries.
indices.recovery.max_concurrent_file_chunks
-
(Dynamic, Expert) Number of file chunk requests sent in parallel for each recovery. Defaults to
2
.You can increase the value of this setting when the recovery of a single shard is not reaching the traffic limit set by
indices.recovery.max_bytes_per_sec
. indices.recovery.max_concurrent_operations
-
(Dynamic, Expert) Number of operations sent in parallel for each recovery. Defaults to
1
.Concurrently replaying operations during recovery can be very resource-intensive and may interfere with indexing, search, and other activities in your cluster. Do not increase this setting without carefully verifying that your cluster has the resources available to handle the extra load that will result.
Indexing buffer settings
The indexing buffer is used to store newly indexed documents. When it fills up, the documents in the buffer are written to a segment on disk. It is divided between all shards on the node.
The following settings are static and must be configured on every data node in the cluster:
indices.memory.index_buffer_size
-
(Static) Accepts either a percentage or a byte size value. It defaults to
10%
, meaning that10%
of the total heap allocated to a node will be used as the indexing buffer size shared across all shards. indices.memory.min_index_buffer_size
-
(Static) If the
index_buffer_size
is specified as a percentage, then this setting can be used to specify an absolute minimum. Defaults to48mb
. indices.memory.max_index_buffer_size
-
(Static) If the
index_buffer_size
is specified as a percentage, then this setting can be used to specify an absolute maximum. Defaults to unbounded.
Local gateway settings
The local gateway stores the cluster state and shard data across full cluster restarts.
The following static settings, which must be set on every master node, control how long a freshly elected master should wait before it tries to recover the cluster state and the cluster’s data.
|
Note
|
These settings only take effect on a full cluster restart. |
gateway.expected_nodes
-
(Static) Deprecated. Use
gateway.expected_data_nodes
instead. Number of data or master nodes expected in the cluster. Recovery of local shards begins when the expected number of nodes join the cluster. Defaults to0
. gateway.expected_master_nodes
-
(Static) Deprecated. Use
gateway.expected_data_nodes
instead. Number of master nodes expected in the cluster. Recovery of local shards begins when the expected number of master nodes join the cluster. Defaults to0
. gateway.expected_data_nodes
-
(Static) Number of data nodes expected in the cluster. Recovery of local shards begins when the expected number of data nodes join the cluster. Defaults to
0
. gateway.recover_after_time
-
(Static) If the expected number of nodes is not achieved, the recovery process waits for the configured amount of time before trying to recover. Defaults to
5m
if one of theexpected_nodes
settings is configured.Once the
recover_after_time
duration has timed out, recovery will start as long as the following conditions are met: gateway.recover_after_nodes
-
(Static) Deprecated. Use
gateway.recover_after_data_nodes
instead. Recover as long as this many data or master nodes have joined the cluster. gateway.recover_after_master_nodes
-
(Static) Deprecated. Use
gateway.recover_after_data_nodes
instead. Recover as long as this many master nodes have joined the cluster. gateway.recover_after_data_nodes
-
(Static) Recover as long as this many data nodes have joined the cluster.
Dangling indices
When a node joins the cluster, if it finds any shards stored in its local data
directory that do not already exist in the cluster, it will consider those
shards to be "dangling". Importing dangling indices
into the cluster using gateway.auto_import_dangling_indices
is not safe.
Instead, use the Dangling indices API. Neither
mechanism provides any guarantees as to whether the imported data truly
represents the latest state of the data when the index was still part of
the cluster.
gateway.auto_import_dangling_indices
-
Deprecated. Use the dedicated dangling indices API instead. Whether to automatically import dangling indices into the cluster state, provided no indices already exist with the same name. Defaults to
false
.
|
Warning
|
The auto-import functionality was intended as a best effort to help users who lose all master nodes. For example, if a new master node were to be started which was unaware of the other indices in the cluster, adding the old nodes would cause the old indices to be imported, instead of being deleted. However there are several issues with automatic importing, and its use is strongly discouraged in favour of the <<dangling-indices-api,dedicated API>. |
|
Warning
|
Losing all master nodes is a situation that should be avoided at all costs, as it puts your cluster’s metadata and data at risk. |
Logging
You can use OpenSearch’s application logs to monitor your cluster and diagnose issues. If you run OpenSearch as a service, the default location of the logs varies based on your platform and installation method:
On Docker, log messages go to the console and are handled by the
configured Docker logging driver. To access logs, run docker logs
.
For Debian installations, OpenSearch writes logs to /var/log/opensearch
.
For RPM installations, OpenSearch writes logs to /var/log/opensearch
.
For macOS .tar.gz
installations, OpenSearch writes logs to
$OPENSEARCH_HOME/logs
.
Files in $OPENSEARCH_HOME
risk deletion during an upgrade. In production, we strongly
recommend you set path.logs
to a location outside of $OPENSEARCH_HOME
.
See Path settings.
For macOS Homebrew installations, OpenSearch writes logs to
/usr/local/var/log/opensearch
.
For Linux .tar.gz
installations, OpenSearch writes logs to
$OPENSEARCH_HOME/logs
.
Files in $OPENSEARCH_HOME
risk deletion during an upgrade. In production, we strongly
recommend you set path.logs
to a location outside of $OPENSEARCH_HOME
.
See Path settings.
For Windows .zip
installations, OpenSearch writes logs to
%OPENSEARCH_HOME%\logs
.
Files in %OPENSEARCH_HOME%
risk deletion during an upgrade. In production, we strongly
recommend you set path.logs
to a location outside of %OPENSEARCH_HOME%`
.
See Path settings.
If you run OpenSearch from the command line, OpenSearch prints logs to the standard output
(stdout
).
Logging configuration
OpenSearch uses Log4j 2 for
logging. Log4j 2 can be configured using the log4j2.properties
file. OpenSearch exposes three properties, ${sys:opensearch.logs.base_path}
,
${sys:opensearch.logs.cluster_name}
, and ${sys:opensearch.logs.node_name}
that can be
referenced in the configuration file to determine the location of the log
files. The property ${sys:opensearch.logs.base_path}
will resolve to the log directory,
${sys:opensearch.logs.cluster_name}
will resolve to the cluster name (used as the
prefix of log filenames in the default configuration), and
${sys:opensearch.logs.node_name}
will resolve to the node name (if the node name is
explicitly set).
For example, if your log directory (path.logs
) is /var/log/opensearch
and
your cluster is named production
then ${sys:opensearch.logs.base_path}
will resolve
to /var/log/opensearch
and
${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}.log
will resolve to /var/log/opensearch/production.log
.
######## Server JSON ############################
appender.rolling.type = RollingFile (1)
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_server.json (2)
appender.rolling.layout.type = OpenSearchJsonLayout (3)
appender.rolling.layout.type_name = server (4)
appender.rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.json.gz (5)
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy (6)
appender.rolling.policies.time.interval = 1 (7)
appender.rolling.policies.time.modulate = true (8)
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy (9)
appender.rolling.policies.size.size = 256MB (10)
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.fileIndex = nomax
appender.rolling.strategy.action.type = Delete (11)
appender.rolling.strategy.action.basepath = ${sys:opensearch.logs.base_path}
appender.rolling.strategy.action.condition.type = IfFileName (12)
appender.rolling.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-* (13)
appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize (14)
appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB (15)
################################################
-
Configure the
RollingFile
appender -
Log to
/var/log/opensearch/production_server.json
-
Use JSON layout.
-
type_name
is a flag populating thetype
field in aOpenSearchJsonLayout
. It can be used to distinguish different types of logs more easily when parsing them. -
Roll logs to
/var/log/opensearch/production-yyyy-MM-dd-i.json
; logs will be compressed on each roll andi
will be incremented -
Use a time-based roll policy
-
Roll logs on a daily basis
-
Align rolls on the day boundary (as opposed to rolling every twenty-four hours)
-
Using a size-based roll policy
-
Roll logs after 256 MB
-
Use a delete action when rolling logs
-
Only delete logs matching a file pattern
-
The pattern is to only delete the main logs
-
Only delete if we have accumulated too many compressed logs
-
The size condition on the compressed logs is 2 GB
######## Server - old style pattern ###########
appender.rolling_old.type = RollingFile
appender.rolling_old.name = rolling_old
appender.rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_server.log (1)
appender.rolling_old.layout.type = PatternLayout
appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
appender.rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.old_log.gz
-
The configuration for
old style
pattern appenders. These logs will be saved in.log
files and if archived will be in.log.gz
files. Note that these should be considered deprecated and will be removed in the future.
|
Note
|
Log4j’s configuration parsing gets confused by any extraneous whitespace; if you copy and paste any Log4j settings on this page, or enter any Log4j configuration in general, be sure to trim any leading and trailing whitespace. |
Note than you can replace .gz
by .zip
in appender.rolling.filePattern
to
compress the rolled logs using the zip format. If you remove the .gz
extension then logs will not be compressed as they are rolled.
If you want to retain log files for a specified period of time, you can use a rollover strategy with a delete action.
appender.rolling.strategy.type = DefaultRolloverStrategy (1)
appender.rolling.strategy.action.type = Delete (2)
appender.rolling.strategy.action.basepath = ${sys:opensearch.logs.base_path} (3)
appender.rolling.strategy.action.condition.type = IfFileName (4)
appender.rolling.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-* (5)
appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified (6)
appender.rolling.strategy.action.condition.nested_condition.age = 7D (7)
-
Configure the
DefaultRolloverStrategy
-
Configure the
Delete
action for handling rollovers -
The base path to the OpenSearch logs
-
The condition to apply when handling rollovers
-
Delete files from the base path matching the glob
${sys:opensearch.logs.cluster_name}-*
; this is the glob that log files are rolled to; this is needed to only delete the rolled OpenSearch logs but not also delete the deprecation and slow logs -
A nested condition to apply to files matching the glob
-
Retain logs for seven days
Multiple configuration files can be loaded (in which case they will get merged)
as long as they are named log4j2.properties
and have the OpenSearch config
directory as an ancestor; this is useful for plugins that expose additional
loggers. The logger section contains the java packages and their corresponding
log level. The appender section contains the destinations for the logs.
Extensive information on how to customize logging and all the supported
appenders can be found on the
Log4j
documentation.
Configuring logging levels
Each Java package in the OpenSearch source code has a related logger. For
example, the org.opensearch.discovery
package has
logger.org.opensearch.discovery
for logs related to the
discovery process.
To get more or less verbose logs, use the cluster
update settings API to change the related logger’s log level. Each logger
accepts Log4j 2’s built-in log levels, from least to most verbose: OFF
,
FATAL
, ERROR
, WARN
, INFO
, DEBUG
, and TRACE
. The default log level is
INFO
. Messages logged at higher verbosity levels (DEBUG
and TRACE
) are
only intended for expert use.
PUT /_cluster/settings
{
"transient": {
"logger.org.opensearch.discovery": "DEBUG"
}
}
Other ways to change log levels include:
-
opensearch.yml
:logger.org.opensearch.discovery: DEBUG
This is most appropriate when debugging a problem on a single node.
-
log4j2.properties
:logger.discovery.name = org.opensearch.discovery logger.discovery.level = debug
This is most appropriate when you already need to change your Log4j 2 configuration for other reasons. For example, you may want to send logs for a particular logger to another file. However, these use cases are rare.
Deprecation logging
In addition to regular logging, OpenSearch allows you to enable logging of deprecated actions. For example this allows you to determine early, if you need to migrate certain functionality in the future. By default, deprecation logging is enabled at the WARN level, the level at which all deprecation log messages will be emitted.
logger.deprecation.level = warn
This will create a daily rolling deprecation log file in your log directory. Check this file regularly, especially when you intend to upgrade to a new major version.
The default logging configuration has set the roll policy for the deprecation logs to roll and compress after 1 GB, and to preserve a maximum of five log files (four rolled logs, and the active log).
You can disable it in the config/log4j2.properties
file by setting the deprecation
log level to error
like this:
logger.deprecation.name = org.opensearch.deprecation
logger.deprecation.level = error
You can identify what is triggering deprecated functionality if X-Opaque-Id
was used as an HTTP header.
The user ID is included in the X-Opaque-ID
field in deprecation JSON logs.
{
"type": "deprecation",
"timestamp": "2019-08-30T12:07:07,126+02:00",
"level": "WARN",
"component": "o.e.d.r.a.a.i.RestCreateIndexAction",
"cluster.name": "distribution_run",
"node.name": "node-0",
"message": "[types removal] Using include_type_name in create index requests is deprecated. The parameter will be removed in the next major version.",
"x-opaque-id": "MY_USER_ID",
"cluster.uuid": "Aq-c-PAeQiK3tfBYtig9Bw",
"node.id": "D7fUYfnfTLa2D7y-xw6tZg"
}
JSON log format
To make parsing OpenSearch logs easier, logs are now printed in a JSON format.
This is configured by a Log4J layout property appender.rolling.layout.type = OpenSearchJsonLayout
.
This layout requires a type_name
attribute to be set which is used to distinguish
logs streams when parsing.
appender.rolling.layout.type = OpenSearchJsonLayoutLayout
appender.rolling.layout.type_name = server
Each line contains a single JSON document with the properties configured in OpenSearchJsonLayout
.
See this class javadoc for more details.
However if a JSON document contains an exception, it will be printed over multiple lines.
The first line will contain regular properties and subsequent lines will contain the
stacktrace formatted as a JSON array.
|
Note
|
You can still use your own custom layout. To do that replace the line
appender.rolling.layout.type with a different layout. See sample below:
|
appender.rolling.type = RollingFile
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_server.log
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %.-10000m%n
appender.rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
Node
Any time that you start an instance of OpenSearch, you are starting a node. A collection of connected nodes is called a cluster. If you are running a single node of OpenSearch, then you have a cluster of one node.
Every node in the cluster can handle HTTP and Transport traffic by default. The transport layer is used exclusively for communication between nodes; the HTTP layer is used by REST clients.
All nodes know about all the other nodes in the cluster and can forward client requests to the appropriate node.
By default, a node is all of the following types: master-eligible, data, ingest, and (if available) machine learning. All data nodes are also transform nodes.
Node roles
You can define the roles of a node by setting node.roles
. If you don’t
configure this setting, then the node has the following roles by default:
-
master
-
data
-
ingest
-
remote_cluster_client
|
Note
|
If you set node.roles , the node is assigned only the roles you specify.
|
- Master-eligible node
-
A node that has the
master
role (default), which makes it eligible to be elected as the master node, which controls the cluster. - Data node
-
A node that has the
data
role (default). Data nodes hold data and perform data related operations such as CRUD, search, and aggregations. A node with thedata
role can fill any of the specialised data node roles. - Ingest node
-
A node that has the
ingest
role (default). Ingest nodes are able to apply an ingest pipeline to a document in order to transform and enrich the document before indexing. With a heavy ingest load, it makes sense to use dedicated ingest nodes and to not include theingest
role from nodes that have themaster
ordata
roles. - Remote-eligible node
-
A node that has the
remote_cluster_client
role (default), which makes it eligible to act as a remote client. By default, any node in the cluster can act as a cross-cluster client and connect to remote clusters.
|
Note
|
Coordinating node
Requests like search requests or bulk-indexing requests may involve data held on different data nodes. A search request, for example, is executed in two phases which are coordinated by the node which receives the client request — the coordinating node. In the scatter phase, the coordinating node forwards the request to the data nodes which hold the data. Each data node executes the request locally and returns its results to the coordinating node. In the gather phase, the coordinating node reduces each data node’s results into a single global result set. Every node is implicitly a coordinating node. This means that a node that has
an explicit empty list of roles via |
Master-eligible node
The master node is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes. It is important for cluster health to have a stable master node.
|
Important
|
Master nodes must have access to the data/ directory (just like
data nodes) as this is where the cluster state is persisted between node
restarts.
|
Dedicated master-eligible node
It is important for the health of the cluster that the elected master node has
the resources it needs to fulfill its responsibilities. If the elected master
node is overloaded with other tasks then the cluster may not operate well. In
particular, indexing and searching your data can be very resource-intensive, so
in large or high-throughput clusters it is a good idea to avoid using the
master-eligible nodes for tasks such as indexing and searching. You can do this
by configuring three of your nodes to be dedicated master-eligible nodes.
Dedicated master-eligible nodes only have the master
role, allowing them to
focus on managing the cluster. While master nodes can also behave as
coordinating nodes and route search and indexing requests
from clients to data nodes, it is better not to use dedicated master nodes for
this purpose.
To create a dedicated master-eligible node, set:
node.roles: [ master ]
Data node
Data nodes hold the shards that contain the documents you have indexed. Data nodes handle data related operations like CRUD, search, and aggregations. These operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more data nodes if they are overloaded.
The main benefit of having dedicated data nodes is the separation of the master and data roles.
To create a dedicated data node, set:
node.roles: [ data ]
Ingest node
Ingest nodes can execute pre-processing pipelines, composed of one or more ingest processors. Depending on the type of operations performed by the ingest processors and the required resources, it may make sense to have dedicated ingest nodes, that will only perform this specific task.
To create a dedicated ingest node, set:
node.roles: [ ingest ]
Coordinating only node
If you take away the ability to be able to handle master duties, to hold data, and pre-process documents, then you are left with a coordinating node that can only route requests, handle the search reduce phase, and distribute bulk indexing. Essentially, coordinating only nodes behave as smart load balancers.
Coordinating only nodes can benefit large clusters by offloading the coordinating node role from data and master-eligible nodes. They join the cluster and receive the full cluster state, like every other node, and they use the cluster state to route requests directly to the appropriate place(s).
|
Warning
|
Adding too many coordinating only nodes to a cluster can increase the burden on the entire cluster because the elected master node must await acknowledgement of cluster state updates from every node! The benefit of coordinating only nodes should not be overstated — data nodes can happily serve the same purpose. |
To create a dedicated coordinating node, set:
node.roles: [ ]
Remote-eligible node
By default, any node in a cluster can act as a cross-cluster client and connect to remote clusters. Once connected, you can search remote clusters using cross-cluster search.
node.roles: [ remote_cluster_client ]
Changing the role of a node
Each data node maintains the following data on disk:
-
the shard data for every shard allocated to that node,
-
the index metadata corresponding with every shard allocated to that node, and
-
the cluster-wide metadata, such as settings and index templates.
Similarly, each master-eligible node maintains the following data on disk:
-
the index metadata for every index in the cluster, and
-
the cluster-wide metadata, such as settings and index templates.
Each node checks the contents of its data path at startup. If it discovers
unexpected data then it will refuse to start. This is to avoid importing
unwanted dangling indices which can lead
to a red cluster health. To be more precise, nodes without the data
role will
refuse to start if they find any shard data on disk at startup, and nodes
without both the master
and data
roles will refuse to start if they have any
index metadata on disk at startup.
It is possible to change the roles of a node by adjusting its
opensearch.yml
file and restarting it. This is known as repurposing a
node. In order to satisfy the checks for unexpected data described above, you
must perform some extra steps to prepare a node for repurposing when starting
the node without the data
or master
roles.
-
If you want to repurpose a data node by removing the
data
role then you should first use an allocation filter to safely migrate all the shard data onto other nodes in the cluster. -
If you want to repurpose a node to have neither the
data
normaster
roles then it is simplest to start a brand-new node with an empty data path and the desired roles. You may find it safest to use an allocation filter to migrate the shard data elsewhere in the cluster first.
If it is not possible to follow these extra steps then you may be able to use
the opensearch-node repurpose
tool to delete any
excess data that prevents a node from starting.
Node data path settings
path.data
Every data and master-eligible node requires access to a data directory where
shards and index and cluster metadata will be stored. The path.data
defaults
to $OPENSEARCH_HOME/data
but can be configured in the opensearch.yml
config
file an absolute path or a path relative to $OPENSEARCH_HOME
as follows:
path.data: /var/opensearch/data
Like all node settings, it can also be specified on the command line as:
./bin/opensearch -Epath.data=/var/opensearch/data
|
Tip
|
When using the .zip or .tar.gz distributions, the path.data setting
should be configured to locate the data directory outside the OpenSearch home
directory, so that the home directory can be deleted without deleting your data!
The RPM and Debian distributions do this for you already.
|
node.max_local_storage_nodes
The data path can be shared by multiple nodes, even by nodes from different clusters. It is recommended however to only run one node of OpenSearch using the same data path. This setting is deprecated.
By default, OpenSearch is configured to prevent more than one node from sharing the
same data path. To allow for more than one node (e.g., on your development
machine), use the setting node.max_local_storage_nodes
and set this to a
positive integer larger than one.
|
Warning
|
Never run different node types (i.e. master, data) from the same data directory. This can lead to unexpected data loss. |
Other node settings
More node settings can be found in Configuring OpenSearch and Important OpenSearch configuration, including:
Network settings
OpenSearch binds to localhost only by default. This is sufficient for you to run a local development server (or even a development cluster, if you start multiple nodes on the same machine), but you will need to configure some basic network settings in order to run a real production cluster across multiple servers.
|
Warning
|
Be careful with the network configuration!
Never expose an unprotected node to the public internet. |
Commonly used network settings
network.host
-
(Static) The node will bind to this hostname or IP address and publish (advertise) this host to other nodes in the cluster. Accepts an IP address, hostname, a special value, or an array of any combination of these. Note that any values containing a
:
(e.g., an IPv6 address or containing one of the special values) must be quoted because:
is a special character in YAML.<IP_ADDRESS>
is an acceptable IP address and will bind to all network interfaces. The value0
has the same effect as the value<IP_ADDRESS>
.Defaults to
local
. discovery.seed_hosts
-
(Static) In order to join a cluster, a node needs to know the hostname or IP address of at least some of the other nodes in the cluster. This setting provides the initial list of addresses this node will try to contact. Accepts IP addresses or hostnames. If a hostname lookup resolves to multiple IP addresses then each IP address will be used for discovery. Round robin DNS — returning a different IP from a list on each lookup — can be used for discovery; non- existent IP addresses will throw exceptions and cause another DNS lookup on the next round of pinging (subject to JVM DNS caching).
Defaults to
["<IP_ADDRESS>", "[::1]"]
. http.port
-
(Static) Port to bind to for incoming HTTP requests. Accepts a single value or a range. If a range is specified, the node will bind to the first available port in the range.
Defaults to
9200-9300
. transport.port
-
(Static) Port to bind for communication between nodes. Accepts a single value or a range. If a range is specified, the node will bind to the first available port in the range.
Defaults to
9300-9400
.
Special values for network.host
The following special values may be passed to network.host
:
local
-
Any loopback addresses on the system, for example
<IP_ADDRESS>
. site
-
Any site-local addresses on the system, for example
<IP_ADDRESS>
. global
-
Any globally-scoped addresses on the system, for example
<IP_ADDRESS>
. [networkInterface]
-
Use the addresses of the network interface called
[networkInterface]
. For example if you wish to use the addresses of an interface calleden0
then setnetwork.host: en0
.
IPv4 vs IPv6
These special values will work over both IPv4 and IPv6 by default, but you can
also limit this with the use of :ipv4
of :ipv6
specifiers. For example,
en0:ipv4
would only bind to the IPv4 addresses of interface en0
.
|
Tip
|
Discovery in the cloud
More special settings are available when running in the cloud with either the EC2 discovery plugin or the Google Compute Engine discovery plugin installed. |
Advanced network settings
The network.host
setting explained in Commonly used network settings
is a shortcut which sets the bind host and the publish host at the same
time. In advanced used cases, such as when running behind a proxy server, you
may need to set these settings to different values:
network.bind_host
-
This specifies which network interface(s) a node should bind to in order to listen for incoming requests. A node can bind to multiple interfaces, e.g. two network cards, or a site-local address and a local address. Defaults to
network.host
. network.publish_host
-
The publish host is the single interface that the node advertises to other nodes in the cluster, so that those nodes can connect to it. Currently an OpenSearch node may be bound to multiple addresses, but only publishes one. If not specified, this defaults to the
`best'' address from `network.host
, sorted by IPv4/IPv6 stack preference, then by reachability. If you set anetwork.host
that results in multiple bind addresses yet rely on a specific address for node-to-node communication, you should explicitly setnetwork.publish_host
.
Both of the above settings can be configured just like network.host
— they
accept IP addresses, host names, and
special values.
Advanced TCP settings
network.tcp.no_delay
-
(Static) Enable or disable the TCP no delay setting. Defaults to
true
. network.tcp.keep_alive
-
(Static) Configures the
SO_KEEPALIVE
option for this socket, which determines whether it sends TCP keepalive probes. network.tcp.keep_idle
-
(Static) Configures the
TCP_KEEPIDLE
option for this socket, which determines the time in seconds that a connection must be idle before starting to send TCP keepalive probes. Defaults to-1
, which uses the system default. This value cannot exceed300
seconds. Only applicable on Linux and macOS, and requires Java 11 or newer. network.tcp.keep_interval
-
(Static) Configures the
TCP_KEEPINTVL
option for this socket, which determines the time in seconds between sending TCP keepalive probes. Defaults to-1
, which uses the system default. This value cannot exceed300
seconds. Only applicable on Linux and macOS, and requires Java 11 or newer. network.tcp.keep_count
-
(Static) Configures the
TCP_KEEPCNT
option for this socket, which determines the number of unacknowledged TCP keepalive probes that may be sent on a connection before it is dropped. Defaults to-1
, which uses the system default. Only applicable on Linux and macOS, and requires Java 11 or newer. network.tcp.reuse_address
-
(Static) Should an address be reused or not. Defaults to
true
on non-windows machines. network.tcp.send_buffer_size
-
(Static) The size of the TCP send buffer (specified with size units). By default not explicitly set.
network.tcp.receive_buffer_size
-
(Static) The size of the TCP receive buffer (specified with size units). By default not explicitly set.
HTTP and transport network communication
Each OpenSearch node uses the network for two different methods of communication:
-
it exposes an HTTP interface for use by clients.
-
it exposes a transport interface for communication between nodes within a cluster, for communication with a remote cluster, and for the Java Transport client to communicate with a cluster.
Node query cache settings
The results of queries used in the filter context are cached in the node query cache for fast lookup. There is one queries cache per node that is shared by all shards. The cache uses an LRU eviction policy: when the cache is full, the least recently used query results are evicted to make way for new data. You cannot inspect the contents of the query cache.
Term queries and queries used outside of a filter context are not eligible for caching.
By default, the cache holds a maximum of 10000 queries in up to 10% of the total heap space. To determine if a query is eligible for caching, OpenSearch maintains a query history to track occurrences.
Caching is done on a per segment basis if a segment contains at least 10000 documents and the segment has at least 3% of the total documents of a shard. Because caching is per segment, merging segments can invalidate cached queries.
The following setting is static and must be configured on every data node in the cluster:
indices.queries.cache.size
-
(Static) Controls the memory size for the filter cache. Accepts either a percentage value, like
5%
, or an exact value, like512mb
. Defaults to10%
.
Query cache index settings
The following setting is an index setting that can be configured on a per-index basis. Can only be set at index creation time or on a closed index:
index.queries.cache.enabled
-
(Static) Controls whether to enable query caching. Accepts
true
(default) orfalse
.
Search settings
The following expert settings can be set to manage global search and aggregation limits.
indices.query.bool.max_clause_count
-
(Static, integer) Maximum number of clauses a Lucene BooleanQuery can contain. Defaults to
1024
.This setting limits the number of clauses a Lucene BooleanQuery can have. The default of 1024 is quite high and should normally be sufficient. This limit does not only affect OpenSearch’s
bool
query, but many other queries are rewritten to Lucene’s BooleanQuery internally. The limit is in place to prevent searches from becoming too large and taking up too much CPU and memory. In case you’re considering increasing this setting, make sure you’ve exhausted all other options to avoid having to do this. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources.
search.max_buckets
-
(Dynamic, integer) Maximum number of aggregation buckets allowed in a single response. Defaults to 65,535.
Requests that attempt to return more than this limit will return an error.
Shard request cache settings
When a search request is run against an index or against many indices, each involved shard executes the search locally and returns its local results to the coordinating node, which combines these shard-level results into a ``global'' result set.
The shard-level request cache module caches the local results on each shard. This allows frequently used (and potentially heavy) search requests to return results almost instantly. The requests cache is a very good fit for the logging use case, where only the most recent index is being actively updated — results from older indices will be served directly from the cache.
|
Important
|
By default, the requests cache will only cache the results of search requests
where Most queries that use Scripted queries that use the API calls which are non-deterministic, such as
|
Cache invalidation
The cache is smart — it keeps the same near real-time promise as uncached search.
Cached results are invalidated automatically whenever the shard refreshes, but only if the data in the shard has actually changed. In other words, you will always get the same results from the cache as you would for an uncached search request.
The longer the refresh interval, the longer that cached entries will remain valid. If the cache is full, the least recently used cache keys will be evicted.
The cache can be expired manually with the clear-cache
API:
POST /my-index-000001,my-index-000002/_cache/clear?request=true
Enabling and disabling caching
The cache is enabled by default, but can be disabled when creating a new index as follows:
PUT /my-index-000001
{
"settings": {
"index.requests.cache.enable": false
}
}
It can also be enabled or disabled dynamically on an existing index with the
update-settings
API:
PUT /my-index-000001/_settings
{ "index.requests.cache.enable": true }
Enabling and disabling caching per request
The request_cache
query-string parameter can be used to enable or disable
caching on a per-request basis. If set, it overrides the index-level setting:
GET /my-index-000001/_search?request_cache=true
{
"size": 0,
"aggs": {
"popular_colors": {
"terms": {
"field": "colors"
}
}
}
}
Requests where size
is greater than 0 will not be cached even if the request cache is
enabled in the index settings. To cache these requests you will need to use the
query-string parameter detailed here.
Cache key
The whole JSON body is used as the cache key. This means that if the JSON changes — for instance if keys are output in a different order — then the cache key will not be recognised.
|
Tip
|
Most JSON libraries support a canonical mode which ensures that JSON keys are always emitted in the same order. This canonical mode can be used in the application to ensure that a request is always serialized in the same way. |
Cache settings
The cache is managed at the node level, and has a default maximum size of 1%
of the heap. This can be changed in the config/opensearch.yml
file with:
indices.requests.cache.size: 2%
Also, you can use the indices.requests.cache.expire setting to specify a TTL for cached results, but there should be no reason to do so. Remember that stale results are automatically invalidated when the index is refreshed. This setting is provided for completeness' sake only.
Monitoring cache usage
The size of the cache (in bytes) and the number of evictions can be viewed
by index, with the indices-stats
API:
GET /_stats/request_cache?human
or by node with the nodes-stats
API:
GET /_nodes/stats/indices/request_cache?human
Transport
REST clients send requests to your OpenSearch cluster over HTTP, but
the node that receives a client request cannot always handle it alone and must
normally pass it on to other nodes for further processing. It does this using
the transport networking layer. The transport layer is used for all internal
communication between nodes within a cluster, all communication with the nodes
of a remote cluster, and also by the
TransportClient
in the OpenSearch Java API.
Transport settings
The following settings can be configured for the internal transport that communicates over TCP. These settings also use the common network settings.
transport.port
-
(Static) A bind port range. Defaults to
9300-9400
. transport.publish_port
-
(Static) The port that other nodes in the cluster should use when communicating with this node. Useful when a cluster node is behind a proxy or firewall and the
transport.port
is not directly addressable from the outside. Defaults to the actual port assigned viatransport.port
. transport.bind_host
-
(Static) The host address to bind the transport service to. Defaults to
transport.host
(if set) ornetwork.bind_host
. transport.publish_host
-
(Static) The host address to publish for nodes in the cluster to connect to. Defaults to
transport.host
(if set) ornetwork.publish_host
. transport.host
-
(Static) Used to set the
transport.bind_host
and thetransport.publish_host
. transport.connect_timeout
-
(Static) The connect timeout for initiating a new connection (in time setting format). Defaults to
30s
. transport.compress
-
(Static) Set to
true
to enable compression (DEFLATE
) between all nodes. Defaults tofalse
. transport.ping_schedule
-
(Static) Schedule a regular application-level ping message to ensure that transport connections between nodes are kept alive. Defaults to
5s
in the transport client and-1
(disabled) elsewhere. It is preferable to correctly configure TCP keep-alives instead of using this feature, because TCP keep-alives apply to all kinds of long-lived connections and not just to transport connections. transport.tcp.no_delay
-
(Static) Enable or disable the TCP no delay setting. Defaults to
network.tcp.no_delay
. transport.tcp.keep_alive
-
(Static) Configures the
SO_KEEPALIVE
option for this socket, which determines whether it sends TCP keepalive probes. Defaults tonetwork.tcp.keep_alive
. transport.tcp.keep_idle
-
(Static) Configures the
TCP_KEEPIDLE
option for this socket, which determines the time in seconds that a connection must be idle before starting to send TCP keepalive probes. Defaults tonetwork.tcp.keep_idle
if set, or the system default otherwise. This value cannot exceed300
seconds. In cases where the system default is higher than300
, the value is automatically lowered to300
. Only applicable on Linux and macOS, and requires Java 11 or newer. transport.tcp.keep_interval
-
(Static) Configures the
TCP_KEEPINTVL
option for this socket, which determines the time in seconds between sending TCP keepalive probes. Defaults tonetwork.tcp.keep_interval
if set, or the system default otherwise. This value cannot exceed300
seconds. In cases where the system default is higher than300
, the value is automatically lowered to300
. Only applicable on Linux and macOS, and requires Java 11 or newer. transport.tcp.keep_count
-
(Static) Configures the
TCP_KEEPCNT
option for this socket, which determines the number of unacknowledged TCP keepalive probes that may be sent on a connection before it is dropped. Defaults tonetwork.tcp.keep_count
if set, or the system default otherwise. Only applicable on Linux and macOS, and requires Java 11 or newer. transport.tcp.reuse_address
-
(Static) Should an address be reused or not. Defaults to
network.tcp.reuse_address
. transport.tcp.send_buffer_size
-
(Static) The size of the TCP send buffer (specified with size units). Defaults to
network.tcp.send_buffer_size
. transport.tcp.receive_buffer_size
-
(Static) The size of the TCP receive buffer (specified with size units). Defaults to
network.tcp.receive_buffer_size
.
Transport profiles
OpenSearch allows you to bind to multiple ports on different interfaces by the use of transport profiles. See this example configuration
transport.profiles.default.port: 9300-9400
transport.profiles.default.bind_host: <IP_ADDRESS>
transport.profiles.client.port: 9500-9600
transport.profiles.client.bind_host: <IP_ADDRESS>
transport.profiles.dmz.port: 9700-9800
transport.profiles.dmz.bind_host: <IP_ADDRESS>
The default
profile is special. It is used as a fallback for any other
profiles, if those do not have a specific configuration setting set, and is how
this node connects to other nodes in the cluster.
The following parameters can be configured on each transport profile, as in the example above:
-
port
: The port to which to bind. -
bind_host
: The host to which to bind. -
publish_host
: The host which is published in informational APIs.
Profiles also support all the other transport settings specified in the
transport settings section, and use these as defaults.
For example, transport.profiles.client.tcp.reuse_address
can be explicitly
configured, and defaults otherwise to transport.tcp.reuse_address
.
Long-lived idle connections
A transport connection between two nodes is made up of a number of long-lived
TCP connections, some of which may be idle for an extended period of time.
Nonetheless, OpenSearch requires these connections to remain open, and it
can disrupt the operation of your cluster if any inter-node connections are
closed by an external influence such as a firewall. It is important to
configure your network to preserve long-lived idle connections between
OpenSearch nodes, for instance by leaving tcp.keep_alive
enabled and
ensuring that the keepalive interval is shorter than any timeout that might
cause idle connections to be closed, or by setting transport.ping_schedule
if
keepalives cannot be configured. Devices which drop connections when they reach
a certain age are a common source of problems to OpenSearch clusters, and
must not be used.
Request compression
By default, the transport.compress
setting is false
and network-level
request compression is disabled between nodes in the cluster. This default
normally makes sense for local cluster communication as compression has a
noticeable CPU cost and local clusters tend to be set up with fast network
connections between nodes.
The transport.compress
setting always configures local cluster request
compression and is the fallback setting for remote cluster request compression.
If you want to configure remote request compression differently than local
request compression, you can set it on a per-remote cluster basis using the
cluster.remote.${cluster_alias}.transport.compress
setting.
Response compression
The compression settings do not configure compression for responses. OpenSearch will compress a response if the inbound request was compressed—even when compression is not enabled. Similarly, OpenSearch will not compress a response if the inbound request was uncompressed—even when compression is enabled.
Transport tracer
The transport layer has a dedicated tracer logger which, when activated, logs incoming and out going requests. The log can be dynamically activated
by setting the level of the org.opensearch.transport.TransportService.tracer
logger to TRACE
:
PUT _cluster/settings
{
"transient" : {
"logger.org.opensearch.transport.TransportService.tracer" : "TRACE"
}
}
You can also control which actions will be traced, using a set of include and exclude wildcard patterns. By default every request will be traced except for fault detection pings:
PUT _cluster/settings
{
"transient" : {
"transport.tracer.include" : "*",
"transport.tracer.exclude" : "internal:coordination/fault_detection/*"
}
}
Thread pools
A node uses several thread pools to manage memory consumption. Queues associated with many of the thread pools enable pending requests to be held instead of discarded.
There are several thread pools, but the important ones include:
generic
-
For generic operations (for example, background node discovery). Thread pool type is
scaling
.
search
-
For count/search/suggest operations. Thread pool type is
fixed_auto_queue_size
with a size ofint((
# of allocated processors
` * 3) / 2) + 1`, and initial queue_size of1000
. search_throttled
-
For count/search/suggest/get operations on
search_throttled indices
. Thread pool type isfixed_auto_queue_size
with a size of1
, and initial queue_size of100
. get
-
For get operations. Thread pool type is
fixed
with a size of# of allocated processors
, queue_size of1000
. analyze
-
For analyze requests. Thread pool type is
fixed
with a size of1
, queue size of16
. write
-
For single-document index/delete/update and bulk requests. Thread pool type is
fixed
with a size of# of allocated processors
, queue_size of10000
. The maximum size for this pool is1 +
# of allocated processors
. snapshot
-
For snapshot/restore operations. Thread pool type is
scaling
with a keep-alive of5m
and a max ofmin(5, (
# of allocated processors
`) / 2)`. warmer
-
For segment warm-up operations. Thread pool type is
scaling
with a keep-alive of5m
and a max ofmin(5, (
# of allocated processors
`) / 2)`. refresh
-
For refresh operations. Thread pool type is
scaling
with a keep-alive of5m
and a max ofmin(10, (
# of allocated processors
`) / 2)`. listener
-
Mainly for java client executing of action when listener threaded is set to
true
. Thread pool type isscaling
with a default max ofmin(10, (
# of allocated processors
`) / 2)`. fetch_shard_started
-
For listing shard states. Thread pool type is
scaling
with keep-alive of5m
and a default maximum size of2 *
# of allocated processors
. fetch_shard_store
-
For listing shard stores. Thread pool type is
scaling
with keep-alive of5m
and a default maximum size of2 *
# of allocated processors
. flush
-
For flush, flush, and translog
fsync
operations. Thread pool type isscaling
with a keep-alive of5m
and a default maximum size ofmin(5, (
# of allocated processors
`) / 2)`. force_merge
-
For force merge operations. Thread pool type is
fixed
with a size of 1 and an unbounded queue size. management
-
For cluster management. Thread pool type is
scaling
with a keep-alive of5m
and a default maximum size of5
. system_read
-
For read operations on system indices. Thread pool type is
fixed
and a default maximum size ofmin(5, (
# of allocated processors
`) / 2)`. system_write
-
For write operations on system indices. Thread pool type is
fixed
and a default maximum size ofmin(5, (
# of allocated processors
`) / 2)`.
Changing a specific thread pool can be done by setting its type-specific
parameters; for example, changing the number of threads in the write
thread
pool:
thread_pool:
write:
size: 30
Thread pool types
The following are the types of thread pools and their respective parameters:
fixed
The fixed
thread pool holds a fixed size of threads to handle the
requests with a queue (optionally bounded) for pending requests that
have no threads to service them.
The size
parameter controls the number of threads.
The queue_size
allows to control the size of the queue of pending
requests that have no threads to execute them. By default, it is set to
-1
which means its unbounded. When a request comes in and the queue is
full, it will abort the request.
thread_pool:
write:
size: 30
queue_size: 1000
fixed_auto_queue_size
Deprecated. The fixed_auto_queue_size
thread pool holds a fixed size of threads to handle
the requests with a bounded queue for pending requests that have no threads to
service them. It’s similar to the fixed
threadpool, however, the queue_size
automatically adjusts according to calculations based on
Little’s Law. These calculations
will potentially adjust the queue_size
up or down by 50 every time
auto_queue_frame_size
operations have been completed.
The size
parameter controls the number of threads.
The queue_size
allows to control the initial size of the queue of pending
requests that have no threads to execute them.
The min_queue_size
setting controls the minimum amount the queue_size
can be
adjusted to.
The max_queue_size
setting controls the maximum amount the queue_size
can be
adjusted to.
The auto_queue_frame_size
setting controls the number of operations during
which measurement is taken before the queue is adjusted. It should be large
enough that a single operation cannot unduly bias the calculation.
The target_response_time
is a time value setting that indicates the targeted
average response time for tasks in the thread pool queue. If tasks are routinely
above this time, the thread pool queue will be adjusted down so that tasks are
rejected.
thread_pool:
search:
size: 30
queue_size: 500
min_queue_size: 10
max_queue_size: 1000
auto_queue_frame_size: 2000
target_response_time: 1s
scaling
The scaling
thread pool holds a dynamic number of threads. This
number is proportional to the workload and varies between the value of
the core
and max
parameters.
The keep_alive
parameter determines how long a thread should be kept
around in the thread pool without it doing any work.
thread_pool:
warmer:
core: 1
max: 8
keep_alive: 2m
Allocated processors setting
The number of processors is automatically detected, and the thread pool settings
are automatically set based on it. In some cases it can be useful to override
the number of detected processors. This can be done by explicitly setting the
node.processors
setting.
node.processors: 2
There are a few use-cases for explicitly overriding the node.processors
setting:
-
If you are running multiple instances of OpenSearch on the same host but want want OpenSearch to size its thread pools as if it only has a fraction of the CPU, you should override the
node.processors
setting to the desired fraction, for example, if you’re running two instances of OpenSearch on a 16-core machine, setnode.processors
to 8. Note that this is an expert-level use case and there’s a lot more involved than just setting thenode.processors
setting as there are other considerations like changing the number of garbage collector threads, pinning processes to cores, and so on. -
Sometimes the number of processors is wrongly detected and in such cases explicitly setting the
node.processors
setting will workaround such issues.
In order to check the number of processors detected, use the nodes info
API with the os
flag.