Correcting gramma

Correcting gramma

[[modules-scripting]]
== Scripting

The scripting module allows to use scripts in order to evaluate custom
expressions. For example, scripts can be used to return "script fields"
as part of a search request, or can be used to evaluate a custom score
for a query and so on.

The scripting module uses by default http://mvel.codehaus.org/[mvel] as
the scripting language with some extensions. mvel is used since it is
extremely fast and very simple to use, and in most cases, simple
expressions are needed (for example, mathematical equations).

Additional `lang` plugins are provided to allow to execute scripts in
different languages. Currently supported plugins are `lang-javascript`
for JavaScript, `lang-groovy` for Groovy, and `lang-python` for Python.
All places where a `script` parameter can be used, a `lang` parameter
(on the same level) can be provided to define the language of the
script. The `lang` options are `mvel`, `js`, `groovy`, `python`, and
`native`.

added[1.2.0, Dynamic scripting is disabled by default since version 1.2.0]

To increase security, Elasticsearch does not allow you to specify scripts with a
request. Instead, scripts must be placed in the `scripts` directory inside the
configuration directory (the directory where elasticsearch.yml is). Scripts
placed into this directory will automatically be picked up and be available to
be used. Once a script has been placed in this directory, it can be referenced
by name. For example, a script called `calculate-score.mvel` can be referenced
in a request like this:

[source,sh]
--------------------------------------------------
$ tree config
config
├── elasticsearch.yml
├── logging.yml
└── scripts
    └── calculate-score.mvel
--------------------------------------------------

[source,sh]
--------------------------------------------------
$ cat config/scripts/calculate-score.mvel
Math.log(_score * 2) + my_modifier
--------------------------------------------------

[source,js]
--------------------------------------------------
curl -XPOST localhost:9200/_search -d '{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "body": "foo"
        }
      },
      "functions": [
        {
          "script_score": {
            "script": "calculate-score",
            "params": {
              "my_modifier": 8
            }
          }
        }
      ]
    }
  }
}'
--------------------------------------------------

The name of the script is derived from the hierarchy of directories it
exists under, and the file name without the lang extension. For example,
a script placed under `config/scripts/group1/group2/test.py` will be
named `group1_group2_test`.

[float]
=== Default Scripting Language

The default scripting language (assuming no `lang` parameter is
provided) is `mvel`. In order to change it set the `script.default_lang`
to the appropriate language.

[float]
=== Enabling dynamic scripting

We recommend running Elasticsearch behind an application or proxy,
which protects Elasticsearch from the outside world. If users are
allowed to run dynamic scripts (even in a search request), then they
have the same access to your box as the user that Elasticsearch is
running as. For this reason dynamic scripting is disabled by default.

First, you should not run Elasticsearch as the `root` user, as this would allow
a script to access or do *anything* on your server, without limitations. Second,
you should not expose Elasticsearch directly to users, but instead have a proxy
application inbetween. If you *do* intend to expose Elasticsearch directly to
your users, then you have to decide whether you trust them enough to run scripts
on your box or not. If you do, you can enable dynamic scripting by adding the
following setting to the `config/elasticsearch.yml` file on every node:

[source,yaml]
-----------------------------------
script.disable_dynamic: false
-----------------------------------

While this still allows execution of named scripts provided in the config, or
_native_ Java scripts registered through plugins, it also allows users to run
arbitrary scripts via the API. Instead of sending the name of the file as the
script, the body of the script can be sent instead.

[float]
=== Automatic Script Reloading

The `config/scripts` directory is scanned periodically for changes.
New and changed scripts are reloaded and deleted script are removed
from preloaded scripts cache. The reload frequency can be specified
using `watcher.interval` setting, which defaults to `60s`.
To disable script reloading completely set `script.auto_reload_enabled`
to `false`.

[float]
=== Native (Java) Scripts

Even though `mvel` is pretty fast, this allows to register native Java based
scripts for faster execution.

In order to allow for scripts, the `NativeScriptFactory` needs to be
implemented that constructs the script that will be executed. There are
two main types, one that extends `AbstractExecutableScript` and one that
extends `AbstractSearchScript` (probably the one most users will extend,
with additional helper classes in `AbstractLongSearchScript`,
`AbstractDoubleSearchScript`, and `AbstractFloatSearchScript`).

Registering them can either be done by settings, for example:
`script.native.my.type` set to `sample.MyNativeScriptFactory` will
register a script named `my`. Another option is in a plugin, access
`ScriptModule` and call `registerScript` on it.

Executing the script is done by specifying the `lang` as `native`, and
the name of the script as the `script`.

Note, the scripts need to be in the classpath of elasticsearch. One
simple way to do it is to create a directory under plugins (choose a
descriptive name), and place the jar / classes files there, they will be
automatically loaded.

[float]
=== Score

In all scripts that can be used in facets, allow to access the current
doc score using `doc.score`.

[float]
=== Computing scores based on terms in scripts

see <<modules-advanced-scripting, advanced scripting documentation>>

[float]
=== Document Fields

Most scripting revolve around the use of specific document fields data.
The `doc['field_name']` can be used to access specific field data within
a document (the document in question is usually derived by the context
the script is used). Document fields are very fast to access since they
end up being loaded into memory (all the relevant field values/tokens
are loaded to memory).

The following data can be extracted from a field:

[cols="<,<",options="header",]
|=======================================================================
|Expression |Description
|`doc['field_name'].value` |The native value of the field. For example,
if its a short type, it will be short.

|`doc['field_name'].values` |The native array values of the field. For
example, if its a short type, it will be short[]. Remember, a field can
have several values within a single doc. Returns an empty array if the
field has no values.

|`doc['field_name'].empty` |A boolean indicating if the field has no
values within the doc.

|`doc['field_name'].multiValued` |A boolean indicating that the field
has several values within the corpus.

|`doc['field_name'].lat` |The latitude of a geo point type.

|`doc['field_name'].lon` |The longitude of a geo point type.

|`doc['field_name'].lats` |The latitudes of a geo point type.

|`doc['field_name'].lons` |The longitudes of a geo point type.

|`doc['field_name'].distance(lat, lon)` |The `plane` distance (in meters)
of this geo point field from the provided lat/lon.

|`doc['field_name'].distanceWithDefault(lat, lon, default)` |The `plane` distance (in meters)
of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].distanceInMiles(lat, lon)` |The `plane` distance (in
miles) of this geo point field from the provided lat/lon.

|`doc['field_name'].distanceInMilesWithDefault(lat, lon, default)` |The `plane` distance (in
miles) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].distanceInKm(lat, lon)` |The `plane` distance (in
km) of this geo point field from the provided lat/lon.

|`doc['field_name'].distanceInKmWithDefault(lat, lon, default)` |The `plane` distance (in
km) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].arcDistance(lat, lon)` |The `arc` distance (in
meters) of this geo point field from the provided lat/lon.

|`doc['field_name'].arcDistanceWithDefault(lat, lon, default)` |The `arc` distance (in
meters) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].arcDistanceInMiles(lat, lon)` |The `arc` distance (in
miles) of this geo point field from the provided lat/lon.

|`doc['field_name'].arcDistanceInMilesWithDefault(lat, lon, default)` |The `arc` distance (in
miles) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].arcDistanceInKm(lat, lon)` |The `arc` distance (in
km) of this geo point field from the provided lat/lon.

|`doc['field_name'].arcDistanceInKmWithDefault(lat, lon, default)` |The `arc` distance (in
km) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].factorDistance(lat, lon)` |The distance factor of this geo point field from the provided lat/lon.

|`doc['field_name'].factorDistance(lat, lon, default)` |The distance factor of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].geohashDistance(geohash)` |The `arc` distance (in meters)
of this geo point field from the provided geohash.

|`doc['field_name'].geohashDistanceInKm(geohash)` |The `arc` distance (in km)
of this geo point field from the provided geohash.

|`doc['field_name'].geohashDistanceInMiles(geohash)` |The `arc` distance (in
miles) of this geo point field from the provided geohash.
|=======================================================================

[float]
=== Stored Fields

Stored fields can also be accessed when executing a script. Note, they
are much slower to access compared with document fields, as they are not
loaded into memory. They can be simply accessed using
`_fields['my_field_name'].value` or `_fields['my_field_name'].values`.

[float]
=== Source Field

The source field can also be accessed when executing a script. The
source field is loaded per doc, parsed, and then provided to the script
for evaluation. The `_source` forms the context under which the source
field can be accessed, for example `_source.obj2.obj1.field3`.

Accessing `_source` is much slower compared to using `_doc`
but the data is not loaded into memory. For a single field access `_fields` may be
faster than using `_source` due to the extra overhead of potentially parsing large documents.
However, `_source` may be faster if you access multiple fields or if the source has already been
loaded for other purposes.

[float]
=== mvel Built In Functions

There are several built in functions that can be used within scripts.
They include:

[cols="<,<",options="header",]
|=======================================================================
|Function |Description
|`time()` |The current time in milliseconds.

|`sin(a)` |Returns the trigonometric sine of an angle.

|`cos(a)` |Returns the trigonometric cosine of an angle.

|`tan(a)` |Returns the trigonometric tangent of an angle.

|`asin(a)` |Returns the arc sine of a value.

|`acos(a)` |Returns the arc cosine of a value.

|`atan(a)` |Returns the arc tangent of a value.

|`toRadians(angdeg)` |Converts an angle measured in degrees to an
approximately equivalent angle measured in radians

|`toDegrees(angrad)` |Converts an angle measured in radians to an
approximately equivalent angle measured in degrees.

|`exp(a)` |Returns Euler's number _e_ raised to the power of value.

|`log(a)` |Returns the natural logarithm (base _e_) of a value.

|`log10(a)` |Returns the base 10 logarithm of a value.

|`sqrt(a)` |Returns the correctly rounded positive square root of a
value.

|`cbrt(a)` |Returns the cube root of a double value.

|`IEEEremainder(f1, f2)` |Computes the remainder operation on two
arguments as prescribed by the IEEE 754 standard.

|`ceil(a)` |Returns the smallest (closest to negative infinity) value
that is greater than or equal to the argument and is equal to a
mathematical integer.

|`floor(a)` |Returns the largest (closest to positive infinity) value
that is less than or equal to the argument and is equal to a
mathematical integer.

|`rint(a)` |Returns the value that is closest in value to the argument
and is equal to a mathematical integer.

|`atan2(y, x)` |Returns the angle _theta_ from the conversion of
rectangular coordinates (_x_, _y_) to polar coordinates (r,_theta_).

|`pow(a, b)` |Returns the value of the first argument raised to the
power of the second argument.

|`round(a)` |Returns the closest _int_ to the argument.

|`random()` |Returns a random _double_ value.

|`abs(a)` |Returns the absolute value of a value.

|`max(a, b)` |Returns the greater of two values.

|`min(a, b)` |Returns the smaller of two values.

|`ulp(d)` |Returns the size of an ulp of the argument.

|`signum(d)` |Returns the signum function of the argument.

|`sinh(x)` |Returns the hyperbolic sine of a value.

|`cosh(x)` |Returns the hyperbolic cosine of a value.

|`tanh(x)` |Returns the hyperbolic tangent of a value.

|`hypot(x, y)` |Returns sqrt(_x2_ + _y2_) without intermediate overflow
or underflow.
|=======================================================================

[float]
=== Arithmetic precision in MVEL

When dividing two numbers using MVEL based scripts, the engine tries to
be smart and adheres to the default behaviour of java. This means if you
divide two integers (you might have configured the fields as integer in
the mapping), the result will also be an integer. This means, if a
calculation like `1/num` is happening in your scripts and `num` is an
integer with the value of `8`, the result is `0` even though you were
expecting it to be `0.125`. You may need to enforce precision by
explicitly using a double like `1.0/num` in order to get the expected
result.

[[modules-scripting]]
== Scripting

The scripting module allows to use scripts in order to evaluate custom
expressions. For example, scripts can be used to return "script fields"
as part of a search request, or can be used to evaluate a custom score
for a query and so on.

The scripting module uses by default http://mvel.codehaus.org/[mvel] as
the scripting language with some extensions. mvel is used since it is
extremely fast and very simple to use, and in most cases, simple
expressions are needed (for example, mathematical equations).

Additional `lang` plugins are provided to allow to execute scripts in
different languages. Currently supported plugins are `lang-javascript`
for JavaScript, `lang-groovy` for Groovy, and `lang-python` for Python.
All places where a `script` parameter can be used, a `lang` parameter
(on the same level) can be provided to define the language of the
script. The `lang` options are `mvel`, `js`, `groovy`, `python`, and
`native`.

added[1.2.0, Dynamic scripting is disabled by default since version 1.2.0]

To increase security, Elasticsearch does not allow you to specify scripts with a
request. Instead, scripts must be placed in the `scripts` directory inside the
configuration directory (the directory where elasticsearch.yml is). Scripts
placed into this directory will automatically be picked up and be available to
be used. Once a script has been placed in this directory, it can be referenced
by name. For example, a script called `calculate-score.mvel` can be referenced
in a request like this:

[source,sh]
--------------------------------------------------
$ tree config
config
├── elasticsearch.yml
├── logging.yml
└── scripts
    └── calculate-score.mvel
--------------------------------------------------

[source,sh]
--------------------------------------------------
$ cat config/scripts/calculate-score.mvel
Math.log(_score * 2) + my_modifier
--------------------------------------------------

[source,js]
--------------------------------------------------
curl -XPOST localhost:9200/_search -d '{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "body": "foo"
        }
      },
      "functions": [
        {
          "script_score": {
            "script": "calculate-score",
            "params": {
              "my_modifier": 8
            }
          }
        }
      ]
    }
  }
}'
--------------------------------------------------

The name of the script is derived from the hierarchy of directories it
exists under, and the file name without the lang extension. For example,
a script placed under `config/scripts/group1/group2/test.py` will be
named `group1_group2_test`.

[float]
=== Default Scripting Language

The default scripting language (assuming no `lang` parameter is
provided) is `mvel`. In order to change it set the `script.default_lang`
to the appropriate language.

[float]
=== Enabling dynamic scripting

We recommend running Elasticsearch behind an application or proxy,
which protects Elasticsearch from the outside world. If users are
allowed to run dynamic scripts (even in a search request), then they
have the same access to your box as the user that Elasticsearch is
running as. For this reason dynamic scripting is disabled by default.

First, you should not run Elasticsearch as the `root` user, as this would allow
a script to access or do *anything* on your server, without limitations. Second,
you should not expose Elasticsearch directly to users, but instead have a proxy
application inbetween. If you *do* intend to expose Elasticsearch directly to
your users, then you have to decide whether you trust them enough to run scripts
on your box or not. If you do, you can enable dynamic scripting by adding the
following setting to the `config/elasticsearch.yml` file on every node:

[source,yaml]
-----------------------------------
script.disable_dynamic: false
-----------------------------------

While this still allows execution of named scripts provided in the config, or
_native_ Java scripts registered through plugins, it also allows users to run
arbitrary scripts via the API. Instead of sending the name of the file as the
script, the body of the script can be sent instead.

[float]
=== Automatic Script Reloading

The `config/scripts` directory is scanned periodically for changes.
New and changed scripts are reloaded and deleted script are removed
from preloaded scripts cache. The reload frequency can be specified
using `watcher.interval` setting, which defaults to `60s`.
To disable script reloading completely set `script.auto_reload_enabled`
to `false`.

[float]
=== Native (Java) Scripts

Even though `mvel` is pretty fast, this allows to register native Java based
scripts for faster execution.

In order to allow for scripts, the `NativeScriptFactory` needs to be
implemented that constructs the script that will be executed. There are
two main types, one that extends `AbstractExecutableScript` and one that
extends `AbstractSearchScript` (probably the one most users will extend,
with additional helper classes in `AbstractLongSearchScript`,
`AbstractDoubleSearchScript`, and `AbstractFloatSearchScript`).

Registering them can either be done by settings, for example:
`script.native.my.type` set to `sample.MyNativeScriptFactory` will
register a script named `my`. Another option is in a plugin, access
`ScriptModule` and call `registerScript` on it.

Executing the script is done by specifying the `lang` as `native`, and
the name of the script as the `script`.

Note, the scripts need to be in the classpath of elasticsearch. One
simple way to do it is to create a directory under plugins (choose a
descriptive name), and place the jar / classes files there, they will be
automatically loaded.

[float]
=== Score

In all scripts that can be used in facets, allow to access the current
doc score using `doc.score`.

[float]
=== Computing scores based on terms in scripts

see <<modules-advanced-scripting, advanced scripting documentation>>

[float]
=== Document Fields

Most scripting revolve around the use of specific document fields data.
The `doc['field_name']` can be used to access specific field data within
a document (the document in question is usually derived by the context
the script is used). Document fields are very fast to access since they
end up being loaded into memory (all the relevant field values/tokens
are loaded to memory).

The following data can be extracted from a field:

[cols="<,<",options="header",]
|=======================================================================
|Expression |Description
|`doc['field_name'].value` |The native value of the field. For example,
if its a short type, it will be short.

|`doc['field_name'].values` |The native array values of the field. For
example, if its a short type, it will be short[]. Remember, a field can
have several values within a single doc. Returns an empty array if the
field has no values.

|`doc['field_name'].empty` |A boolean indicating if the field has no
values within the doc.

|`doc['field_name'].multiValued` |A boolean indicating that the field
has several values within the corpus.

|`doc['field_name'].lat` |The latitude of a geo point type.

|`doc['field_name'].lon` |The longitude of a geo point type.

|`doc['field_name'].lats` |The latitudes of a geo point type.

|`doc['field_name'].lons` |The longitudes of a geo point type.

|`doc['field_name'].distance(lat, lon)` |The `plane` distance (in meters)
of this geo point field from the provided lat/lon.

|`doc['field_name'].distanceWithDefault(lat, lon, default)` |The `plane` distance (in meters)
of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].distanceInMiles(lat, lon)` |The `plane` distance (in
miles) of this geo point field from the provided lat/lon.

|`doc['field_name'].distanceInMilesWithDefault(lat, lon, default)` |The `plane` distance (in
miles) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].distanceInKm(lat, lon)` |The `plane` distance (in
km) of this geo point field from the provided lat/lon.

|`doc['field_name'].distanceInKmWithDefault(lat, lon, default)` |The `plane` distance (in
km) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].arcDistance(lat, lon)` |The `arc` distance (in
meters) of this geo point field from the provided lat/lon.

|`doc['field_name'].arcDistanceWithDefault(lat, lon, default)` |The `arc` distance (in
meters) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].arcDistanceInMiles(lat, lon)` |The `arc` distance (in
miles) of this geo point field from the provided lat/lon.

|`doc['field_name'].arcDistanceInMilesWithDefault(lat, lon, default)` |The `arc` distance (in
miles) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].arcDistanceInKm(lat, lon)` |The `arc` distance (in
km) of this geo point field from the provided lat/lon.

|`doc['field_name'].arcDistanceInKmWithDefault(lat, lon, default)` |The `arc` distance (in
km) of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].factorDistance(lat, lon)` |The distance factor of this geo point field from the provided lat/lon.

|`doc['field_name'].factorDistance(lat, lon, default)` |The distance factor of this geo point field from the provided lat/lon with a default value.

|`doc['field_name'].geohashDistance(geohash)` |The `arc` distance (in meters)
of this geo point field from the provided geohash.

|`doc['field_name'].geohashDistanceInKm(geohash)` |The `arc` distance (in km)
of this geo point field from the provided geohash.

|`doc['field_name'].geohashDistanceInMiles(geohash)` |The `arc` distance (in
miles) of this geo point field from the provided geohash.
|=======================================================================

[float]
=== Stored Fields

Stored fields can also be accessed when executing a script. Note, they
are much slower to access compared with document fields, but are not
loaded into memory. They can be simply accessed using
`_fields['my_field_name'].value` or `_fields['my_field_name'].values`.

[float]
=== Source Field

The source field can also be accessed when executing a script. The
source field is loaded per doc, parsed, and then provided to the script
for evaluation. The `_source` forms the context under which the source
field can be accessed, for example `_source.obj2.obj1.field3`.

Accessing `_source` is much slower compared to using `_doc`
but the data is not loaded into memory. For a single field access `_fields` may be
faster than using `_source` due to the extra overhead of potentially parsing large documents.
However, `_source` may be faster if you access multiple fields or if the source has already been
loaded for other purposes.

[float]
=== mvel Built In Functions

There are several built in functions that can be used within scripts.
They include:

[cols="<,<",options="header",]
|=======================================================================
|Function |Description
|`time()` |The current time in milliseconds.

|`sin(a)` |Returns the trigonometric sine of an angle.

|`cos(a)` |Returns the trigonometric cosine of an angle.

|`tan(a)` |Returns the trigonometric tangent of an angle.

|`asin(a)` |Returns the arc sine of a value.

|`acos(a)` |Returns the arc cosine of a value.

|`atan(a)` |Returns the arc tangent of a value.

|`toRadians(angdeg)` |Converts an angle measured in degrees to an
approximately equivalent angle measured in radians

|`toDegrees(angrad)` |Converts an angle measured in radians to an
approximately equivalent angle measured in degrees.

|`exp(a)` |Returns Euler's number _e_ raised to the power of value.

|`log(a)` |Returns the natural logarithm (base _e_) of a value.

|`log10(a)` |Returns the base 10 logarithm of a value.

|`sqrt(a)` |Returns the correctly rounded positive square root of a
value.

|`cbrt(a)` |Returns the cube root of a double value.

|`IEEEremainder(f1, f2)` |Computes the remainder operation on two
arguments as prescribed by the IEEE 754 standard.

|`ceil(a)` |Returns the smallest (closest to negative infinity) value
that is greater than or equal to the argument and is equal to a
mathematical integer.

|`floor(a)` |Returns the largest (closest to positive infinity) value
that is less than or equal to the argument and is equal to a
mathematical integer.

|`rint(a)` |Returns the value that is closest in value to the argument
and is equal to a mathematical integer.

|`atan2(y, x)` |Returns the angle _theta_ from the conversion of
rectangular coordinates (_x_, _y_) to polar coordinates (r,_theta_).

|`pow(a, b)` |Returns the value of the first argument raised to the
power of the second argument.

|`round(a)` |Returns the closest _int_ to the argument.

|`random()` |Returns a random _double_ value.

|`abs(a)` |Returns the absolute value of a value.

|`max(a, b)` |Returns the greater of two values.

|`min(a, b)` |Returns the smaller of two values.

|`ulp(d)` |Returns the size of an ulp of the argument.

|`signum(d)` |Returns the signum function of the argument.

|`sinh(x)` |Returns the hyperbolic sine of a value.

|`cosh(x)` |Returns the hyperbolic cosine of a value.

|`tanh(x)` |Returns the hyperbolic tangent of a value.

|`hypot(x, y)` |Returns sqrt(_x2_ + _y2_) without intermediate overflow
or underflow.
|=======================================================================

[float]
=== Arithmetic precision in MVEL

When dividing two numbers using MVEL based scripts, the engine tries to
be smart and adheres to the default behaviour of java. This means if you
divide two integers (you might have configured the fields as integer in
the mapping), the result will also be an integer. This means, if a
calculation like `1/num` is happening in your scripts and `num` is an
integer with the value of `8`, the result is `0` even though you were
expecting it to be `0.125`. You may need to enforce precision by
explicitly using a double like `1.0/num` in order to get the expected
result.


Fixed references to Multi Index Syntax (#27283)

[[modules-snapshots]]
== Snapshot And Restore

You can store snapshots of individual indices or an entire cluster in
a remote repository like a shared file system, S3, or HDFS. These snapshots
are great for backups because they can be restored relatively quickly. However,
snapshots can only be restored to versions of Elasticsearch that can read the
indices:

* A snapshot of an index created in 5.x can be restored to 6.x.
* A snapshot of an index created in 2.x can be restored to 5.x.
* A snapshot of an index created in 1.x can be restored to 2.x.

Conversely, snapshots of indices created in 1.x **cannot** be restored to
5.x or 6.x, and snapshots of indices created in 2.x **cannot** be restored
to 6.x.

Snapshots are incremental and can contain indices created in various
versions of Elasticsearch. If any indices in a snapshot were created in an
incompatible version, you will not be able restore the snapshot.

IMPORTANT: When backing up your data prior to an upgrade, keep in mind that you
won't be able to restore snapshots after you upgrade if they contain indices
created in a version that's incompatible with the upgrade version.

If you end up in a situation where you need to restore a snapshot of an index
that is incompatible with the version of the cluster you are currently running,
you can restore it on the latest compatible version and use
<<reindex-from-remote,reindex-from-remote>> to rebuild the index on the current
version. Reindexing from remote is only possible if the original index has
source enabled. Retrieving and reindexing the data can take significantly longer
than simply restoring a snapshot. If you have a large amount of data, we
recommend testing the reindex from remote process with a subset of your data to
understand the time requirements before proceeding.

[float]
=== Repositories

You must register a snapshot repository before you can perform snapshot and
restore operations. We recommend creating a new snapshot repository for each
major version. The valid repository settings depend on the repository type.

If you register same snapshot repository with multiple clusters, only
one cluster should have write access to the repository. All other clusters
connected to that repository should set the repository to `readonly` mode.

NOTE: The snapshot format can change across major versions, so if you have
clusters on different major versions trying to write the same repository,
new snapshots written by one version will not be visible to the other. While
setting the repository to `readonly` on all but one of the clusters should work
with multiple clusters differing by one major version, it is not a supported
configuration.

[source,js]
-----------------------------------
PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": {
    "location": "my_backup_location"
  }
}
-----------------------------------
// CONSOLE
// TESTSETUP

To retrieve information about a registered repository, use a GET request:

[source,js]
-----------------------------------
GET /_snapshot/my_backup
-----------------------------------
// CONSOLE

which returns:

[source,js]
-----------------------------------
{
  "my_backup": {
    "type": "fs",
    "settings": {
      "location": "my_backup_location"
    }
  }
}
-----------------------------------
// TESTRESPONSE

To retrieve information about multiple repositories, specify a
a comma-delimited list of repositories. You can also use the * wildcard when
specifying repository names. For example, the following request retrieves
information about all of the snapshot repositories that start with `repo` or
contain `backup`:

[source,js]
-----------------------------------
GET /_snapshot/repo*,*backup*
-----------------------------------
// CONSOLE

To retrieve information about all registered snapshot repositories, omit the
repository name or specify `_all`:

[source,js]
-----------------------------------
GET /_snapshot
-----------------------------------
// CONSOLE

or

[source,js]
-----------------------------------
GET /_snapshot/_all
-----------------------------------
// CONSOLE

[float]
===== Shared File System Repository

The shared file system repository (`"type": "fs"`) uses the shared file system to store snapshots. In order to register
the shared file system repository it is necessary to mount the same shared filesystem to the same location on all
master and data nodes. This location (or one of its parent directories) must be registered in the `path.repo`
setting on all master and data nodes.

Assuming that the shared filesystem is mounted to `/mount/backups/my_backup`, the following setting should be added to
`elasticsearch.yml` file:

[source,yaml]
--------------
path.repo: ["/mount/backups", "/mount/longterm_backups"]
--------------

The `path.repo` setting supports Microsoft Windows UNC paths as long as at least server name and share are specified as
a prefix and back slashes are properly escaped:

[source,yaml]
--------------
path.repo: ["\\\\MY_SERVER\\Snapshots"]
--------------

After all nodes are restarted, the following command can be used to register the shared file system repository with
the name `my_backup`:

[source,js]
-----------------------------------
PUT /_snapshot/my_fs_backup
{
    "type": "fs",
    "settings": {
        "location": "/mount/backups/my_fs_backup_location",
        "compress": true
    }
}
-----------------------------------
// CONSOLE
// TEST[skip:no access to absolute path]

If the repository location is specified as a relative path this path will be resolved against the first path specified
in `path.repo`:

[source,js]
-----------------------------------
PUT /_snapshot/my_fs_backup
{
    "type": "fs",
    "settings": {
        "location": "my_fs_backup_location",
        "compress": true
    }
}
-----------------------------------
// CONSOLE
// TEST[continued]

The following settings are supported:

[horizontal]
`location`:: Location of the snapshots. Mandatory.
`compress`:: Turns on compression of the snapshot files. Compression is applied only to metadata files (index mapping and settings). Data files are not compressed. Defaults to `true`.
`chunk_size`:: Big files can be broken down into chunks during snapshotting if needed. The chunk size can be specified in bytes or by
 using size value notation, i.e. 1g, 10m, 5k. Defaults to `null` (unlimited chunk size).
`max_restore_bytes_per_sec`:: Throttles per node restore rate. Defaults to `40mb` per second.
`max_snapshot_bytes_per_sec`:: Throttles per node snapshot rate. Defaults to `40mb` per second.
`readonly`:: Makes repository read-only.  Defaults to `false`.

[float]
===== Read-only URL Repository

The URL repository (`"type": "url"`) can be used as an alternative read-only way to access data created by the shared file
system repository. The URL specified in the `url` parameter should point to the root of the shared filesystem repository.
The following settings are supported:

[horizontal]
`url`:: Location of the snapshots. Mandatory.

URL Repository supports the following protocols: "http", "https", "ftp", "file" and "jar". URL repositories with `http:`,
`https:`, and `ftp:` URLs has to be whitelisted by specifying allowed URLs in the `repositories.url.allowed_urls` setting.
This setting supports wildcards in the place of host, path, query, and fragment. For example:

[source,yaml]
-----------------------------------
repositories.url.allowed_urls: ["http://www.example.org/root/*", "https://*.mydomain.com/*?*#*"]
-----------------------------------

URL repositories with `file:` URLs can only point to locations registered in the `path.repo` setting similar to
shared file system repository.

[float]
===== Repository plugins

Other repository backends are available in these official plugins:

* {plugins}/repository-s3.html[repository-s3] for S3 repository support
* {plugins}/repository-hdfs.html[repository-hdfs] for HDFS repository support in Hadoop environments
* {plugins}/repository-azure.html[repository-azure] for Azure storage repositories
* {plugins}/repository-gcs.html[repository-gcs] for Google Cloud Storage repositories

[float]
===== Repository Verification
When a repository is registered, it's immediately verified on all master and data nodes to make sure that it is functional
on all nodes currently present in the cluster. The `verify` parameter can be used to explicitly disable the repository
verification when registering or updating a repository:

[source,js]
-----------------------------------
PUT /_snapshot/my_unverified_backup?verify=false
{
  "type": "fs",
  "settings": {
    "location": "my_unverified_backup_location"
  }
}
-----------------------------------
// CONSOLE
// TEST[continued]

The verification process can also be executed manually by running the following command:

[source,js]
-----------------------------------
POST /_snapshot/my_unverified_backup/_verify
-----------------------------------
// CONSOLE
// TEST[continued]

It returns a list of nodes where repository was successfully verified or an error message if verification process failed.

[float]
=== Snapshot

A repository can contain multiple snapshots of the same cluster. Snapshots are identified by unique names within the
cluster. A snapshot with the name `snapshot_1` in the repository `my_backup` can be created by executing the following
command:

[source,js]
-----------------------------------
PUT /_snapshot/my_backup/snapshot_1?wait_for_completion=true
-----------------------------------
// CONSOLE
// TEST[continued]

The `wait_for_completion` parameter specifies whether or not the request should return immediately after snapshot
initialization (default) or wait for snapshot completion. During snapshot initialization, information about all
previous snapshots is loaded into the memory, which means that in large repositories it may take several seconds (or
even minutes) for this command to return even if the `wait_for_completion` parameter is set to `false`.

By default a snapshot of all open and started indices in the cluster is created. This behavior can be changed by
specifying the list of indices in the body of the snapshot request.

[source,js]
-----------------------------------
PUT /_snapshot/my_backup/snapshot_2?wait_for_completion=true
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": false
}
-----------------------------------
// CONSOLE
// TEST[continued]

The list of indices that should be included into the snapshot can be specified using the `indices` parameter that
supports <<multi-index,multi index syntax>>. The snapshot request also supports the
`ignore_unavailable` option. Setting it to `true` will cause indices that do not exist to be ignored during snapshot
creation. By default, when `ignore_unavailable` option is not set and an index is missing the snapshot request will fail.
By setting `include_global_state` to false it's possible to prevent the cluster global state to be stored as part of
the snapshot. By default, the entire snapshot will fail if one or more indices participating in the snapshot don't have
all primary shards available. This behaviour can be changed by setting `partial` to `true`.

The index snapshot process is incremental. In the process of making the index snapshot Elasticsearch analyses
the list of the index files that are already stored in the repository and copies only files that were created or
changed since the last snapshot. That allows multiple snapshots to be preserved in the repository in a compact form.
Snapshotting process is executed in non-blocking fashion. All indexing and searching operation can continue to be
executed against the index that is being snapshotted. However, a snapshot represents the point-in-time view of the index
at the moment when snapshot was created, so no records that were added to the index after the snapshot process was started
will be present in the snapshot. The snapshot process starts immediately for the primary shards that has been started
and are not relocating at the moment. Before version 1.2.0, the snapshot operation fails if the cluster has any relocating or
initializing primaries of indices participating in the snapshot. Starting with version 1.2.0, Elasticsearch waits for
relocation or initialization of shards to complete before snapshotting them.

Besides creating a copy of each index the snapshot process can also store global cluster metadata, which includes persistent
cluster settings and templates. The transient settings and registered snapshot repositories are not stored as part of
the snapshot.

Only one snapshot process can be executed in the cluster at any time. While snapshot of a particular shard is being
created this shard cannot be moved to another node, which can interfere with rebalancing process and allocation
filtering. Elasticsearch will only be able to move a shard to another node (according to the current allocation
filtering settings and rebalancing algorithm) once the snapshot is finished.

Once a snapshot is created information about this snapshot can be obtained using the following command:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1
-----------------------------------
// CONSOLE
// TEST[continued]

This command returns basic information about the snapshot including start and end time, version of
elasticsearch that created the snapshot, the list of included indices, the current state of the
snapshot and the list of failures that occurred during the snapshot. The snapshot `state` can be

[horizontal]
`IN_PROGRESS`::

  The snapshot is currently running.

`SUCCESS`::

  The snapshot finished and all shards were stored successfully.

`FAILED`::

  The snapshot finished with an error and failed to store any data.

`PARTIAL`::

  The global cluster state was stored, but data of at least one shard wasn't stored successfully.
  The `failure` section in this case should contain more detailed information about shards
  that were not processed correctly.

`INCOMPATIBLE`::

  The snapshot was created with an old version of elasticsearch and therefore is incompatible with
  the current version of the cluster.

Similar as for repositories, information about multiple snapshots can be queried in one go, supporting wildcards as well:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_*,some_other_snapshot
-----------------------------------
// CONSOLE
// TEST[continued]

All snapshots currently stored in the repository can be listed using the following command:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/_all
-----------------------------------
// CONSOLE
// TEST[continued]

The command fails if some of the snapshots are unavailable. The boolean parameter `ignore_unavailable` can be used to
return all snapshots that are currently available.

Getting all snapshots in the repository can be costly on cloud-based repositories,
both from a cost and performance perspective.  If the only information required is
the snapshot names/uuids in the repository and the indices in each snapshot, then
the optional boolean parameter `verbose` can be set to `false` to execute a more
performant and cost-effective retrieval of the snapshots in the repository.  Note
that setting `verbose` to `false` will omit all other information about the snapshot
such as status information, the number of snapshotted shards, etc.  The default
value of the `verbose` parameter is `true`.

A currently running snapshot can be retrieved using the following command:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/_current
-----------------------------------
// CONSOLE
// TEST[continued]

A snapshot can be deleted from the repository using the following command:

[source,sh]
-----------------------------------
DELETE /_snapshot/my_backup/snapshot_2
-----------------------------------
// CONSOLE
// TEST[continued]

When a snapshot is deleted from a repository, Elasticsearch deletes all files that are associated with the deleted
snapshot and not used by any other snapshots. If the deleted snapshot operation is executed while the snapshot is being
created the snapshotting process will be aborted and all files created as part of the snapshotting process will be
cleaned. Therefore, the delete snapshot operation can be used to cancel long running snapshot operations that were
started by mistake.

A repository can be deleted using the following command:

[source,sh]
-----------------------------------
DELETE /_snapshot/my_fs_backup
-----------------------------------
// CONSOLE
// TEST[continued]

When a repository is deleted, Elasticsearch only removes the reference to the location where the repository is storing
the snapshots. The snapshots themselves are left untouched and in place.

[float]
=== Restore

A snapshot can be restored using the following command:

[source,sh]
-----------------------------------
POST /_snapshot/my_backup/snapshot_1/_restore
-----------------------------------
// CONSOLE
// TEST[continued]

By default, all indices in the snapshot are restored, and the cluster state is
*not* restored. It's possible to select indices that should be restored as well
as to allow the global cluster state from being restored by using `indices` and
`include_global_state` options in the restore request body. The list of indices
supports <<multi-index,multi index syntax>>. The `rename_pattern`
and `rename_replacement` options can be also used to rename indices on restore
using regular expression that supports referencing the original text as
explained
http://docs.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html#appendReplacement(java.lang.StringBuffer,%20java.lang.String)[here].
Set `include_aliases` to `false` to prevent aliases from being restored together
with associated indices

[source,js]
-----------------------------------
POST /_snapshot/my_backup/snapshot_1/_restore
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": true,
  "rename_pattern": "index_(.+)",
  "rename_replacement": "restored_index_$1"
}
-----------------------------------
// CONSOLE
// TEST[continued]

The restore operation can be performed on a functioning cluster. However, an
existing index can be only restored if it's <<indices-open-close,closed>> and
has the same number of shards as the index in the snapshot. The restore
operation automatically opens restored indices if they were closed and creates
new indices if they didn't exist in the cluster. If cluster state is restored
with `include_global_state` (defaults to `false`), the restored templates that
don't currently exist in the cluster are added and existing templates with the
same name are replaced by the restored templates. The restored persistent
settings are added to the existing persistent settings.

[float]
==== Partial restore

By default, the entire restore operation will fail if one or more indices participating in the operation don't have
snapshots of all shards available. It can occur if some shards failed to snapshot for example. It is still possible to
restore such indices by setting `partial` to `true`. Please note, that only successfully snapshotted shards will be
restored in this case and all missing shards will be recreated empty.

[float]
==== Changing index settings during restore

Most of index settings can be overridden during the restore process. For example, the following command will restore
the index `index_1` without creating any replicas while switching back to default refresh interval:

[source,js]
-----------------------------------
POST /_snapshot/my_backup/snapshot_1/_restore
{
  "indices": "index_1",
  "index_settings": {
    "index.number_of_replicas": 0
  },
  "ignore_index_settings": [
    "index.refresh_interval"
  ]
}
-----------------------------------
// CONSOLE
// TEST[continued]

Please note, that some settings such as `index.number_of_shards` cannot be changed during restore operation.

[float]
==== Restoring to a different cluster

The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore it's possible to
restore a snapshot made from one cluster into another cluster. All that is required is registering the repository
containing the snapshot in the new cluster and starting the restore process. The new cluster doesn't have to have the
same size or topology.  However, the version of the new cluster should be the same or newer (only 1 major version newer) than the cluster that was used to create the snapshot.  For example, you can restore a 1.x snapshot to a 2.x cluster, but not a 1.x snapshot to a 5.x cluster.

If the new cluster has a smaller size additional considerations should be made. First of all it's necessary to make sure
that new cluster have enough capacity to store all indices in the snapshot. It's possible to change indices settings
during restore to reduce the number of replicas, which can help with restoring snapshots into smaller cluster. It's also
possible to select only subset of the indices using the `indices` parameter.

If indices in the original cluster were assigned to particular nodes using
<<shard-allocation-filtering,shard allocation filtering>>, the same rules will be enforced in the new cluster. Therefore
if the new cluster doesn't contain nodes with appropriate attributes that a restored index can be allocated on, such
index will not be successfully restored unless these index allocation settings are changed during restore operation.

The restore operation also checks that restored persistent settings are compatible with the current cluster to avoid accidentally
restoring an incompatible settings such as `discovery.zen.minimum_master_nodes` and as a result disable a smaller cluster until the
required number of master eligible nodes is added. If you need to restore a snapshot with incompatible persistent settings, try
restoring it without the global cluster state.

[float]
=== Snapshot status

A list of currently running snapshots with their detailed status information can be obtained using the following command:

[source,sh]
-----------------------------------
GET /_snapshot/_status
-----------------------------------
// CONSOLE
// TEST[continued]

In this format, the command will return information about all currently running snapshots. By specifying a repository name, it's possible
to limit the results to a particular repository:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/_status
-----------------------------------
// CONSOLE
// TEST[continued]

If both repository name and snapshot id are specified, this command will return detailed status information for the given snapshot even
if it's not currently running:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1/_status
-----------------------------------
// CONSOLE
// TEST[continued]

Multiple ids are also supported:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1,snapshot_2/_status
-----------------------------------
// CONSOLE
// TEST[continued]

[float]
=== Monitoring snapshot/restore progress

There are several ways to monitor the progress of the snapshot and restores processes while they are running. Both
operations support `wait_for_completion` parameter that would block client until the operation is completed. This is
the simplest method that can be used to get notified about operation completion.

The snapshot operation can be also monitored by periodic calls to the snapshot info:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1
-----------------------------------
// CONSOLE
// TEST[continued]

Please note that snapshot info operation uses the same resources and thread pool as the snapshot operation. So,
executing a snapshot info operation while large shards are being snapshotted can cause the snapshot info operation to wait
for available resources before returning the result. On very large shards the wait time can be significant.

To get more immediate and complete information about snapshots the snapshot status command can be used instead:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1/_status
-----------------------------------
// CONSOLE
// TEST[continued]

While snapshot info method returns only basic information about the snapshot in progress, the snapshot status returns
complete breakdown of the current state for each shard participating in the snapshot.

The restore process piggybacks on the standard recovery mechanism of the Elasticsearch. As a result, standard recovery
monitoring services can be used to monitor the state of restore. When restore operation is executed the cluster
typically goes into `red` state. It happens because the restore operation starts with "recovering" primary shards of the
restored indices. During this operation the primary shards become unavailable which manifests itself in the `red` cluster
state. Once recovery of primary shards is completed Elasticsearch is switching to standard replication process that
creates the required number of replicas at this moment cluster switches to the `yellow` state. Once all required replicas
are created, the cluster switches to the `green` states.

The cluster health operation provides only a high level status of the restore process. It's possible to get more
detailed insight into the current state of the recovery process by using <<indices-recovery, indices recovery>> and
<<cat-recovery, cat recovery>> APIs.

[float]
=== Stopping currently running snapshot and restore operations

The snapshot and restore framework allows running only one snapshot or one restore operation at a time. If a currently
running snapshot was executed by mistake, or takes unusually long, it can be terminated using the snapshot delete operation.
The snapshot delete operation checks if the deleted snapshot is currently running and if it does, the delete operation stops
that snapshot before deleting the snapshot data from the repository.

[source,sh]
-----------------------------------
DELETE /_snapshot/my_backup/snapshot_1
-----------------------------------
// CONSOLE
// TEST[continued]

The restore operation uses the standard shard recovery mechanism. Therefore, any currently running restore operation can
be canceled by deleting indices that are being restored. Please note that data for all deleted indices will be removed
from the cluster as a result of this operation.

[float]
=== Effect of cluster blocks on snapshot and restore operations
Many snapshot and restore operations are affected by cluster and index blocks. For example, registering and unregistering
repositories require write global metadata access. The snapshot operation requires that all indices and their metadata as
well as the global metadata were readable. The restore operation requires the global metadata to be writable, however
the index level blocks are ignored during restore because indices are essentially recreated during restore.
Please note that a repository content is not part of the cluster and therefore cluster blocks don't affect internal
repository operations such as listing or deleting snapshots from an already registered repository.

[[modules-snapshots]]
== Snapshot And Restore

You can store snapshots of individual indices or an entire cluster in
a remote repository like a shared file system, S3, or HDFS. These snapshots
are great for backups because they can be restored relatively quickly. However,
snapshots can only be restored to versions of Elasticsearch that can read the
indices:

* A snapshot of an index created in 5.x can be restored to 6.x.
* A snapshot of an index created in 2.x can be restored to 5.x.
* A snapshot of an index created in 1.x can be restored to 2.x.

Conversely, snapshots of indices created in 1.x **cannot** be restored to
5.x or 6.x, and snapshots of indices created in 2.x **cannot** be restored
to 6.x.

Snapshots are incremental and can contain indices created in various
versions of Elasticsearch. If any indices in a snapshot were created in an
incompatible version, you will not be able restore the snapshot.

IMPORTANT: When backing up your data prior to an upgrade, keep in mind that you
won't be able to restore snapshots after you upgrade if they contain indices
created in a version that's incompatible with the upgrade version.

If you end up in a situation where you need to restore a snapshot of an index
that is incompatible with the version of the cluster you are currently running,
you can restore it on the latest compatible version and use
<<reindex-from-remote,reindex-from-remote>> to rebuild the index on the current
version. Reindexing from remote is only possible if the original index has
source enabled. Retrieving and reindexing the data can take significantly longer
than simply restoring a snapshot. If you have a large amount of data, we
recommend testing the reindex from remote process with a subset of your data to
understand the time requirements before proceeding.

[float]
=== Repositories

You must register a snapshot repository before you can perform snapshot and
restore operations. We recommend creating a new snapshot repository for each
major version. The valid repository settings depend on the repository type.

If you register same snapshot repository with multiple clusters, only
one cluster should have write access to the repository. All other clusters
connected to that repository should set the repository to `readonly` mode.

NOTE: The snapshot format can change across major versions, so if you have
clusters on different major versions trying to write the same repository,
new snapshots written by one version will not be visible to the other. While
setting the repository to `readonly` on all but one of the clusters should work
with multiple clusters differing by one major version, it is not a supported
configuration.

[source,js]
-----------------------------------
PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": {
    "location": "my_backup_location"
  }
}
-----------------------------------
// CONSOLE
// TESTSETUP

To retrieve information about a registered repository, use a GET request:

[source,js]
-----------------------------------
GET /_snapshot/my_backup
-----------------------------------
// CONSOLE

which returns:

[source,js]
-----------------------------------
{
  "my_backup": {
    "type": "fs",
    "settings": {
      "location": "my_backup_location"
    }
  }
}
-----------------------------------
// TESTRESPONSE

To retrieve information about multiple repositories, specify a
a comma-delimited list of repositories. You can also use the * wildcard when
specifying repository names. For example, the following request retrieves
information about all of the snapshot repositories that start with `repo` or
contain `backup`:

[source,js]
-----------------------------------
GET /_snapshot/repo*,*backup*
-----------------------------------
// CONSOLE

To retrieve information about all registered snapshot repositories, omit the
repository name or specify `_all`:

[source,js]
-----------------------------------
GET /_snapshot
-----------------------------------
// CONSOLE

or

[source,js]
-----------------------------------
GET /_snapshot/_all
-----------------------------------
// CONSOLE

[float]
===== Shared File System Repository

The shared file system repository (`"type": "fs"`) uses the shared file system to store snapshots. In order to register
the shared file system repository it is necessary to mount the same shared filesystem to the same location on all
master and data nodes. This location (or one of its parent directories) must be registered in the `path.repo`
setting on all master and data nodes.

Assuming that the shared filesystem is mounted to `/mount/backups/my_backup`, the following setting should be added to
`elasticsearch.yml` file:

[source,yaml]
--------------
path.repo: ["/mount/backups", "/mount/longterm_backups"]
--------------

The `path.repo` setting supports Microsoft Windows UNC paths as long as at least server name and share are specified as
a prefix and back slashes are properly escaped:

[source,yaml]
--------------
path.repo: ["\\\\MY_SERVER\\Snapshots"]
--------------

After all nodes are restarted, the following command can be used to register the shared file system repository with
the name `my_backup`:

[source,js]
-----------------------------------
PUT /_snapshot/my_fs_backup
{
    "type": "fs",
    "settings": {
        "location": "/mount/backups/my_fs_backup_location",
        "compress": true
    }
}
-----------------------------------
// CONSOLE
// TEST[skip:no access to absolute path]

If the repository location is specified as a relative path this path will be resolved against the first path specified
in `path.repo`:

[source,js]
-----------------------------------
PUT /_snapshot/my_fs_backup
{
    "type": "fs",
    "settings": {
        "location": "my_fs_backup_location",
        "compress": true
    }
}
-----------------------------------
// CONSOLE
// TEST[continued]

The following settings are supported:

[horizontal]
`location`:: Location of the snapshots. Mandatory.
`compress`:: Turns on compression of the snapshot files. Compression is applied only to metadata files (index mapping and settings). Data files are not compressed. Defaults to `true`.
`chunk_size`:: Big files can be broken down into chunks during snapshotting if needed. The chunk size can be specified in bytes or by
 using size value notation, i.e. 1g, 10m, 5k. Defaults to `null` (unlimited chunk size).
`max_restore_bytes_per_sec`:: Throttles per node restore rate. Defaults to `40mb` per second.
`max_snapshot_bytes_per_sec`:: Throttles per node snapshot rate. Defaults to `40mb` per second.
`readonly`:: Makes repository read-only.  Defaults to `false`.

[float]
===== Read-only URL Repository

The URL repository (`"type": "url"`) can be used as an alternative read-only way to access data created by the shared file
system repository. The URL specified in the `url` parameter should point to the root of the shared filesystem repository.
The following settings are supported:

[horizontal]
`url`:: Location of the snapshots. Mandatory.

URL Repository supports the following protocols: "http", "https", "ftp", "file" and "jar". URL repositories with `http:`,
`https:`, and `ftp:` URLs has to be whitelisted by specifying allowed URLs in the `repositories.url.allowed_urls` setting.
This setting supports wildcards in the place of host, path, query, and fragment. For example:

[source,yaml]
-----------------------------------
repositories.url.allowed_urls: ["http://www.example.org/root/*", "https://*.mydomain.com/*?*#*"]
-----------------------------------

URL repositories with `file:` URLs can only point to locations registered in the `path.repo` setting similar to
shared file system repository.

[float]
===== Repository plugins

Other repository backends are available in these official plugins:

* {plugins}/repository-s3.html[repository-s3] for S3 repository support
* {plugins}/repository-hdfs.html[repository-hdfs] for HDFS repository support in Hadoop environments
* {plugins}/repository-azure.html[repository-azure] for Azure storage repositories
* {plugins}/repository-gcs.html[repository-gcs] for Google Cloud Storage repositories

[float]
===== Repository Verification
When a repository is registered, it's immediately verified on all master and data nodes to make sure that it is functional
on all nodes currently present in the cluster. The `verify` parameter can be used to explicitly disable the repository
verification when registering or updating a repository:

[source,js]
-----------------------------------
PUT /_snapshot/my_unverified_backup?verify=false
{
  "type": "fs",
  "settings": {
    "location": "my_unverified_backup_location"
  }
}
-----------------------------------
// CONSOLE
// TEST[continued]

The verification process can also be executed manually by running the following command:

[source,js]
-----------------------------------
POST /_snapshot/my_unverified_backup/_verify
-----------------------------------
// CONSOLE
// TEST[continued]

It returns a list of nodes where repository was successfully verified or an error message if verification process failed.

[float]
=== Snapshot

A repository can contain multiple snapshots of the same cluster. Snapshots are identified by unique names within the
cluster. A snapshot with the name `snapshot_1` in the repository `my_backup` can be created by executing the following
command:

[source,js]
-----------------------------------
PUT /_snapshot/my_backup/snapshot_1?wait_for_completion=true
-----------------------------------
// CONSOLE
// TEST[continued]

The `wait_for_completion` parameter specifies whether or not the request should return immediately after snapshot
initialization (default) or wait for snapshot completion. During snapshot initialization, information about all
previous snapshots is loaded into the memory, which means that in large repositories it may take several seconds (or
even minutes) for this command to return even if the `wait_for_completion` parameter is set to `false`.

By default a snapshot of all open and started indices in the cluster is created. This behavior can be changed by
specifying the list of indices in the body of the snapshot request.

[source,js]
-----------------------------------
PUT /_snapshot/my_backup/snapshot_2?wait_for_completion=true
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": false
}
-----------------------------------
// CONSOLE
// TEST[continued]

The list of indices that should be included into the snapshot can be specified using the `indices` parameter that
supports <<search-multi-index-type,multi index syntax>>. The snapshot request also supports the
`ignore_unavailable` option. Setting it to `true` will cause indices that do not exist to be ignored during snapshot
creation. By default, when `ignore_unavailable` option is not set and an index is missing the snapshot request will fail.
By setting `include_global_state` to false it's possible to prevent the cluster global state to be stored as part of
the snapshot. By default, the entire snapshot will fail if one or more indices participating in the snapshot don't have
all primary shards available. This behaviour can be changed by setting `partial` to `true`.

The index snapshot process is incremental. In the process of making the index snapshot Elasticsearch analyses
the list of the index files that are already stored in the repository and copies only files that were created or
changed since the last snapshot. That allows multiple snapshots to be preserved in the repository in a compact form.
Snapshotting process is executed in non-blocking fashion. All indexing and searching operation can continue to be
executed against the index that is being snapshotted. However, a snapshot represents the point-in-time view of the index
at the moment when snapshot was created, so no records that were added to the index after the snapshot process was started
will be present in the snapshot. The snapshot process starts immediately for the primary shards that has been started
and are not relocating at the moment. Before version 1.2.0, the snapshot operation fails if the cluster has any relocating or
initializing primaries of indices participating in the snapshot. Starting with version 1.2.0, Elasticsearch waits for
relocation or initialization of shards to complete before snapshotting them.

Besides creating a copy of each index the snapshot process can also store global cluster metadata, which includes persistent
cluster settings and templates. The transient settings and registered snapshot repositories are not stored as part of
the snapshot.

Only one snapshot process can be executed in the cluster at any time. While snapshot of a particular shard is being
created this shard cannot be moved to another node, which can interfere with rebalancing process and allocation
filtering. Elasticsearch will only be able to move a shard to another node (according to the current allocation
filtering settings and rebalancing algorithm) once the snapshot is finished.

Once a snapshot is created information about this snapshot can be obtained using the following command:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1
-----------------------------------
// CONSOLE
// TEST[continued]

This command returns basic information about the snapshot including start and end time, version of
elasticsearch that created the snapshot, the list of included indices, the current state of the
snapshot and the list of failures that occurred during the snapshot. The snapshot `state` can be

[horizontal]
`IN_PROGRESS`::

  The snapshot is currently running.

`SUCCESS`::

  The snapshot finished and all shards were stored successfully.

`FAILED`::

  The snapshot finished with an error and failed to store any data.

`PARTIAL`::

  The global cluster state was stored, but data of at least one shard wasn't stored successfully.
  The `failure` section in this case should contain more detailed information about shards
  that were not processed correctly.

`INCOMPATIBLE`::

  The snapshot was created with an old version of elasticsearch and therefore is incompatible with
  the current version of the cluster.

Similar as for repositories, information about multiple snapshots can be queried in one go, supporting wildcards as well:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_*,some_other_snapshot
-----------------------------------
// CONSOLE
// TEST[continued]

All snapshots currently stored in the repository can be listed using the following command:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/_all
-----------------------------------
// CONSOLE
// TEST[continued]

The command fails if some of the snapshots are unavailable. The boolean parameter `ignore_unavailable` can be used to
return all snapshots that are currently available.

Getting all snapshots in the repository can be costly on cloud-based repositories,
both from a cost and performance perspective.  If the only information required is
the snapshot names/uuids in the repository and the indices in each snapshot, then
the optional boolean parameter `verbose` can be set to `false` to execute a more
performant and cost-effective retrieval of the snapshots in the repository.  Note
that setting `verbose` to `false` will omit all other information about the snapshot
such as status information, the number of snapshotted shards, etc.  The default
value of the `verbose` parameter is `true`.

A currently running snapshot can be retrieved using the following command:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/_current
-----------------------------------
// CONSOLE
// TEST[continued]

A snapshot can be deleted from the repository using the following command:

[source,sh]
-----------------------------------
DELETE /_snapshot/my_backup/snapshot_2
-----------------------------------
// CONSOLE
// TEST[continued]

When a snapshot is deleted from a repository, Elasticsearch deletes all files that are associated with the deleted
snapshot and not used by any other snapshots. If the deleted snapshot operation is executed while the snapshot is being
created the snapshotting process will be aborted and all files created as part of the snapshotting process will be
cleaned. Therefore, the delete snapshot operation can be used to cancel long running snapshot operations that were
started by mistake.

A repository can be deleted using the following command:

[source,sh]
-----------------------------------
DELETE /_snapshot/my_fs_backup
-----------------------------------
// CONSOLE
// TEST[continued]

When a repository is deleted, Elasticsearch only removes the reference to the location where the repository is storing
the snapshots. The snapshots themselves are left untouched and in place.

[float]
=== Restore

A snapshot can be restored using the following command:

[source,sh]
-----------------------------------
POST /_snapshot/my_backup/snapshot_1/_restore
-----------------------------------
// CONSOLE
// TEST[continued]

By default, all indices in the snapshot are restored, and the cluster state is
*not* restored. It's possible to select indices that should be restored as well
as to allow the global cluster state from being restored by using `indices` and
`include_global_state` options in the restore request body. The list of indices
supports <<search-multi-index-type,multi index syntax>>. The `rename_pattern`
and `rename_replacement` options can be also used to rename indices on restore
using regular expression that supports referencing the original text as
explained
http://docs.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html#appendReplacement(java.lang.StringBuffer,%20java.lang.String)[here].
Set `include_aliases` to `false` to prevent aliases from being restored together
with associated indices

[source,js]
-----------------------------------
POST /_snapshot/my_backup/snapshot_1/_restore
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": true,
  "rename_pattern": "index_(.+)",
  "rename_replacement": "restored_index_$1"
}
-----------------------------------
// CONSOLE
// TEST[continued]

The restore operation can be performed on a functioning cluster. However, an
existing index can be only restored if it's <<indices-open-close,closed>> and
has the same number of shards as the index in the snapshot. The restore
operation automatically opens restored indices if they were closed and creates
new indices if they didn't exist in the cluster. If cluster state is restored
with `include_global_state` (defaults to `false`), the restored templates that
don't currently exist in the cluster are added and existing templates with the
same name are replaced by the restored templates. The restored persistent
settings are added to the existing persistent settings.

[float]
==== Partial restore

By default, the entire restore operation will fail if one or more indices participating in the operation don't have
snapshots of all shards available. It can occur if some shards failed to snapshot for example. It is still possible to
restore such indices by setting `partial` to `true`. Please note, that only successfully snapshotted shards will be
restored in this case and all missing shards will be recreated empty.

[float]
==== Changing index settings during restore

Most of index settings can be overridden during the restore process. For example, the following command will restore
the index `index_1` without creating any replicas while switching back to default refresh interval:

[source,js]
-----------------------------------
POST /_snapshot/my_backup/snapshot_1/_restore
{
  "indices": "index_1",
  "index_settings": {
    "index.number_of_replicas": 0
  },
  "ignore_index_settings": [
    "index.refresh_interval"
  ]
}
-----------------------------------
// CONSOLE
// TEST[continued]

Please note, that some settings such as `index.number_of_shards` cannot be changed during restore operation.

[float]
==== Restoring to a different cluster

The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore it's possible to
restore a snapshot made from one cluster into another cluster. All that is required is registering the repository
containing the snapshot in the new cluster and starting the restore process. The new cluster doesn't have to have the
same size or topology.  However, the version of the new cluster should be the same or newer (only 1 major version newer) than the cluster that was used to create the snapshot.  For example, you can restore a 1.x snapshot to a 2.x cluster, but not a 1.x snapshot to a 5.x cluster.

If the new cluster has a smaller size additional considerations should be made. First of all it's necessary to make sure
that new cluster have enough capacity to store all indices in the snapshot. It's possible to change indices settings
during restore to reduce the number of replicas, which can help with restoring snapshots into smaller cluster. It's also
possible to select only subset of the indices using the `indices` parameter.

If indices in the original cluster were assigned to particular nodes using
<<shard-allocation-filtering,shard allocation filtering>>, the same rules will be enforced in the new cluster. Therefore
if the new cluster doesn't contain nodes with appropriate attributes that a restored index can be allocated on, such
index will not be successfully restored unless these index allocation settings are changed during restore operation.

The restore operation also checks that restored persistent settings are compatible with the current cluster to avoid accidentally
restoring an incompatible settings such as `discovery.zen.minimum_master_nodes` and as a result disable a smaller cluster until the
required number of master eligible nodes is added. If you need to restore a snapshot with incompatible persistent settings, try
restoring it without the global cluster state.

[float]
=== Snapshot status

A list of currently running snapshots with their detailed status information can be obtained using the following command:

[source,sh]
-----------------------------------
GET /_snapshot/_status
-----------------------------------
// CONSOLE
// TEST[continued]

In this format, the command will return information about all currently running snapshots. By specifying a repository name, it's possible
to limit the results to a particular repository:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/_status
-----------------------------------
// CONSOLE
// TEST[continued]

If both repository name and snapshot id are specified, this command will return detailed status information for the given snapshot even
if it's not currently running:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1/_status
-----------------------------------
// CONSOLE
// TEST[continued]

Multiple ids are also supported:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1,snapshot_2/_status
-----------------------------------
// CONSOLE
// TEST[continued]

[float]
=== Monitoring snapshot/restore progress

There are several ways to monitor the progress of the snapshot and restores processes while they are running. Both
operations support `wait_for_completion` parameter that would block client until the operation is completed. This is
the simplest method that can be used to get notified about operation completion.

The snapshot operation can be also monitored by periodic calls to the snapshot info:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1
-----------------------------------
// CONSOLE
// TEST[continued]

Please note that snapshot info operation uses the same resources and thread pool as the snapshot operation. So,
executing a snapshot info operation while large shards are being snapshotted can cause the snapshot info operation to wait
for available resources before returning the result. On very large shards the wait time can be significant.

To get more immediate and complete information about snapshots the snapshot status command can be used instead:

[source,sh]
-----------------------------------
GET /_snapshot/my_backup/snapshot_1/_status
-----------------------------------
// CONSOLE
// TEST[continued]

While snapshot info method returns only basic information about the snapshot in progress, the snapshot status returns
complete breakdown of the current state for each shard participating in the snapshot.

The restore process piggybacks on the standard recovery mechanism of the Elasticsearch. As a result, standard recovery
monitoring services can be used to monitor the state of restore. When restore operation is executed the cluster
typically goes into `red` state. It happens because the restore operation starts with "recovering" primary shards of the
restored indices. During this operation the primary shards become unavailable which manifests itself in the `red` cluster
state. Once recovery of primary shards is completed Elasticsearch is switching to standard replication process that
creates the required number of replicas at this moment cluster switches to the `yellow` state. Once all required replicas
are created, the cluster switches to the `green` states.

The cluster health operation provides only a high level status of the restore process. It's possible to get more
detailed insight into the current state of the recovery process by using <<indices-recovery, indices recovery>> and
<<cat-recovery, cat recovery>> APIs.

[float]
=== Stopping currently running snapshot and restore operations

The snapshot and restore framework allows running only one snapshot or one restore operation at a time. If a currently
running snapshot was executed by mistake, or takes unusually long, it can be terminated using the snapshot delete operation.
The snapshot delete operation checks if the deleted snapshot is currently running and if it does, the delete operation stops
that snapshot before deleting the snapshot data from the repository.

[source,sh]
-----------------------------------
DELETE /_snapshot/my_backup/snapshot_1
-----------------------------------
// CONSOLE
// TEST[continued]

The restore operation uses the standard shard recovery mechanism. Therefore, any currently running restore operation can
be canceled by deleting indices that are being restored. Please note that data for all deleted indices will be removed
from the cluster as a result of this operation.

[float]
=== Effect of cluster blocks on snapshot and restore operations
Many snapshot and restore operations are affected by cluster and index blocks. For example, registering and unregistering
repositories require write global metadata access. The snapshot operation requires that all indices and their metadata as
well as the global metadata were readable. The restore operation requires the global metadata to be writable, however
the index level blocks are ignored during restore because indices are essentially recreated during restore.
Please note that a repository content is not part of the cluster and therefore cluster blocks don't affect internal
repository operations such as listing or deleting snapshots from an already registered repository.