Update 2016-01-18-LTSP-Images-servidas-por-Servidor-de-Aula.adoc

= LTSP Images servidas por Servidor de Aula

:published_at: 2016-01-18
:hp-tags: LTSP, aula, aula-servidor
:hp-image: http://www.greaterlatrobe.net/seniorcenter/Computer%20Classroom%2010-05.jpg

En las aulas los Servidores de Aula _(ordenador del profesor)_ se encargan de ofrecer la imagen a arrancar a los LTSP _(ordenadores de los alumnos)_

Los LTSP tienen activada la funcionalidad para pedir una imagen a través de la Red, el Servidor de Aula es el encargado de proveer internet a los LTSP al mismo tiempo que hace de servidor DHCP.

== Imagen LTSP

Los archivos que hacen de imagen para los LTSP están en

----
/opt/i386
----

Se puede montar la imagen en el servidor de aula con

----
chroot /opt/i386
----

== Procesar imagen para LTSP

La imagen que se envía no es exactamente lo que hay en esa carpeta, hay que hacer un proceso de procesado y compresión, el archivo resultante es lo que se enviará a los LTSP cada vez que se arranquen.

Para ello hay que ejecutar

----
ltsp-update-image --arch i386
----

Una vez termine ese proceso cuando los LTSP vuelvan a arrancar tendrán la nueva imagen con los cambios realizados

= LTSP Images servidas por Servidor de Aula

:published_at: 2016-01-18
:hp-tags: LTSP, aula, aula-servidor
:hp-image: http://www.greaterlatrobe.net/seniorcenter/Computer%20Classroom%2010-05.jpg

Durante este post vamos a comentar como se inician los LTSP en las aulas


Fixing typo in the first JSON example

Fixing typo in the first JSON example

Closes #7172

[[mapping-core-types]]
=== Core Types

Each JSON field can be mapped to a specific core type. JSON itself
already provides us with some typing, with its support for `string`,
`integer`/`long`, `float`/`double`, `boolean`, and `null`.

The following sample tweet JSON document will be used to explain the
core types:

[source,js]
--------------------------------------------------
{
    "tweet" {
        "user" : "kimchy",
        "message" : "This is a tweet!",
        "postDate" : "2009-11-15T14:12:12",
        "priority" : 4,
        "rank" : 12.3
    }
}
--------------------------------------------------

Explicit mapping for the above JSON tweet can be:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "user" : {"type" : "string", "index" : "not_analyzed"},
            "message" : {"type" : "string", "null_value" : "na"},
            "postDate" : {"type" : "date"},
            "priority" : {"type" : "integer"},
            "rank" : {"type" : "float"}
        }
    }
}
--------------------------------------------------

[float]
[[string]]
==== String

The text based string type is the most basic type, and contains one or
more characters. An example mapping can be:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "message" : {
                "type" : "string",
                "store" : true,
                "index" : "analyzed",
                "null_value" : "na"
            },
            "user" : {
                "type" : "string",
                "index" : "not_analyzed",
                "norms" : {
                    "enabled" : false
                }
            }
        }
    }
}
--------------------------------------------------

The above mapping defines a `string` `message` property/field within the
`tweet` type. The field is stored in the index (so it can later be
retrieved using selective loading when searching), and it gets analyzed
(broken down into searchable terms). If the message has a `null` value,
then the value that will be stored is `na`. There is also a `string` `user`
which is indexed as-is (not broken down into tokens) and has norms
disabled (so that matching this field is a binary decision, no match is
better than another one).

The following table lists all the attributes that can be used with the
`string` type:

[cols="<,<",options="header",]
|=======================================================================
|Attribute |Description
|`index_name` |The name of the field that will be stored in the index.
Defaults to the property/field name.

|`store` |Set to `true` to actually store the field in the index, `false` to not
store it. Defaults to `false` (note, the JSON document itself is stored,
and it can be retrieved from it).

|`index` |Set to `analyzed` for the field to be indexed and searchable
after being broken down into token using an analyzer. `not_analyzed`
means that its still searchable, but does not go through any analysis
process or broken down into tokens. `no` means that it won't be
searchable at all (as an individual field; it may still be included in
`_all`). Setting to `no` disables `include_in_all`. Defaults to
`analyzed`.

|`doc_values` |Set to `true` to store field values in a column-stride fashion.
Automatically set to `true` when the <<fielddata-formats,`fielddata` format>> is `doc_values`.

|`term_vector` |Possible values are `no`, `yes`, `with_offsets`,
`with_positions`, `with_positions_offsets`. Defaults to `no`.

|`boost` |The boost value. Defaults to `1.0`.

|`null_value` |When there is a (JSON) null value for the field, use the
`null_value` as the field value. Defaults to not adding the field at
all.

|`norms: {enabled: <value>}` |Boolean value if norms should be enabled or
not. Defaults to `true` for `analyzed` fields, and to `false` for
`not_analyzed` fields. See the <<norms,section about norms>>.

|`norms: {loading: <value>}` |Describes how norms should be loaded, possible values are
`eager` and `lazy` (default). It is possible to change the default value to
eager for all fields by configuring the index setting `index.norms.loading`
to `eager`.

|`index_options` | Allows to set the indexing
options, possible values are `docs` (only doc numbers are indexed),
`freqs` (doc numbers and term frequencies), and `positions` (doc
numbers, term frequencies and positions). Defaults to `positions` for
`analyzed` fields, and to `docs` for `not_analyzed` fields. It
is also possible to set it to `offsets` (doc numbers, term
frequencies, positions and offsets).

|`analyzer` |The analyzer used to analyze the text contents when
`analyzed` during indexing and when searching using a query string.
Defaults to the globally configured analyzer.

|`index_analyzer` |The analyzer used to analyze the text contents when
`analyzed` during indexing.

|`search_analyzer` |The analyzer used to analyze the field when part of
a query string. Can be updated on an existing field.

|`include_in_all` |Should the field be included in the `_all` field (if
enabled). If `index` is set to `no` this defaults to `false`, otherwise,
defaults to `true` or to the parent `object` type setting.

|`ignore_above` |The analyzer will ignore strings larger than this size.
Useful for generic `not_analyzed` fields that should ignore long text.

|`position_offset_gap` |Position increment gap between field instances
with the same field name. Defaults to 0.
|=======================================================================

The `string` type also support custom indexing parameters associated
with the indexed value. For example:

[source,js]
--------------------------------------------------
{
    "message" : {
        "_value":  "boosted value",
        "_boost":  2.0
    }
}
--------------------------------------------------

The mapping is required to disambiguate the meaning of the document.
Otherwise, the structure would interpret "message" as a value of type
"object". The key `_value` (or `value`) in the inner document specifies
the real string content that should eventually be indexed. The `_boost`
(or `boost`) key specifies the per field document boost (here 2.0).

[float]
[[norms]]
===== Norms

Norms store various normalization factors that are later used (at query time)
in order to compute the score of a document relatively to a query.

Although useful for scoring, norms also require quite a lot of memory
(typically in the order of one byte per document per field in your index,
even for documents that don't have this specific field). As a consequence, if
you don't need scoring on a specific field, it is highly recommended to disable
norms on it. In  particular, this is the case for fields that are used solely
for filtering or aggregations.

added[1.2.0]
In case you would like to disable norms after the fact, it is possible to do so
by using the <<indices-put-mapping,PUT mapping API>>. Please however note that
norms won't be removed instantly, but as your index will receive new insertions
or updates, and segments get merged. Any score computation on a field that got
norms removed might return inconsistent results since some documents won't have
norms anymore while other documents might still have norms.

[float]
[[number]]
==== Number

A number based type supporting `float`, `double`, `byte`, `short`,
`integer`, and `long`. It uses specific constructs within Lucene in
order to support numeric values. The number types have the same ranges
as corresponding
http://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html[Java
types]. An example mapping can be:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "rank" : {
                "type" : "float",
                "null_value" : 1.0
            }
        }
    }
}
--------------------------------------------------

The following table lists all the attributes that can be used with a
numbered type:

[cols="<,<",options="header",]
|=======================================================================
|Attribute |Description
|`type` |The type of the number. Can be `float`, `double`, `integer`,
`long`, `short`, `byte`. Required.

|`index_name` |The name of the field that will be stored in the index.
Defaults to the property/field name.

|`store` |Set to `true` to store actual field in the index, `false` to not
store it. Defaults to `false` (note, the JSON document itself is stored,
and it can be retrieved from it).

|`index` |Set to `no` if the value should not be indexed. Setting to
`no` disables `include_in_all`. If set to `no` the field should be either stored
in `_source`, have `include_in_all` enabled, or `store` be set to
`true` for this to be useful.

|`doc_values` |Set to `true` to store field values in a column-stride fashion.
Automatically set to `true` when the fielddata format is `doc_values`.

|`precision_step` |The precision step (influences the number of terms
generated for each number value). Defaults to `16` for `long`, `double`,
`8` for `short`, `integer`, `float`, and `2147483647` for `byte`.

|`boost` |The boost value. Defaults to `1.0`.

|`null_value` |When there is a (JSON) null value for the field, use the
`null_value` as the field value. Defaults to not adding the field at
all.

|`include_in_all` |Should the field be included in the `_all` field (if
enabled). If `index` is set to `no` this defaults to `false`, otherwise,
defaults to `true` or to the parent `object` type setting.

|`ignore_malformed` |Ignored a malformed number. Defaults to `false`.

|`coerce` |Try convert strings to numbers and truncate fractions for integers. Defaults to `true`.

|=======================================================================

[float]
[[token_count]]
==== Token Count
The `token_count` type maps to the JSON string type but indexes and stores
the number of tokens in the string rather than the string itself.  For
example:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "name" : {
                "type" : "string",
                "fields" : {
                    "word_count": {
                        "type" : "token_count",
                        "store" : "yes",
                        "analyzer" : "standard"
                    }
                }
            }
        }
    }
}
--------------------------------------------------

All the configuration that can be specified for a number can be specified
for a token_count.  The only extra configuration is the required
`analyzer` field which specifies which analyzer to use to break the string
into tokens.  For best performance, use an analyzer with no token filters.

[NOTE]
===================================================================
Technically the `token_count` type sums position increments rather than
counting tokens. This means that even if the analyzer filters out stop
words they are included in the count.
===================================================================

[float]
[[date]]
==== Date

The date type is a special type which maps to JSON string type. It
follows a specific format that can be explicitly set. All dates are
`UTC`. Internally, a date maps to a number type `long`, with the added
parsing stage from string to long and from long to string. An example
mapping:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "postDate" : {
                "type" : "date",
                "format" : "YYYY-MM-dd"
            }
        }
    }
}
--------------------------------------------------

The date type will also accept a long number representing UTC
milliseconds since the epoch, regardless of the format it can handle.

The following table lists all the attributes that can be used with a
date type:

[cols="<,<",options="header",]
|=======================================================================
|Attribute |Description
|`index_name` |The name of the field that will be stored in the index.
Defaults to the property/field name.

|`format` |The <<mapping-date-format,date
format>>. Defaults to `dateOptionalTime`.

|`store` |Set to `true` to store actual field in the index, `false` to not
store it. Defaults to `false` (note, the JSON document itself is stored,
and it can be retrieved from it).

|`index` |Set to `no` if the value should not be indexed. Setting to
`no` disables `include_in_all`. If set to `no` the field should be either stored
in `_source`, have `include_in_all` enabled, or `store` be set to
`true` for this to be useful.

|`doc_values` |Set to `true` to store field values in a column-stride fashion.
Automatically set to `true` when the fielddata format is `doc_values`.

|`precision_step` |The precision step (influences the number of terms
generated for each number value). Defaults to `16`.

|`boost` |The boost value. Defaults to `1.0`.

|`null_value` |When there is a (JSON) null value for the field, use the
`null_value` as the field value. Defaults to not adding the field at
all.

|`include_in_all` |Should the field be included in the `_all` field (if
enabled). If `index` is set to `no` this defaults to `false`, otherwise,
defaults to `true` or to the parent `object` type setting.

|`ignore_malformed` |Ignored a malformed number. Defaults to `false`.

|=======================================================================

[float]
[[boolean]]
==== Boolean

The boolean type Maps to the JSON boolean type. It ends up storing
within the index either `T` or `F`, with automatic translation to `true`
and `false` respectively.

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "hes_my_special_tweet" : {
                "type" : "boolean"
            }
        }
    }
}
--------------------------------------------------

The boolean type also supports passing the value as a number or a string
(in this case `0`, an empty string, `false`, `off` and `no` are
`false`, all other values are `true`).

The following table lists all the attributes that can be used with the
boolean type:

[cols="<,<",options="header",]
|=======================================================================
|Attribute |Description
|`index_name` |The name of the field that will be stored in the index.
Defaults to the property/field name.

|`store` |Set to `true` to store actual field in the index, `false` to not
store it. Defaults to `false` (note, the JSON document itself is stored,
and it can be retrieved from it).

|`index` |Set to `no` if the value should not be indexed. Setting to
`no` disables `include_in_all`. If set to `no` the field should be either stored
in `_source`, have `include_in_all` enabled, or `store` be set to
`true` for this to be useful.

|`boost` |The boost value. Defaults to `1.0`.

|`null_value` |When there is a (JSON) null value for the field, use the
`null_value` as the field value. Defaults to not adding the field at
all.
|=======================================================================

[float]
[[binary]]
==== Binary

The binary type is a base64 representation of binary data that can be
stored in the index. The field is not stored by default and not indexed at
all.

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "image" : {
                "type" : "binary"
            }
        }
    }
}
--------------------------------------------------

The following table lists all the attributes that can be used with the
binary type:

[horizontal]

`index_name`::

    The name of the field that will be stored in the index. Defaults to the
    property/field name.

`store`::

    Set to `true` to store actual field in the index, `false` to not store it.
    Defaults to `false` (note, the JSON document itself is already stored, so
    the binary field can be retrieved from there).

`doc_values`::

    Set to `true` to store field values in a column-stride fashion.

`compress`::

    Set to `true` to compress the stored binary value.

`compress_threshold`::

    Compression will only be applied to stored binary fields that are greater
    than this size. Defaults to `-1`

NOTE: Enabling compression on stored binary fields only makes sense on large
and highly-compressible values. Otherwise per-field compression is usually not
worth doing as the space savings do not compensate for the overhead of the
compression format. Normally, you should not configure any compression and
just rely on the block compression of stored fields (which is enabled by
default and can't be disabled).

[float]
[[fielddata-filters]]
==== Fielddata filters

It is possible to control which field values are loaded into memory,
which is particularly useful for faceting on string fields, using
fielddata filters, which are explained in detail in the
<<index-modules-fielddata,Fielddata>> section.

Fielddata filters can exclude terms which do not match a regex, or which
don't fall between a `min` and `max` frequency range:

[source,js]
--------------------------------------------------
{
    tweet: {
        type:      "string",
        analyzer:  "whitespace"
        fielddata: {
            filter: {
                regex: {
                    "pattern":        "^#.*"
                },
                frequency: {
                    min:              0.001,
                    max:              0.1,
                    min_segment_size: 500
                }
            }
        }
    }
}
--------------------------------------------------

These filters can be updated on an existing field mapping and will take
effect the next time the fielddata for a segment is loaded. Use the
<<indices-clearcache,Clear Cache>> API
to reload the fielddata using the new filters.

[float]
==== Similarity

Elasticsearch allows you to configure a similarity (scoring algorithm) per field.
The `similarity` setting provides a simple way of choosing a similarity algorithm
other than the default TF/IDF, such as `BM25`.

You can configure similarities via the
<<index-modules-similarity,similarity module>>

[float]
===== Configuring Similarity per Field

Defining the Similarity for a field is done via the `similarity` mapping
property, as this example shows:

[source,js]
--------------------------------------------------
{
   "book":{
      "properties":{
         "title":{
            "type":"string", "similarity":"BM25"
         }
      }
   }
}
--------------------------------------------------

The following Similarities are configured out-of-box:

`default`::
        The Default TF/IDF algorithm used by Elasticsearch and
        Lucene in previous versions.

`BM25`::
        The BM25 algorithm.
        http://en.wikipedia.org/wiki/Okapi_BM25[See Okapi_BM25] for more
        details.

[[copy-to]]
[float]
===== Copy to field

added[1.0.0.RC2]

Adding `copy_to` parameter to any field mapping will cause all values of this field to be copied to fields specified in
the parameter. In the following example all values from fields `title` and `abstract` will be copied to the field
`meta_data`.

[source,js]
--------------------------------------------------
{
  "book" : {
    "properties" : {
      "title" : { "type" : "string", "copy_to" : "meta_data" },
      "abstract" : { "type" : "string", "copy_to" : "meta_data" },
      "meta_data" : { "type" : "string" }
    }
}
--------------------------------------------------

Multiple fields are also supported:

[source,js]
--------------------------------------------------
{
  "book" : {
    "properties" : {
      "title" : { "type" : "string", "copy_to" : ["meta_data", "article_info"] }
    }
}
--------------------------------------------------

[float]
===== Multi fields

added[1.0.0.RC1]

The `fields` options allows to map several core types fields into a single
json source field. This can be useful if a single field need to be
used in different ways. For example a single field is to be used for both
free text search and sorting.

[source,js]
--------------------------------------------------
{
  "tweet" : {
    "properties" : {
      "name" : {
        "type" : "string",
        "index" : "analyzed",
        "fields" : {
          "raw" : {"type" : "string", "index" : "not_analyzed"}
        }
      }
    }
  }
}
--------------------------------------------------

In the above example the field `name` gets processed twice. The first time it gets
processed as an analyzed string and this version is accessible under the field name
`name`, this is the main field and is in fact just like any other field. The second time
it gets processed as a not analyzed string and is accessible under the name `name.raw`.

[float]
==== Include in All

The `include_in_all` setting is ignored on any field that is defined in
the `fields` options. Setting the `include_in_all` only makes sense on
the main field, since the raw field value is copied to the `_all` field,
the tokens aren't copied.

[float]
==== Updating a field

In the essence a field can't be updated. However multi fields can be
added to existing fields. This allows for example to have a different
`index_analyzer` configuration in addition to the already configured
`index_analyzer` configuration specified in the main and other multi fields.

Also the new multi field will only be applied on document that have been
added after the multi field has been added and in fact the new multi field
doesn't exist in existing documents.

Another important note is that new multi fields will be merged into the
list of existing multi fields, so when adding new multi fields for a field
previous added multi fields don't need to be specified.

[[mapping-core-types]]
=== Core Types

Each JSON field can be mapped to a specific core type. JSON itself
already provides us with some typing, with its support for `string`,
`integer`/`long`, `float`/`double`, `boolean`, and `null`.

The following sample tweet JSON document will be used to explain the
core types:

[source,js]
--------------------------------------------------
{
    "tweet" {
        "user" : "kimchy"
        "message" : "This is a tweet!",
        "postDate" : "2009-11-15T14:12:12",
        "priority" : 4,
        "rank" : 12.3
    }
}
--------------------------------------------------

Explicit mapping for the above JSON tweet can be:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "user" : {"type" : "string", "index" : "not_analyzed"},
            "message" : {"type" : "string", "null_value" : "na"},
            "postDate" : {"type" : "date"},
            "priority" : {"type" : "integer"},
            "rank" : {"type" : "float"}
        }
    }
}
--------------------------------------------------

[float]
[[string]]
==== String

The text based string type is the most basic type, and contains one or
more characters. An example mapping can be:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "message" : {
                "type" : "string",
                "store" : true,
                "index" : "analyzed",
                "null_value" : "na"
            },
            "user" : {
                "type" : "string",
                "index" : "not_analyzed",
                "norms" : {
                    "enabled" : false
                }
            }
        }
    }
}
--------------------------------------------------

The above mapping defines a `string` `message` property/field within the
`tweet` type. The field is stored in the index (so it can later be
retrieved using selective loading when searching), and it gets analyzed
(broken down into searchable terms). If the message has a `null` value,
then the value that will be stored is `na`. There is also a `string` `user`
which is indexed as-is (not broken down into tokens) and has norms
disabled (so that matching this field is a binary decision, no match is
better than another one).

The following table lists all the attributes that can be used with the
`string` type:

[cols="<,<",options="header",]
|=======================================================================
|Attribute |Description
|`index_name` |The name of the field that will be stored in the index.
Defaults to the property/field name.

|`store` |Set to `true` to actually store the field in the index, `false` to not
store it. Defaults to `false` (note, the JSON document itself is stored,
and it can be retrieved from it).

|`index` |Set to `analyzed` for the field to be indexed and searchable
after being broken down into token using an analyzer. `not_analyzed`
means that its still searchable, but does not go through any analysis
process or broken down into tokens. `no` means that it won't be
searchable at all (as an individual field; it may still be included in
`_all`). Setting to `no` disables `include_in_all`. Defaults to
`analyzed`.

|`doc_values` |Set to `true` to store field values in a column-stride fashion.
Automatically set to `true` when the <<fielddata-formats,`fielddata` format>> is `doc_values`.

|`term_vector` |Possible values are `no`, `yes`, `with_offsets`,
`with_positions`, `with_positions_offsets`. Defaults to `no`.

|`boost` |The boost value. Defaults to `1.0`.

|`null_value` |When there is a (JSON) null value for the field, use the
`null_value` as the field value. Defaults to not adding the field at
all.

|`norms: {enabled: <value>}` |Boolean value if norms should be enabled or
not. Defaults to `true` for `analyzed` fields, and to `false` for
`not_analyzed` fields. See the <<norms,section about norms>>.

|`norms: {loading: <value>}` |Describes how norms should be loaded, possible values are
`eager` and `lazy` (default). It is possible to change the default value to
eager for all fields by configuring the index setting `index.norms.loading`
to `eager`.

|`index_options` | Allows to set the indexing
options, possible values are `docs` (only doc numbers are indexed),
`freqs` (doc numbers and term frequencies), and `positions` (doc
numbers, term frequencies and positions). Defaults to `positions` for
`analyzed` fields, and to `docs` for `not_analyzed` fields. It
is also possible to set it to `offsets` (doc numbers, term
frequencies, positions and offsets).

|`analyzer` |The analyzer used to analyze the text contents when
`analyzed` during indexing and when searching using a query string.
Defaults to the globally configured analyzer.

|`index_analyzer` |The analyzer used to analyze the text contents when
`analyzed` during indexing.

|`search_analyzer` |The analyzer used to analyze the field when part of
a query string. Can be updated on an existing field.

|`include_in_all` |Should the field be included in the `_all` field (if
enabled). If `index` is set to `no` this defaults to `false`, otherwise,
defaults to `true` or to the parent `object` type setting.

|`ignore_above` |The analyzer will ignore strings larger than this size.
Useful for generic `not_analyzed` fields that should ignore long text.

|`position_offset_gap` |Position increment gap between field instances
with the same field name. Defaults to 0.
|=======================================================================

The `string` type also support custom indexing parameters associated
with the indexed value. For example:

[source,js]
--------------------------------------------------
{
    "message" : {
        "_value":  "boosted value",
        "_boost":  2.0
    }
}
--------------------------------------------------

The mapping is required to disambiguate the meaning of the document.
Otherwise, the structure would interpret "message" as a value of type
"object". The key `_value` (or `value`) in the inner document specifies
the real string content that should eventually be indexed. The `_boost`
(or `boost`) key specifies the per field document boost (here 2.0).

[float]
[[norms]]
===== Norms

Norms store various normalization factors that are later used (at query time)
in order to compute the score of a document relatively to a query.

Although useful for scoring, norms also require quite a lot of memory
(typically in the order of one byte per document per field in your index,
even for documents that don't have this specific field). As a consequence, if
you don't need scoring on a specific field, it is highly recommended to disable
norms on it. In  particular, this is the case for fields that are used solely
for filtering or aggregations.

added[1.2.0]
In case you would like to disable norms after the fact, it is possible to do so
by using the <<indices-put-mapping,PUT mapping API>>. Please however note that
norms won't be removed instantly, but as your index will receive new insertions
or updates, and segments get merged. Any score computation on a field that got
norms removed might return inconsistent results since some documents won't have
norms anymore while other documents might still have norms.

[float]
[[number]]
==== Number

A number based type supporting `float`, `double`, `byte`, `short`,
`integer`, and `long`. It uses specific constructs within Lucene in
order to support numeric values. The number types have the same ranges
as corresponding
http://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html[Java
types]. An example mapping can be:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "rank" : {
                "type" : "float",
                "null_value" : 1.0
            }
        }
    }
}
--------------------------------------------------

The following table lists all the attributes that can be used with a
numbered type:

[cols="<,<",options="header",]
|=======================================================================
|Attribute |Description
|`type` |The type of the number. Can be `float`, `double`, `integer`,
`long`, `short`, `byte`. Required.

|`index_name` |The name of the field that will be stored in the index.
Defaults to the property/field name.

|`store` |Set to `true` to store actual field in the index, `false` to not
store it. Defaults to `false` (note, the JSON document itself is stored,
and it can be retrieved from it).

|`index` |Set to `no` if the value should not be indexed. Setting to
`no` disables `include_in_all`. If set to `no` the field should be either stored
in `_source`, have `include_in_all` enabled, or `store` be set to
`true` for this to be useful.

|`doc_values` |Set to `true` to store field values in a column-stride fashion.
Automatically set to `true` when the fielddata format is `doc_values`.

|`precision_step` |The precision step (influences the number of terms
generated for each number value). Defaults to `16` for `long`, `double`,
`8` for `short`, `integer`, `float`, and `2147483647` for `byte`.

|`boost` |The boost value. Defaults to `1.0`.

|`null_value` |When there is a (JSON) null value for the field, use the
`null_value` as the field value. Defaults to not adding the field at
all.

|`include_in_all` |Should the field be included in the `_all` field (if
enabled). If `index` is set to `no` this defaults to `false`, otherwise,
defaults to `true` or to the parent `object` type setting.

|`ignore_malformed` |Ignored a malformed number. Defaults to `false`.

|`coerce` |Try convert strings to numbers and truncate fractions for integers. Defaults to `true`.

|=======================================================================

[float]
[[token_count]]
==== Token Count
The `token_count` type maps to the JSON string type but indexes and stores
the number of tokens in the string rather than the string itself.  For
example:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "name" : {
                "type" : "string",
                "fields" : {
                    "word_count": {
                        "type" : "token_count",
                        "store" : "yes",
                        "analyzer" : "standard"
                    }
                }
            }
        }
    }
}
--------------------------------------------------

All the configuration that can be specified for a number can be specified
for a token_count.  The only extra configuration is the required
`analyzer` field which specifies which analyzer to use to break the string
into tokens.  For best performance, use an analyzer with no token filters.

[NOTE]
===================================================================
Technically the `token_count` type sums position increments rather than
counting tokens. This means that even if the analyzer filters out stop
words they are included in the count.
===================================================================

[float]
[[date]]
==== Date

The date type is a special type which maps to JSON string type. It
follows a specific format that can be explicitly set. All dates are
`UTC`. Internally, a date maps to a number type `long`, with the added
parsing stage from string to long and from long to string. An example
mapping:

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "postDate" : {
                "type" : "date",
                "format" : "YYYY-MM-dd"
            }
        }
    }
}
--------------------------------------------------

The date type will also accept a long number representing UTC
milliseconds since the epoch, regardless of the format it can handle.

The following table lists all the attributes that can be used with a
date type:

[cols="<,<",options="header",]
|=======================================================================
|Attribute |Description
|`index_name` |The name of the field that will be stored in the index.
Defaults to the property/field name.

|`format` |The <<mapping-date-format,date
format>>. Defaults to `dateOptionalTime`.

|`store` |Set to `true` to store actual field in the index, `false` to not
store it. Defaults to `false` (note, the JSON document itself is stored,
and it can be retrieved from it).

|`index` |Set to `no` if the value should not be indexed. Setting to
`no` disables `include_in_all`. If set to `no` the field should be either stored
in `_source`, have `include_in_all` enabled, or `store` be set to
`true` for this to be useful.

|`doc_values` |Set to `true` to store field values in a column-stride fashion.
Automatically set to `true` when the fielddata format is `doc_values`.

|`precision_step` |The precision step (influences the number of terms
generated for each number value). Defaults to `16`.

|`boost` |The boost value. Defaults to `1.0`.

|`null_value` |When there is a (JSON) null value for the field, use the
`null_value` as the field value. Defaults to not adding the field at
all.

|`include_in_all` |Should the field be included in the `_all` field (if
enabled). If `index` is set to `no` this defaults to `false`, otherwise,
defaults to `true` or to the parent `object` type setting.

|`ignore_malformed` |Ignored a malformed number. Defaults to `false`.

|=======================================================================

[float]
[[boolean]]
==== Boolean

The boolean type Maps to the JSON boolean type. It ends up storing
within the index either `T` or `F`, with automatic translation to `true`
and `false` respectively.

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "hes_my_special_tweet" : {
                "type" : "boolean"
            }
        }
    }
}
--------------------------------------------------

The boolean type also supports passing the value as a number or a string
(in this case `0`, an empty string, `false`, `off` and `no` are
`false`, all other values are `true`).

The following table lists all the attributes that can be used with the
boolean type:

[cols="<,<",options="header",]
|=======================================================================
|Attribute |Description
|`index_name` |The name of the field that will be stored in the index.
Defaults to the property/field name.

|`store` |Set to `true` to store actual field in the index, `false` to not
store it. Defaults to `false` (note, the JSON document itself is stored,
and it can be retrieved from it).

|`index` |Set to `no` if the value should not be indexed. Setting to
`no` disables `include_in_all`. If set to `no` the field should be either stored
in `_source`, have `include_in_all` enabled, or `store` be set to
`true` for this to be useful.

|`boost` |The boost value. Defaults to `1.0`.

|`null_value` |When there is a (JSON) null value for the field, use the
`null_value` as the field value. Defaults to not adding the field at
all.
|=======================================================================

[float]
[[binary]]
==== Binary

The binary type is a base64 representation of binary data that can be
stored in the index. The field is not stored by default and not indexed at
all.

[source,js]
--------------------------------------------------
{
    "tweet" : {
        "properties" : {
            "image" : {
                "type" : "binary"
            }
        }
    }
}
--------------------------------------------------

The following table lists all the attributes that can be used with the
binary type:

[horizontal]

`index_name`::

    The name of the field that will be stored in the index. Defaults to the
    property/field name.

`store`::

    Set to `true` to store actual field in the index, `false` to not store it.
    Defaults to `false` (note, the JSON document itself is already stored, so
    the binary field can be retrieved from there).

`doc_values`::

    Set to `true` to store field values in a column-stride fashion.

`compress`::

    Set to `true` to compress the stored binary value.

`compress_threshold`::

    Compression will only be applied to stored binary fields that are greater
    than this size. Defaults to `-1`

NOTE: Enabling compression on stored binary fields only makes sense on large
and highly-compressible values. Otherwise per-field compression is usually not
worth doing as the space savings do not compensate for the overhead of the
compression format. Normally, you should not configure any compression and
just rely on the block compression of stored fields (which is enabled by
default and can't be disabled).

[float]
[[fielddata-filters]]
==== Fielddata filters

It is possible to control which field values are loaded into memory,
which is particularly useful for faceting on string fields, using
fielddata filters, which are explained in detail in the
<<index-modules-fielddata,Fielddata>> section.

Fielddata filters can exclude terms which do not match a regex, or which
don't fall between a `min` and `max` frequency range:

[source,js]
--------------------------------------------------
{
    tweet: {
        type:      "string",
        analyzer:  "whitespace"
        fielddata: {
            filter: {
                regex: {
                    "pattern":        "^#.*"
                },
                frequency: {
                    min:              0.001,
                    max:              0.1,
                    min_segment_size: 500
                }
            }
        }
    }
}
--------------------------------------------------

These filters can be updated on an existing field mapping and will take
effect the next time the fielddata for a segment is loaded. Use the
<<indices-clearcache,Clear Cache>> API
to reload the fielddata using the new filters.

[float]
==== Similarity

Elasticsearch allows you to configure a similarity (scoring algorithm) per field.
The `similarity` setting provides a simple way of choosing a similarity algorithm
other than the default TF/IDF, such as `BM25`.

You can configure similarities via the
<<index-modules-similarity,similarity module>>

[float]
===== Configuring Similarity per Field

Defining the Similarity for a field is done via the `similarity` mapping
property, as this example shows:

[source,js]
--------------------------------------------------
{
   "book":{
      "properties":{
         "title":{
            "type":"string", "similarity":"BM25"
         }
      }
   }
}
--------------------------------------------------

The following Similarities are configured out-of-box:

`default`::
        The Default TF/IDF algorithm used by Elasticsearch and
        Lucene in previous versions.

`BM25`::
        The BM25 algorithm.
        http://en.wikipedia.org/wiki/Okapi_BM25[See Okapi_BM25] for more
        details.

[[copy-to]]
[float]
===== Copy to field

added[1.0.0.RC2]

Adding `copy_to` parameter to any field mapping will cause all values of this field to be copied to fields specified in
the parameter. In the following example all values from fields `title` and `abstract` will be copied to the field
`meta_data`.

[source,js]
--------------------------------------------------
{
  "book" : {
    "properties" : {
      "title" : { "type" : "string", "copy_to" : "meta_data" },
      "abstract" : { "type" : "string", "copy_to" : "meta_data" },
      "meta_data" : { "type" : "string" }
    }
}
--------------------------------------------------

Multiple fields are also supported:

[source,js]
--------------------------------------------------
{
  "book" : {
    "properties" : {
      "title" : { "type" : "string", "copy_to" : ["meta_data", "article_info"] }
    }
}
--------------------------------------------------

[float]
===== Multi fields

added[1.0.0.RC1]

The `fields` options allows to map several core types fields into a single
json source field. This can be useful if a single field need to be
used in different ways. For example a single field is to be used for both
free text search and sorting.

[source,js]
--------------------------------------------------
{
  "tweet" : {
    "properties" : {
      "name" : {
        "type" : "string",
        "index" : "analyzed",
        "fields" : {
          "raw" : {"type" : "string", "index" : "not_analyzed"}
        }
      }
    }
  }
}
--------------------------------------------------

In the above example the field `name` gets processed twice. The first time it gets
processed as an analyzed string and this version is accessible under the field name
`name`, this is the main field and is in fact just like any other field. The second time
it gets processed as a not analyzed string and is accessible under the name `name.raw`.

[float]
==== Include in All

The `include_in_all` setting is ignored on any field that is defined in
the `fields` options. Setting the `include_in_all` only makes sense on
the main field, since the raw field value is copied to the `_all` field,
the tokens aren't copied.

[float]
==== Updating a field

In the essence a field can't be updated. However multi fields can be
added to existing fields. This allows for example to have a different
`index_analyzer` configuration in addition to the already configured
`index_analyzer` configuration specified in the main and other multi fields.

Also the new multi field will only be applied on document that have been
added after the multi field has been added and in fact the new multi field
doesn't exist in existing documents.

Another important note is that new multi fields will be merged into the
list of existing multi fields, so when adding new multi fields for a field
previous added multi fields don't need to be specified.


docs: add best practises for wildcard queries inside percolator queries

[[percolator]]
=== Percolator type

The `percolator` field type parses a json structure into a native query and
stores that query, so that the <<query-dsl-percolate-query,percolate query>>
can use it to match provided documents.

Any field that contains a json object can be configured to be a percolator
field. The percolator field type has no settings. Just configuring the `percolator`
field type is sufficient to instruct Elasticsearch to treat a field as a
query.

If the following mapping configures the `percolator` field type for the
`query` field:

[source,js]
--------------------------------------------------
PUT my_index
{
    "mappings": {
        "_doc": {
            "properties": {
                "query": {
                    "type": "percolator"
                },
                "field": {
                    "type": "text"
                }
            }
        }
    }
}
--------------------------------------------------
// CONSOLE
// TESTSETUP

Then you can index a query:

[source,js]
--------------------------------------------------
PUT my_index/_doc/match_value
{
    "query" : {
        "match" : {
            "field" : "value"
        }
    }
}
--------------------------------------------------
// CONSOLE

[IMPORTANT]
=====================================

Fields referred to in a percolator query must *already* exist in the mapping
associated with the index used for percolation. In order to make sure these fields exist,
add or update a mapping via the <<indices-create-index,create index>> or <<indices-put-mapping,put mapping>> APIs.
Fields referred in a percolator query may exist in any type of the index containing the `percolator` field type.

=====================================

[float]
==== Reindexing your percolator queries

Reindexing percolator queries is sometimes required to benefit from improvements made to the `percolator` field type in
new releases.

Reindexing percolator queries can be reindexed by using the <<docs-reindex,reindex api>>.
Lets take a look at the following index with a percolator field type:

[source,js]
--------------------------------------------------
PUT index
{
  "mappings": {
    "_doc" : {
      "properties": {
        "query" : {
          "type" : "percolator"
        },
        "body" : {
          "type": "text"
        }
      }
    }
  }
}

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "index",
        "alias": "queries" <1>
      }
    }
  ]
}

PUT queries/_doc/1?refresh
{
  "query" : {
    "match" : {
      "body" : "quick brown fox"
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> It is always recommended to define an alias for your index, so that in case of a reindex systems / applications
    don't need to be changed to know that the percolator queries are now in a different index.

Lets say you're going to upgrade to a new major version and in order for the new Elasticsearch version to still be able
to read your queries you need to reindex your queries into a new index on the current Elasticsearch version:

[source,js]
--------------------------------------------------
PUT new_index
{
  "mappings": {
    "_doc" : {
      "properties": {
        "query" : {
          "type" : "percolator"
        },
        "body" : {
          "type": "text"
        }
      }
    }
  }
}

POST /_reindex?refresh
{
  "source": {
    "index": "index"
  },
  "dest": {
    "index": "new_index"
  }
}

POST _aliases
{
  "actions": [ <1>
    {
      "remove": {
        "index" : "index",
        "alias": "queries"
      }
    },
    {
      "add": {
        "index": "new_index",
        "alias": "queries"
      }
    }
  ]
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> If you have an alias don't forget to point it to the new index.

Executing the `percolate` query via the `queries` alias:

[source,js]
--------------------------------------------------
GET /queries/_search
{
  "query": {
    "percolate" : {
      "field" : "query",
      "document" : {
        "body" : "fox jumps over the lazy dog"
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

now returns matches from the new index:

[source,js]
--------------------------------------------------
{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped" : 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "new_index", <1>
        "_type": "_doc",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "query": {
            "match": {
              "body": "quick brown fox"
            }
          }
        },
        "fields" : {
          "_percolator_document_slot" : [0]
        }
      }
    ]
  }
}
--------------------------------------------------
// TESTRESPONSE[s/"took": 3,/"took": "$body.took",/]

<1> Percolator query hit is now being presented from the new index.

[float]
==== Optimizing query time text analysis

When the percolator verifies a percolator candidate match it is going to parse, perform query time text analysis and actually run
the percolator query on the document being percolated. This is done for each candidate match and every time the `percolate` query executes.
If your query time text analysis is relatively expensive part of query parsing then text analysis can become the
dominating factor time is being spent on when percolating. This query parsing overhead can become noticeable when the
percolator ends up verifying many candidate percolator query matches.

To avoid the most expensive part of text analysis at percolate time. One can choose to do the expensive part of text analysis
when indexing the percolator query. This requires using two different analyzers. The first analyzer actually performs
text analysis that needs be performed (expensive part). The second analyzer (usually whitespace) just splits the generated tokens
that the first analyzer has produced. Then before indexing a percolator query, the analyze api should be used to analyze the query
text with the more expensive analyzer. The result of the analyze api, the tokens, should be used to substitute the original query
text in the percolator query. It is important that the query should now be configured to override the analyzer from the mapping and
just the second analyzer. Most text based queries support an `analyzer` option (`match`, `query_string`, `simple_query_string`).
Using this approach the expensive text analysis is performed once instead of many times.

Lets demonstrate this workflow via a simplified example.

Lets say we want to index the following percolator query:

[source,js]
--------------------------------------------------
{
  "query" : {
    "match" : {
      "body" : {
        "query" : "missing bicycles"
      }
    }
  }
}
--------------------------------------------------
// NOTCONSOLE

with these settings and mapping:

[source,js]
--------------------------------------------------
PUT /test_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer" : {
          "tokenizer": "standard",
          "filter" : ["lowercase", "porter_stem"]
        }
      }
    }
  },
  "mappings": {
    "_doc" : {
      "properties": {
        "query" : {
          "type": "percolator"
        },
        "body" : {
          "type": "text",
          "analyzer": "my_analyzer" <1>
        }
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> For the purpose of this example, this analyzer is considered expensive.

First we need to use the analyze api to perform the text analysis prior to indexing:

[source,js]
--------------------------------------------------
POST /test_index/_analyze
{
  "analyzer" : "my_analyzer",
  "text" : "missing bicycles"
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

This results the following response:

[source,js]
--------------------------------------------------
{
  "tokens": [
    {
      "token": "miss",
      "start_offset": 0,
      "end_offset": 7,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "bicycl",
      "start_offset": 8,
      "end_offset": 16,
      "type": "<ALPHANUM>",
      "position": 1
    }
  ]
}
--------------------------------------------------
// TESTRESPONSE

All the tokens in the returned order need to replace the query text in the percolator query:

[source,js]
--------------------------------------------------
PUT /test_index/_doc/1?refresh
{
  "query" : {
    "match" : {
      "body" : {
        "query" : "miss bicycl",
        "analyzer" : "whitespace" <1>
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> It is important to select a whitespace analyzer here, otherwise the analyzer defined in the mapping will be used,
which defeats the point of using this workflow. Note that `whitespace` is a built-in analyzer, if a different analyzer
needs to be used, it needs to be configured first in the index's settings.

The analyze api prior to the indexing the percolator flow should be done for each percolator query.

At percolate time nothing changes and the `percolate` query can be defined normally:

[source,js]
--------------------------------------------------
GET /test_index/_search
{
  "query": {
    "percolate" : {
      "field" : "query",
      "document" : {
        "body" : "Bycicles are missing"
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

This results in a response like this:

[source,js]
--------------------------------------------------
{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped" : 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "test_index",
        "_type": "_doc",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "query": {
            "match": {
              "body": {
                "query": "miss bicycl",
                "analyzer": "whitespace"
              }
            }
          }
        },
        "fields" : {
          "_percolator_document_slot" : [0]
        }
      }
    ]
  }
}
--------------------------------------------------
// TESTRESPONSE[s/"took": 6,/"took": "$body.took",/]

[float]
==== Optimizing wildcard queries.

Wildcard queries are  more expensive than other queries for the percolator,
especially if the wildcard expressions are large.

In the case of `wildcard` queries with prefix wildcard expressions or just the `prefix` query,
the `edge_ngram` token filter can be used to replace these queries with regular `term`
query on a field where the `edge_ngram` token filter is configured.

Creating an index with custom analysis settings:

[source,js]
--------------------------------------------------
PUT my_queries1
{
  "settings": {
    "analysis": {
      "analyzer": {
        "wildcard_prefix": { <1>
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "standard",
            "lowercase",
            "wildcard_edge_ngram"
          ]
        }
      },
      "filter": {
        "wildcard_edge_ngram": { <2>
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 32
        }
      }
    }
  },
  "mappings": {
    "query": {
      "properties": {
        "query": {
          "type": "percolator"
        },
        "my_field": {
          "type": "text",
          "fields": {
            "prefix": { <3>
              "type": "text",
              "analyzer": "wildcard_prefix",
              "search_analyzer": "standard"
            }
          }
        }
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> The analyzer that generates the prefix tokens to be used at index time only.
<2> Increase the `min_gram` and decrease `max_gram` settings based on your prefix search needs.
<3> This multifield should be used to do the prefix search
    with a `term` or `match` query instead of a `prefix` or `wildcard` query.

Then instead of indexing the following query:

[source,js]
--------------------------------------------------
{
  "query": {
    "wildcard": {
      "my_field": "abc*"
    }
  }
}
--------------------------------------------------
// NOTCONSOLE

this query below should be indexed:

[source,js]
--------------------------------------------------
PUT /my_queries1/query/1?refresh
{
  "query": {
    "term": {
      "my_field.prefix": "abc"
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

This way can handle the second query more efficiently than the first query.

The following search request will match with the previously indexed
percolator query:

[source,js]
--------------------------------------------------
GET /my_queries1/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "my_field": "abcd"
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

[source,js]
--------------------------------------------------
{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.41501677,
    "hits": [
      {
        "_index": "my_queries1",
        "_type": "query",
        "_id": "1",
        "_score": 0.41501677,
        "_source": {
          "query": {
            "term": {
              "my_field.prefix": "abc"
            }
          }
        },
        "fields": {
          "_percolator_document_slot": [
            0
          ]
        }
      }
    ]
  }
}
--------------------------------------------------
// TESTRESPONSE[s/"took": 6,/"took": "$body.took",/]

The same technique can also be used to speed up suffix
wildcard searches. By using the `reverse` token filter
before the `edge_ngram` token filter.

[source,js]
--------------------------------------------------
PUT my_queries2
{
  "settings": {
    "analysis": {
      "analyzer": {
        "wildcard_suffix": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "standard",
            "lowercase",
            "reverse",
            "wildcard_edge_ngram"
          ]
        },
        "wildcard_suffix_search_time": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "standard",
            "lowercase",
            "reverse"
          ]
        }
      },
      "filter": {
        "wildcard_edge_ngram": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 32
        }
      }
    }
  },
  "mappings": {
    "query": {
      "properties": {
        "query": {
          "type": "percolator"
        },
        "my_field": {
          "type": "text",
          "fields": {
            "suffix": {
              "type": "text",
              "analyzer": "wildcard_suffix",
              "search_analyzer": "wildcard_suffix_search_time" <1>
            }
          }
        }
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> A custom analyzer is needed at search time too, because otherwise
    the query terms are not being reversed and would otherwise not match
    with the reserved suffix tokens.

Then instead of indexing the following query:

[source,js]
--------------------------------------------------
{
  "query": {
    "wildcard": {
      "my_field": "*xyz"
    }
  }
}
--------------------------------------------------
// NOTCONSOLE

the following query below should be indexed:

[source,js]
--------------------------------------------------
PUT /my_queries2/query/2?refresh
{
  "query": {
    "match": { <1>
      "my_field.suffix": "xyz"
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> The `match` query should be used instead of the `term` query,
    because text analysis needs to reverse the query terms.

The following search request will match with the previously indexed
percolator query:

[source,js]
--------------------------------------------------
GET /my_queries2/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "my_field": "wxyz"
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

[float]
==== Dedicated Percolator Index

Percolate queries can be added to any index. Instead of adding percolate queries to the index the data resides in,
these queries can also be added to a dedicated index. The advantage of this is that this dedicated percolator index
can have its own index settings (For example the number of primary and replica shards). If you choose to have a dedicated
percolate index, you need to make sure that the mappings from the normal index are also available on the percolate index.
Otherwise percolate queries can be parsed incorrectly.

[float]
==== Forcing Unmapped Fields to be Handled as Strings

In certain cases it is unknown what kind of percolator queries do get registered, and if no field mapping exists for fields
that are referred by percolator queries then adding a percolator query fails. This means the mapping needs to be updated
to have the field with the appropriate settings, and then the percolator query can be added. But sometimes it is sufficient
if all unmapped fields are handled as if these were default text fields. In those cases one can configure the
`index.percolator.map_unmapped_fields_as_text` setting to `true` (default to `false`) and then if a field referred in
a percolator query does not exist, it will be handled as a default text field so that adding the percolator query doesn't
fail.

[float]
==== Limitations

[float]
===== Parent/child

Because the `percolate` query is processing one document at a time, it doesn't support queries and filters that run
against child documents such as `has_child` and `has_parent`.

[float]
===== Fetching queries

There are a number of queries that fetch data via a get call during query parsing. For example the `terms` query when
using terms lookup, `template` query when using indexed scripts and `geo_shape` when using pre-indexed shapes. When these
queries are indexed by the `percolator` field type then the get call is executed once. So each time the `percolator`
query evaluates these queries, the fetches terms, shapes etc. as the were upon index time will be used. Important to note
is that fetching of terms that these queries do, happens both each time the percolator query gets indexed on both primary
and replica shards, so the terms that are actually indexed can be different between shard copies, if the source index
changed while indexing.

[float]
===== Script query

The script inside a `script` query can only access doc values fields. The `percolate` query indexes the provided document
into an in-memory index. This in-memory index doesn't support stored fields and because of that the `_source` field and
other stored fields are not stored. This is the reason why in the `script` query the `_source` and other stored fields
aren't available.

[[percolator]]
=== Percolator type

The `percolator` field type parses a json structure into a native query and
stores that query, so that the <<query-dsl-percolate-query,percolate query>>
can use it to match provided documents.

Any field that contains a json object can be configured to be a percolator
field. The percolator field type has no settings. Just configuring the `percolator`
field type is sufficient to instruct Elasticsearch to treat a field as a
query.

If the following mapping configures the `percolator` field type for the
`query` field:

[source,js]
--------------------------------------------------
PUT my_index
{
    "mappings": {
        "_doc": {
            "properties": {
                "query": {
                    "type": "percolator"
                },
                "field": {
                    "type": "text"
                }
            }
        }
    }
}
--------------------------------------------------
// CONSOLE
// TESTSETUP

Then you can index a query:

[source,js]
--------------------------------------------------
PUT my_index/_doc/match_value
{
    "query" : {
        "match" : {
            "field" : "value"
        }
    }
}
--------------------------------------------------
// CONSOLE

[IMPORTANT]
=====================================

Fields referred to in a percolator query must *already* exist in the mapping
associated with the index used for percolation. In order to make sure these fields exist,
add or update a mapping via the <<indices-create-index,create index>> or <<indices-put-mapping,put mapping>> APIs.
Fields referred in a percolator query may exist in any type of the index containing the `percolator` field type.

=====================================

[float]
==== Reindexing your percolator queries

Reindexing percolator queries is sometimes required to benefit from improvements made to the `percolator` field type in
new releases.

Reindexing percolator queries can be reindexed by using the <<docs-reindex,reindex api>>.
Lets take a look at the following index with a percolator field type:

[source,js]
--------------------------------------------------
PUT index
{
  "mappings": {
    "_doc" : {
      "properties": {
        "query" : {
          "type" : "percolator"
        },
        "body" : {
          "type": "text"
        }
      }
    }
  }
}

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "index",
        "alias": "queries" <1>
      }
    }
  ]
}

PUT queries/_doc/1?refresh
{
  "query" : {
    "match" : {
      "body" : "quick brown fox"
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> It is always recommended to define an alias for your index, so that in case of a reindex systems / applications
    don't need to be changed to know that the percolator queries are now in a different index.

Lets say you're going to upgrade to a new major version and in order for the new Elasticsearch version to still be able
to read your queries you need to reindex your queries into a new index on the current Elasticsearch version:

[source,js]
--------------------------------------------------
PUT new_index
{
  "mappings": {
    "_doc" : {
      "properties": {
        "query" : {
          "type" : "percolator"
        },
        "body" : {
          "type": "text"
        }
      }
    }
  }
}

POST /_reindex?refresh
{
  "source": {
    "index": "index"
  },
  "dest": {
    "index": "new_index"
  }
}

POST _aliases
{
  "actions": [ <1>
    {
      "remove": {
        "index" : "index",
        "alias": "queries"
      }
    },
    {
      "add": {
        "index": "new_index",
        "alias": "queries"
      }
    }
  ]
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> If you have an alias don't forget to point it to the new index.

Executing the `percolate` query via the `queries` alias:

[source,js]
--------------------------------------------------
GET /queries/_search
{
  "query": {
    "percolate" : {
      "field" : "query",
      "document" : {
        "body" : "fox jumps over the lazy dog"
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

now returns matches from the new index:

[source,js]
--------------------------------------------------
{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped" : 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "new_index", <1>
        "_type": "_doc",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "query": {
            "match": {
              "body": "quick brown fox"
            }
          }
        },
        "fields" : {
          "_percolator_document_slot" : [0]
        }
      }
    ]
  }
}
--------------------------------------------------
// TESTRESPONSE[s/"took": 3,/"took": "$body.took",/]

<1> Percolator query hit is now being presented from the new index.

[float]
==== Optimizing query time text analysis

When the percolator verifies a percolator candidate match it is going to parse, perform query time text analysis and actually run
the percolator query on the document being percolated. This is done for each candidate match and every time the `percolate` query executes.
If your query time text analysis is relatively expensive part of query parsing then text analysis can become the
dominating factor time is being spent on when percolating. This query parsing overhead can become noticeable when the
percolator ends up verifying many candidate percolator query matches.

To avoid the most expensive part of text analysis at percolate time. One can choose to do the expensive part of text analysis
when indexing the percolator query. This requires using two different analyzers. The first analyzer actually performs
text analysis that needs be performed (expensive part). The second analyzer (usually whitespace) just splits the generated tokens
that the first analyzer has produced. Then before indexing a percolator query, the analyze api should be used to analyze the query
text with the more expensive analyzer. The result of the analyze api, the tokens, should be used to substitute the original query
text in the percolator query. It is important that the query should now be configured to override the analyzer from the mapping and
just the second analyzer. Most text based queries support an `analyzer` option (`match`, `query_string`, `simple_query_string`).
Using this approach the expensive text analysis is performed once instead of many times.

Lets demonstrate this workflow via a simplified example.

Lets say we want to index the following percolator query:

[source,js]
--------------------------------------------------
{
  "query" : {
    "match" : {
      "body" : {
        "query" : "missing bicycles"
      }
    }
  }
}
--------------------------------------------------
// NOTCONSOLE

with these settings and mapping:

[source,js]
--------------------------------------------------
PUT /test_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer" : {
          "tokenizer": "standard",
          "filter" : ["lowercase", "porter_stem"]
        }
      }
    }
  },
  "mappings": {
    "_doc" : {
      "properties": {
        "query" : {
          "type": "percolator"
        },
        "body" : {
          "type": "text",
          "analyzer": "my_analyzer" <1>
        }
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> For the purpose of this example, this analyzer is considered expensive.

First we need to use the analyze api to perform the text analysis prior to indexing:

[source,js]
--------------------------------------------------
POST /test_index/_analyze
{
  "analyzer" : "my_analyzer",
  "text" : "missing bicycles"
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

This results the following response:

[source,js]
--------------------------------------------------
{
  "tokens": [
    {
      "token": "miss",
      "start_offset": 0,
      "end_offset": 7,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "bicycl",
      "start_offset": 8,
      "end_offset": 16,
      "type": "<ALPHANUM>",
      "position": 1
    }
  ]
}
--------------------------------------------------
// TESTRESPONSE

All the tokens in the returned order need to replace the query text in the percolator query:

[source,js]
--------------------------------------------------
PUT /test_index/_doc/1?refresh
{
  "query" : {
    "match" : {
      "body" : {
        "query" : "miss bicycl",
        "analyzer" : "whitespace" <1>
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

<1> It is important to select a whitespace analyzer here, otherwise the analyzer defined in the mapping will be used,
which defeats the point of using this workflow. Note that `whitespace` is a built-in analyzer, if a different analyzer
needs to be used, it needs to be configured first in the index's settings.

The analyze api prior to the indexing the percolator flow should be done for each percolator query.

At percolate time nothing changes and the `percolate` query can be defined normally:

[source,js]
--------------------------------------------------
GET /test_index/_search
{
  "query": {
    "percolate" : {
      "field" : "query",
      "document" : {
        "body" : "Bycicles are missing"
      }
    }
  }
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

This results in a response like this:

[source,js]
--------------------------------------------------
{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped" : 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "test_index",
        "_type": "_doc",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "query": {
            "match": {
              "body": {
                "query": "miss bicycl",
                "analyzer": "whitespace"
              }
            }
          }
        },
        "fields" : {
          "_percolator_document_slot" : [0]
        }
      }
    ]
  }
}
--------------------------------------------------
// TESTRESPONSE[s/"took": 6,/"took": "$body.took",/]

[float]
==== Dedicated Percolator Index

Percolate queries can be added to any index. Instead of adding percolate queries to the index the data resides in,
these queries can also be added to a dedicated index. The advantage of this is that this dedicated percolator index
can have its own index settings (For example the number of primary and replica shards). If you choose to have a dedicated
percolate index, you need to make sure that the mappings from the normal index are also available on the percolate index.
Otherwise percolate queries can be parsed incorrectly.

[float]
==== Forcing Unmapped Fields to be Handled as Strings

In certain cases it is unknown what kind of percolator queries do get registered, and if no field mapping exists for fields
that are referred by percolator queries then adding a percolator query fails. This means the mapping needs to be updated
to have the field with the appropriate settings, and then the percolator query can be added. But sometimes it is sufficient
if all unmapped fields are handled as if these were default text fields. In those cases one can configure the
`index.percolator.map_unmapped_fields_as_text` setting to `true` (default to `false`) and then if a field referred in
a percolator query does not exist, it will be handled as a default text field so that adding the percolator query doesn't
fail.

[float]
==== Limitations

[float]
===== Parent/child

Because the `percolate` query is processing one document at a time, it doesn't support queries and filters that run
against child documents such as `has_child` and `has_parent`.

[float]
===== Fetching queries

There are a number of queries that fetch data via a get call during query parsing. For example the `terms` query when
using terms lookup, `template` query when using indexed scripts and `geo_shape` when using pre-indexed shapes. When these
queries are indexed by the `percolator` field type then the get call is executed once. So each time the `percolator`
query evaluates these queries, the fetches terms, shapes etc. as the were upon index time will be used. Important to note
is that fetching of terms that these queries do, happens both each time the percolator query gets indexed on both primary
and replica shards, so the terms that are actually indexed can be different between shard copies, if the source index
changed while indexing.

[float]
===== Script query

The script inside a `script` query can only access doc values fields. The `percolate` query indexes the provided document
into an in-memory index. This in-memory index doesn't support stored fields and because of that the `_source` field and
other stored fields are not stored. This is the reason why in the `script` query the `_source` and other stored fields
aren't available.


Groovy is no longer the default scripting language

Closes #21208

[[modules-scripting-groovy]]
=== Groovy Scripting Language

deprecated[5.0.0,Groovy will be replaced by the new scripting language <<modules-scripting-painless, `Painless`>>]

Groovy is available in Elasticsearch by default.  Although
limited by the <<java-security-manager,Java Security Manager>>, it is not a
sandboxed language and only `file` scripts may be used by default.

Enabling `inline` or `stored` Groovy scripting is a security risk and should
only be considered if your Elasticsearch cluster is protected from the outside
world. Even a simple `while (true) { }` loop could behave as a denial-of-
service attack on your cluster.

See <<modules-scripting-security, Scripting and Security>> for details
on security issues with scripts, including how to customize class
whitelisting.

[float]
=== Doc value properties and methods

Doc values in Groovy support the following properties and methods (depending
on the underlying field type):

`doc['field_name'].value`::
    The native value of the field. For example, if its a short type, it will be short.

`doc['field_name'].values`::
    The native array values of the field. For example, if its a short type,
     it will be short[]. Remember, a field can have several values within a
     single doc. Returns an empty array if the field has no values.

`doc['field_name'].empty`::
    A boolean indicating if the field has no values within the doc.

`doc['field_name'].lat`::
    The latitude of a geo point type, or `null`.

`doc['field_name'].lon`::
    The longitude of a geo point type, or `null`.

`doc['field_name'].lats`::
    The latitudes of a geo point type, or an empty array.

`doc['field_name'].lons`::
    The longitudes of a geo point type, or an empty array.

`doc['field_name'].arcDistance(lat, lon)`::
    The `arc` distance (in meters) of this geo point field from the provided lat/lon.

`doc['field_name'].arcDistanceWithDefault(lat, lon, default)`::
    The `arc` distance (in meters) of this geo point field from the provided lat/lon with a default value
    for empty fields.

`doc['field_name'].planeDistance(lat, lon)`::
    The `plane` distance (in meters) of this geo point field from the provided lat/lon.

`doc['field_name'].planeDistanceWithDefault(lat, lon, default)`::
    The `plane` distance (in meters) of this geo point field from the provided lat/lon with a default value
    for empty fields.

`doc['field_name'].geohashDistance(geohash)`::
    The `arc` distance (in meters) of this geo point field from the provided geohash.

`doc['field_name'].geohashDistanceWithDefault(geohash, default)`::
    The `arc` distance (in meters) of this geo point field from the provided geohash with a default value
    for empty fields.

[float]
=== Groovy Built In Functions

There are several built in functions that can be used within scripts.
They include:

[cols="<,<",options="header",]
|=======================================================================
|Function |Description
|`sin(a)` |Returns the trigonometric sine of an angle.

|`cos(a)` |Returns the trigonometric cosine of an angle.

|`tan(a)` |Returns the trigonometric tangent of an angle.

|`asin(a)` |Returns the arc sine of a value.

|`acos(a)` |Returns the arc cosine of a value.

|`atan(a)` |Returns the arc tangent of a value.

|`toRadians(angdeg)` |Converts an angle measured in degrees to an
approximately equivalent angle measured in radians

|`toDegrees(angrad)` |Converts an angle measured in radians to an
approximately equivalent angle measured in degrees.

|`exp(a)` |Returns Euler's number _e_ raised to the power of value.

|`log(a)` |Returns the natural logarithm (base _e_) of a value.

|`log10(a)` |Returns the base 10 logarithm of a value.

|`sqrt(a)` |Returns the correctly rounded positive square root of a
value.

|`cbrt(a)` |Returns the cube root of a double value.

|`IEEEremainder(f1, f2)` |Computes the remainder operation on two
arguments as prescribed by the IEEE 754 standard.

|`ceil(a)` |Returns the smallest (closest to negative infinity) value
that is greater than or equal to the argument and is equal to a
mathematical integer.

|`floor(a)` |Returns the largest (closest to positive infinity) value
that is less than or equal to the argument and is equal to a
mathematical integer.

|`rint(a)` |Returns the value that is closest in value to the argument
and is equal to a mathematical integer.

|`atan2(y, x)` |Returns the angle _theta_ from the conversion of
rectangular coordinates (_x_, _y_) to polar coordinates (r,_theta_).

|`pow(a, b)` |Returns the value of the first argument raised to the
power of the second argument.

|`round(a)` |Returns the closest _int_ to the argument.

|`random()` |Returns a random _double_ value.

|`abs(a)` |Returns the absolute value of a value.

|`max(a, b)` |Returns the greater of two values.

|`min(a, b)` |Returns the smaller of two values.

|`ulp(d)` |Returns the size of an ulp of the argument.

|`signum(d)` |Returns the signum function of the argument.

|`sinh(x)` |Returns the hyperbolic sine of a value.

|`cosh(x)` |Returns the hyperbolic cosine of a value.

|`tanh(x)` |Returns the hyperbolic tangent of a value.

|`hypot(x, y)` |Returns sqrt(_x2_ + _y2_) without intermediate overflow
or underflow.
|=======================================================================

[[modules-scripting-groovy]]
=== Groovy Scripting Language

deprecated[5.0.0,Groovy will be replaced by the new scripting language <<modules-scripting-painless, `Painless`>>]

Groovy is the default scripting language available in Elasticsearch.  Although
limited by the <<java-security-manager,Java Security Manager>>, it is not a
sandboxed language and only `file` scripts may be used by default.

Enabling `inline` or `stored` Groovy scripting is a security risk and should
only be considered if your Elasticsearch cluster is protected from the outside
world. Even a simple `while (true) { }` loop could behave as a denial-of-
service attack on your cluster.

See <<modules-scripting-security, Scripting and Security>> for details
on security issues with scripts, including how to customize class
whitelisting.

[float]
=== Doc value properties and methods

Doc values in Groovy support the following properties and methods (depending
on the underlying field type):

`doc['field_name'].value`::
    The native value of the field. For example, if its a short type, it will be short.

`doc['field_name'].values`::
    The native array values of the field. For example, if its a short type,
     it will be short[]. Remember, a field can have several values within a
     single doc. Returns an empty array if the field has no values.

`doc['field_name'].empty`::
    A boolean indicating if the field has no values within the doc.

`doc['field_name'].lat`::
    The latitude of a geo point type, or `null`.

`doc['field_name'].lon`::
    The longitude of a geo point type, or `null`.

`doc['field_name'].lats`::
    The latitudes of a geo point type, or an empty array.

`doc['field_name'].lons`::
    The longitudes of a geo point type, or an empty array.

`doc['field_name'].arcDistance(lat, lon)`::
    The `arc` distance (in meters) of this geo point field from the provided lat/lon.

`doc['field_name'].arcDistanceWithDefault(lat, lon, default)`::
    The `arc` distance (in meters) of this geo point field from the provided lat/lon with a default value
    for empty fields.

`doc['field_name'].planeDistance(lat, lon)`::
    The `plane` distance (in meters) of this geo point field from the provided lat/lon.

`doc['field_name'].planeDistanceWithDefault(lat, lon, default)`::
    The `plane` distance (in meters) of this geo point field from the provided lat/lon with a default value
    for empty fields.

`doc['field_name'].geohashDistance(geohash)`::
    The `arc` distance (in meters) of this geo point field from the provided geohash.

`doc['field_name'].geohashDistanceWithDefault(geohash, default)`::
    The `arc` distance (in meters) of this geo point field from the provided geohash with a default value
    for empty fields.

[float]
=== Groovy Built In Functions

There are several built in functions that can be used within scripts.
They include:

[cols="<,<",options="header",]
|=======================================================================
|Function |Description
|`sin(a)` |Returns the trigonometric sine of an angle.

|`cos(a)` |Returns the trigonometric cosine of an angle.

|`tan(a)` |Returns the trigonometric tangent of an angle.

|`asin(a)` |Returns the arc sine of a value.

|`acos(a)` |Returns the arc cosine of a value.

|`atan(a)` |Returns the arc tangent of a value.

|`toRadians(angdeg)` |Converts an angle measured in degrees to an
approximately equivalent angle measured in radians

|`toDegrees(angrad)` |Converts an angle measured in radians to an
approximately equivalent angle measured in degrees.

|`exp(a)` |Returns Euler's number _e_ raised to the power of value.

|`log(a)` |Returns the natural logarithm (base _e_) of a value.

|`log10(a)` |Returns the base 10 logarithm of a value.

|`sqrt(a)` |Returns the correctly rounded positive square root of a
value.

|`cbrt(a)` |Returns the cube root of a double value.

|`IEEEremainder(f1, f2)` |Computes the remainder operation on two
arguments as prescribed by the IEEE 754 standard.

|`ceil(a)` |Returns the smallest (closest to negative infinity) value
that is greater than or equal to the argument and is equal to a
mathematical integer.

|`floor(a)` |Returns the largest (closest to positive infinity) value
that is less than or equal to the argument and is equal to a
mathematical integer.

|`rint(a)` |Returns the value that is closest in value to the argument
and is equal to a mathematical integer.

|`atan2(y, x)` |Returns the angle _theta_ from the conversion of
rectangular coordinates (_x_, _y_) to polar coordinates (r,_theta_).

|`pow(a, b)` |Returns the value of the first argument raised to the
power of the second argument.

|`round(a)` |Returns the closest _int_ to the argument.

|`random()` |Returns a random _double_ value.

|`abs(a)` |Returns the absolute value of a value.

|`max(a, b)` |Returns the greater of two values.

|`min(a, b)` |Returns the smaller of two values.

|`ulp(d)` |Returns the size of an ulp of the argument.

|`signum(d)` |Returns the signum function of the argument.

|`sinh(x)` |Returns the hyperbolic sine of a value.

|`cosh(x)` |Returns the hyperbolic cosine of a value.

|`tanh(x)` |Returns the hyperbolic tangent of a value.

|`hypot(x, y)` |Returns sqrt(_x2_ + _y2_) without intermediate overflow
or underflow.
|=======================================================================