Field data types
Each field has a field data type, or field type. This type indicates the
kind of data the field contains, such as strings or boolean values, and its
intended use. For example, you can index strings to both text
and keyword
fields. However, text
field values are analyzed for full-text
search while keyword
strings are left as-is for filtering and sorting.
Field types are grouped by family. Types in the same family support the same search functionality but may have different space usage or performance characteristics.
Currently, the only type family is keyword
, which consists of the keyword
,
constant_keyword
, and wildcard
field types. Other type families have only a
single field type. For example, the boolean
type family consists of one field
type: boolean
.
Common types
binary
-
Binary value encoded as a Base64 string.
boolean
-
true
andfalse
values. - Keywords
-
The keyword family, including
keyword
,constant_keyword
, andwildcard
. - Numbers
-
Numeric types, such as
long
anddouble
, used to express amounts. - Dates
-
Date types, including
date
anddate_nanos
. alias
-
Defines an alias for an existing field.
Objects and relational types
Structured data types
Aggregate data types
Text search types
text
-
Analyzed, unstructured text.
annotated-text
-
Text containing special markup. Used for identifying named entities.
completion
-
Used for auto-complete suggestions.
search_as_you_type
-
text
-like type for as-you-type completion. token_count
-
A count of tokens in a text.
Document ranking types
rank_feature
-
Records a numeric feature to boost hits at query time.
rank_features
-
Records numeric features to boost hits at query time.
Spatial data types
Other types
percolator
-
Indexes queries written in Query DSL.
Arrays
In OpenSearch, arrays do not require a dedicated field data type. Any field can contain zero or more values by default, however, all values in the array must be of the same field type. See Arrays.
Multi-fields
It is often useful to index the same field in different ways for different
purposes. For instance, a string
field could be mapped as
a text
field for full-text search, and as a keyword
field for
sorting or aggregations. Alternatively, you could index a text field with
the standard
analyzer, the
english
analyzer, and the
french
analyzer.
This is the purpose of multi-fields. Most field types support multi-fields
via the fields
parameter.
Alias field type
An alias
mapping defines an alternate name for a field in the index.
The alias can be used in place of the target field in search requests,
and selected other APIs like field capabilities.
PUT trips
{
"mappings": {
"properties": {
"distance": {
"type": "long"
},
"route_length_miles": {
"type": "alias",
"path": "distance" (1)
},
"transit_mode": {
"type": "keyword"
}
}
}
}
GET _search
{
"query": {
"range" : {
"route_length_miles" : {
"gte" : 39
}
}
}
}
-
The path to the target field. Note that this must be the full path, including any parent objects (e.g.
object1.object2.field
).
Almost all components of the search request accept field aliases. In particular, aliases can be
used in queries, aggregations, and sort fields, as well as when requesting docvalue_fields
,
stored_fields
, suggestions, and highlights. Scripts also support aliases when accessing
field values. Please see the section on unsupported APIs for exceptions.
In some parts of the search request and when requesting field capabilities, field wildcard patterns can be provided. In these cases, the wildcard pattern will match field aliases in addition to concrete fields:
GET trips/_field_caps?fields=route_*,transit_mode
Alias targets
There are a few restrictions on the target of an alias:
-
The target must be a concrete field, and not an object or another field alias.
-
The target field must exist at the time the alias is created.
-
If nested objects are defined, a field alias must have the same nested scope as its target.
Additionally, a field alias can only have one target. This means that it is not possible to use a field alias to query over multiple target fields in a single clause.
An alias can be changed to refer to a new target through a mappings update. A known limitation is that if any stored percolator queries contain the field alias, they will still refer to its original target. More information can be found in the percolator documentation.
Unsupported APIs
Writes to field aliases are not supported: attempting to use an alias in an index or update request
will result in a failure. Likewise, aliases cannot be used as the target of copy_to
or in multi-fields.
Because alias names are not present in the document source, aliases cannot be used when performing
source filtering. For example, the following request will return an empty result for _source
:
GET /_search
{
"query" : {
"match_all": {}
},
"_source": "route_length_miles"
}
Currently only the search and field capabilities APIs will accept and resolve field aliases. Other APIs that accept field names, such as term vectors, cannot be used with field aliases.
Finally, some queries, such as terms
, geo_shape
, and more_like_this
, allow for fetching query
information from an indexed document. Because field aliases aren’t supported when fetching documents,
the part of the query that specifies the lookup path cannot refer to a field by its alias.
Arrays
In OpenSearch, there is no dedicated array
data type. Any field can contain
zero or more values by default, however, all values in the array must be of the
same data type. For instance:
-
an array of strings: [
"one"
,"two"
] -
an array of integers: [
1
,2
] -
an array of arrays: [
1
, [2
,3
]] which is the equivalent of [1
,2
,3
] -
an array of objects: [
{ "name": "Mary", "age": 12 }
,{ "name": "John", "age": 10 }
]
|
Note
|
Arrays of objects
Arrays of objects do not work as you would expect: you cannot query each
object independently of the other objects in the array. If you need to be
able to do this then you should use the This is explained in more detail in Nested field type. |
When adding a field dynamically, the first value in the array determines the
field type
. All subsequent values must be of the same data type or it must
at least be possible to coerce subsequent values to the same
data type.
Arrays with a mixture of data types are not supported: [ 10
, "some string"
]
An array may contain null
values, which are either replaced by the
configured null_value
or skipped entirely. An empty array
[]
is treated as a missing field — a field with no values.
Nothing needs to be pre-configured in order to use arrays in documents, they are supported out of the box:
PUT my-index-000001/_doc/1
{
"message": "some arrays in this document...",
"tags": [ "opensearch", "wow" ], (1)
"lists": [ (2)
{
"name": "prog_list",
"description": "programming list"
},
{
"name": "cool_list",
"description": "cool stuff list"
}
]
}
PUT my-index-000001/_doc/2 (3)
{
"message": "no arrays in this document...",
"tags": "opensearch",
"lists": {
"name": "prog_list",
"description": "programming list"
}
}
GET my-index-000001/_search
{
"query": {
"match": {
"tags": "opensearch" (4)
}
}
}
-
The
tags
field is dynamically added as astring
field. -
The
lists
field is dynamically added as anobject
field. -
The second document contains no arrays, but can be indexed into the same fields.
-
The query looks for
opensearch
in thetags
field, and matches both documents.
Binary field type
The binary
type accepts a binary value as a
Base64 encoded string. The field is not
stored by default and is not searchable:
PUT my-index-000001
{
"mappings": {
"properties": {
"name": {
"type": "text"
},
"blob": {
"type": "binary"
}
}
}
}
PUT my-index-000001/_doc/1
{
"name": "Some binary blob",
"blob": "U29tZSBiaW5hcnkgYmxvYg==" (1)
}
-
The Base64 encoded binary value must not have embedded newlines
\n
.
Parameters for binary
fields
The following parameters are accepted by binary
fields:
doc_values
|
Should the field be stored on disk in a column-stride fashion, so that it
can later be used for sorting, aggregations, or scripting? Accepts |
store
|
Whether the field value should be stored and retrievable separately from
the |
Boolean field type
Boolean fields accept JSON true
and false
values, but can also accept
strings which are interpreted as either true or false:
| False values |
|
| True values |
|
For example:
PUT my-index-000001
{
"mappings": {
"properties": {
"is_published": {
"type": "boolean"
}
}
}
}
POST my-index-000001/_doc/1
{
"is_published": "true" (1)
}
GET my-index-000001/_search
{
"query": {
"term": {
"is_published": true (2)
}
}
}
-
Indexing a document with
"true"
, which is interpreted astrue
. -
Searching for documents with a JSON
true
.
Aggregations like the terms
aggregation use 1
and 0
for the key
, and the strings "true"
and
"false"
for the key_as_string
. Boolean fields when used in scripts,
return 1
and 0
:
POST my-index-000001/_doc/1
{
"is_published": true
}
POST my-index-000001/_doc/2
{
"is_published": false
}
GET my-index-000001/_search
{
"aggs": {
"publish_state": {
"terms": {
"field": "is_published"
}
}
},
"script_fields": {
"is_published": {
"script": {
"lang": "painless",
"source": "doc['is_published'].value"
}
}
}
}
Parameters for boolean
fields
The following parameters are accepted by boolean
fields:
boost
|
Mapping field-level query time boosting. Accepts a floating point number, defaults
to |
doc_values
|
Should the field be stored on disk in a column-stride fashion, so that it
can later be used for sorting, aggregations, or scripting? Accepts |
index
|
Should the field be searchable? Accepts |
null_value
|
Accepts any of the true or false values listed above. The value is
substituted for any explicit |
store
|
Whether the field value should be stored and retrievable separately from
the |
meta
|
Metadata about the field. |
Date field type
JSON doesn’t have a date data type, so dates in OpenSearch can either be:
-
strings containing formatted dates, e.g.
"2015-01-01"
or"2015/01/01 12:10:30"
. -
a long number representing milliseconds-since-the-epoch.
-
an integer representing seconds-since-the-epoch.
|
Note
|
Values for milliseconds-since-the-epoch and seconds-since-the-epoch must be non-negative. Use a formatted date to represent dates before 1970. |
Internally, dates are converted to UTC (if the time-zone is specified) and stored as a long number representing milliseconds-since-the-epoch.
Queries on dates are internally converted to range queries on this long representation, and the result of aggregations and stored fields is converted back to a string depending on the date format that is associated with the field.
|
Note
|
Dates will always be rendered as strings, even if they were initially supplied as a long in the JSON document. |
Date formats can be customised, but if no format
is specified then it uses
the default:
"strict_date_optional_time||epoch_millis"
This means that it will accept dates with optional timestamps, which conform
to the formats supported by strict_date_optional_time
or milliseconds-since-the-epoch.
For instance:
PUT my-index-000001
{
"mappings": {
"properties": {
"date": {
"type": "date" (1)
}
}
}
}
PUT my-index-000001/_doc/1
{ "date": "2015-01-01" } (2)
PUT my-index-000001/_doc/2
{ "date": "2015-01-01T12:10:30Z" } (3)
PUT my-index-000001/_doc/3
{ "date":<PHONE_NUMBER>001 } (4)
GET my-index-000001/_search
{
"sort": { "date": "asc"} (5)
}
-
The
date
field uses the defaultformat
. -
This document uses a plain date.
-
This document includes a time.
-
This document uses milliseconds-since-the-epoch.
-
Note that the
sort
values that are returned are all in milliseconds-since-the-epoch.
Multiple date formats
Multiple formats can be specified by separating them with ||
as a separator.
Each format will be tried in turn until a matching format is found. The first
format will be used to convert the milliseconds-since-the-epoch value back
into a string.
PUT my-index-000001
{
"mappings": {
"properties": {
"date": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
}
}
}
}
Parameters for date
fields
The following parameters are accepted by date
fields:
boost
|
Mapping field-level query time boosting. Accepts a floating point number, defaults
to |
doc_values
|
Should the field be stored on disk in a column-stride fashion, so that it
can later be used for sorting, aggregations, or scripting? Accepts |
format
|
The date format(s) that can be parsed. Defaults to
|
locale
|
The locale to use when parsing dates since months do not have the same names
and/or abbreviations in all languages. The default is the
|
ignore_malformed
|
If |
index
|
Should the field be searchable? Accepts |
null_value
|
Accepts a date value in one of the configured format's as the field
which is substituted for any explicit |
store
|
Whether the field value should be stored and retrievable separately from
the |
meta
|
Metadata about the field. |
Date nanoseconds field type
This data type is an addition to the date
data type. However there is an
important distinction between the two. The existing date
data type stores
dates in millisecond resolution. The date_nanos
data type stores dates
in nanosecond resolution, which limits its range of dates from roughly
1970 to 2262, as dates are still stored as a long representing nanoseconds
since the epoch.
Queries on nanoseconds are internally converted to range queries on this long representation, and the result of aggregations and stored fields is converted back to a string depending on the date format that is associated with the field.
Date formats can be customised, but if no format
is specified then it uses
the default:
"strict_date_optional_time||epoch_millis"
This means that it will accept dates with optional timestamps, which conform
to the formats supported by
strict_date_optional_time
including up to nine second
fractionals or milliseconds-since-the-epoch (thus losing precision on the
nano second part). Using strict_date_optional_time
will
format the result up to only three second fractionals. To
print and parse up to nine digits of resolution, use strict_date_optional_time_nanos
.
For instance:
PUT my-index-000001?include_type_name=true
{
"mappings": {
"_doc": {
"properties": {
"date": {
"type": "date_nanos" (1)
}
}
}
}
}
PUT my-index-000001/_doc/1
{ "date": "2015-01-01" } (2)
PUT my-index-000001/_doc/2
{ "date": "2015-01-01T12:10:30.123456789Z" } (3)
PUT my-index-000001/_doc/3
{ "date":<PHONE_NUMBER> } (4)
GET my-index-000001/_search
{
"sort": { "date": "asc"} (5)
}
GET my-index-000001/_search
{
"script_fields" : {
"my_field" : {
"script" : {
"lang" : "painless",
"source" : "doc['date'].value.nano" (6)
}
}
}
}
GET my-index-000001/_search
{
"docvalue_fields" : [
{
"field" : "date",
"format": "strict_date_time" (7)
}
]
}
-
The
date
field uses the defaultformat
. -
This document uses a plain date.
-
This document includes a time.
-
This document uses milliseconds-since-the-epoch.
-
Note that the
sort
values that are returned are all in nanoseconds-since-the-epoch. -
Access the nanosecond part of the date in a script
-
Use doc value fields, which can be formatted in nanosecond resolution
You can also specify multiple date formats separated by ||
. The
same mapping parameters than with the date
field can be used.
Limitations
Aggregations are still on millisecond resolution, even when using a date_nanos
field.
Geo-point field type
Fields of type geo_point
accept latitude-longitude pairs, which can be used:
-
to find geo-points within a bounding box, within a certain distance of a central point, or within a polygon or within a geo_shape query.
-
to aggregate documents geographically or by distance from a central point.
-
to integrate distance into a document’s relevance score.
-
to sort documents by distance.
There are five ways that a geo-point may be specified, as demonstrated below:
PUT my-index-000001
{
"mappings": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
PUT my-index-000001/_doc/1
{
"text": "Geo-point as an object",
"location": { (1)
"lat": 41.12,
"lon": -71.34
}
}
PUT my-index-000001/_doc/2
{
"text": "Geo-point as a string",
"location": "41.12,-71.34" (2)
}
PUT my-index-000001/_doc/3
{
"text": "Geo-point as a geohash",
"location": "drm3btev3e86" (3)
}
PUT my-index-000001/_doc/4
{
"text": "Geo-point as an array",
"location": [ -71.34, 41.12 ] (4)
}
PUT my-index-000001/_doc/5
{
"text": "Geo-point as a WKT POINT primitive",
"location" : "POINT (-71.34 41.12)" (5)
}
GET my-index-000001/_search
{
"query": {
"geo_bounding_box": { (6)
"location": {
"top_left": {
"lat": 42,
"lon": -72
},
"bottom_right": {
"lat": 40,
"lon": -74
}
}
}
}
}
-
Geo-point expressed as an object, with
lat
andlon
keys. -
Geo-point expressed as a string with the format:
"lat,lon"
. -
Geo-point expressed as a geohash.
-
Geo-point expressed as an array with the format: [
lon
,lat
] -
Geo-point expressed as a Well-Known Text POINT with the format:
"POINT(lon lat)"
-
A geo-bounding box query which finds all geo-points that fall inside the box.
|
Important
|
Geo-points expressed as an array or string
Please note that string geo-points are ordered as Originally, |
|
Note
|
A point can be expressed as a geohash. Geohashes are base32 encoded strings of the bits of the latitude and longitude interleaved. Each character in a geohash adds additional 5 bits to the precision. So the longer the hash, the more precise it is. For the indexing purposed geohashs are translated into latitude-longitude pairs. During this process only first 12 characters are used, so specifying more than 12 characters in a geohash doesn’t increase the precision. The 12 characters provide 60 bits, which should reduce a possible error to less than 2cm. |
Parameters for geo_point
fields
The following parameters are accepted by geo_point
fields:
ignore_malformed
|
If |
ignore_z_value
|
If |
null_value
|
Accepts an geopoint value which is substituted for any explicit |
Using geo-points in scripts
When accessing the value of a geo-point in a script, the value is returned as
a GeoPoint
object, which allows access to the .lat
and .lon
values
respectively:
def geopoint = doc['location'].value;
def lat = geopoint.lat;
def lon = geopoint.lon;
For performance reasons, it is better to access the lat/lon values directly:
def lat = doc['location'].lat;
def lon = doc['location'].lon;
Geo-shape field type
The geo_shape
data type facilitates the indexing of and searching
with arbitrary geo shapes such as rectangles and polygons. It should be
used when either the data being indexed or the queries being executed
contain shapes other than just points.
You can query documents using this type using geo_shape Query.
Mapping Options
The geo_shape mapping maps geo_json geometry objects to the geo_shape type. To enable it, users must explicitly map fields to the geo_shape type.
| Option | Description | Default |
|---|---|---|
|
Deprecated. Name of the PrefixTree
implementation to be used: |
|
|
Deprecated. This parameter may
be used instead of |
|
|
Deprecated. Maximum number
of layers to be used by the PrefixTree. This can be used to control the
precision of shape representations andtherefore how many terms are
indexed. Defaults to the default value of the chosen PrefixTree
implementation. Since this parameter requires a certain level of
understanding of the underlying implementation, users may use the
|
various |
|
Deprecated. The strategy
parameter defines the approach for how to represent shapes at indexing
and search time. It also influences the capabilities available so it
is recommended to let OpenSearch set this parameter automatically.
There are two strategies available: |
|
|
Deprecated. Used as a
hint to the PrefixTree about how precise it should be. Defaults to 0.025 (2.5%)
with 0.5 as the maximum supported value. PERFORMANCE NOTE: This value will
default to 0 if a |
|
|
Optional. Vertex order for the shape’s coordinates list. This parameter sets and returns only a To set * To set * Defaults to Individual GeoJSON or WKT documents can override this parameter. |
|
|
Deprecated. Setting this option to
|
|
|
If true, malformed GeoJSON or WKT shapes are ignored. If false (default), malformed GeoJSON and WKT shapes throw an exception and reject the entire document. |
|
|
If |
|
|
If |
|
Indexing approach
GeoShape types are indexed by decomposing the shape into a triangular mesh and
indexing each triangle as a 7 dimension point in a BKD tree. This provides
near perfect spatial resolution (down to 1e-7 decimal degree precision) since all
spatial relations are computed using an encoded vector representation of the
original shape instead of a raster-grid representation as used by the
Prefix trees indexing approach. Performance of the tessellator primarily
depends on the number of vertices that define the polygon/multi-polygon. While
this is the default indexing technique prefix trees can still be used by setting
the tree
or strategy
parameters according to the appropriate
Mapping Options. Note that these parameters are now deprecated
and will be removed in a future version.
IMPORTANT NOTES
CONTAINS
relation query - when using the new default vector indexing strategy, geo_shape
queries with relation
defined as contains
are supported for indices created with
ElasticSearch 7.5.0 or higher.
Prefix trees
Deprecated. To efficiently represent shapes in an inverted index, Shapes are converted into a series of hashes representing grid squares (commonly referred to as "rasters") using implementations of a PrefixTree. The tree notion comes from the fact that the PrefixTree uses multiple grid layers, each with an increasing level of precision to represent the Earth. This can be thought of as increasing the level of detail of a map or image at higher zoom levels. Since this approach causes precision issues with indexed shape, it has been deprecated in favor of a vector indexing approach that indexes the shapes as a triangular mesh (see Indexing approach).
Multiple PrefixTree implementations are provided:
-
GeohashPrefixTree - Uses geohashes for grid squares. Geohashes are base32 encoded strings of the bits of the latitude and longitude interleaved. So the longer the hash, the more precise it is. Each character added to the geohash represents another tree level and adds 5 bits of precision to the geohash. A geohash represents a rectangular area and has 32 sub rectangles. The maximum number of levels in OpenSearch is 24; the default is 9.
-
QuadPrefixTree - Uses a quadtree for grid squares. Similar to geohash, quad trees interleave the bits of the latitude and longitude the resulting hash is a bit set. A tree level in a quad tree represents 2 bits in this bit set, one for each coordinate. The maximum number of levels for the quad trees in OpenSearch is 29; the default is 21.
Spatial strategies
Deprecated. The indexing implementation selected relies on a SpatialStrategy for choosing how to decompose the shapes (either as grid squares or a tessellated triangular mesh). Each strategy answers the following:
-
What type of Shapes can be indexed?
-
What types of Query Operations and Shapes can be used?
-
Does it support more than one Shape per field?
The following Strategy implementations (with corresponding capabilities) are provided:
| Strategy | Supported Shapes | Supported Queries | Multiple Shapes |
|---|---|---|---|
|
|
Yes |
|
|
|
Yes |
Accuracy
Recursive
and Term
strategies do not provide 100% accuracy and depending on
how they are configured it may return some false positives for INTERSECTS
,
WITHIN
and CONTAINS
queries, and some false negatives for DISJOINT
queries.
To mitigate this, it is important to select an appropriate value for the tree_levels
parameter and to adjust expectations accordingly. For example, a point may be near
the border of a particular grid cell and may thus not match a query that only matches
the cell right next to it — even though the shape is very close to the point.
Example
PUT /example
{
"mappings": {
"properties": {
"location": {
"type": "geo_shape"
}
}
}
}
This mapping definition maps the location field to the geo_shape type using the default vector implementation. It provides approximately 1e-7 decimal degree precision.
Performance considerations with Prefix Trees
Deprecated. With prefix trees, OpenSearch uses the paths in the tree as terms in the inverted index and in queries. The higher the level (and thus the precision), the more terms are generated. Of course, calculating the terms, keeping them in memory, and storing them on disk all have a price. Especially with higher tree levels, indices can become extremely large even with a modest amount of data. Additionally, the size of the features also matters. Big, complex polygons can take up a lot of space at higher tree levels. Which setting is right depends on the use case. Generally one trades off accuracy against index size and query performance.
The defaults in OpenSearch for both implementations are a compromise between index size and a reasonable level of precision of 50m at the equator. This allows for indexing tens of millions of shapes without overly bloating the resulting index too much relative to the input size.
|
Note
|
Geo-shape queries on geo-shapes implemented with PrefixTrees will not be executed if
search.allow_expensive_queries is set to false.
|
Input Structure
Shapes can be represented using either the GeoJSON or Well-Known Text (WKT) format. The following table provides a mapping of GeoJSON and WKT to OpenSearch types:
| GeoJSON Type | WKT Type | OpenSearch Type | Description |
|---|---|---|---|
|
|
|
A single geographic coordinate. Note: OpenSearch uses WGS-84 coordinates only. |
|
|
|
An arbitrary line given two or more points. |
|
|
|
A closed polygon whose first and last point
must match, thus requiring |
|
|
|
An array of unconnected, but likely related points. |
|
|
|
An array of separate linestrings. |
|
|
|
An array of separate polygons. |
|
|
|
A GeoJSON shape similar to the
|
|
|
|
A bounding rectangle, or envelope, specified by specifying only the top left and bottom right points. |
|
|
|
A circle specified by a center point and radius with
units, which default to |
|
Note
|
For all types, both the inner In GeoJSON and WKT, and therefore OpenSearch, the correct coordinate order is longitude, latitude (X, Y) within coordinate arrays. This differs from many Geospatial APIs (e.g., Google Maps) that generally use the colloquial latitude, longitude (Y, X). |
Point
A point is a single geographic coordinate, such as the location of a building or the current position given by a smartphone’s Geolocation API. The following is an example of a point in GeoJSON.
POST /example/_doc
{
"location" : {
"type" : "point",
"coordinates" : [-77.03653, 38.897676]
}
}
The following is an example of a point in WKT:
POST /example/_doc
{
"location" : "POINT (-77.03653 38.897676)"
}
LineString
A linestring
defined by an array of two or more positions. By
specifying only two points, the linestring
will represent a straight
line. Specifying more than two points creates an arbitrary path. The
following is an example of a LineString in GeoJSON.
POST /example/_doc
{
"location" : {
"type" : "linestring",
"coordinates" : [[-77.03653, 38.897676], [-77.009051, 38.889939]]
}
}
The following is an example of a LineString in WKT:
POST /example/_doc
{
"location" : "LINESTRING (-77.03653 38.897676, -77.009051 38.889939)"
}
The above linestring
would draw a straight line starting at the White
House to the US Capitol Building.
Polygon
A polygon is defined by a list of a list of points. The first and last points in each (outer) list must be the same (the polygon must be closed). The following is an example of a Polygon in GeoJSON.
POST /example/_doc
{
"location" : {
"type" : "polygon",
"coordinates" : [
[ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ]
]
}
}
The following is an example of a Polygon in WKT:
POST /example/_doc
{
"location" : "POLYGON ((100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0, 100.0 0.0))"
}
The first array represents the outer boundary of the polygon, the other arrays represent the interior shapes ("holes"). The following is a GeoJSON example of a polygon with a hole:
POST /example/_doc
{
"location" : {
"type" : "polygon",
"coordinates" : [
[ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ],
[ [100.2, 0.2], [100.8, 0.2], [100.8, 0.8], [100.2, 0.8], [100.2, 0.2] ]
]
}
}
The following is an example of a Polygon with a hole in WKT:
POST /example/_doc
{
"location" : "POLYGON ((100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0, 100.0 0.0), (100.2 0.2, 100.8 0.2, 100.8 0.8, 100.2 0.8, 100.2 0.2))"
}
IMPORTANT NOTE: WKT does not enforce a specific order for vertices thus ambiguous polygons around the dateline and poles are possible. GeoJSON mandates that the outer polygon must be counterclockwise and interior shapes must be clockwise, which agrees with the Open Geospatial Consortium (OGC) Simple Feature Access specification for vertex ordering.
OpenSearch accepts both clockwise and counterclockwise polygons if they appear not to cross the dateline (i.e. they cross less than 180° of longitude), but for polygons that do cross the dateline (or for other polygons wider than 180°) OpenSearch requires the vertex ordering to comply with the OGC and GeoJSON specifications. Otherwise, an unintended polygon may be created and unexpected query/filter results will be returned.
The following provides an example of an ambiguous polygon. OpenSearch will apply the GeoJSON standard to eliminate ambiguity resulting in a polygon that crosses the dateline.
POST /example/_doc
{
"location" : {
"type" : "polygon",
"coordinates" : [
[ [-177.0, 10.0], [176.0, 15.0], [172.0, 0.0], [176.0, -15.0], [-177.0, -10.0], [-177.0, 10.0] ],
[ [178.2, 8.2], [-178.8, 8.2], [-180.8, -8.8], [178.2, 8.8] ]
]
}
}
An orientation
parameter can be defined when setting the geo_shape mapping (see Mapping Options). This will define vertex
order for the coordinate list on the mapped geo_shape field. It can also be overridden on each document. The following is an example for
overriding the orientation on a document:
POST /example/_doc
{
"location" : {
"type" : "polygon",
"orientation" : "clockwise",
"coordinates" : [
[ [100.0, 0.0], [100.0, 1.0], [101.0, 1.0], [101.0, 0.0], [100.0, 0.0] ]
]
}
}
MultiPoint
The following is an example of a list of geojson points:
POST /example/_doc
{
"location" : {
"type" : "multipoint",
"coordinates" : [
[102.0, 2.0], [103.0, 2.0]
]
}
}
The following is an example of a list of WKT points:
POST /example/_doc
{
"location" : "MULTIPOINT (102.0 2.0, 103.0 2.0)"
}
MultiLineString
The following is an example of a list of geojson linestrings:
POST /example/_doc
{
"location" : {
"type" : "multilinestring",
"coordinates" : [
[ [102.0, 2.0], [103.0, 2.0], [103.0, 3.0], [102.0, 3.0] ],
[ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0] ],
[ [100.2, 0.2], [100.8, 0.2], [100.8, 0.8], [100.2, 0.8] ]
]
}
}
The following is an example of a list of WKT linestrings:
POST /example/_doc
{
"location" : "MULTILINESTRING ((102.0 2.0, 103.0 2.0, 103.0 3.0, 102.0 3.0), (100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0), (100.2 0.2, 100.8 0.2, 100.8 0.8, 100.2 0.8))"
}
MultiPolygon
The following is an example of a list of geojson polygons (second polygon contains a hole):
POST /example/_doc
{
"location" : {
"type" : "multipolygon",
"coordinates" : [
[ [[102.0, 2.0], [103.0, 2.0], [103.0, 3.0], [102.0, 3.0], [102.0, 2.0]] ],
[ [[100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0]],
[[100.2, 0.2], [100.8, 0.2], [100.8, 0.8], [100.2, 0.8], [100.2, 0.2]] ]
]
}
}
The following is an example of a list of WKT polygons (second polygon contains a hole):
POST /example/_doc
{
"location" : "MULTIPOLYGON (((102.0 2.0, 103.0 2.0, 103.0 3.0, 102.0 3.0, 102.0 2.0)), ((100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0, 100.0 0.0), (100.2 0.2, 100.8 0.2, 100.8 0.8, 100.2 0.8, 100.2 0.2)))"
}
Geometry Collection
The following is an example of a collection of geojson geometry objects:
POST /example/_doc
{
"location" : {
"type": "geometrycollection",
"geometries": [
{
"type": "point",
"coordinates": [100.0, 0.0]
},
{
"type": "linestring",
"coordinates": [ [101.0, 0.0], [102.0, 1.0] ]
}
]
}
}
The following is an example of a collection of WKT geometry objects:
POST /example/_doc
{
"location" : "GEOMETRYCOLLECTION (POINT (100.0 0.0), LINESTRING (101.0 0.0, 102.0 1.0))"
}
Envelope
OpenSearch supports an envelope
type, which consists of coordinates
for upper left and lower right points of the shape to represent a
bounding rectangle in the format :
POST /example/_doc
{
"location" : {
"type" : "envelope",
"coordinates" : [ [100.0, 1.0], [101.0, 0.0] ]
}
}
The following is an example of an envelope using the WKT BBOX format:
NOTE: WKT specification expects the following order: minLon, maxLon, maxLat, minLat.
POST /example/_doc
{
"location" : "BBOX (100.0, 102.0, 2.0, 0.0)"
}
Circle
OpenSearch supports a circle
type, which consists of a center
point with a radius. Note that this circle representation can only
be indexed when using the recursive
Prefix Tree strategy. For
the default Indexing approach circles should be approximated using
a POLYGON
.
POST /example/_doc
{
"location" : {
"type" : "circle",
"coordinates" : [101.0, 1.0],
"radius" : "100m"
}
}
Note: The inner radius
field is required. If not specified, then
the units of the radius
will default to METERS
.
NOTE: Neither GeoJSON or WKT support a point-radius circle type.
Sorting and Retrieving index Shapes
Due to the complex input structure and index representation of shapes,
it is not currently possible to sort shapes or retrieve their fields
directly. The geo_shape value is only retrievable through the _source
field.
IP field type
PUT my-index-000001
{
"mappings": {
"properties": {
"ip_addr": {
"type": "ip"
}
}
}
}
PUT my-index-000001/_doc/1
{
"ip_addr": "<IP_ADDRESS>"
}
GET my-index-000001/_search
{
"query": {
"term": {
"ip_addr": "<IP_ADDRESS>/16"
}
}
}
|
Note
|
You can also store ip ranges in a single field using an ip_range data type. |
Parameters for ip
fields
The following parameters are accepted by ip
fields:
boost
|
Mapping field-level query time boosting. Accepts a floating point number, defaults
to |
doc_values
|
Should the field be stored on disk in a column-stride fashion, so that it
can later be used for sorting, aggregations, or scripting? Accepts |
ignore_malformed
|
If |
index
|
Should the field be searchable? Accepts |
null_value
|
Accepts an IPv4 or IPv6 value which is substituted for any explicit |
store
|
Whether the field value should be stored and retrievable separately from
the |
Querying ip
fields
The most common way to query ip addresses is to use the
CIDR
notation: [ip_address]/[prefix_length]
. For instance:
GET my-index-000001/_search
{
"query": {
"term": {
"ip_addr": "<IP_ADDRESS>/16"
}
}
}
or
GET my-index-000001/_search
{
"query": {
"term": {
"ip_addr": "2001:db8::/48"
}
}
}
Also beware that colons are special characters to the
query_string
query, so ipv6 addresses will
need to be escaped. The easiest way to do so is to put quotes around the
searched value:
GET my-index-000001/_search
{
"query": {
"query_string" : {
"query": "ip_addr:\"2001:db8::/48\""
}
}
}
Join field type
The join
data type is a special field that creates
parent/child relation within documents of the same index.
The relations
section defines a set of possible relations within the documents,
each relation being a parent name and a child name.
A parent/child relation can be defined as follows:
PUT my-index-000001
{
"mappings": {
"properties": {
"my_id": {
"type": "keyword"
},
"my_join_field": { (1)
"type": "join",
"relations": {
"question": "answer" (2)
}
}
}
}
}
-
The name for the field
-
Defines a single relation where
question
is parent ofanswer
.
To index a document with a join, the name of the relation and the optional parent
of the document must be provided in the source
.
For instance the following example creates two parent
documents in the question
context:
PUT my-index-000001/_doc/1?refresh
{
"my_id": "1",
"text": "This is a question",
"my_join_field": {
"name": "question" (1)
}
}
PUT my-index-000001/_doc/2?refresh
{
"my_id": "2",
"text": "This is another question",
"my_join_field": {
"name": "question"
}
}
-
This document is a
question
document.
When indexing parent documents, you can choose to specify just the name of the relation as a shortcut instead of encapsulating it in the normal object notation:
PUT my-index-000001/_doc/1?refresh
{
"my_id": "1",
"text": "This is a question",
"my_join_field": "question" (1)
}
PUT my-index-000001/_doc/2?refresh
{
"my_id": "2",
"text": "This is another question",
"my_join_field": "question"
}
-
Simpler notation for a parent document just uses the relation name.
When indexing a child, the name of the relation as well as the parent id of the document
must be added in the _source
.
|
Warning
|
It is required to index the lineage of a parent in the same shard so you must always route child documents using their greater parent id. |
For instance the following example shows how to index two child
documents:
PUT my-index-000001/_doc/3?routing=1&refresh (1)
{
"my_id": "3",
"text": "This is an answer",
"my_join_field": {
"name": "answer", (2)
"parent": "1" (3)
}
}
PUT my-index-000001/_doc/4?routing=1&refresh
{
"my_id": "4",
"text": "This is another answer",
"my_join_field": {
"name": "answer",
"parent": "1"
}
}
-
The routing value is mandatory because parent and child documents must be indexed on the same shard
-
answer
is the name of the join for this document -
The parent id of this child document
Parent-join and performance
The join field shouldn’t be used like joins in a relation database. In OpenSearch the key to good performance
is to de-normalize your data into documents. Each join field, has_child
or has_parent
query adds a
significant tax to your query performance. It can also trigger global ordinals to be built.
The only case where the join field makes sense is if your data contains a one-to-many relationship where one entity significantly outnumbers the other entity. An example of such case is a use case with products and offers for these products. In the case that offers significantly outnumbers the number of products then it makes sense to model the product as parent document and the offer as child document.
Parent-join restrictions
-
Only one
join
field mapping is allowed per index. -
Parent and child documents must be indexed on the same shard. This means that the same
routing
value needs to be provided when getting, deleting, or updating a child document. -
An element can have multiple children but only one parent.
-
It is possible to add a new relation to an existing
join
field. -
It is also possible to add a child to an existing element but only if the element is already a parent.
Searching with parent-join
The parent-join creates one field to index the name of the relation
within the document (my_parent
, my_child
, …).
It also creates one field per parent/child relation.
The name of this field is the name of the join
field followed by #
and the
name of the parent in the relation.
So for instance for the my_parent
→ [my_child
, another_child
] relation,
the join
field creates an additional field named my_join_field#my_parent
.
This field contains the parent _id
that the document links to
if the document is a child (my_child
or another_child
) and the _id
of
document if it’s a parent (my_parent
).
When searching an index that contains a join
field, these two fields are always
returned in the search response:
GET my-index-000001/_search
{
"query": {
"match_all": {}
},
"sort": ["my_id"]
}
Will return:
{
...,
"hits": {
"total": {
"value": 4,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": "my-index-000001",
"_type": "_doc",
"_id": "1",
"_score": null,
"_source": {
"my_id": "1",
"text": "This is a question",
"my_join_field": "question" (1)
},
"sort": [
"1"
]
},
{
"_index": "my-index-000001",
"_type": "_doc",
"_id": "2",
"_score": null,
"_source": {
"my_id": "2",
"text": "This is another question",
"my_join_field": "question" (2)
},
"sort": [
"2"
]
},
{
"_index": "my-index-000001",
"_type": "_doc",
"_id": "3",
"_score": null,
"_routing": "1",
"_source": {
"my_id": "3",
"text": "This is an answer",
"my_join_field": {
"name": "answer", (3)
"parent": "1" (4)
}
},
"sort": [
"3"
]
},
{
"_index": "my-index-000001",
"_type": "_doc",
"_id": "4",
"_score": null,
"_routing": "1",
"_source": {
"my_id": "4",
"text": "This is another answer",
"my_join_field": {
"name": "answer",
"parent": "1"
}
},
"sort": [
"4"
]
}
]
}
}
-
This document belongs to the
question
join -
This document belongs to the
question
join -
This document belongs to the
answer
join -
The linked parent id for the child document
Parent-join queries and aggregations
See the has_child
and
has_parent
queries,
the children
aggregation,
and inner hits for more information.
The value of the join
field is accessible in aggregations
and scripts, and may be queried with the
parent_id
query:
GET my-index-000001/_search
{
"query": {
"parent_id": { (1)
"type": "answer",
"id": "1"
}
},
"aggs": {
"parents": {
"terms": {
"field": "my_join_field#question", (2)
"size": 10
}
}
},
"script_fields": {
"parent": {
"script": {
"source": "doc['my_join_field#question']" (3)
}
}
}
}
-
Querying the
parent id
field (also see thehas_parent
query and thehas_child
query) -
Aggregating on the
parent id
field (also see thechildren
aggregation) -
Accessing the parent id` field in scripts
Global ordinals
The join
field uses global ordinals to speed up joins.
Global ordinals need to be rebuilt after any change to a shard. The more
parent id values are stored in a shard, the longer it takes to rebuild the
global ordinals for the join
field.
Global ordinals, by default, are built eagerly: if the index has changed,
global ordinals for the join
field will be rebuilt as part of the refresh.
This can add significant time to the refresh. However most of the times this is the
right trade-off, otherwise global ordinals are rebuilt when the first parent-join
query or aggregation is used. This can introduce a significant latency spike for
your users and usually this is worse as multiple global ordinals for the join
field may be attempt rebuilt within a single refresh interval when many writes
are occurring.
When the join
field is used infrequently and writes occur frequently it may
make sense to disable eager loading:
PUT my-index-000001
{
"mappings": {
"properties": {
"my_join_field": {
"type": "join",
"relations": {
"question": "answer"
},
"eager_global_ordinals": false
}
}
}
}
The amount of heap used by global ordinals can be checked per parent relation as follows:
# Per-index
GET _stats/fielddata?human&fields=my_join_field#question
# Per-node per-index
GET _nodes/stats/indices/fielddata?human&fields=my_join_field#question
Multiple children per parent
It is also possible to define multiple children for a single parent:
PUT my-index-000001
{
"mappings": {
"properties": {
"my_join_field": {
"type": "join",
"relations": {
"question": ["answer", "comment"] (1)
}
}
}
}
}
-
question
is parent ofanswer
andcomment
.
Multiple levels of parent join
|
Warning
|
Using multiple levels of relations to replicate a relational model is not recommended. Each level of relation adds an overhead at query time in terms of memory and computation. You should de-normalize your data if you care about performance. |
Multiple levels of parent/child:
PUT my-index-000001
{
"mappings": {
"properties": {
"my_join_field": {
"type": "join",
"relations": {
"question": ["answer", "comment"], (1)
"answer": "vote" (2)
}
}
}
}
}
-
question
is parent ofanswer
andcomment
-
answer
is parent ofvote
The mapping above represents the following tree:
question / \ / \ comment answer | | vote
Indexing a grandchild document requires a routing
value equals
to the grand-parent (the greater parent of the lineage):
PUT my-index-000001/_doc/3?routing=1&refresh (1)
{
"text": "This is a vote",
"my_join_field": {
"name": "vote",
"parent": "2" (2)
}
}
-
This child document must be on the same shard than its grand-parent and parent
-
The parent id of this document (must points to an
answer
document)
Keyword type family
The keyword family includes the following field types:
-
keyword
, which is used for structured content such as IDs, email addresses, hostnames, status codes, zip codes, or tags.
Keyword fields are often used in sorting,
aggregations, and term-level
queries, such as term
.
|
Tip
|
Avoid using keyword fields for full-text search. Use the text
field type instead.
|
Keyword field type
Below is an example of a mapping for a basic keyword
field:
PUT my-index-000001
{
"mappings": {
"properties": {
"tags": {
"type": "keyword"
}
}
}
}
|
Tip
|
Mapping numeric identifiers
Not all numeric data should be mapped as a numeric field data type.
OpenSearch optimizes numeric fields, such as Identifiers, such as an ISBN or a product ID, are rarely used in Consider mapping a numeric identifier as a
If you’re unsure which to use, you can use a multi-field to map
the data as both a |
Parameters for basic keyword fields
The following parameters are accepted by keyword
fields:
boost
|
Mapping field-level query time boosting. Accepts a floating point number, defaults
to |
doc_values
|
Should the field be stored on disk in a column-stride fashion, so that it
can later be used for sorting, aggregations, or scripting? Accepts |
eager_global_ordinals
|
Should global ordinals be loaded eagerly on refresh? Accepts |
fields
|
Multi-fields allow the same string value to be indexed in multiple ways for different purposes, such as one field for search and a multi-field for sorting and aggregations. |
ignore_above
|
Do not index any string longer than this value. Defaults to |
index
|
Should the field be searchable? Accepts |
index_options
|
What information should be stored in the index, for scoring purposes.
Defaults to |
norms
|
Whether field-length should be taken into account when scoring queries.
Accepts |
null_value
|
Accepts a string value which is substituted for any explicit |
store
|
Whether the field value should be stored and retrievable separately from
the |
similarity
|
Which scoring algorithm or similarity should be used. Defaults
to |
normalizer
|
How to pre-process the keyword prior to indexing. Defaults to |
split_queries_on_whitespace
|
Whether full text queries should split the input on whitespace
when building a query for this field.
Accepts |
meta
|
Metadata about the field. |
Nested field type
The nested
type is a specialised version of the object
data type
that allows arrays of objects to be indexed in a way that they can be queried
independently of each other.
How arrays of objects are flattened
OpenSearch has no concept of inner objects. Therefore, it flattens object hierarchies into a simple list of field names and values. For instance, consider the following document:
PUT my-index-000001/_doc/1
{
"group" : "fans",
"user" : [ (1)
{
"first" : "John",
"last" : "Smith"
},
{
"first" : "Alice",
"last" : "White"
}
]
}
-
The
user
field is dynamically added as a field of typeobject
.
The previous document would be transformed internally into a document that looks more like this:
{
"group" : "fans",
"user.first" : [ "alice", "john" ],
"user.last" : [ "smith", "white" ]
}
The user.first
and user.last
fields are flattened into multi-value fields,
and the association between alice
and white
is lost. This document would
incorrectly match a query for alice AND smith
:
GET my-index-000001/_search
{
"query": {
"bool": {
"must": [
{ "match": { "user.first": "Alice" }},
{ "match": { "user.last": "Smith" }}
]
}
}
}
Using nested
fields for arrays of objects
If you need to index arrays of objects and to maintain the independence of
each object in the array, use the nested
data type instead of the
object
data type.
Internally, nested objects index each object in
the array as a separate hidden document, meaning that each nested object can be
queried independently of the others with the nested
query:
PUT my-index-000001
{
"mappings": {
"properties": {
"user": {
"type": "nested" (1)
}
}
}
}
PUT my-index-000001/_doc/1
{
"group" : "fans",
"user" : [
{
"first" : "John",
"last" : "Smith"
},
{
"first" : "Alice",
"last" : "White"
}
]
}
GET my-index-000001/_search
{
"query": {
"nested": {
"path": "user",
"query": {
"bool": {
"must": [
{ "match": { "user.first": "Alice" }},
{ "match": { "user.last": "Smith" }} (2)
]
}
}
}
}
}
GET my-index-000001/_search
{
"query": {
"nested": {
"path": "user",
"query": {
"bool": {
"must": [
{ "match": { "user.first": "Alice" }},
{ "match": { "user.last": "White" }} (3)
]
}
},
"inner_hits": { (4)
"highlight": {
"fields": {
"user.first": {}
}
}
}
}
}
}
-
The
user
field is mapped as typenested
instead of typeobject
. -
This query doesn’t match because
Alice
andSmith
are not in the same nested object. -
This query matches because
Alice
andWhite
are in the same nested object. -
inner_hits
allow us to highlight the matching nested documents.
Interacting with nested
documents
Nested documents can be:
-
queried with the
nested
query. -
analyzed with the
nested
andreverse_nested
aggregations. -
sorted with nested sorting.
-
retrieved and highlighted with nested inner hits.
|
Important
|
Because nested documents are indexed as separate documents, they can only be
accessed within the scope of the For instance, if a string field within a nested document has
|
Parameters for nested
fields
The following parameters are accepted by nested
fields:
dynamic
-
(Optional, string) Whether or not new
properties
should be added dynamically to an existing nested object. Acceptstrue
(default),false
andstrict
. properties
-
(Optional, object) The fields within the nested object, which can be of any data type, including
nested
. New properties may be added to an existing nested object.
include_in_parent
-
(Optional, Boolean) If
true
, all fields in the nested object are also added to the parent document as standard (flat) fields. Defaults tofalse
.
include_in_root
-
(Optional, Boolean) If
true
, all fields in the nested object are also added to the root document as standard (flat) fields. Defaults tofalse
.
Limits on nested
mappings and objects
As described earlier, each nested object is indexed as a separate Lucene document.
Continuing with the previous example, if we indexed a single document containing 100 user
objects,
then 101 Lucene documents would be created: one for the parent document, and one for each
nested object. Because of the expense associated with nested
mappings, OpenSearch puts
settings in place to guard against performance problems:
index.mapping.nested_fields.limit
-
The maximum number of distinct
nested
mappings in an index. Thenested
type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of uniquenested
types per index. Default is50
.
In the previous example, the user
mapping would count as only 1 towards this limit.
index.mapping.nested_objects.limit
-
The maximum number of nested JSON objects that a single document can contain across all
nested
types. This limit helps to prevent out of memory errors when a document contains too many nested objects. Default is10000
.
To illustrate how this setting works, consider adding another nested
type called comments
to the previous example mapping. For each document, the combined number of user
and comment
objects it contains must be below the limit.
See Settings to prevent mappings explosion regarding additional settings for preventing mappings explosion.
Numeric field types
The following numeric types are supported:
long
|
A signed 64-bit integer with a minimum value of -2^63^ and a maximum value of 2^63^-1. |
integer
|
A signed 32-bit integer with a minimum value of -2^31^ and a maximum value of 2^31^-1. |
short
|
A signed 16-bit integer with a minimum value of -32,768 and a maximum value of 32,767. |
byte
|
A signed 8-bit integer with a minimum value of -128 and a maximum value of 127. |
double
|
A double-precision 64-bit IEEE 754 floating point number, restricted to finite values. |
float
|
A single-precision 32-bit IEEE 754 floating point number, restricted to finite values. |
half_float
|
A half-precision 16-bit IEEE 754 floating point number, restricted to finite values. |
scaled_float
|
A floating point number that is backed by a |
unsigned_long
|
An unsigned 64-bit integer with a minimum value of 0 and a maximum value of 2^64^-1. |
Below is an example of configuring a mapping with numeric fields:
PUT my-index-000001
{
"mappings": {
"properties": {
"number_of_bytes": {
"type": "integer"
},
"time_in_seconds": {
"type": "float"
},
"price": {
"type": "scaled_float",
"scaling_factor": 100
}
}
}
}
|
Note
|
The double , float and half_float types consider that -0.0 and
+0.0 are different values. As a consequence, doing a term query on
-0.0 will not match +0.0 and vice-versa. Same is true for range queries:
if the upper bound is -0.0 then +0.0 will not match, and if the lower
bound is +0.0 then -0.0 will not match.
|
Which type should I use?
As far as integer types (byte
, short
, integer
and long
) are concerned,
you should pick the smallest type which is enough for your use-case. This will
help indexing and searching be more efficient. Note however that storage is
optimized based on the actual values that are stored, so picking one type over
another one will have no impact on storage requirements.
For floating-point types, it is often more efficient to store floating-point
data into an integer using a scaling factor, which is what the scaled_float
type does under the hood. For instance, a price
field could be stored in a
scaled_float
with a scaling_factor
of 100. All APIs would work as if
the field was stored as a double, but under the hood OpenSearch would be
working with the number of cents, price*100, which is an integer. This is
mostly helpful to save disk space since integers are way easier to compress
than floating points. scaled_float
is also fine to use in order to trade
accuracy for disk space. For instance imagine that you are tracking cpu
utilization as a number between 0 and 1. It usually does not matter much
whether cpu utilization is 12.7% or 13%, so you could use a scaled_float
with a scaling_factor
of 100 in order to round cpu utilization to the
closest percent in order to save space.
If scaled_float
is not a good fit, then you should pick the smallest type
that is enough for the use-case among the floating-point types: double
,
float
and half_float
. Here is a table that compares these types in order
to help make a decision.
| Type | Minimum value | Maximum value | Significant bits / digits |
|---|---|---|---|
|
2^-1074^ |
(2-2^-52^)·2^1023^ |
53 / 15.95 |
|
2^-149^ |
(2-2^-23^)·2^127^ |
24 / 7.22 |
|
2^-24^ |
65504 |
11 / 3.31 |
|
Tip
|
Mapping numeric identifiers
Not all numeric data should be mapped as a numeric field data type.
OpenSearch optimizes numeric fields, such as Identifiers, such as an ISBN or a product ID, are rarely used in Consider mapping a numeric identifier as a
If you’re unsure which to use, you can use a multi-field to map
the data as both a |
Parameters for numeric fields
The following parameters are accepted by numeric types:
coerce
|
Try to convert strings to numbers and truncate fractions for integers.
Accepts |
boost
|
Mapping field-level query time boosting. Accepts a floating point number, defaults
to |
doc_values
|
Should the field be stored on disk in a column-stride fashion, so that it
can later be used for sorting, aggregations, or scripting? Accepts |
ignore_malformed
|
If |
index
|
Should the field be searchable? Accepts |
null_value
|
Accepts a numeric value of the same |
store
|
Whether the field value should be stored and retrievable separately from
the |
meta
|
Metadata about the field. |
Parameters for scaled_float
scaled_float
accepts an additional parameter:
scaling_factor
|
The scaling factor to use when encoding values. Values will be multiplied
by this factor at index time and rounded to the closest long value. For
instance, a |
Object field type
JSON documents are hierarchical in nature: the document may contain inner objects which, in turn, may contain inner objects themselves:
PUT my-index-000001/_doc/1
{ (1)
"region": "US",
"manager": { (2)
"age": 30,
"name": { (3)
"first": "John",
"last": "Smith"
}
}
}
-
The outer document is also a JSON object.
-
It contains an inner object called
manager
. -
Which in turn contains an inner object called
name
.
Internally, this document is indexed as a simple, flat list of key-value pairs, something like this:
{
"region": "US",
"manager.age": 30,
"manager.name.first": "John",
"manager.name.last": "Smith"
}
An explicit mapping for the above document could look like this:
PUT my-index-000001
{
"mappings": {
"properties": { (1)
"region": {
"type": "keyword"
},
"manager": { (2)
"properties": {
"age": { "type": "integer" },
"name": { (3)
"properties": {
"first": { "type": "text" },
"last": { "type": "text" }
}
}
}
}
}
}
}
-
Properties in the top-level mappings definition.
-
The
manager
field is an innerobject
field. -
The
manager.name
field is an innerobject
field within themanager
field.
You are not required to set the field type
to object
explicitly, as this is the default value.
Parameters for object
fields
The following parameters are accepted by object
fields:
dynamic
|
Whether or not new |
enabled
|
Whether the JSON value given for the object field should be
parsed and indexed ( |
properties
|
The fields within the object, which can be of any
data type, including |
|
Important
|
If you need to index arrays of objects instead of single objects, read Nested field type first. |
Percolator field type
The percolator
field type parses a json structure into a native query and
stores that query, so that the percolate query
can use it to match provided documents.
Any field that contains a json object can be configured to be a percolator
field. The percolator field type has no settings. Just configuring the percolator
field type is sufficient to instruct OpenSearch to treat a field as a
query.
If the following mapping configures the percolator
field type for the
query
field:
PUT my-index-000001
{
"mappings": {
"properties": {
"query": {
"type": "percolator"
},
"field": {
"type": "text"
}
}
}
}
Then you can index a query:
PUT my-index-000001/_doc/match_value
{
"query": {
"match": {
"field": "value"
}
}
}
|
Important
|
Fields referred to in a percolator query must already exist in the mapping associated with the index used for percolation. In order to make sure these fields exist, add or update a mapping via the create index or put mapping APIs. |
Reindexing your percolator queries
Reindexing percolator queries is sometimes required to benefit from improvements made to the percolator
field type in
new releases.
Reindexing percolator queries can be reindexed by using the reindex api. Lets take a look at the following index with a percolator field type:
PUT index
{
"mappings": {
"properties": {
"query" : {
"type" : "percolator"
},
"body" : {
"type": "text"
}
}
}
}
POST _aliases
{
"actions": [
{
"add": {
"index": "index",
"alias": "queries" (1)
}
}
]
}
PUT queries/_doc/1?refresh
{
"query" : {
"match" : {
"body" : "quick brown fox"
}
}
}
-
It is always recommended to define an alias for your index, so that in case of a reindex systems / applications don’t need to be changed to know that the percolator queries are now in a different index.
Lets say you’re going to upgrade to a new major version and in order for the new OpenSearch version to still be able to read your queries you need to reindex your queries into a new index on the current OpenSearch version:
PUT new_index
{
"mappings": {
"properties": {
"query" : {
"type" : "percolator"
},
"body" : {
"type": "text"
}
}
}
}
POST /_reindex?refresh
{
"source": {
"index": "index"
},
"dest": {
"index": "new_index"
}
}
POST _aliases
{
"actions": [ (1)
{
"remove": {
"index" : "index",
"alias": "queries"
}
},
{
"add": {
"index": "new_index",
"alias": "queries"
}
}
]
}
-
If you have an alias don’t forget to point it to the new index.
Executing the percolate
query via the queries
alias:
GET /queries/_search
{
"query": {
"percolate" : {
"field" : "query",
"document" : {
"body" : "fox jumps over the lazy dog"
}
}
}
}
now returns matches from the new index:
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped" : 0,
"failed": 0
},
"hits": {
"total" : {
"value": 1,
"relation": "eq"
},
"max_score": 0.13076457,
"hits": [
{
"_index": "new_index", (1)
"_type": "_doc",
"_id": "1",
"_score": 0.13076457,
"_source": {
"query": {
"match": {
"body": "quick brown fox"
}
}
},
"fields" : {
"_percolator_document_slot" : [0]
}
}
]
}
}
-
Percolator query hit is now being presented from the new index.
Optimizing query time text analysis
When the percolator verifies a percolator candidate match it is going to parse, perform query time text analysis and actually run
the percolator query on the document being percolated. This is done for each candidate match and every time the percolate
query executes.
If your query time text analysis is relatively expensive part of query parsing then text analysis can become the
dominating factor time is being spent on when percolating. This query parsing overhead can become noticeable when the
percolator ends up verifying many candidate percolator query matches.
To avoid the most expensive part of text analysis at percolate time. One can choose to do the expensive part of text analysis
when indexing the percolator query. This requires using two different analyzers. The first analyzer actually performs
text analysis that needs be performed (expensive part). The second analyzer (usually whitespace) just splits the generated tokens
that the first analyzer has produced. Then before indexing a percolator query, the analyze api should be used to analyze the query
text with the more expensive analyzer. The result of the analyze api, the tokens, should be used to substitute the original query
text in the percolator query. It is important that the query should now be configured to override the analyzer from the mapping and
just the second analyzer. Most text based queries support an analyzer
option (match
, query_string
, simple_query_string
).
Using this approach the expensive text analysis is performed once instead of many times.
Lets demonstrate this workflow via a simplified example.
Lets say we want to index the following percolator query:
{
"query" : {
"match" : {
"body" : {
"query" : "missing bicycles"
}
}
}
}
with these settings and mapping:
PUT /test_index
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer" : {
"tokenizer": "standard",
"filter" : ["lowercase", "porter_stem"]
}
}
}
},
"mappings": {
"properties": {
"query" : {
"type": "percolator"
},
"body" : {
"type": "text",
"analyzer": "my_analyzer" (1)
}
}
}
}
-
For the purpose of this example, this analyzer is considered expensive.
First we need to use the analyze api to perform the text analysis prior to indexing:
POST /test_index/_analyze
{
"analyzer" : "my_analyzer",
"text" : "missing bicycles"
}
This results the following response:
{
"tokens": [
{
"token": "miss",
"start_offset": 0,
"end_offset": 7,
"type": "<ALPHANUM>",
"position": 0
},
{
"token": "bicycl",
"start_offset": 8,
"end_offset": 16,
"type": "<ALPHANUM>",
"position": 1
}
]
}
All the tokens in the returned order need to replace the query text in the percolator query:
PUT /test_index/_doc/1?refresh
{
"query" : {
"match" : {
"body" : {
"query" : "miss bicycl",
"analyzer" : "whitespace" (1)
}
}
}
}
-
It is important to select a whitespace analyzer here, otherwise the analyzer defined in the mapping will be used, which defeats the point of using this workflow. Note that
whitespace
is a built-in analyzer, if a different analyzer needs to be used, it needs to be configured first in the index’s settings.
The analyze api prior to the indexing the percolator flow should be done for each percolator query.
At percolate time nothing changes and the percolate
query can be defined normally:
GET /test_index/_search
{
"query": {
"percolate" : {
"field" : "query",
"document" : {
"body" : "Bycicles are missing"
}
}
}
}
This results in a response like this:
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped" : 0,
"failed": 0
},
"hits": {
"total" : {
"value": 1,
"relation": "eq"
},
"max_score": 0.13076457,
"hits": [
{
"_index": "test_index",
"_type": "_doc",
"_id": "1",
"_score": 0.13076457,
"_source": {
"query": {
"match": {
"body": {
"query": "miss bicycl",
"analyzer": "whitespace"
}
}
}
},
"fields" : {
"_percolator_document_slot" : [0]
}
}
]
}
}
Optimizing wildcard queries.
Wildcard queries are more expensive than other queries for the percolator, especially if the wildcard expressions are large.
In the case of wildcard
queries with prefix wildcard expressions or just the prefix
query,
the edge_ngram
token filter can be used to replace these queries with regular term
query on a field where the edge_ngram
token filter is configured.
Creating an index with custom analysis settings:
PUT my_queries1
{
"settings": {
"analysis": {
"analyzer": {
"wildcard_prefix": { (1)
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"wildcard_edge_ngram"
]
}
},
"filter": {
"wildcard_edge_ngram": { (2)
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 32
}
}
}
},
"mappings": {
"properties": {
"query": {
"type": "percolator"
},
"my_field": {
"type": "text",
"fields": {
"prefix": { (3)
"type": "text",
"analyzer": "wildcard_prefix",
"search_analyzer": "standard"
}
}
}
}
}
}
-
The analyzer that generates the prefix tokens to be used at index time only.
-
Increase the
min_gram
and decreasemax_gram
settings based on your prefix search needs. -
This multifield should be used to do the prefix search with a
term
ormatch
query instead of aprefix
orwildcard
query.
Then instead of indexing the following query:
{
"query": {
"wildcard": {
"my_field": "abc*"
}
}
}
this query below should be indexed:
PUT /my_queries1/_doc/1?refresh
{
"query": {
"term": {
"my_field.prefix": "abc"
}
}
}
This way can handle the second query more efficiently than the first query.
The following search request will match with the previously indexed percolator query:
GET /my_queries1/_search
{
"query": {
"percolate": {
"field": "query",
"document": {
"my_field": "abcd"
}
}
}
}
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total" : {
"value": 1,
"relation": "eq"
},
"max_score": 0.18864399,
"hits": [
{
"_index": "my_queries1",
"_type": "_doc",
"_id": "1",
"_score": 0.18864399,
"_source": {
"query": {
"term": {
"my_field.prefix": "abc"
}
}
},
"fields": {
"_percolator_document_slot": [
0
]
}
}
]
}
}
The same technique can also be used to speed up suffix
wildcard searches. By using the reverse
token filter
before the edge_ngram
token filter.
PUT my_queries2
{
"settings": {
"analysis": {
"analyzer": {
"wildcard_suffix": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"reverse",
"wildcard_edge_ngram"
]
},
"wildcard_suffix_search_time": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"reverse"
]
}
},
"filter": {
"wildcard_edge_ngram": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 32
}
}
}
},
"mappings": {
"properties": {
"query": {
"type": "percolator"
},
"my_field": {
"type": "text",
"fields": {
"suffix": {
"type": "text",
"analyzer": "wildcard_suffix",
"search_analyzer": "wildcard_suffix_search_time" (1)
}
}
}
}
}
}
-
A custom analyzer is needed at search time too, because otherwise the query terms are not being reversed and would otherwise not match with the reserved suffix tokens.
Then instead of indexing the following query:
{
"query": {
"wildcard": {
"my_field": "*xyz"
}
}
}
the following query below should be indexed:
PUT /my_queries2/_doc/2?refresh
{
"query": {
"match": { (1)
"my_field.suffix": "xyz"
}
}
}
-
The
match
query should be used instead of theterm
query, because text analysis needs to reverse the query terms.
The following search request will match with the previously indexed percolator query:
GET /my_queries2/_search
{
"query": {
"percolate": {
"field": "query",
"document": {
"my_field": "wxyz"
}
}
}
}
Dedicated Percolator Index
Percolate queries can be added to any index. Instead of adding percolate queries to the index the data resides in, these queries can also be added to a dedicated index. The advantage of this is that this dedicated percolator index can have its own index settings (For example the number of primary and replica shards). If you choose to have a dedicated percolate index, you need to make sure that the mappings from the normal index are also available on the percolate index. Otherwise percolate queries can be parsed incorrectly.
Forcing Unmapped Fields to be Handled as Strings
In certain cases it is unknown what kind of percolator queries do get registered, and if no field mapping exists for fields
that are referred by percolator queries then adding a percolator query fails. This means the mapping needs to be updated
to have the field with the appropriate settings, and then the percolator query can be added. But sometimes it is sufficient
if all unmapped fields are handled as if these were default text fields. In those cases one can configure the
index.percolator.map_unmapped_fields_as_text
setting to true
(default to false
) and then if a field referred in
a percolator query does not exist, it will be handled as a default text field so that adding the percolator query doesn’t
fail.
Limitations
Parent/child
Because the percolate
query is processing one document at a time, it doesn’t support queries and filters that run
against child documents such as has_child
and has_parent
.
Fetching queries
There are a number of queries that fetch data via a get call during query parsing. For example the terms
query when
using terms lookup, template
query when using indexed scripts and geo_shape
when using pre-indexed shapes. When these
queries are indexed by the percolator
field type then the get call is executed once. So each time the percolator
query evaluates these queries, the fetches terms, shapes etc. as the were upon index time will be used. Important to note
is that fetching of terms that these queries do, happens both each time the percolator query gets indexed on both primary
and replica shards, so the terms that are actually indexed can be different between shard copies, if the source index
changed while indexing.
Script query
The script inside a script
query can only access doc values fields. The percolate
query indexes the provided document
into an in-memory index. This in-memory index doesn’t support stored fields and because of that the _source
field and
other stored fields are not stored. This is the reason why in the script
query the _source
and other stored fields
aren’t available.
Field aliases
Percolator queries that contain field aliases may not always behave as expected. In particular, if a percolator query is registered that contains a field alias, and then that alias is updated in the mappings to refer to a different field, the stored query will still refer to the original target field. To pick up the change to the field alias, the percolator query must be explicitly reindexed.
Range field types
Range field types represent a continuous range of values between an upper and lower
bound. For example, a range can represent any date in October or any
integer from 0 to 9. They are defined using the operators
gt
or gte
for the lower bound, and lt
or lte
for the upper bound.
They can be used for querying, and have
limited support for aggregations. The only supported aggregations are
histogram,
cardinality.
The following range types are supported:
integer_range
|
A range of signed 32-bit integers with a minimum value of -2^31^ and maximum of 2^31^-1. |
float_range
|
A range of single-precision 32-bit IEEE 754 floating point values. |
long_range
|
A range of signed 64-bit integers with a minimum value of -2^63^ and maximum of 2^63^-1. |
double_range
|
A range of double-precision 64-bit IEEE 754 floating point values. |
date_range
|
A range of |
ip_range
|
A range of ip values supporting either IPv4 or IPv6 (or mixed) addresses. |
Below is an example of configuring a mapping with various range fields followed by an example that indexes several range types.
PUT range_index
{
"settings": {
"number_of_shards": 2
},
"mappings": {
"properties": {
"expected_attendees": {
"type": "integer_range"
},
"time_frame": {
"type": "date_range", (1)
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
}
}
}
}
PUT range_index/_doc/1?refresh
{
"expected_attendees" : { (2)
"gte" : 10,
"lt" : 20
},
"time_frame" : {
"gte" : "2015-10-31 12:00:00", (3)
"lte" : "2015-11-01"
}
}
-
date_range
types accept the same field parameters defined by thedate
type. -
Example indexing a meeting with 10 to 20 attendees, not including 20.
-
Example date range using date time stamp.
The following is an example of a term query on the integer_range
field named "expected_attendees".
12 is a value inside the range, so it will match.
GET range_index/_search
{
"query" : {
"term" : {
"expected_attendees" : {
"value": 12
}
}
}
}
The result produced by the above query.
{
"took": 13,
"timed_out": false,
"_shards" : {
"total": 2,
"successful": 2,
"skipped" : 0,
"failed": 0
},
"hits" : {
"total" : {
"value": 1,
"relation": "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "range_index",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"expected_attendees" : {
"gte" : 10, "lt" : 20
},
"time_frame" : {
"gte" : "2015-10-31 12:00:00", "lte" : "2015-11-01"
}
}
}
]
}
}
The following is an example of a date_range
query over the date_range
field named "time_frame".
GET range_index/_search
{
"query" : {
"range" : {
"time_frame" : { (1)
"gte" : "2015-10-31",
"lte" : "2015-11-01",
"relation" : "within" (2)
}
}
}
}
-
Range queries work the same as described in range query.
-
Range queries over range fields support a
relation
parameter which can be one ofWITHIN
,CONTAINS
,INTERSECTS
(default).
This query produces a similar result:
{
"took": 13,
"timed_out": false,
"_shards" : {
"total": 2,
"successful": 2,
"skipped" : 0,
"failed": 0
},
"hits" : {
"total" : {
"value": 1,
"relation": "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "range_index",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"expected_attendees" : {
"gte" : 10, "lt" : 20
},
"time_frame" : {
"gte" : "2015-10-31 12:00:00", "lte" : "2015-11-01"
}
}
}
]
}
}
IP Range
In addition to the range format above, IP ranges can be provided in CIDR notation:
PUT range_index/_mapping
{
"properties": {
"ip_allowlist": {
"type": "ip_range"
}
}
}
PUT range_index/_doc/2
{
"ip_allowlist" : "<IP_ADDRESS>/16"
}
Parameters for range fields
The following parameters are accepted by range types:
coerce
|
Try to convert strings to numbers and truncate fractions for integers.
Accepts |
boost
|
Mapping field-level query time boosting. Accepts a floating point number, defaults
to |
index
|
Should the field be searchable? Accepts |
store
|
Whether the field value should be stored and retrievable separately from
the |
Rank feature field type
A rank_feature
field can index numbers so that they can later be used to boost
documents in queries with a rank_feature
query.
PUT my-index-000001
{
"mappings": {
"properties": {
"pagerank": {
"type": "rank_feature" (1)
},
"url_length": {
"type": "rank_feature",
"positive_score_impact": false (2)
}
}
}
}
PUT my-index-000001/_doc/1
{
"pagerank": 8,
"url_length": 22
}
GET my-index-000001/_search
{
"query": {
"rank_feature": {
"field": "pagerank"
}
}
}
-
Rank feature fields must use the
rank_feature
field type -
Rank features that correlate negatively with the score need to declare it
|
Note
|
rank_feature fields only support single-valued fields and strictly positive
values. Multi-valued fields and negative values will be rejected.
|
|
Note
|
rank_feature fields do not support querying, sorting or aggregating. They may
only be used within rank_feature queries.
|
|
Note
|
rank_feature fields only preserve 9 significant bits for the precision, which
translates to a relative error of about 0.4%.
|
Rank features that correlate negatively with the score should set
positive_score_impact
to false
(defaults to true
). This will be used by
the rank_feature
query to modify the scoring formula
in such a way that the score decreases with the value of the feature instead of
increasing. For instance in web search, the url length is a commonly used
feature which correlates negatively with scores.
Rank features field type
A rank_features
field can index numeric feature vectors, so that they can
later be used to boost documents in queries with a
rank_feature
query.
It is analogous to the rank_feature
data type but is better suited
when the list of features is sparse so that it wouldn’t be reasonable to add
one field to the mappings for each of them.
PUT my-index-000001
{
"mappings": {
"properties": {
"topics": {
"type": "rank_features" (1)
}
}
}
}
PUT my-index-000001/_doc/1
{
"topics": { (2)
"politics": 20,
"economics": 50.8
}
}
PUT my-index-000001/_doc/2
{
"topics": {
"politics": 5.2,
"sports": 80.1
}
}
GET my-index-000001/_search
{
"query": {
"rank_feature": {
"field": "topics.politics"
}
}
}
-
Rank features fields must use the
rank_features
field type -
Rank features fields must be a hash with string keys and strictly positive numeric values
|
Note
|
rank_features fields only support single-valued features and strictly
positive values. Multi-valued fields and zero or negative values will be rejected.
|
|
Note
|
rank_features fields do not support sorting or aggregating and may
only be queried using rank_feature queries.
|
|
Note
|
rank_features fields only preserve 9 significant bits for the
precision, which translates to a relative error of about 0.4%.
|
Search-as-you-type field type
The search_as_you_type
field type is a text-like field that is optimized to
provide out-of-the-box support for queries that serve an as-you-type completion
use case. It creates a series of subfields that are analyzed to index terms
that can be efficiently matched by a query that partially matches the entire
indexed text value. Both prefix completion (i.e matching terms starting at the
beginning of the input) and infix completion (i.e. matching terms at any
position within the input) are supported.
When adding a field of this type to a mapping
PUT my-index-000001
{
"mappings": {
"properties": {
"my_field": {
"type": "search_as_you_type"
}
}
}
}
This creates the following fields
my_field
|
Analyzed as configured in the mapping. If an analyzer is not configured, the default analyzer for the index is used |
my_field._2gram
|
Wraps the analyzer of |
my_field._3gram
|
Wraps the analyzer of |
my_field._index_prefix
|
Wraps the analyzer of |
The size of shingles in subfields can be configured with the max_shingle_size
mapping parameter. The default is 3, and valid values for this parameter are
integer values 2 - 4 inclusive. Shingle subfields will be created for each
shingle size from 2 up to and including the max_shingle_size
. The
my_field._index_prefix
subfield will always use the analyzer from the shingle
subfield with the max_shingle_size
when constructing its own analyzer.
Increasing the max_shingle_size
will improve matches for queries with more
consecutive terms, at the cost of larger index size. The default
max_shingle_size
should usually be sufficient.
The same input text is indexed into each of these fields automatically, with
their differing analysis chains, when an indexed document has a value for the
root field my_field
.
PUT my-index-000001/_doc/1?refresh
{
"my_field": "quick brown fox jump lazy dog"
}
The most efficient way of querying to serve a search-as-you-type use case is
usually a multi_match
query of type
bool_prefix
that targets the root
search_as_you_type
field and its shingle subfields. This can match the query
terms in any order, but will score documents higher if they contain the terms
in order in a shingle subfield.
GET my-index-000001/_search
{
"query": {
"multi_match": {
"query": "brown f",
"type": "bool_prefix",
"fields": [
"my_field",
"my_field._2gram",
"my_field._3gram"
]
}
}
}
{
"took" : 44,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 0.8630463,
"hits" : [
{
"_index" : "my-index-000001",
"_type" : "_doc",
"_id" : "1",
"_score" : 0.8630463,
"_source" : {
"my_field" : "quick brown fox jump lazy dog"
}
}
]
}
}
To search for documents that strictly match the query terms in order, or to
search using other properties of phrase queries, use a
match_phrase_prefix
query on the root
field. A match_phrase
query can also be used
if the last term should be matched exactly, and not as a prefix. Using phrase
queries may be less efficient than using the match_bool_prefix
query.
GET my-index-000001/_search
{
"query": {
"match_phrase_prefix": {
"my_field": "brown f"
}
}
}
Parameters specific to the search_as_you_type
field
The following parameters are accepted in a mapping for the search_as_you_type
field and are specific to this field type
max_shingle_size
-
(Optional, integer) Largest shingle size to create. Valid values are
2
(inclusive) to4
(inclusive). Defaults to3
.A subfield is created for each integer between
2
and this value. For example, a value of3
creates two subfields:my_field._2gram
andmy_field._3gram
More subfields enables more specific queries but increases index size.
Parameters of the field type as a text field
The following parameters are accepted in a mapping for the search_as_you_type
field due to its nature as a text-like field, and behave similarly to their
behavior when configuring a field of the text
data type. Unless
otherwise noted, these options configure the root fields subfields in
the same way.
analyzer
-
The analyzer which should be used for
text
fields, both at index-time and at search-time (unless overridden by thesearch_analyzer
). Defaults to the default index analyzer, or thestandard
analyzer. index
-
Should the field be searchable? Accepts
true
(default) orfalse
. index_options
-
What information should be stored in the index, for search and highlighting purposes. Defaults to
positions
. norms
-
Whether field-length should be taken into account when scoring queries. Accepts
true
orfalse
. This option configures the root field and shingle subfields, where its default istrue
. It does not configure the prefix subfield, where it itfalse
. store
-
Whether the field value should be stored and retrievable separately from the
_source
field. Acceptstrue
orfalse
(default). This option only configures the root field, and does not configure any subfields. search_analyzer
-
The
analyzer
that should be used at search time ontext
fields. Defaults to theanalyzer
setting. search_quote_analyzer
-
The
analyzer
that should be used at search time when a phrase is encountered. Defaults to thesearch_analyzer
setting. similarity
-
Which scoring algorithm or similarity should be used. Defaults to
BM25
. term_vector
-
Whether term vectors should be stored for the field. Defaults to
no
. This option configures the root field and shingle subfields, but not the prefix subfield.
Optimization of prefix queries
When making a prefix
query to the root field or
any of its subfields, the query will be rewritten to a
term
query on the ._index_prefix
subfield. This
matches more efficiently than is typical of prefix
queries on text fields,
as prefixes up to a certain length of each shingle are indexed directly as
terms in the ._index_prefix
subfield.
The analyzer of the ._index_prefix
subfield slightly modifies the
shingle-building behavior to also index prefixes of the terms at the end of the
field’s value that normally would not be produced as shingles. For example, if
the value quick brown fox
is indexed into a search_as_you_type
field with
max_shingle_size
of 3, prefixes for brown fox
and fox
are also indexed
into the ._index_prefix
subfield even though they do not appear as terms in
the ._3gram
subfield. This allows for completion of all the terms in the
field’s input.
Text field type
A field to index full-text values, such as the body of an email or the
description of a product. These fields are analyzed
, that is they are passed through an
analyzer to convert the string into a list of individual terms
before being indexed. The analysis process allows OpenSearch to search for
individual words within each full text field. Text fields are not
used for sorting and seldom used for aggregations (although the
significant text aggregation
is a notable exception).
If you need to index structured content such as email addresses, hostnames, status
codes, or tags, it is likely that you should rather use a keyword
field.
Below is an example of a mapping for a text field:
PUT my-index-000001
{
"mappings": {
"properties": {
"full_name": {
"type": "text"
}
}
}
}
Use a field as both text and keyword
Sometimes it is useful to have both a full text (text
) and a keyword
(keyword
) version of the same field: one for full text search and the
other for aggregations and sorting. This can be achieved with
multi-fields.
Parameters for text fields
The following parameters are accepted by text
fields:
analyzer
|
The analyzer which should be used for
the |
boost
|
Mapping field-level query time boosting. Accepts a floating point number, defaults
to |
eager_global_ordinals
|
Should global ordinals be loaded eagerly on refresh? Accepts |
fielddata
|
Can the field use in-memory fielddata for sorting, aggregations,
or scripting? Accepts |
fielddata_frequency_filter
|
Expert settings which allow to decide which values to load in memory when |
fields
|
Multi-fields allow the same string value to be indexed in multiple ways for different purposes, such as one field for search and a multi-field for sorting and aggregations, or the same string value analyzed by different analyzers. |
index
|
Should the field be searchable? Accepts |
index_options
|
What information should be stored in the index, for search and highlighting purposes.
Defaults to |
index_prefixes
|
If enabled, term prefixes of between 2 and 5 characters are indexed into a separate field. This allows prefix searches to run more efficiently, at the expense of a larger index. |
index_phrases
|
If enabled, two-term word combinations ('shingles') are indexed into a separate
field. This allows exact phrase queries (no slop) to run more efficiently, at the expense
of a larger index. Note that this works best when stopwords are not removed,
as phrases containing stopwords will not use the subsidiary field and will fall
back to a standard phrase query. Accepts |
norms
|
Whether field-length should be taken into account when scoring queries.
Accepts |
position_increment_gap
|
The number of fake term position which should be inserted between each
element of an array of strings. Defaults to the |
store
|
Whether the field value should be stored and retrievable separately from
the |
search_analyzer
|
The |
search_quote_analyzer
|
The |
similarity
|
Which scoring algorithm or similarity should be used. Defaults
to |
term_vector
|
Whether term vectors should be stored for the field. Defaults to |
meta
|
Metadata about the field. |
fielddata
mapping parameter
text
fields are searchable by default, but by default are not available for
aggregations, sorting, or scripting. If you try to sort, aggregate, or access
values from a script on a text
field, you will see this exception:
Fielddata is disabled on text fields by default. Set fielddata=true
on
your_field_name
in order to load fielddata in memory by uninverting the
inverted index. Note that this can however use significant memory.
Field data is the only way to access the analyzed tokens from a full text field
in aggregations, sorting, or scripting. For example, a full text field like New York
would get analyzed as new
and york
. To aggregate on these tokens requires field data.
Before enabling fielddata
It usually doesn’t make sense to enable fielddata on text fields. Field data is stored in the heap with the field data cache because it is expensive to calculate. Calculating the field data can cause latency spikes, and increasing heap usage is a cause of cluster performance issues.
Most users who want to do more with text fields use multi-field mappings
by having both a text
field for full text searches, and an
unanalyzed keyword
field for aggregations, as follows:
PUT my-index-000001
{
"mappings": {
"properties": {
"my_field": { (1)
"type": "text",
"fields": {
"keyword": { (2)
"type": "keyword"
}
}
}
}
}
}
-
Use the
my_field
field for searches. -
Use the
my_field.keyword
field for aggregations, sorting, or in scripts.
Enabling fielddata on text
fields
You can enable fielddata on an existing text
field using the
PUT mapping API as follows:
PUT my-index-000001/_mapping
{
"properties": {
"my_field": { (1)
"type": "text",
"fielddata": true
}
}
}
-
The mapping that you specify for
my_field
should consist of the existing mapping for that field, plus thefielddata
parameter.
fielddata_frequency_filter
mapping parameter
Fielddata filtering can be used to reduce the number of terms loaded into memory, and thus reduce memory usage. Terms can be filtered by frequency:
The frequency filter allows you to only load terms whose document frequency falls
between a min
and max
value, which can be expressed an absolute
number (when the number is bigger than 1.0) or as a percentage
(eg 0.01
is 1%
and 1.0
is 100%
). Frequency is calculated
per segment. Percentages are based on the number of docs which have a
value for the field, as opposed to all docs in the segment.
Small segments can be excluded completely by specifying the minimum
number of docs that the segment should contain with min_segment_size
:
PUT my-index-000001
{
"mappings": {
"properties": {
"tag": {
"type": "text",
"fielddata": true,
"fielddata_frequency_filter": {
"min": 0.001,
"max": 0.1,
"min_segment_size": 500
}
}
}
}
}
Token count field type
A field of type token_count
is really an integer
field which
accepts string values, analyzes them, then indexes the number of tokens in the
string.
For instance:
PUT my-index-000001
{
"mappings": {
"properties": {
"name": { (1)
"type": "text",
"fields": {
"length": { (2)
"type": "token_count",
"analyzer": "standard"
}
}
}
}
}
}
PUT my-index-000001/_doc/1
{ "name": "John Smith" }
PUT my-index-000001/_doc/2
{ "name": "Rachel Alice Williams" }
GET my-index-000001/_search
{
"query": {
"term": {
"name.length": 3 (3)
}
}
}
-
The
name
field is atext
field which uses the defaultstandard
analyzer. -
The
name.length
field is atoken_count
multi-field which will index the number of tokens in thename
field. -
This query matches only the document containing
Rachel Alice Williams
, as it contains three tokens.
Parameters for token_count
fields
The following parameters are accepted by token_count
fields:
analyzer
|
The analyzer which should be used to analyze the string value. Required. For best performance, use an analyzer without token filters. |
enable_position_increments
|
Indicates if position increments should be counted.
Set to |
boost
|
Mapping field-level query time boosting. Accepts a floating point number, defaults
to |
doc_values
|
Should the field be stored on disk in a column-stride fashion, so that it
can later be used for sorting, aggregations, or scripting? Accepts |
index
|
Should the field be searchable? Accepts |
null_value
|
Accepts a numeric value of the same |
store
|
Whether the field value should be stored and retrievable separately from
the |


SHORTTITLE PROFILEPICTURE
| Other Names | OTHERNAMES |
| Surname | SURNAME |
| Forename | FORENAME |
| Rank | RANK |
| Addressed as | ``ADDRESSEDAS'' |
SHORTISHTITLE
BIOGRAPHY
Cyprian Data
Cyprian Legal Name
CYPRIANLEGALNAME
Other Titles
OTHERTITLES
ACCOLADES ARMSOther Data
OTHERDATA


Publications
-
B. Bengfort and P. Keleher, “Brief Announcement: Hierarchical Consensus,” in Proceedings of the 2017 ACM Symposium on Principles of Distributed Computing, Washington, D.C., 2017, pp. 355–357.
-
B. Bengfort and P. Keleher, “Federating Consistency for Partition-Prone Networks,” Proceedings of the 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, GA, 2017.
-
B. Bengfort, R. Bilbro, and T. Ojeda, Applied Text Analysis with Python: Enabling Language Aware Data Products with Machine Learning. O’Reilly Media, Inc., 2017.
-
V. Eyorokon, B. Bengfort, P. Uday, and M. Cox, “Goal Trajectories for Knowledge Investigation,” in Twenty-Forth International Conference on Case-Based Reasoning Workshop Proceedings: Synergies between CBR and Knowledge Discovery, Atlanta, GA, 2016, pp. 202–211.
-
B. Bengfort and M. T. Cox, “Interactive Knowledge-Goal Reasoning,” in Goal Reasoning: Papers from the ACS Workshop, 2015, pp. 10. [PDF]
-
B. Bengfort and J. Kim, Hadoop Fundamentals for Data Scientists. O’Reilly Media, 2015.
-
B. Bengfort, P. Y. Kim, K. Harrison, and J. A. Reggia, “Evolutionary design of self-organizing particle systems for collective problem solving,” in Swarm Intelligence (SIS), 2014 IEEE Symposium on, 2014, pp. 1–8. [PDF]
-
T. Ojeda, S. P. Murphy, B. Bengfort, and A. Dasgupta, Practical Data Science Cookbook. Packt Publishing Ltd, 2014.
-
B. Bengfort, “A Survey of Stochastic and Gazetteer Based Approaches for Named Entity Recognition,” University of Maryland Baltimore County, Nov. 2012. [PDF]
-
B. Bengfort, W. Zhang, and X. Du, “Efficient resource allocation in Hybrid Wireless Networks,” in Wireless Communications and Networking Conference (WCNC), 2011 IEEE, 2011, pp. 820–825. [PDF]
Proposal
- B. Bengfort, P. Keleher, “Responsive Consistency in User-Centric Dynamic Network Environments,” proposal for the degree of Doctor of Philosophy.
[PDF] [Slides]
Talks
-
B. Bengfort, “Visual Pipelines for Text Analysis,” presented at the The Data Intelligence Conference, Tyson’s Corner, Virginia, 24-Jun-2017.
-
B. Bengfort, “Dynamics in Graph Analysis: Adding Time as a Structure for Visual and Statistical Insight,” presented at the PyData DC 2016, Washington D.C., 09-Oct-2016. [Video]
-
B. Bengfort, “Dynamics in Graph Analysis: Adding Time as a Structure for Visual and Statistical Insight,” presented at the PyData Carolinas 2016, Raleigh, NC, 15-Sep-2016. [Video] [Slides]
-
B. Bengfort, “Visualizing the Model Selection Process,” presented at the Data Day Seattle 2016, Seattle, Washington, 23-Jul-2016. [Slides]
-
B. Bengfort, “Data Product Architectures,” presented at the Seattle Data Geeks Meetup, Seattle, Washington, 21-Jul-2016. [Slides]
-
B. Bengfort, “Natural Language Processing with NLTK and Gensim,” presented at the PyCon 2016, Portland, Oregon, 29-May-2016. [Video] [Slides]
-
B. Bengfort and S. P. Murphy, “Teaching the Elephant to Read: Hadoop, Python, and NLP,” presented at the Strata Conference + Hadoop World, New York, NY, USA, 28-Oct-2013. [Video] [Slides]
-
B. Bengfort, “Natural Language Processing of Big Data using NLTK and Hadoop Streaming,” presented at the Data Science DC: Big Data Week, George Washington University, Washington DC, 23-Apr-2013.
Posters
-
B. Bengfort and P. Keleher, “Federating Consistency for Partition-Prone Networks,” Poster Presented at 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, GA, 2017. [Poster]
-
R. Bilbro and B. Bengfort, “Yellowbrick: Steering Scikit-Learn with Visual Transformers,” presented at the PyCon 2017 Poster Session, Portland, Oregon, 21-May-2017. [Poster]
-
B. Bengfort, W. Voorhees, R. Bilbro, and T. Ojeda, “On the Hour Data Ingestion from the Web to a Mongo Database,” presented at the PyCon 2017 Poster Session, Portland, Oregon, 21-May-2017. [Poster]
-
B. Bengfort, L. Lorenz, and R. Bilbro, “Model Management Systems: Scikit-Learn and Django,” presented at the PyCon 2017, Portland, Oregon, 21-May-2017. [Poster]
-
B. Bengfort and R. Bilbro, “An Architecture for Machine Learning in Django: Web Applications that Learn by Example,” presented at the PyCon 2016 Poster Session, Portland, Oregon, 01-Jun-2016. [Poster] [Poster]
-
B. Bengfort, “Evolutionary Design of Particle Swarms: Science with Python, Celery, and PyGame,” presented at the PyCon 2016 Poster Session, Portland, Oregon, 01-Jun-2016. [Poster] [Poster]


AP Top Political News at 1:43 a.m. EST
by EndPlayAbove-ground power lines grow in risk as climate changes
As more men run, Klobuchar makes case for ‘woman candidate'
Pentagon chief fires Navy secretary over SEAL controversy
Michael Bloomberg launches Democratic presidential bid
Supreme Court says Ginsburg released from hospital
Report: Review details effort to justify Ukraine decision
She can't vote, but 2020 Democrats want her support anyway
Sarah Sanders eyes possible run for governor of Arkansas
Schiff says more hearings, witnesses possible
Feds: Give Chinese woman prison for Mar-a-Lago trespass


Agilla Documentation
-
Want to jump start your way into using Agilla? Work through the tutorials!
-
A list of instructions that an agent can execute.
-
A collection of tips on how to get around potential problems
-
Specifies the various types of variables supported by Agilla.
-
Specifies the active message numbers of every message used in Agilla.
-
Describes the abstractions provided by
an agent. This includes the user-viewable abstractions like the
stack and heap as well as a description of the various data types
available.
-
For experimental purposes, Agilla utilized a grid topology and filters
to mimic a multi-hop network in a small area. This documentation
describes how to change the dimensions of the grid.
-
Agilla maintains special tuples in each tuplespace that tell agents about their context.
These tuples include sensor types and AgentIDs. This documentation lists these tuples and
defines their format.
-
Agilla puts a mote into an error state if an agent attempts to execute
an illegal instruction. This documentation explains how to interpret the error and offers suggestions on how to fix them.
-
Agilla has been ported over to the Tyndall25 motes. These are very small 25mm modules that have the Atmel 128L processor.
This work is supported by the ONR MURI Project CONTESSA
and the NSF under grant number CCR-9970939.