Metrics aggregations
The aggregations in this family compute metrics based on values extracted in one way or another from the documents that are being aggregated. The values are typically extracted from the fields of the document (using the field data), but can also be generated using scripts.
Numeric metrics aggregations are a special type of metrics aggregation which output numeric values. Some aggregations output
a single numeric metric (e.g. avg
) and are called single-value numeric metrics aggregation
, others generate multiple
metrics (e.g. stats
) and are called multi-value numeric metrics aggregation
. The distinction between single-value and
multi-value numeric metrics aggregations plays a role when these aggregations serve as direct sub-aggregations of some
bucket aggregations (some bucket aggregations enable you to sort the returned buckets based on the numeric metrics in each bucket).
Avg aggregation
A single-value
metrics aggregation that computes the average of numeric values that are extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.
Assuming the data consists of documents representing exams grades (between 0 and 100) of students we can average their scores with:
POST /exams/_search?size=0
{
"aggs": {
"avg_grade": { "avg": { "field": "grade" } }
}
}
The above aggregation computes the average grade over all documents. The aggregation type is avg
and the field
setting defines the numeric field of the documents the average will be computed on. The above will return the following:
{
...
"aggregations": {
"avg_grade": {
"value": 75.0
}
}
}
The name of the aggregation (avg_grade
above) also serves as the key by which the aggregation result can be retrieved from the returned response.
Script
Computing the average grade based on a script:
POST /exams/_search?size=0
{
"aggs": {
"avg_grade": {
"avg": {
"script": {
"source": "doc.grade.value"
}
}
}
}
}
This will interpret the script
parameter as an inline
script with the painless
script language and no script parameters. To use a stored script use the following syntax:
POST /exams/_search?size=0
{
"aggs": {
"avg_grade": {
"avg": {
"script": {
"id": "my_script",
"params": {
"field": "grade"
}
}
}
}
}
}
Value Script
It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new average:
POST /exams/_search?size=0
{
"aggs": {
"avg_corrected_grade": {
"avg": {
"field": "grade",
"script": {
"lang": "painless",
"source": "_value * params.correction",
"params": {
"correction": 1.2
}
}
}
}
}
}
Missing value
The missing
parameter defines how documents that are missing a value should be treated.
By default they will be ignored but it is also possible to treat them as if they
had a value.
POST /exams/_search?size=0
{
"aggs": {
"grade_avg": {
"avg": {
"field": "grade",
"missing": 10 (1)
}
}
}
}
-
Documents without a value in the
grade
field will fall into the same bucket as documents that have the value10
.
Cardinality aggregation
A single-value
metrics aggregation that calculates an approximate count of
distinct values. Values can be extracted either from specific fields in the
document or generated by a script.
Assume you are indexing store sales and would like to count the unique number of sold products that match a query:
POST /sales/_search?size=0
{
"aggs": {
"type_count": {
"cardinality": {
"field": "type"
}
}
}
}
Response:
{
...
"aggregations": {
"type_count": {
"value": 3
}
}
}
Precision control
This aggregation also supports the precision_threshold
option:
POST /sales/_search?size=0
{
"aggs": {
"type_count": {
"cardinality": {
"field": "type",
"precision_threshold": 100 (1)
}
}
}
}
-
The
precision_threshold
options allows to trade memory for accuracy, and defines a unique count below which counts are expected to be close to accurate. Above this value, counts might become a bit more fuzzy. The maximum supported value is 40000, thresholds above this number will have the same effect as a threshold of 40000. The default value is 3000.
Counts are approximate
Computing exact counts requires loading values into a hash set and returning its size. This doesn’t scale when working on high-cardinality sets and/or large values as the required memory usage and the need to communicate those per-shard sets between nodes would utilize too many resources of the cluster.
This cardinality
aggregation is based on the
HyperLogLog++
algorithm, which counts based on the hashes of the values with some interesting
properties:
-
configurable precision, which decides on how to trade memory for accuracy,
-
excellent accuracy on low-cardinality sets,
-
fixed memory usage: no matter if there are tens or billions of unique values, memory usage only depends on the configured precision.
For a precision threshold of c
, the implementation that we are using requires
about c * 8
bytes.
The following chart shows how the error varies before and after the threshold:
For all 3 thresholds, counts have been accurate up to the configured threshold. Although not guaranteed, this is likely to be the case. Accuracy in practice depends on the dataset in question. In general, most datasets show consistently good accuracy. Also note that even with a threshold as low as 100, the error remains very low (1-6% as seen in the above graph) even when counting millions of items.
The HyperLogLog++ algorithm depends on the leading zeros of hashed values, the exact distributions of hashes in a dataset can affect the accuracy of the cardinality.
Please also note that even with a threshold as low as 100, the error remains very low, even when counting millions of items.
Pre-computed hashes
On string fields that have a high cardinality, it might be faster to store the
hash of your field values in your index and then run the cardinality aggregation
on this field. This can either be done by providing hash values from client-side
or by letting OpenSearch compute hash values for you by using the
mapper-murmur3
plugin.
|
Note
|
Pre-computing hashes is usually only useful on very large and/or high-cardinality fields as it saves CPU and memory. However, on numeric fields, hashing is very fast and storing the original values requires as much or less memory than storing the hashes. This is also true on low-cardinality string fields, especially given that those have an optimization in order to make sure that hashes are computed at most once per unique value per segment. |
Script
The cardinality
metric supports scripting, with a noticeable performance hit
however since hashes need to be computed on the fly.
POST /sales/_search?size=0
{
"aggs": {
"type_promoted_count": {
"cardinality": {
"script": {
"lang": "painless",
"source": "doc['type'].value + ' ' + doc['promoted'].value"
}
}
}
}
}
This will interpret the script
parameter as an inline
script with the painless
script language and no script parameters. To use a stored script use the following syntax:
POST /sales/_search?size=0
{
"aggs": {
"type_promoted_count": {
"cardinality": {
"script": {
"id": "my_script",
"params": {
"type_field": "type",
"promoted_field": "promoted"
}
}
}
}
}
}
Missing value
The missing
parameter defines how documents that are missing a value should be treated.
By default they will be ignored but it is also possible to treat them as if they
had a value.
POST /sales/_search?size=0
{
"aggs": {
"tag_cardinality": {
"cardinality": {
"field": "tag",
"missing": "N/A" (1)
}
}
}
}
-
Documents without a value in the
tag
field will fall into the same bucket as documents that have the valueN/A
.
Extended stats aggregation
A multi-value
metrics aggregation that computes stats over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.
The extended_stats
aggregations is an extended version of the stats
aggregation, where additional metrics are added such as sum_of_squares
, variance
, std_deviation
and std_deviation_bounds
.
Assuming the data consists of documents representing exams grades (between 0 and 100) of students
GET /exams/_search
{
"size": 0,
"aggs": {
"grades_stats": { "extended_stats": { "field": "grade" } }
}
}
The above aggregation computes the grades statistics over all documents. The aggregation type is extended_stats
and the field
setting defines the numeric field of the documents the stats will be computed on. The above will return the following:
The std_deviation
and variance
are calculated as population metrics so they are always the same as std_deviation_population
and variance_population
respectively.
{
...
"aggregations": {
"grades_stats": {
"count": 2,
"min": 50.0,
"max": 100.0,
"avg": 75.0,
"sum": 150.0,
"sum_of_squares": 12500.0,
"variance": 625.0,
"variance_population": 625.0,
"variance_sampling": 1250.0,
"std_deviation": 25.0,
"std_deviation_population": 25.0,
"std_deviation_sampling": 35.35533905932738,
"std_deviation_bounds": {
"upper": 125.0,
"lower": 25.0,
"upper_population": 125.0,
"lower_population": 25.0,
"upper_sampling":<PHONE_NUMBER>1865476,
"lower_sampling": 4.289321881345245
}
}
}
}
The name of the aggregation (grades_stats
above) also serves as the key by which the aggregation result can be retrieved from the returned response.
Standard Deviation Bounds
By default, the extended_stats
metric will return an object called std_deviation_bounds
, which provides an interval of plus/minus two standard
deviations from the mean. This can be a useful way to visualize variance of your data. If you want a different boundary, for example
three standard deviations, you can set sigma
in the request:
GET /exams/_search
{
"size": 0,
"aggs": {
"grades_stats": {
"extended_stats": {
"field": "grade",
"sigma": 3 (1)
}
}
}
}
-
sigma
controls how many standard deviations +/- from the mean should be displayed
sigma
can be any non-negative double, meaning you can request non-integer values such as 1.5
. A value of 0
is valid, but will simply
return the average for both upper
and lower
bounds.
The upper
and lower
bounds are calculated as population metrics so they are always the same as upper_population
and
lower_population
respectively.
|
Note
|
Standard Deviation and Bounds require normality
The standard deviation and its bounds are displayed by default, but they are not always applicable to all data-sets. Your data must be normally distributed for the metrics to make sense. The statistics behind standard deviations assumes normally distributed data, so if your data is skewed heavily left or right, the value returned will be misleading. |
Script
Computing the grades stats based on a script:
GET /exams/_search
{
"size": 0,
"aggs": {
"grades_stats": {
"extended_stats": {
"script": {
"source": "doc['grade'].value",
"lang": "painless"
}
}
}
}
}
This will interpret the script
parameter as an inline
script with the painless
script language and no script parameters. To use a stored script use the following syntax:
GET /exams/_search
{
"size": 0,
"aggs": {
"grades_stats": {
"extended_stats": {
"script": {
"id": "my_script",
"params": {
"field": "grade"
}
}
}
}
}
}
Value Script
It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new stats:
GET /exams/_search
{
"size": 0,
"aggs": {
"grades_stats": {
"extended_stats": {
"field": "grade",
"script": {
"lang": "painless",
"source": "_value * params.correction",
"params": {
"correction": 1.2
}
}
}
}
}
}
Missing value
The missing
parameter defines how documents that are missing a value should be treated.
By default they will be ignored but it is also possible to treat them as if they
had a value.
GET /exams/_search
{
"size": 0,
"aggs": {
"grades_stats": {
"extended_stats": {
"field": "grade",
"missing": 0 (1)
}
}
}
}
-
Documents without a value in the
grade
field will fall into the same bucket as documents that have the value0
.
Geo-bounds aggregation
A metric aggregation that computes the bounding box containing all geo values for a field.
Example:
PUT /museums
{
"mappings": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
POST /museums/_bulk?refresh
{"index":{"_id":1}}
{"location": "52.374081,4.912350", "name": "NEMO Science Museum"}
{"index":{"_id":2}}
{"location": "52.369219,4.901618", "name": "Museum Het Rembrandthuis"}
{"index":{"_id":3}}
{"location": "52.371667,4.914722", "name": "Nederlands Scheepvaartmuseum"}
{"index":{"_id":4}}
{"location": "51.222900,4.405200", "name": "Letterenhuis"}
{"index":{"_id":5}}
{"location": "48.861111,2.336389", "name": "Musée du Louvre"}
{"index":{"_id":6}}
{"location": "48.860000,2.327000", "name": "Musée d'Orsay"}
POST /museums/_search?size=0
{
"query": {
"match": { "name": "musée" }
},
"aggs": {
"viewport": {
"geo_bounds": {
"field": "location", (1)
"wrap_longitude": true (2)
}
}
}
}
The above aggregation demonstrates how one would compute the bounding box of the location field for all documents with a business type of shop
The response for the above aggregation:
{
...
"aggregations": {
"viewport": {
"bounds": {
"top_left": {
"lat": 48.86111099738628,
"lon": 2.3269999679178
},
"bottom_right": {
"lat": 48.85999997612089,
"lon": 2.3363889567553997
}
}
}
}
}
Geo-centroid aggregation
A metric aggregation that computes the weighted centroid from all coordinate values for geo fields.
Example:
PUT /museums
{
"mappings": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
POST /museums/_bulk?refresh
{"index":{"_id":1}}
{"location": "52.374081,4.912350", "city": "Amsterdam", "name": "NEMO Science Museum"}
{"index":{"_id":2}}
{"location": "52.369219,4.901618", "city": "Amsterdam", "name": "Museum Het Rembrandthuis"}
{"index":{"_id":3}}
{"location": "52.371667,4.914722", "city": "Amsterdam", "name": "Nederlands Scheepvaartmuseum"}
{"index":{"_id":4}}
{"location": "51.222900,4.405200", "city": "Antwerp", "name": "Letterenhuis"}
{"index":{"_id":5}}
{"location": "48.861111,2.336389", "city": "Paris", "name": "Musée du Louvre"}
{"index":{"_id":6}}
{"location": "48.860000,2.327000", "city": "Paris", "name": "Musée d'Orsay"}
POST /museums/_search?size=0
{
"aggs": {
"centroid": {
"geo_centroid": {
"field": "location" (1)
}
}
}
}
-
The
geo_centroid
aggregation specifies the field to use for computing the centroid. (NOTE: field must be a Geo-point field type type)
The above aggregation demonstrates how one would compute the centroid of the location field for all documents with a crime type of burglary.
The response for the above aggregation:
{
...
"aggregations": {
"centroid": {
"location": {
"lat": 51.00982965203002,
"lon": 3.9662131341174245
},
"count": 6
}
}
}
The geo_centroid
aggregation is more interesting when combined as a sub-aggregation to other bucket aggregations.
Example:
POST /museums/_search?size=0
{
"aggs": {
"cities": {
"terms": { "field": "city.keyword" },
"aggs": {
"centroid": {
"geo_centroid": { "field": "location" }
}
}
}
}
}
The above example uses geo_centroid
as a sub-aggregation to a
terms bucket aggregation
for finding the central location for museums in each city.
The response for the above aggregation:
{
...
"aggregations": {
"cities": {
"sum_other_doc_count": 0,
"doc_count_error_upper_bound": 0,
"buckets": [
{
"key": "Amsterdam",
"doc_count": 3,
"centroid": {
"location": {
"lat": 52.371655656024814,
"lon": 4.909563297405839
},
"count": 3
}
},
{
"key": "Paris",
"doc_count": 2,
"centroid": {
"location": {
"lat": 48.86055548675358,
"lon": 2.3316944623366
},
"count": 2
}
},
{
"key": "Antwerp",
"doc_count": 1,
"centroid": {
"location": {
"lat": 51.22289997059852,
"lon": 4.40519998781383
},
"count": 1
}
}
]
}
}
}
|
Warning
|
Using
geo_centroid as a sub-aggregation of geohash_grid The If a |
Matrix stats aggregation
The matrix_stats
aggregation is a numeric aggregation that computes the following statistics over a set of document fields:
count
|
Number of per field samples included in the calculation. |
mean
|
The average value for each field. |
variance
|
Per field Measurement for how spread out the samples are from the mean. |
skewness
|
Per field measurement quantifying the asymmetric distribution around the mean. |
kurtosis
|
Per field measurement quantifying the shape of the distribution. |
covariance
|
A matrix that quantitatively describes how changes in one field are associated with another. |
correlation
|
The covariance matrix scaled to a range of -1 to 1, inclusive. Describes the relationship between field distributions. |
|
Important
|
Unlike other metric aggregations, the matrix_stats aggregation does
not support scripting.
|
The following example demonstrates the use of matrix stats to describe the relationship between income and poverty.
GET /_search
{
"aggs": {
"statistics": {
"matrix_stats": {
"fields": [ "poverty", "income" ]
}
}
}
}
The aggregation type is matrix_stats
and the fields
setting defines the set of fields (as an array) for computing
the statistics. The above request returns the following response:
{
...
"aggregations": {
"statistics": {
"doc_count": 50,
"fields": [ {
"name": "income",
"count": 50,
"mean": 51985.1,
"variance": 7.383377037755103E7,
"skewness": 0.5595114003506483,
"kurtosis": 2.5692365287787124,
"covariance": {
"income": 7.383377037755103E7,
"poverty": -21093.65836734694
},
"correlation": {
"income": 1.0,
"poverty": -0.8352655256272504
}
}, {
"name": "poverty",
"count": 50,
"mean": 12.732000000000001,
"variance": 8.637730612244896,
"skewness": 0.4516049811903419,
"kurtosis": 2.8615929677997767,
"covariance": {
"income": -21093.65836734694,
"poverty": 8.637730612244896
},
"correlation": {
"income": -0.8352655256272504,
"poverty": 1.0
}
} ]
}
}
}
The doc_count
field indicates the number of documents involved in the computation of the statistics.
Multi Value Fields
The matrix_stats
aggregation treats each document field as an independent sample. The mode
parameter controls what
array value the aggregation will use for array or multi-valued fields. This parameter can take one of the following:
avg
|
(default) Use the average of all values. |
min
|
Pick the lowest value. |
max
|
Pick the highest value. |
sum
|
Use the sum of all values. |
median
|
Use the median of all values. |
Missing Values
The missing
parameter defines how documents that are missing a value should be treated.
By default they will be ignored but it is also possible to treat them as if they had a value.
This is done by adding a set of fieldname : value mappings to specify default values per field.
GET /_search
{
"aggs": {
"matrixstats": {
"matrix_stats": {
"fields": [ "poverty", "income" ],
"missing": { "income": 50000 } (1)
}
}
}
}
-
Documents without a value in the
income
field will have the default value50000
.
Script
This aggregation family does not yet support scripting.
Max aggregation
A single-value
metrics aggregation that keeps track and returns the maximum
value among the numeric values extracted from the aggregated documents. These
values can be extracted either from specific numeric fields in the documents,
or be generated by a provided script.
|
Note
|
The min and max aggregation operate on the double representation of
the data. As a consequence, the result may be approximate when running on longs
whose absolute value is greater than 2^53.
|
Computing the max price value across all documents
POST /sales/_search?size=0
{
"aggs": {
"max_price": { "max": { "field": "price" } }
}
}
Response:
{
...
"aggregations": {
"max_price": {
"value": 200.0
}
}
}
As can be seen, the name of the aggregation (max_price
above) also serves as
the key by which the aggregation result can be retrieved from the returned
response.
Script
The max
aggregation can also calculate the maximum of a script. The example
below computes the maximum price:
POST /sales/_search
{
"aggs" : {
"max_price" : {
"max" : {
"script" : {
"source" : "doc.price.value"
}
}
}
}
}
This will use the Painless scripting language and no script parameters. To use a stored script use the following syntax:
POST /sales/_search
{
"aggs" : {
"max_price" : {
"max" : {
"script" : {
"id": "my_script",
"params": {
"field": "price"
}
}
}
}
}
}
Value Script
Let’s say that the prices of the documents in our index are in USD, but we would like to compute the max in EURO (and for the sake of this example, let’s say the conversion rate is 1.2). We can use a value script to apply the conversion rate to every value before it is aggregated:
POST /sales/_search
{
"aggs" : {
"max_price_in_euros" : {
"max" : {
"field" : "price",
"script" : {
"source" : "_value * params.conversion_rate",
"params" : {
"conversion_rate" : 1.2
}
}
}
}
}
}
Missing value
The missing
parameter defines how documents that are missing a value should
be treated. By default they will be ignored but it is also possible to treat
them as if they had a value.
POST /sales/_search
{
"aggs" : {
"grade_max" : {
"max" : {
"field" : "grade",
"missing": 10 (1)
}
}
}
}
-
Documents without a value in the
grade
field will fall into the same bucket as documents that have the value10
.
Median absolute deviation aggregation
This single-value
aggregation approximates the median absolute deviation
of its search results.
Median absolute deviation is a measure of variability. It is a robust statistic, meaning that it is useful for describing data that may have outliers, or may not be normally distributed. For such data it can be more descriptive than standard deviation.
It is calculated as the median of each data point’s deviation from the median of the entire sample. That is, for a random variable X, the median absolute deviation is median(|median(X) - Xi|).
Example
Assume our data represents product reviews on a one to five star scale. Such reviews are usually summarized as a mean, which is easily understandable but doesn’t describe the reviews' variability. Estimating the median absolute deviation can provide insight into how much reviews vary from one another.
In this example we have a product which has an average rating of 3 stars. Let’s look at its ratings' median absolute deviation to determine how much they vary
GET reviews/_search
{
"size": 0,
"aggs": {
"review_average": {
"avg": {
"field": "rating"
}
},
"review_variability": {
"median_absolute_deviation": {
"field": "rating" (1)
}
}
}
}
-
rating
must be a numeric field
The resulting median absolute deviation of 2
tells us that there is a fair
amount of variability in the ratings. Reviewers must have diverse opinions about
this product.
{
...
"aggregations": {
"review_average": {
"value": 3.0
},
"review_variability": {
"value": 2.0
}
}
}
Approximation
The naive implementation of calculating median absolute deviation stores the entire sample in memory, so this aggregation instead calculates an approximation. It uses the TDigest data structure to approximate the sample median and the median of deviations from the sample median. For more about the approximation characteristics of TDigests, see Percentiles are (usually) approximate.
The tradeoff between resource usage and accuracy of a TDigest’s quantile
approximation, and therefore the accuracy of this aggregation’s approximation
of median absolute deviation, is controlled by the compression
parameter. A
higher compression
setting provides a more accurate approximation at the
cost of higher memory usage. For more about the characteristics of the TDigest
compression
parameter see
Compression.
GET reviews/_search
{
"size": 0,
"aggs": {
"review_variability": {
"median_absolute_deviation": {
"field": "rating",
"compression": 100
}
}
}
}
The default compression
value for this aggregation is 1000
. At this
compression level this aggregation is usually within 5% of the exact result,
but observed performance will depend on the sample data.
Script
This metric aggregation supports scripting. In our example above, product reviews are on a scale of one to five. If we wanted to modify them to a scale of one to ten, we can using scripting.
To provide an inline script:
GET reviews/_search
{
"size": 0,
"aggs": {
"review_variability": {
"median_absolute_deviation": {
"script": {
"lang": "painless",
"source": "doc['rating'].value * params.scaleFactor",
"params": {
"scaleFactor": 2
}
}
}
}
}
}
To provide a stored script:
GET reviews/_search
{
"size": 0,
"aggs": {
"review_variability": {
"median_absolute_deviation": {
"script": {
"id": "my_script",
"params": {
"field": "rating"
}
}
}
}
}
}
Missing value
The missing
parameter defines how documents that are missing a value should be
treated. By default they will be ignored but it is also possible to treat them
as if they had a value.
Let’s be optimistic and assume some reviewers loved the product so much that they forgot to give it a rating. We’ll assign them five stars
GET reviews/_search
{
"size": 0,
"aggs": {
"review_variability": {
"median_absolute_deviation": {
"field": "rating",
"missing": 5
}
}
}
}
Min aggregation
A single-value
metrics aggregation that keeps track and returns the minimum
value among numeric values extracted from the aggregated documents. These
values can be extracted either from specific numeric fields in the documents,
or be generated by a provided script.
|
Note
|
The min and max aggregation operate on the double representation of
the data. As a consequence, the result may be approximate when running on longs
whose absolute value is greater than 2^53.
|
Computing the min price value across all documents:
POST /sales/_search?size=0
{
"aggs": {
"min_price": { "min": { "field": "price" } }
}
}
Response:
{
...
"aggregations": {
"min_price": {
"value": 10.0
}
}
}
As can be seen, the name of the aggregation (min_price
above) also serves as
the key by which the aggregation result can be retrieved from the returned
response.
Script
The min
aggregation can also calculate the minimum of a script. The example
below computes the minimum price:
POST /sales/_search
{
"aggs": {
"min_price": {
"min": {
"script": {
"source": "doc.price.value"
}
}
}
}
}
This will use the Painless scripting language and no script parameters. To use a stored script use the following syntax:
POST /sales/_search
{
"aggs": {
"min_price": {
"min": {
"script": {
"id": "my_script",
"params": {
"field": "price"
}
}
}
}
}
}
Value Script
Let’s say that the prices of the documents in our index are in USD, but we would like to compute the min in EURO (and for the sake of this example, let’s say the conversion rate is 1.2). We can use a value script to apply the conversion rate to every value before it is aggregated:
POST /sales/_search
{
"aggs": {
"min_price_in_euros": {
"min": {
"field": "price",
"script": {
"source": "_value * params.conversion_rate",
"params": {
"conversion_rate": 1.2
}
}
}
}
}
}
Missing value
The missing
parameter defines how documents that are missing a value should
be treated. By default they will be ignored but it is also possible to treat
them as if they had a value.
POST /sales/_search
{
"aggs": {
"grade_min": {
"min": {
"field": "grade",
"missing": 10 (1)
}
}
}
}
-
Documents without a value in the
grade
field will fall into the same bucket as documents that have the value10
.
Percentile ranks aggregation
A multi-value
metrics aggregation that calculates one or more percentile ranks
over numeric values extracted from the aggregated documents. These values
can be extracted either from specific numeric fields in the documents, or
be generated by a provided script.
|
Note
|
Please see Percentiles are (usually) approximate and Compression for advice regarding approximation and memory use of the percentile ranks aggregation |
Percentile rank show the percentage of observed values which are below certain value. For example, if a value is greater than or equal to 95% of the observed values it is said to be at the 95th percentile rank.
Assume your data consists of website load times. You may have a service agreement that 95% of page loads complete within 500ms and 99% of page loads complete within 600ms.
Let’s look at a range of percentiles representing load time:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_ranks": {
"percentile_ranks": {
"field": "load_time", (1)
"values": [ 500, 600 ]
}
}
}
}
-
The field
load_time
must be a numeric field
The response will look like this:
{
...
"aggregations": {
"load_time_ranks": {
"values": {
"500.0": 90.01,
"600.0": 100.0
}
}
}
}
From this information you can determine you are hitting the 99% load time target but not quite hitting the 95% load time target
Keyed Response
By default the keyed
flag is set to true
associates a unique string key with each bucket and returns the ranges as a hash rather than an array. Setting the keyed
flag to false
will disable this behavior:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_ranks": {
"percentile_ranks": {
"field": "load_time",
"values": [ 500, 600 ],
"keyed": false
}
}
}
}
Response:
{
...
"aggregations": {
"load_time_ranks": {
"values": [
{
"key": 500.0,
"value": 90.01
},
{
"key": 600.0,
"value": 100.0
}
]
}
}
}
Script
The percentile rank metric supports scripting. For example, if our load times are in milliseconds but we want to specify values in seconds, we could use a script to convert them on-the-fly:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_ranks": {
"percentile_ranks": {
"values": [ 500, 600 ],
"script": {
"lang": "painless",
"source": "doc['load_time'].value / params.timeUnit", (1)
"params": {
"timeUnit": 1000 (2)
}
}
}
}
}
}
-
The
field
parameter is replaced with ascript
parameter, which uses the script to generate values which percentile ranks are calculated on -
Scripting supports parameterized input just like any other script
This will interpret the script
parameter as an inline
script with the painless
script language and no script parameters. To use a stored script use the following syntax:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_ranks": {
"percentile_ranks": {
"values": [ 500, 600 ],
"script": {
"id": "my_script",
"params": {
"field": "load_time"
}
}
}
}
}
}
HDR Histogram
|
Note
|
This setting exposes the internal implementation of HDR Histogram and the syntax may change in the future. |
HDR Histogram (High Dynamic Range Histogram) is an alternative implementation that can be useful when calculating percentile ranks for latency measurements as it can be faster than the t-digest implementation with the trade-off of a larger memory footprint. This implementation maintains a fixed worse-case percentage error (specified as a number of significant digits). This means that if data is recorded with values from 1 microsecond up to 1 hour (3,600,000,000 microseconds) in a histogram set to 3 significant digits, it will maintain a value resolution of 1 microsecond for values up to 1 millisecond and 3.6 seconds (or better) for the maximum tracked value (1 hour).
The HDR Histogram can be used by specifying the hdr
object in the request:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_ranks": {
"percentile_ranks": {
"field": "load_time",
"values": [ 500, 600 ],
"hdr": { (1)
"number_of_significant_value_digits": 3 (2)
}
}
}
}
}
-
hdr
object indicates that HDR Histogram should be used to calculate the percentiles and specific settings for this algorithm can be specified inside the object -
number_of_significant_value_digits
specifies the resolution of values for the histogram in number of significant digits
The HDRHistogram only supports positive values and will error if it is passed a negative value. It is also not a good idea to use the HDRHistogram if the range of values is unknown as this could lead to high memory usage.
Missing value
The missing
parameter defines how documents that are missing a value should be treated.
By default they will be ignored but it is also possible to treat them as if they
had a value.
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_ranks": {
"percentile_ranks": {
"field": "load_time",
"values": [ 500, 600 ],
"missing": 10 (1)
}
}
}
}
-
Documents without a value in the
load_time
field will fall into the same bucket as documents that have the value10
.
Percentiles aggregation
A multi-value
metrics aggregation that calculates one or more percentiles
over numeric values extracted from the aggregated documents. These values
can be extracted either from specific numeric fields in the documents, or
be generated by a provided script.
Percentiles show the point at which a certain percentage of observed values occur. For example, the 95th percentile is the value which is greater than 95% of the observed values.
Percentiles are often used to find outliers. In normal distributions, the 0.13th and 99.87th percentiles represents three standard deviations from the mean. Any data which falls outside three standard deviations is often considered an anomaly.
When a range of percentiles are retrieved, they can be used to estimate the data distribution and determine if the data is skewed, bimodal, etc.
Assume your data consists of website load times. The average and median load times are not overly useful to an administrator. The max may be interesting, but it can be easily skewed by a single slow response.
Let’s look at a range of percentiles representing load time:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_outlier": {
"percentiles": {
"field": "load_time" (1)
}
}
}
}
-
The field
load_time
must be a numeric field
By default, the percentile
metric will generate a range of
percentiles: [ 1, 5, 25, 50, 75, 95, 99 ]
. The response will look like this:
{
...
"aggregations": {
"load_time_outlier": {
"values": {
"1.0": 5.0,
"5.0": 25.0,
"25.0": 165.0,
"50.0": 445.0,
"75.0": 725.0,
"95.0": 945.0,
"99.0": 985.0
}
}
}
}
As you can see, the aggregation will return a calculated value for each percentile in the default range. If we assume response times are in milliseconds, it is immediately obvious that the webpage normally loads in 10-725ms, but occasionally spikes to 945-985ms.
Often, administrators are only interested in outliers — the extreme percentiles. We can specify just the percents we are interested in (requested percentiles must be a value between 0-100 inclusive):
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_outlier": {
"percentiles": {
"field": "load_time",
"percents": [ 95, 99, 99.9 ] (1)
}
}
}
}
-
Use the
percents
parameter to specify particular percentiles to calculate
Keyed Response
By default the keyed
flag is set to true
which associates a unique string key with each bucket and returns the ranges as a hash rather than an array. Setting the keyed
flag to false
will disable this behavior:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_outlier": {
"percentiles": {
"field": "load_time",
"keyed": false
}
}
}
}
Response:
{
...
"aggregations": {
"load_time_outlier": {
"values": [
{
"key": 1.0,
"value": 5.0
},
{
"key": 5.0,
"value": 25.0
},
{
"key": 25.0,
"value": 165.0
},
{
"key": 50.0,
"value": 445.0
},
{
"key": 75.0,
"value": 725.0
},
{
"key": 95.0,
"value": 945.0
},
{
"key": 99.0,
"value": 985.0
}
]
}
}
}
Script
The percentile metric supports scripting. For example, if our load times are in milliseconds but we want percentiles calculated in seconds, we could use a script to convert them on-the-fly:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_outlier": {
"percentiles": {
"script": {
"lang": "painless",
"source": "doc['load_time'].value / params.timeUnit", (1)
"params": {
"timeUnit": 1000 (2)
}
}
}
}
}
}
-
The
field
parameter is replaced with ascript
parameter, which uses the script to generate values which percentiles are calculated on -
Scripting supports parameterized input just like any other script
This will interpret the script
parameter as an inline
script with the painless
script language and no script parameters. To use a stored script use the following syntax:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_outlier": {
"percentiles": {
"script": {
"id": "my_script",
"params": {
"field": "load_time"
}
}
}
}
}
}
Percentiles are (usually) approximate
There are many different algorithms to calculate percentiles. The naive
implementation simply stores all the values in a sorted array. To find the 50th
percentile, you simply find the value that is at my_array[count(my_array) * 0.5]
.
Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an OpenSearch cluster, approximate percentiles are calculated.
The algorithm used by the percentile
metric is called TDigest (introduced by
Ted Dunning in
Computing Accurate Quantiles using T-Digests).
When using this metric, there are a few guidelines to keep in mind:
-
Accuracy is proportional to
q(1-q)
. This means that extreme percentiles (e.g. 99%) are more accurate than less extreme percentiles, such as the median -
For small sets of values, percentiles are highly accurate (and potentially 100% accurate if the data is small enough).
-
As the quantity of values in a bucket grows, the algorithm begins to approximate the percentiles. It is effectively trading accuracy for memory savings. The exact level of inaccuracy is difficult to generalize, since it depends on your data distribution and volume of data being aggregated
The following chart shows the relative error on a uniform distribution depending on the number of collected values and the requested percentile:
It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.
|
Warning
|
Percentile aggregations are also non-deterministic. This means you can get slightly different results using the same data. |
Compression
Approximate algorithms must balance memory utilization with estimation accuracy.
This balance can be controlled using a compression
parameter:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_outlier": {
"percentiles": {
"field": "load_time",
"tdigest": {
"compression": 200 (1)
}
}
}
}
}
-
Compression controls memory usage and approximation error
The TDigest algorithm uses a number of "nodes" to approximate percentiles — the
more nodes available, the higher the accuracy (and large memory footprint) proportional
to the volume of data. The compression
parameter limits the maximum number of
nodes to 20 * compression
.
Therefore, by increasing the compression value, you can increase the accuracy of
your percentiles at the cost of more memory. Larger compression values also
make the algorithm slower since the underlying tree data structure grows in size,
resulting in more expensive operations. The default compression value is
100
.
A "node" uses roughly 32 bytes of memory, so under worst-case scenarios (large amount of data which arrives sorted and in-order) the default settings will produce a TDigest roughly 64KB in size. In practice data tends to be more random and the TDigest will use less memory.
HDR Histogram
|
Note
|
This setting exposes the internal implementation of HDR Histogram and the syntax may change in the future. |
HDR Histogram (High Dynamic Range Histogram) is an alternative implementation that can be useful when calculating percentiles for latency measurements as it can be faster than the t-digest implementation with the trade-off of a larger memory footprint. This implementation maintains a fixed worse-case percentage error (specified as a number of significant digits). This means that if data is recorded with values from 1 microsecond up to 1 hour (3,600,000,000 microseconds) in a histogram set to 3 significant digits, it will maintain a value resolution of 1 microsecond for values up to 1 millisecond and 3.6 seconds (or better) for the maximum tracked value (1 hour).
The HDR Histogram can be used by specifying the method
parameter in the request:
GET latency/_search
{
"size": 0,
"aggs": {
"load_time_outlier": {
"percentiles": {
"field": "load_time",
"percents": [ 95, 99, 99.9 ],
"hdr": { (1)
"number_of_significant_value_digits": 3 (2)
}
}
}
}
}
-
hdr
object indicates that HDR Histogram should be used to calculate the percentiles and specific settings for this algorithm can be specified inside the object -
number_of_significant_value_digits
specifies the resolution of values for the histogram in number of significant digits
The HDRHistogram only supports positive values and will error if it is passed a negative value. It is also not a good idea to use the HDRHistogram if the range of values is unknown as this could lead to high memory usage.
Missing value
The missing
parameter defines how documents that are missing a value should be treated.
By default they will be ignored but it is also possible to treat them as if they
had a value.
GET latency/_search
{
"size": 0,
"aggs": {
"grade_percentiles": {
"percentiles": {
"field": "grade",
"missing": 10 (1)
}
}
}
}
-
Documents without a value in the
grade
field will fall into the same bucket as documents that have the value10
.
Scripted metric aggregation
A metric aggregation that executes using scripts to provide a metric output.
|
Warning
|
Using scripts can result in slower search speeds. See Scripts and search speed. |
Example:
POST ledger/_search?size=0
{
"query": {
"match_all": {}
},
"aggs": {
"profit": {
"scripted_metric": {
"init_script": "state.transactions = []", (1)
"map_script": "state.transactions.add(doc.type.value == 'sale' ? doc.amount.value : -1 * doc.amount.value)",
"combine_script": "double profit = 0; for (t in state.transactions) { profit += t } return profit",
"reduce_script": "double profit = 0; for (a in states) { profit += a } return profit"
}
}
}
}
-
init_script
is an optional parameter, all other scripts are required.
The above aggregation demonstrates how one would use the script aggregation compute the total profit from sale and cost transactions.
The response for the above aggregation:
{
"took": 218,
...
"aggregations": {
"profit": {
"value": 240.0
}
}
}
The above example can also be specified using stored scripts as follows:
POST ledger/_search?size=0
{
"aggs": {
"profit": {
"scripted_metric": {
"init_script": {
"id": "my_init_script"
},
"map_script": {
"id": "my_map_script"
},
"combine_script": {
"id": "my_combine_script"
},
"params": {
"field": "amount" (1)
},
"reduce_script": {
"id": "my_reduce_script"
}
}
}
}
}
-
script parameters for
init
,map
andcombine
scripts must be specified in a globalparams
object so that it can be shared between the scripts.
For more details on specifying scripts see script documentation.
Allowed return types
Whilst any valid script object can be used within a single script, the scripts must return or store in the state
object only the following types:
-
primitive types
-
String
-
Map (containing only keys and values of the types listed here)
-
Array (containing elements of only the types listed here)
Scope of scripts
The scripted metric aggregation uses scripts at 4 stages of its execution:
- init_script
-
Executed prior to any collection of documents. Allows the aggregation to set up any initial state.
In the above example, the
init_script
creates an arraytransactions
in thestate
object. - map_script
-
Executed once per document collected. This is a required script. If no combine_script is specified, the resulting state needs to be stored in the
state
object.In the above example, the
map_script
checks the value of the type field. If the value is 'sale' the value of the amount field is added to the transactions array. If the value of the type field is not 'sale' the negated value of the amount field is added to transactions. - combine_script
-
Executed once on each shard after document collection is complete. This is a required script. Allows the aggregation to consolidate the state returned from each shard.
In the above example, the
combine_script
iterates through all the stored transactions, summing the values in theprofit
variable and finally returnsprofit
. - reduce_script
-
Executed once on the coordinating node after all shards have returned their results. This is a required script. The script is provided with access to a variable
states
which is an array of the result of the combine_script on each shard.In the above example, the
reduce_script
iterates through theprofit
returned by each shard summing the values before returning the final combined profit which will be returned in the response of the aggregation.
Worked example
Imagine a situation where you index the following documents into an index with 2 shards:
PUT /transactions/_bulk?refresh
{"index":{"_id":1}}
{"type": "sale","amount": 80}
{"index":{"_id":2}}
{"type": "cost","amount": 10}
{"index":{"_id":3}}
{"type": "cost","amount": 30}
{"index":{"_id":4}}
{"type": "sale","amount": 130}
Lets say that documents 1 and 3 end up on shard A and documents 2 and 4 end up on shard B. The following is a breakdown of what the aggregation result is at each stage of the example above.
Before init_script
state
is initialized as a new empty object.
"state" : {}
After init_script
This is run once on each shard before any document collection is performed, and so we will have a copy on each shard:
- Shard A
-
"state" : { "transactions" : [] }
- Shard B
-
"state" : { "transactions" : [] }
After map_script
Each shard collects its documents and runs the map_script on each document that is collected:
- Shard A
-
"state" : { "transactions" : [ 80, -30 ] }
- Shard B
-
"state" : { "transactions" : [ -10, 130 ] }
After combine_script
The combine_script is executed on each shard after document collection is complete and reduces all the transactions down to a single profit figure for each shard (by summing the values in the transactions array) which is passed back to the coordinating node:
- Shard A
-
50
- Shard B
-
120
After reduce_script
The reduce_script receives a states
array containing the result of the combine script for each shard:
"states" : [
50,
120
]
It reduces the responses for the shards down to a final overall profit figure (by summing the values) and returns this as the result of the aggregation to produce the response:
{
...
"aggregations": {
"profit": {
"value": 170
}
}
}
Other parameters
| params |
Optional. An object whose contents will be passed as variables to the
|
Empty buckets
If a parent bucket of the scripted metric aggregation does not collect any documents an empty aggregation response will be returned from the
shard with a null
value. In this case the reduce_script’s `states
variable will contain null
as a response from that shard.
reduce_script’s should therefore expect and deal with `null
responses from shards.
Stats aggregation
A multi-value
metrics aggregation that computes stats over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.
The stats that are returned consist of: min
, max
, sum
, count
and avg
.
Assuming the data consists of documents representing exams grades (between 0 and 100) of students
POST /exams/_search?size=0
{
"aggs": {
"grades_stats": { "stats": { "field": "grade" } }
}
}
The above aggregation computes the grades statistics over all documents. The aggregation type is stats
and the field
setting defines the numeric field of the documents the stats will be computed on. The above will return the following:
{
...
"aggregations": {
"grades_stats": {
"count": 2,
"min": 50.0,
"max": 100.0,
"avg": 75.0,
"sum": 150.0
}
}
}
The name of the aggregation (grades_stats
above) also serves as the key by which the aggregation result can be retrieved from the returned response.
Script
Computing the grades stats based on a script:
POST /exams/_search?size=0
{
"aggs": {
"grades_stats": {
"stats": {
"script": {
"lang": "painless",
"source": "doc['grade'].value"
}
}
}
}
}
This will interpret the script
parameter as an inline
script with the painless
script language and no script parameters. To use a stored script use the following syntax:
POST /exams/_search?size=0
{
"aggs": {
"grades_stats": {
"stats": {
"script": {
"id": "my_script",
"params": {
"field": "grade"
}
}
}
}
}
}
Value Script
It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use a value script to get the new stats:
POST /exams/_search?size=0
{
"aggs": {
"grades_stats": {
"stats": {
"field": "grade",
"script": {
"lang": "painless",
"source": "_value * params.correction",
"params": {
"correction": 1.2
}
}
}
}
}
}
Missing value
The missing
parameter defines how documents that are missing a value should be treated.
By default they will be ignored but it is also possible to treat them as if they
had a value.
POST /exams/_search?size=0
{
"aggs": {
"grades_stats": {
"stats": {
"field": "grade",
"missing": 0 (1)
}
}
}
}
-
Documents without a value in the
grade
field will fall into the same bucket as documents that have the value0
.
Sum aggregation
A single-value
metrics aggregation that sums up numeric values that are extracted from the aggregated documents.
These values can be extracted either from specific numeric fields in the documents,
or be generated by a provided script.
Assuming the data consists of documents representing sales records we can sum the sale price of all hats with:
POST /sales/_search?size=0
{
"query": {
"constant_score": {
"filter": {
"match": { "type": "hat" }
}
}
},
"aggs": {
"hat_prices": { "sum": { "field": "price" } }
}
}
Resulting in:
{
...
"aggregations": {
"hat_prices": {
"value": 450.0
}
}
}
The name of the aggregation (hat_prices
above) also serves as the key by which the aggregation result can be retrieved from the returned response.
Script
We could also use a script to fetch the sales price:
POST /sales/_search?size=0
{
"query": {
"constant_score": {
"filter": {
"match": { "type": "hat" }
}
}
},
"aggs": {
"hat_prices": {
"sum": {
"script": {
"source": "doc.price.value"
}
}
}
}
}
This will interpret the script
parameter as an inline
script with the painless
script language and no script parameters. To use a stored script use the following syntax:
POST /sales/_search?size=0
{
"query": {
"constant_score": {
"filter": {
"match": { "type": "hat" }
}
}
},
"aggs": {
"hat_prices": {
"sum": {
"script": {
"id": "my_script",
"params": {
"field": "price"
}
}
}
}
}
}
Value Script
It is also possible to access the field value from the script using _value
.
For example, this will sum the square of the prices for all hats:
POST /sales/_search?size=0
{
"query": {
"constant_score": {
"filter": {
"match": { "type": "hat" }
}
}
},
"aggs": {
"square_hats": {
"sum": {
"field": "price",
"script": {
"source": "_value * _value"
}
}
}
}
}
Missing value
The missing
parameter defines how documents that are missing a value should
be treated. By default documents missing the value will be ignored but it is
also possible to treat them as if they had a value. For example, this treats
all hat sales without a price as being 100
.
POST /sales/_search?size=0
{
"query": {
"constant_score": {
"filter": {
"match": { "type": "hat" }
}
}
},
"aggs": {
"hat_prices": {
"sum": {
"field": "price",
"missing": 100 (1)
}
}
}
}
Top hits aggregation
A top_hits
metric aggregator keeps track of the most relevant document being aggregated. This aggregator is intended
to be used as a sub aggregator, so that the top matching documents can be aggregated per bucket.
|
Tip
|
We do not recommend using top_hits as a top-level aggregation. If you
want to group search hits, use the collapse
parameter instead.
|
The top_hits
aggregator can effectively be used to group result sets by certain fields via a bucket aggregator.
One or more bucket aggregators determines by which properties a result set get sliced into.
Options
-
from
- The offset from the first result you want to fetch. -
size
- The maximum number of top matching hits to return per bucket. By default the top three matching hits are returned. -
sort
- How the top matching hits should be sorted. By default the hits are sorted by the score of the main query.
Supported per hit features
The top_hits aggregation returns regular search hits, because of this many per hit features can be supported:
top_hits
does not support the rescore
parameter. Query rescoring
applies only to search hits, not aggregation results. To change the scores used
by aggregations, use a function_score
or
script_score
query.
Example
In the following example we group the sales by type and per type we show the last sale. For each sale only the date and price fields are being included in the source.
POST /sales/_search?size=0
{
"aggs": {
"top_tags": {
"terms": {
"field": "type",
"size": 3
},
"aggs": {
"top_sales_hits": {
"top_hits": {
"sort": [
{
"date": {
"order": "desc"
}
}
],
"_source": {
"includes": [ "date", "price" ]
},
"size": 1
}
}
}
}
}
}
Possible response:
{
...
"aggregations": {
"top_tags": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "hat",
"doc_count": 3,
"top_sales_hits": {
"hits": {
"total" : {
"value": 3,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": "sales",
"_type": "_doc",
"_id": "AVnNBmauCQpcRyxw6ChK",
"_source": {
"date": "2015/03/01 00:00:00",
"price": 200
},
"sort": [
1425168000000
],
"_score": null
}
]
}
}
},
{
"key": "t-shirt",
"doc_count": 3,
"top_sales_hits": {
"hits": {
"total" : {
"value": 3,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": "sales",
"_type": "_doc",
"_id": "AVnNBmauCQpcRyxw6ChL",
"_source": {
"date": "2015/03/01 00:00:00",
"price": 175
},
"sort": [
1425168000000
],
"_score": null
}
]
}
}
},
{
"key": "bag",
"doc_count": 1,
"top_sales_hits": {
"hits": {
"total" : {
"value": 1,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": "sales",
"_type": "_doc",
"_id": "AVnNBmatCQpcRyxw6ChH",
"_source": {
"date": "2015/01/01 00:00:00",
"price": 150
},
"sort": [
1420070400000
],
"_score": null
}
]
}
}
}
]
}
}
}
Field collapse example
Field collapsing or result grouping is a feature that logically groups a result set into groups and per group returns
top documents. The ordering of the groups is determined by the relevancy of the first document in a group. In
OpenSearch this can be implemented via a bucket aggregator that wraps a top_hits
aggregator as sub-aggregator.
In the example below we search across crawled webpages. For each webpage we store the body and the domain the webpage
belong to. By defining a terms
aggregator on the domain
field we group the result set of webpages by domain. The
top_hits
aggregator is then defined as sub-aggregator, so that the top matching hits are collected per bucket.
Also a max
aggregator is defined which is used by the terms
aggregator’s order feature to return the buckets by
relevancy order of the most relevant document in a bucket.
POST /sales/_search
{
"query": {
"match": {
"body": "elections"
}
},
"aggs": {
"top_sites": {
"terms": {
"field": "domain",
"order": {
"top_hit": "desc"
}
},
"aggs": {
"top_tags_hits": {
"top_hits": {}
},
"top_hit" : {
"max": {
"script": {
"source": "_score"
}
}
}
}
}
}
}
At the moment the max
(or min
) aggregator is needed to make sure the buckets from the terms
aggregator are
ordered according to the score of the most relevant webpage per domain. Unfortunately the top_hits
aggregator
can’t be used in the order
option of the terms
aggregator yet.
top_hits support in a nested or reverse_nested aggregator
If the top_hits
aggregator is wrapped in a nested
or reverse_nested
aggregator then nested hits are being returned.
Nested hits are in a sense hidden mini documents that are part of regular document where in the mapping a nested field type
has been configured. The top_hits
aggregator has the ability to un-hide these documents if it is wrapped in a nested
or reverse_nested
aggregator. Read more about nested in the nested type mapping.
If nested type has been configured a single document is actually indexed as multiple Lucene documents and they share
the same id. In order to determine the identity of a nested hit there is more needed than just the id, so that is why
nested hits also include their nested identity. The nested identity is kept under the _nested
field in the search hit
and includes the array field and the offset in the array field the nested hit belongs to. The offset is zero based.
Let’s see how it works with a real sample. Considering the following mapping:
PUT /sales
{
"mappings": {
"properties": {
"tags": { "type": "keyword" },
"comments": { (1)
"type": "nested",
"properties": {
"username": { "type": "keyword" },
"comment": { "type": "text" }
}
}
}
}
}
-
The
comments
is an array that holds nested documents under theproduct
object.
And some documents:
PUT /sales/_doc/1?refresh
{
"tags": [ "car", "auto" ],
"comments": [
{ "username": "baddriver007", "comment": "This car could have better brakes" },
{ "username": "dr_who", "comment": "Where's the autopilot? Can't find it" },
{ "username": "ilovemotorbikes", "comment": "This car has two extra wheels" }
]
}
It’s now possible to execute the following top_hits
aggregation (wrapped in a nested
aggregation):
POST /sales/_search
{
"query": {
"term": { "tags": "car" }
},
"aggs": {
"by_sale": {
"nested": {
"path": "comments"
},
"aggs": {
"by_user": {
"terms": {
"field": "comments.username",
"size": 1
},
"aggs": {
"by_nested": {
"top_hits": {}
}
}
}
}
}
}
}
Top hits response snippet with a nested hit, which resides in the first slot of array field comments
:
{
...
"aggregations": {
"by_sale": {
"by_user": {
"buckets": [
{
"key": "baddriver007",
"doc_count": 1,
"by_nested": {
"hits": {
"total" : {
"value": 1,
"relation": "eq"
},
"max_score": 0.3616575,
"hits": [
{
"_index": "sales",
"_type" : "_doc",
"_id": "1",
"_nested": {
"field": "comments", (1)
"offset": 0 (2)
},
"_score": 0.3616575,
"_source": {
"comment": "This car could have better brakes", (3)
"username": "baddriver007"
}
}
]
}
}
}
...
]
}
}
}
}
-
Name of the array field containing the nested hit
-
Position if the nested hit in the containing array
-
Source of the nested hit
If _source
is requested then just the part of the source of the nested object is returned, not the entire source of the document.
Also stored fields on the nested inner object level are accessible via top_hits
aggregator residing in a nested
or reverse_nested
aggregator.
Only nested hits will have a _nested
field in the hit, non nested (regular) hits will not have a _nested
field.
The information in _nested
can also be used to parse the original source somewhere else if _source
isn’t enabled.
If there are multiple levels of nested object types defined in mappings then the _nested
information can also be hierarchical
in order to express the identity of nested hits that are two layers deep or more.
In the example below a nested hit resides in the first slot of the field nested_grand_child_field
which then resides in
the second slow of the nested_child_field
field:
...
"hits": {
"total" : {
"value": 2565,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "a",
"_type": "b",
"_id": "1",
"_score": 1,
"_nested" : {
"field" : "nested_child_field",
"offset" : 1,
"_nested" : {
"field" : "nested_grand_child_field",
"offset" : 0
}
}
"_source": ...
},
...
]
}
...
Value count aggregation
A single-value
metrics aggregation that counts the number of values that are extracted from the aggregated documents.
These values can be extracted either from specific fields in the documents, or be generated by a provided script. Typically,
this aggregator will be used in conjunction with other single-value aggregations. For example, when computing the avg
one might be interested in the number of values the average is computed over.
value_count
does not de-duplicate values, so even if a field has duplicates (or a script generates multiple
identical values for a single document), each value will be counted individually.
POST /sales/_search?size=0
{
"aggs" : {
"types_count" : { "value_count" : { "field" : "type" } }
}
}
Response:
{
...
"aggregations": {
"types_count": {
"value": 7
}
}
}
The name of the aggregation (types_count
above) also serves as the key by which the aggregation result can be
retrieved from the returned response.
Script
Counting the values generated by a script:
POST /sales/_search?size=0
{
"aggs": {
"type_count": {
"value_count": {
"script": {
"source": "doc['type'].value"
}
}
}
}
}
This will interpret the script
parameter as an inline
script with the painless
script language and no script parameters. To use a stored script use the following syntax:
POST /sales/_search?size=0
{
"aggs": {
"types_count": {
"value_count": {
"script": {
"id": "my_script",
"params": {
"field": "type"
}
}
}
}
}
}
- NOTE
-
Because
value_count
is designed to work with any field it internally treats all values as simple bytes. Due to this implementation, if_value
script variable is used to fetch a value instead of accessing the field directly (e.g. a "value script"), the field value will be returned as a string instead of it’s native format.
Weighted avg aggregation
A single-value
metrics aggregation that computes the weighted average of numeric values that are extracted from the aggregated documents.
These values can be extracted either from specific numeric fields in the documents.
When calculating a regular average, each datapoint has an equal "weight" … it contributes equally to the final value. Weighted averages, on the other hand, weight each datapoint differently. The amount that each datapoint contributes to the final value is extracted from the document, or provided by a script.
As a formula, a weighted average is the ∑(value * weight) / ∑(weight)
A regular average can be thought of as a weighted average where every value has an implicit weight of 1
.
| Parameter Name | Description | Required | Default Value |
|---|---|---|---|
|
The configuration for the field or script that provides the values |
Required |
|
|
The configuration for the field or script that provides the weights |
Required |
|
|
The numeric response formatter |
Optional |
|
|
A hint about the values for pure scripts or unmapped fields |
Optional |
The value
and weight
objects have per-field specific configuration:
| Parameter Name | Description | Required | Default Value |
|---|---|---|---|
|
The field that values should be extracted from |
Required |
|
|
A value to use if the field is missing entirely |
Optional |
| Parameter Name | Description | Required | Default Value |
|---|---|---|---|
|
The field that weights should be extracted from |
Required |
|
|
A weight to use if the field is missing entirely |
Optional |
Examples
If our documents have a "grade"
field that holds a 0-100 numeric score, and a "weight"
field which holds an arbitrary numeric weight,
we can calculate the weighted average using:
POST /exams/_search
{
"size": 0,
"aggs": {
"weighted_grade": {
"weighted_avg": {
"value": {
"field": "grade"
},
"weight": {
"field": "weight"
}
}
}
}
}
Which yields a response like:
{
...
"aggregations": {
"weighted_grade": {
"value": 70.0
}
}
}
While multiple values-per-field are allowed, only one weight is allowed. If the aggregation encounters
a document that has more than one weight (e.g. the weight field is a multi-valued field) it will throw an exception.
If you have this situation, you will need to specify a script
for the weight field, and use the script
to combine the multiple values into a single value to be used.
This single weight will be applied independently to each value extracted from the value
field.
This example show how a single document with multiple values will be averaged with a single weight:
POST /exams/_doc?refresh
{
"grade": [1, 2, 3],
"weight": 2
}
POST /exams/_search
{
"size": 0,
"aggs": {
"weighted_grade": {
"weighted_avg": {
"value": {
"field": "grade"
},
"weight": {
"field": "weight"
}
}
}
}
}
The three values (1
, 2
, and 3
) will be included as independent values, all with the weight of 2
:
{
...
"aggregations": {
"weighted_grade": {
"value": 2.0
}
}
}
The aggregation returns 2.0
as the result, which matches what we would expect when calculating by hand:
1*2) + (2*2) + (3*2 / (2+2+2) == 2
Script
Both the value and the weight can be derived from a script, instead of a field. As a simple example, the following will add one to the grade and weight in the document using a script:
POST /exams/_search
{
"size": 0,
"aggs": {
"weighted_grade": {
"weighted_avg": {
"value": {
"script": "doc.grade.value + 1"
},
"weight": {
"script": "doc.weight.value + 1"
}
}
}
}
}
Missing values
The missing
parameter defines how documents that are missing a value should be treated.
The default behavior is different for value
and weight
:
By default, if the value
field is missing the document is ignored and the aggregation moves on to the next document.
If the weight
field is missing, it is assumed to have a weight of 1
(like a normal average).
Both of these defaults can be overridden with the missing
parameter:
POST /exams/_search
{
"size": 0,
"aggs": {
"weighted_grade": {
"weighted_avg": {
"value": {
"field": "grade",
"missing": 2
},
"weight": {
"field": "weight",
"missing": 3
}
}
}
}
}


The Magnificent Seven
Miss Peregrine's Home for Peculiar Children
Passengers
Deepwater Horizon
Suicide Squad
Doctor Strange
Sully
Swiss Army Man
Ghostbusters
The Girl on the Train
Star Trek Beyond
Bridget Jones's Baby
Storks
Masterminds
The Shallows
Snowden
Don't Breathe
Mike and Dave Need Wedding Dates
The Legend of Tarzan
The Infiltrator
Blair Witch
Teenage Mutant Ninja Turtles: Out of the Shadows
31
I.T.
The Jungle Book
The Secret Life of Pets
Rogue One: A Star Wars Story
Me Before You
The Nice Guys
ARQ
Alice Through the Looking Glass
Sausage Party
Free State of Jones
The Purge: Election Year
Goat
Paterson
Now You See Me 2
Arrival
Hell or High Water
The Neon Demon
American Honey
Central Intelligence
Mechanic: Resurrection
Jason Bourne
The Space Between Us
Batman v Superman: Dawn of Justice
Inferno
The Girl with All the Gifts
True Memoirs of an International Assassin
Bad Moms
Café Society
Hacksaw Ridge
Trespass Against Us
Jack Reacher: Never Go Back
The Accountant
Blood Father
Money Monster
Kubo and the Two Strings
Nerve
War Dogs
Pink
Keeping Up with the Joneses
Friend Request
20th Century Women
The Birth of a Nation
The Light Between Oceans
Ice Age: Collision Course
Laid in America
London Has Fallen
When the Bough Breaks
13 Hours
The Good Neighbor
Dirty 30
La La Land
Pete's Dragon
Imperium


AWS::Cognito::UserPoolUser
The AWS::Cognito::UserPoolUser
resource creates an Amazon Cognito user pool user.
Syntax
To declare this entity in your AWS CloudFormation template, use the following syntax:
JSON
{ "Type" : "AWS::Cognito::UserPoolUser", "Properties" : { "DesiredDeliveryMediums" : [
String, ...
], "ForceAliasCreation" :Boolean
, "UserAttributes" : [AttributeType
, ... ], "MessageAction" :String
, "Username" :String
, "UserPoolId" :String
, "ValidationData" : [AttributeType
, ...] } }
YAML
Type: AWS::Cognito::UserPoolUser Properties: DesiredDeliveryMediums: -
String
ForceAliasCreation:Boolean
UserAttributes: -AttributeType
MessageAction:String
Username:String
UserPoolId:String
ValidationData: -AttributeType
Properties
DesiredDeliveryMediums
-
Specifies how the welcome message will be sent. For email, specify
EMAIL
. To use a phone number, specifySMS
. You can specify more than one value. The default value isSMS
.Required: No
Type: List of String values
Update requires: Replacement
ForceAliasCreation
-
Use this parameter only if the
phone_number_verified
attribute or theemail_verified
attribute is set toTrue
. Otherwise, it is ignored. The default value isFalse
.If this parameter is set to
True
and the phone number or email address specified in theUserAttributes
parameter already exists as an alias with a different user, the API call migrates the alias from the previous user to the newly created user. The previous user can no longer log in using that alias.If this parameter is set to
False
and the alias already exists, the API throws anAliasExistsException
error.Required: No
Type: Boolean
Update requires: Replacement
UserAttributes
-
A list of name-value pairs that contain user attributes and attribute values to be set for the user that you are creating. You can create a user without specifying any attributes other than
Username
. However, any attributes that you specify as required (inCreateUserPool
or in the Attributes tab of the console) must be supplied either by you (in your call toAdminCreateUser
) or by the user (when signing up in response to your welcome message).Required: No
Type: List of Amazon Cognito UserPoolUser AttributeType
Update requires: Replacement
MessageAction
-
Specifies the action you'd like to take for the message. Valid values are
RESEND
andSUPPRESS
.To resend the invitation message to a user that already exists and reset the expiration limit on the user's account, set this parameter to
RESEND
. To suppress sending the message, set it toSUPPRESS
. You can specify only one value.Required: No
Type: String
Update requires: Replacement
Username
-
The user name for the user.
Username
must be unique within the user pool. It must be a UTF-8 string between 1 and 128 characters. You can't change the username.Required: No
Type: String
Update requires: Replacement
UserPoolId
-
The ID for the user pool where the user will be created.
Required: Yes
Type: String
Update requires: Replacement
ValidationData
-
The user's validation data. This is a list of name-value pairs that contain user attributes and attribute values that you can use for custom validation, such as restricting the types of user accounts that can be registered. For example, you might choose to allow or disallow user sign-up based on the user's domain.
To configure custom validation, you must create a Pre Sign-up Lambda trigger for the user pool. The Lambda trigger receives the validation data and uses it in the validation process. For more information, see Customizing User Pool Workflows by Using AWS Lambda Triggers in the Amazon Cognito Developer Guide.
Required: No
Type: List of Amazon Cognito UserPoolUser AttributeType
Update requires: Replacement
Return Value
Ref
When the logical ID of this resource is provided to the Ref
intrinsic function, Ref
returns the name of the user. For example, admin
.
For more information about using the Ref
function, see Ref.


isambard.ampal.specifications.assembly_specs.nucleic_acid_duplex module¶
Contains code for generating nucleic acid duplexes.
-
class
DNADuplex
(strand)[source]¶ Bases:
ampal.assembly.Assembly
Creates a DNA duplex from a single strand of helical DNA.
Parameters: strand (NucleicAcidStrand) – DNA single strand helix. -
classmethod
from_sequence
(sequence, phos_3_prime=False)[source]¶ Creates a DNA duplex from a nucleotide sequence.
Parameters:
-
classmethod


Infinite Replay Value
Air Traffic Control Simulation
and the Business Value
of Event Sourcing
Overview
- SimSuite ATC Simulation Software
- Air Traffic Control (ATC) Overview
- New Customer Requirements
- Traditional OOP State Management
- What is Event Sourcing
- Event Sourcing Benefits
- Code Example
- Business Value from Event Sourcing
About the SimSuite™ Software
-
Domain:
Air Traffic Control (ATC) Simulation
-
Summary:
Comprehensive ATC training and simulation platform with components for ground, tower, terminal and enroute training.
-
Users:
Instructors, ATC Students, Pilot Students
-
Environment:
ATC Simulation labs at colleges & universities with ATC training programs
-
Components:
SimCab (Tower/Ground), SimScope (Radar), SimTarget (Traffic Generation) & SimVox (Communication)
SimCab Tower & Ground
Tower controllers manage traffic on the runways, takeoffs and landings. They can physically see the grounds of the airport out the tower window. They give planes instructions based on this information (e.g. taxi to runway 7, follow that plane).
SimScope Radar
Radar ATCs give planes instructions based on radar info (e.g. Climb to 5000 feet).
Approach/Departure Radar ATCs control the plane during approach/departure.
En Route Radar ATCs control the plane during flight. Planes may be handed off between multiple en route ATCs throughout their flight.
SimTarget Traffic Generation
Remote Pilot Operator (RPO or Ghost Pilots) Instructor or student who generates aircraft and adds them to the simulation at various points and issues commands to aircraft (altitude, speed changes). The auto-pilot that flies the plane based on commands from the RPO.
SimVox Communication
Voice communication system that supports the various types of radio and interphone channels used in modern air traffic control facilities. Installed on a tablet which can be used to issue commands via voice or buttons.
Sessions & Scenarios
-
Session:
An isolated simulation instance to which multiple lab stations (tower, radar, rpo, etc.) can connect. (analog: game server).
-
Scenario:
A set of aircraft in pre-defined positions (altitude, lat/lng) and behaviors (taxiing, turning, landing) from which a session can be started. Created to give students the opportunity to practice specific skills such as landing a plane.
New Requirements
-
Replay
When a session includes a failed or successful attempt that has educational value, students
and instructors can watch the session as a class.
-
Retry
When a student fails a scenario, the instructor can rewind the recorded scenario to a chosen point in time to allow the student to retry without the need to start from the beginning.
-
Scenario from Session
When a session contains a moment that can serve as a learning experience, instructors can create a
new scenario from a certain point in the session to use as the basis of future simulations.
State Management in SimSuite
-
Models: Objects or data structures which model our domain. State is stored as properties
of models. Example: Aircraft with an altitude property.
-
State Change: Mutate the state of the objects through public methods which change
the values of the model's properties. Example: setAltitude(int newAltitude)
-
Persistence: Serialized version of the model stored in a persistent storage layer (e.g. database, filesystem)
-
How would we implement these new requirements with this approach? What challenges do we encounter?
What is Event Sourcing?
- Each change to a model's state is recorded in an event object.
- The ordered event stream is persisted instead of the model.
- The model can be reconstituted from it's event history.
Event Sourcing: Step By Step
Instead of directly modifying properties or using setters, use public mutator methods that:
-
create an event object and push onto a stack (e.g. new AircraftAltitudeChangedEvent(newAltitude))
-
call an event-specific method to mutate the model's state (e.g. ApplyAircraftAltitudeChangedEvent)
-
increment a version number so we can tell the model has changed (and which version of it we have)
Event Classes
abstract class ModelChangedEvent
{
public string modelId;
public int version = 0;
}
class AltitudeChangedEvent : ModelChangedEvent
{
public int altitude;
}
class Aircraft
{
public string id;
public int version;
public int altitude;
public List<ModelChangedEvent> events = new List<ModelChangedEvent>();
public void SetAltitude(int newAltitude) {
RecordThat(new AltitudeChangedEvent(id, newAltitude));
}
protected void RecordThat(ModelChangedEvent evt) {
evt.version = version + 1;
events.Add(evt);
Apply(evt);
}
protected void Apply(ModelChangedEvent evt) {
switch (evt.GetType().Name) {
case "AltitudeChangedEvent":
ApplyAltitudeChangedEvent((AltitudeChangedEvent) evt);
break;
}
version = evt.version;
}
protected void ApplyAltitudeChangedEvent(AltitudeChangedEvent evt) {
altitude = evt.altitude;
}
}
Persisting Models
To persist a model, flush any events from the stack and save them to the storage engine (database, eventstore).
class Aircraft
{
public List<ModelChangedEvent> ReleaseEvents() {
List<ModelChangedEvent> releasedEvents = events;
events = new List<ModelChangedEvent>();
return releasedEvents;
}
}
Reconsituting Models
To load a model from an event stream, pull the events history from the storage engine and re-apply them in order.
class Aircraft
{
public void ReconstituteFromEvents(List<ModelChangedEvent> evts) {
foreach (var evt in evts) {
Apply(evt);
}
}
}
Benefits of Event Sourcing
-
Auditing:
Event stream gives the "Why" of state change
-
Versioning:
Tell whether a model has changed and if its different from your copy
-
Replay/rewind:
Free redo/undo functionality through replay of only some events (all but the last one).
-
Debugability:
Developers can load state from a bug report's event stream to reproduce bugs and look at the history to see where things went wrong.
-
Real-Time Updates:
Push small event messages to clients to trigger updates
Problems with Event Sourcing
-
Large # of Events:
Recording and storing large numbers of events could be slow or take a lot of storage space.
Solution: Measure first, might not be a problem
-
Frequently Reconsituting Models:
Replaying large numbers of events on larger numbers of models could be slow.
Solution: Snapshots
-
Querying & Reporting:
Event streams don't lend themselves to querying.
Solution: CQRS/Read Models
Business Value
-
Better learning experience for future classes to learn from past classes mistakes/successes, instructor can teach entire class concurrently
-
Software developers and schools can easily demo the software at trade shows (dog and pony show in the lab for important visitors such as senators who might fund their training program)
-
Instructors can easily create scenarios based on past simulations makes it easy for them to generate new materials for a specific issue


 That’s why.
inappropriate behavior is often laughed off as boys will be boys,” women face higher conduct standards especially in the workplace. That’s why.
inappropriate behavior is often laughed off as boys will be boys,” women face higher conduct standards especially in the workplace. That’s why.
inappropriate behavior is often laughed off as boys will be boys,” women face higher conduct standards especially in the workplace. That’s why.
inappropriate behavior is often laughed off as “boys will be boys,” women face higher conduct standards especially in the workplace. That’s why it’s crucial that, as women, our behavior on the job is beyond reproach. inappropriate behavior is often laughed off.
See Detailsinappropriate behavior is often laughed off as “boys will be boys,” women face higher conduct standards especially in the workplace. That’s why it’s crucial that, as women, our behavior on the job is beyond reproach. inappropriate behavior is often laughed off.
See DetailsLorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.


Xiaomi Mi Note 10 Pro is launching in Malaysia on 3rd December
by Alexander WongThe penta-camera Xiaomi Mi Note 10 (aka Xiaomi CC9 Pro) is coming to Malaysia very soon and it is the world’s first smartphone to feature an ultra high-resolution 108MP main camera. After it was spotted on SIRIM, Xiaomi Malaysia has officially revealed its launch event that’s taking place on the 3rd December 2019.
The launch details were revealed on a digital display at Pavilion Mall in Kuala Lumpur. What’s interesting is that Xiaomi will be launching not just the Mi Note 10 but also the Mi Note 10 Pro.
The Xiaomi Mi Note 10 Pro isn’t launched yet and it is speculated to be a high-spec variant of the Snapdragon 730G powered Mi Note 10. We can’t confirm the specs just yet but it could be powered by a Snapdragon 855+ processor with more RAM and storage options.
To recap, the device features a 6.47″ dual-curved Full HD+ OLED display and it has a 32MP selfie camera on a tiny notch. At the rear, it has a 108MP f/1.69 main camera, a 5MP telephoto lens with 5X optical zoom, a 12MP telephoto camera with 2x optical zoom, a 20MP ultra-wide-angle shooter and a dedicated macro lens.
The Xiaomi CC9 Pro Premium Edition also has the highest DxOMark Mobile score of 121 points which is on par with the Huawei Mate 30 Pro. Unlike the Huawei flagship, the Mi Note 10 will come with Google apps and services pre-installed.
Powering the device is a huge 5,260mAh battery that charges via USB-C. Included is a 30W fast charger and it also comes with NFC and IR blaster support. Stay tuned to us for more details.
For those who can’t wait, the Xiaomi CC9 Pro, which is the Chinese market version, is now available in Malaysia with a retail price tag of RM2,188. This is an import set and it comes with 2-years in-house warranty.


new Entity(manager, id, tag)
Entity provides identity to all game objects.
Parameters:
| Name | Type | Argument | Description |
|---|---|---|---|
manager |
EntityManager | EntityManager that created the Entity |
|
id |
number | Unique numeric Id assigned by the EntityManager |
|
tag |
string |
<optional> |
Entity Tag |
- Source:
Extends
Members
-
<readonly> children :cog.Node[]
-
Gets an array of the Node's children.
Type:
- cog.Node[]
- Inherited From:
- Source:
-
components :cog.Entity~components
-
Type:
- Source:
-
<readonly> id :number
-
Gets the entity id
Type:
- number
- Source:
-
<readonly> manager :EntityManager
-
Gets the EntityManager
Type:
- EntityManager
- Source:
-
<readonly> parent :cog.Node
Gets the the Node's parent.
-
Type:
- Inherited From:
- Source:
-
tag :string
Gets the entity tag
-
Type:
- string
- Source:
-
<readonly> valid :boolean
True after the Entity is initialized. False once destroyed.
-
Type:
- boolean
- Source:
Methods
-
addChild(node) → {this}
-
Adds a child to the node.
Parameters:
Name Type Description node
cog.Node The node to be added
- Inherited From:
- Source:
Returns:
- Type
- this
-
clone() → {Entity}
-
Creates a new instance of the Entity. Components and enumerable properties are copied to the new Entity.
- Source:
Returns:
- Type
- Entity
Example
// Creates a clone of the entity. var clonedEntity = entity.clone();
-
components(Component) → {component|Array}
Gets component(s) from the entity
-
The components method gets a component or an array of components. If a Component constructor is passed to the function, this method will find and return the matching component. If no arguments are passed, this method will create an array containing all of the components on the entity.
Parameters:
Name Type Argument Description Component
Component <optional>
If passed, components will get the matching component.
- Source:
Returns:
- The component that matches the Constructor or an Array of all components
- Type
- component | Array
-
destroy()
-
Destroys the instance of the Entity
- Source:
-
init(manager, id, tag)
-
Initializes the Entity
Parameters:
Name Type Argument Description manager
EntityManager <optional>
EntityManager that created the Entity.
id
number <optional>
Id assigned by the EntityManager.
tag
string <optional>
Entity Tag.
- Source:
-
removeAllChildren() → {this}
-
Removes all children.
- Inherited From:
- Source:
Returns:
- Type
- this
-
removeChild(node) → {this}
-
Removes a child from the node.
Parameters:
Name Type Description node
cog.Node The node to be removed
- Inherited From:
- Source:
Returns:
- Type
- this


The Mountian Spoke
The Mountian Spoke
This project is a website that was made for a fictitious mountian biking company.
This project is a website that was made for a fictitious mountian biking company.
This website is a weather site that pulls in API data to fill information on cities in the U.S., this also has a form to report a storm as well as a gallery.


Workshop at ISMB 2018
The first workshop on single-cell cancer genomics (SCANGEN) will take place on July 10th at ISMB 2018 in Chicago, USA. The workshop aims to cover computational and statistical methods, models, and applications in single-cell cancer genomics.
| Start | End | Presenter | Title |
|---|---|---|---|
| 8:35 | 8:40 | Introduction / Welcome | |
| 8:40 | 9:00 | Jean Fan, Harvard University (invited) | Integrated genetic and transcriptional analysis at the single-cell level |
| 9:00 | 9:20 | Kieran O'Neill, BC Cancer Agency | Determing the Mechanism of 5-Azacytidine Response in Myeloid Malignancies Using Single-cell DNA Methylation Sequencing Paired With Flow Cytometry |
| 9:20 | 9:40 | Sombeet Sahu, Mission Bio | Methods for Identifying Tumor Heterogeneity and Rare Subclones in Single Cell DNA Sequence Data |
| 10:15 | 10:20 | Welcome Back | |
| 10:20 | 10:40 | Chris Yau, University of Birmingham (invited) | Scalable Bayesian Tensor Factorization for single-cell Genomics |
| 10:40 | 11:00 | Huidong Chen, Massachusetts General Hospital Research Institute | STREAM: Single-cell Trajectories Reconstruction, Exploration And Mapping of omics data |
| 11:00 | 11:20 | Sahar Ansari, University of Michigan | Leveraging transcription factor networks to identify cell types from single-cell transcriptomes |
| 11:20 | 11:40 | Xun Zhu, University of Hawaii Cancer Center | Granatum X: A community engaging and flexible scRNA-Seq analysis environment connecting tool developers to bench scientists |
| 11:40 | 12:00 | Wei Vivian Li, University of California, Los Angeles | scImpute: An accurate and robust imputation method for single-cell RNA-seq data |
| 12:00 | 12:20 | Davis McCarthy, EMBL-EBI (invited) | An approach to studying clonal cell populations using bulk exome and single-cell RNA sequencing data |
| 12:20 | 12:40 | Simone Ciccolella, Università degli studi Milano Bicocca | Inferring Cancer Progression from Single Cell Sequencing while allowing loss of mutations |
| Lunch and posters | |||
| 14:00 | 14:20 | Florian Markowetz, CRUK Cambridge Institute (invited) | Inferring cancer evolution and mutational processes |
| 14:20 | 14:40 | Mohammed El-Kebir, University of Illinois at Urbana-Champaign | SPhyR: Tumor Phylogeny Estimation from Single-Cell Sequencing Data under Loss and Error |
| 14:40 | 15:00 | Violeta Beleva Guthrie, Johns Hopkins University | Single-cell sequencing analysis of pancreatic cancer precursor lesions reveals convergent evolution and polyclonal origins |
| 15:00 | 15:20 | Haoyun Lei, Carnegie Mellon University | Deconvolution of tumor copy number data using bulk and single-cell sequencing data |
| 15:20 | 15:40 | Katharina Jahn, ETH Zurich (invited) | Modelling tumour evolution from single-cell sequencing data |
| 15:40 | 16:00 | Bojan Losic, Icahn Institute for Muliscale Biology | Regulatory underpinnings of intra-tumor heterogeneity in HCC liver cancer |
| 16:00 | 16:20 | Sumon Ahmed, University of Manchester | Investigating Haematopoietic Stem Cells development by single cell sequencing |
| 16:20 | 16:40 | Rui Kuang, University of Minnesota Twin Cities | A multitask learning approach for clustering multiple single cell RNA-seq datasets |
The call for submissions is now closed.
We are pleased to offer two travel awards of $1000USD to PhD students and postdocs, generously supported by Memorial Sloan Kettering Cancer Centre. To apply, first submit your work to SCANGEN then email kicampbell _at_ bccrc _dot_ ca indicating that you wish to be considered for an award.
The workshop is co-located at ISMB 2018 in Chicago, USA. It will be held on July 10th. For information on the venue, hotels, registration, and travel, see the main ISMB conference page.