qm
Source: qminerdoc.
QMiner module.
Example
// import module
var qm = require('qminer');
Child classes
Methods
Abstract types
- BaseConstructorParam
- BaseLoadCSVParam
- DetailKeyObject
- FeatureExtractor
- FeatureExtractorCategorical
- FeatureExtractorConstant
- FeatureExtractorDateWindow
- FeatureExtractorJoin
- FeatureExtractorJsfunc
- FeatureExtractorMultinomial
- FeatureExtractorNumeric
- FeatureExtractorPair
- FeatureExtractorRandom
- FeatureExtractorSparseVector
- FeatureExtractorText
- FeatureTokenizer
- PerformanceStat
- PerformanceStatBase
- PerformanceStatStore
- QMinerFlags
- QueryObject
- SchemaDef
- SchemaFieldDef
- SchemaJoinDef
- SchemaKeyDef
- SchemaTimeWindowDef
- StreamAggrAggrResampler
- StreamAggrAnomalyDetectorNN
- StreamAggregator
- StreamAggrEMA
- StreamAggrEMASpVec
- StreamAggrFeatureSpace
- StreamAggrHistogram
- StreamAggrMax
- StreamAggrMerger
- StreamAggrMin
- StreamAggrMovingAverage
- StreamAggrMovingCorrelation
- StreamAggrMovingCovariance
- StreamAggrMovingVariance
- StreamAggrPageHinkley
- StreamAggrRecordBuffer
- StreamAggrRecordSwitch
- StreamAggrResampler
- StreamAggrSimpleLinearRegression
- StreamAggrSimpleLinearRegressionResult
- StreamAggrSlottedHistogram
- StreamAggrSparseVecSum
- StreamAggrSum
- StreamAggrTDigest
- StreamAggrThreshold
- StreamAggrTimeSeriesTick
- StreamAggrTimeSeriesWindow
- StreamAggrTimeSeriesWindowVector
- StreamAggrVecDiff
- StreamAggrWindowQuantiles
Classes
Base
CircularRecordBuffer
FeatureSpace
Iterator
Record
RecordSet
RecordVector
Store
StreamAggr
Properties
flags
Returns an object with all compile flags. Type module:qm~QMinerFlags.
Example
// import qm module
var qm = require('qminer');
// get the compile flags
var flags = qm.flags;
version
Returns the module version.
Example
// import qm module
var qm = require('qminer');
// get the module version
var version = qm.version;
- Returns
string
B The module version.
Methods
config([configPath][, overwrite][, portN][, cacheSize])
Creates a directory structure.
Parameters
| Name | Type | Optional | Description |
|---|---|---|---|
|
configPath |
string |
Yes |
The path to configuration file. Defaults to |
|
overwrite |
boolean |
Yes |
If you want to overwrite the configuration file. Defaults to |
|
portN |
number |
Yes |
The number of the port. Currently not used. Defaults to |
|
cacheSize |
number |
Yes |
Sets available memory for indexing (in MB). Defaults to |
create([configPath][, schemaPath][, clear]) → module:qm.Base
Creates an empty base.
Parameters
| Name | Type | Optional | Description |
|---|---|---|---|
|
configPath |
string |
Yes |
Configuration file path. Defaults to |
|
schemaPath |
string |
Yes |
Schema file path. Defaults to |
|
clear |
boolean |
Yes |
Clear the existing db folder. Defaults to |
- Returns
module:qm.Base
B The newly created base.
open([configPath][, readOnly]) → module:qm.Base
Opens a base.
Parameters
| Name | Type | Optional | Description |
|---|---|---|---|
|
configPath |
string |
Yes |
The configuration file path. Defaults to |
|
readOnly |
boolean |
Yes |
Open in read-only mode. Defaults to |
- Returns
module:qm.Base
B The loaded base.
stats()
Returns an JSON with two properties: "byClass" and "total". The "byClass" value is a JSON where each key is a class ID and each value is of the form { newFromCpp: number, newFromJs: number, destructorCalls: number} and the value of "total" is of the same form (aggregated over "byClass")
verbosity([level])
Set verbosity of QMiner internals.
Parameter
| Name | Type | Optional | Description |
|---|---|---|---|
|
level |
number |
Yes |
verbosity level. Possible options:
Defaults to |
Abstract types
BaseConstructorParam Object
Base constructor parameter used for module:qm.Base.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
mode |
string |
Yes |
Base access mode. Can be one of the following:
Defaults to |
|
indexCache |
number |
Yes |
The ammount of memory reserved for indexing (in MB). Defaults to |
|
storeCache |
number |
Yes |
The ammount of memory reserved for store cache (in MB). Defaults to |
|
schemaPath |
string |
Yes |
The path to schema definition file. Defaults to |
|
schema |
Array of module:qm~SchemaDef |
Yes |
Schema definition object array. Defaults to |
|
dbPath |
string |
Yes |
The path to db directory. Defaults to |
BaseLoadCSVParam object
The parameter given to module:qm.Base#loadCSV.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
file |
string |
|
The name of the input file. |
|
store |
string |
|
Name of the store which will be created. |
|
base |
|
QMiner base object that creates the store. |
|
|
delimiter |
string |
Yes |
Optional delimiter. Defaults to |
|
quote |
string |
Yes |
Optional character to escape values that contain a delimiter. Defaults to |
DetailKeyObject object
The details about the key object used in module:qm.Store#key and module:qm.Store#keys.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
fq |
|
The frequency. |
|
|
vocabulary |
|
The vocabulary. |
|
|
name |
string |
|
The key name. |
|
store |
|
The store. |
FeatureExtractor Object
Feature extractor types. Used for constructing module:qm.FeatureSpace objects.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
constant |
|
The constant type. Adds a constant value as a feature. |
|
|
random |
|
The random type. Adds a random value as a feature. |
|
|
numeric |
|
The numeric type. Adds the numeric value as a feature. |
|
|
categorical |
|
The categorical type. |
|
|
multinomial |
|
The multinomial type. |
|
|
text |
|
The text type. Creates the bag-of-words text representation. |
|
|
join |
|
The join type. |
|
|
pair |
|
The pair type. |
|
|
jsfunc |
|
The jsfunc type. Allows creating a custom feature extractor. |
|
|
dateWindow |
|
The date window type. |
|
|
sparseVector |
|
The sparse vector type. |
FeatureExtractorCategorical Object
The feature extractor of type 'categorical'
. Used for constructing module:qm.FeatureSpace objects.
Example
var qm = require('qminer');
// create a simple base, where each record contains the student name and it's study group
// here we know the student is part of only one study group
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: "Class",
fields: [
{ name: "Name", type: "string" },
{ name: "StudyGroup", type: "string" }
]
}]
});
// create a feature space containing the categorical extractor, where it's values
// are taken from the field "StudyGroup": "A", "B", "C" and "D"
var ftr = new qm.FeatureSpace(base, { type: "categorical", source: "Class", field: "StudyGroup" });
// add a few records to the store
base.store("Class").push({ Name: "Fred", StudyGroup: "A" });
base.store("Class").push({ Name: "Wilma", StudyGroup: "B" });
base.store("Class").push({ Name: "Barney", StudyGroup: "C" });
// update the feature space to get the categories
ftr.updateRecords(base.store("Class").allRecords);
// get the feature vector for the first record
var vec = ftr.extractVector(base.store("Class")[0]); // returns vector [1, 0, 0]
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
values |
Array of Object |
Yes |
A fixed set of values, which form a fixed feature set. No dimensionality changes if new values are seen in the upgrades. |
|
hashDimension |
number |
Yes |
A hashing code to set the fixed dimensionality. All values are hashed and divided modulo hasDimension to get the corresponding dimension. |
|
field |
string |
|
The name of the field form which to take the values. |
|
source |
string |
|
The store name. |
FeatureExtractorConstant Object
The feature extractor of type 'contant'
. Used for constructing module:qm.FeatureSpace objects.
Example
var qm = require('qminer');
// create a simple base, where each record contains only a persons name
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: "Person",
fields: [{ name: "Name", type: "string" }]
}]
});
// create a feature space containing the constant extractor, where the constant is equal 5
var ftr = new qm.FeatureSpace(base, { type: "constant", source: "Person", const: 5 });
// add a new record to the base
base.store("Person").push({ Name: "Peterson" });
// get the features of the record
var vec = ftr.extractVector(base.store("Person")[0]); // the vector [5]
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
const |
number |
Yes |
A constant number. Defaults to |
|
source |
string |
|
The store name. |
FeatureExtractorDateWindow Object
The feature extractor of type 'dateWindow'
. Used for constructing module:qm.FeatureSpace objects.
Example
// import qm module
var qm = require('qminer');
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
unit |
string |
Yes |
How granular is the time window. Possible options are Defaults to |
|
window |
number |
Yes |
The size of the window. Defaults to |
|
normalize |
boolean |
Yes |
Normalize the resulting vector of the extractor to have L2 norm 1.0. Defaults to |
|
start |
number |
|
//TODO |
|
end |
number |
|
//TODO |
|
source |
string |
|
The store name. |
FeatureExtractorJoin Object
The feature extractor of type 'join'
. Used for constructing module:qm.FeatureSpace objects.
Example
// import qm module
var qm = require('qminer');
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
bucketSize |
number |
Yes |
The size of the bucket in which we group consecutive records. Defaults to |
|
source |
string |
|
The store name. |
FeatureExtractorJsfunc Object
The feature extractor of type 'jsfunc'
. Used for constructing module:qm.FeatureSpace objects.
Example
var qm = require('qminer');
// create a simple base, where each record contains the name of the student and his study groups
// each student is part of multiple study groups
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: "Class",
fields: [
{ name: "Name", type: "string" },
{ name: "StudyGroups", type: "string_v" }
]
}]
});
// create a feature space containing the jsfunc extractor, where the function counts the number
// of study groups each student is part of. The functions name is "NumberOFGroups", it's dimension
// is 1 (returns only one value, not an array)
var ftr = new qm.FeatureSpace(base, {
type: "jsfunc", source: "Class", name: "NumberOfGroups", dim: 1,
fun: function (rec) { return rec.StudyGroups.length; }
});
// add a record to the store
base.store("Class").push({ Name: "Gaben", StudyGroups: ["A", "B", "D"] });
// get the feature vector of the record
var vec = ftr.extractVector(base.store("Class")[0]); // returns vector [3]
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
name |
string |
|
The features name. |
|
fun |
function() |
|
The javascript function callback. It takes one parameter:
|
|
dim |
number |
Yes |
The dimension of the feature extractor. Defaults to |
|
source |
string |
|
The store name. |
FeatureExtractorMultinomial Object
The feature extractor of type 'multinomial'
. Used for constructing module:qm.FeatureSpace objects.
Example
var qm = require('qminer');
// create a simple base, where each record contains the student name and an array of study groups
// here we know a student can be part of multiple study groups
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: "Class",
fields: [
{ name: "Name", type: "string" },
{ name: "StudyGroups", type: "string_v" }
]
}]
});
// create a feature space containing the multinomial extractor, where the values are normalized,
// and taken from the field "StudyGroup": "A", "B", "C", "D", "E", "F"
var ftr = new qm.FeatureSpace(base, {
type: "multinomial", source: "Class", field: "StudyGroups", normalize: true
});
// add a few records to the store
base.store("Class").push({ Name: "Fred", StudyGroups: ["A", "B"] });
base.store("Class").push({ Name: "Wilma", StudyGroups: ["B", "C"] });
base.store("Class").push({ Name: "Barney", StudyGroups: ["C", "A"] });
// update the feature space to get the categories
ftr.updateRecords(base.store("Class").allRecords);
// get the feature vector for the first record
var vec = ftr.extractVector(base.store("Class")[0]); // returns vector [0.707, 0.707, 0]
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
normalize |
boolean |
Yes |
Normalize the resulting vector of the extractor to have L2 norm 1.0. Defaults to |
|
transform |
string |
Yes |
Transformation to apply to each dimension of the feature vector.
|
|
values |
Array of Object |
Yes |
A fixed set of values, which form a fixed feature set, no dimensionality changes if new values are seen in the updates. Cannot be used the same time as datetime. |
|
hashDimension |
number |
Yes |
A hashing code to set the fixed dimensionality. All values are hashed and divided modulo hashDimension to get the corresponding dimension. |
|
datetime |
Object |
Yes |
Same as Defaults to |
|
field |
(string or Array of String) |
|
The name of the field from which to take the key value. |
|
valueField |
(string or Array of String) |
Yes |
The name of the field from which to take the numeric value.
|
|
source |
string |
|
The store name. |
FeatureExtractorNumeric Object
The feature extractor of type 'numeric'
. Used for constructing module:qm.FeatureSpace objects.
Example
var qm = require('qminer');
// create a simple base, where each record contains the student name and it's grade
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: "Class",
fields: [
{ name: "Name", type: "string" },
{ name: "Grade", type: "int" }
]
}]
});
// create a feature space containing the numeric extractor, where the values are
// normalized, the values are taken from the field "Grade"
var ftr = new qm.FeatureSpace(base, { type: "numeric", source: "Class", normalize: "scale", field: "Grade" });
// add a new record to the base
base.store("Class").push({ Name: "Peterson", Grade: 9 });
base.store("Class").push({ Name: "Ericsson", Grade: 8 });
// update the feature space for scaling
ftr.updateRecords(base.store("Class").allRecords);
// get the features of the first record
var vec = ftr.extractVector(base.store("Class")[0]); // the vector with the random value
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
normalize |
string |
Yes |
Normalize values between 0.0 and 1.0 if set to Defaults to |
|
min |
number |
Yes |
The minimal value used to form the normalization. |
|
max |
number |
Yes |
The maximal value used to form the normalization. |
|
field |
string |
|
The name of the field from which to take the value. |
|
source |
string |
|
The store name. |
FeatureExtractorPair Object
The feature extractor of type 'pair'
. Used for constructing module:qm.FeatureSpace objects.
Example
var qm = require('qminer');
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
first |
module:qm~FeatureExtractors |
|
The first feature extractor. |
|
second |
module:qm~FeatureExtractors |
|
The second feature extractor. |
|
source |
source |
|
The store name. |
FeatureExtractorRandom Object
The feature extractor of type 'random'
. Used for constructing module:qm.FeatureSpace objects.
Example
var qm = require('qminer');
// create a simple base, where each record contains only a persons name
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: "Person",
fields: [{ name: "Name", type: "string" }]
}]
});
// create a feature space containing the random extractor
var ftr = new qm.FeatureSpace(base, { type: "random", source: "Person" });
// add a new record to the base
base.store("Person").push({ Name: "Peterson" });
// get the features of the record
var vec = ftr.extractVector(base.store("Person")[0]); // the vector with the random value
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
seed |
number |
Yes |
The seed number used to construct the random number. Defaults to |
|
source |
string |
|
The store name. |
FeatureExtractorSparseVector Object
The feature extractor of type 'num_sp_v'
. Used for constructing module:qm.FeatureSpace objects.
Example
var qm = require('qminer');
// create a simple base, where each record contains the student name and it's grade
var base = new qm.Base({
mode: 'createClean',
schema: [{
"name": "Class",
"fields": [
{ "name": "Name", "type": "string" },
{ "name": "Features", "type": "num_sp_v" }
]
}]
});
// create a feature space containing the numeric extractor, where the values are
// normalized, the values are taken from the field "Grade"
var ftr = new qm.FeatureSpace(base, { type: "num_sp_v", source: "Class", normalize: false, field: "Features" });
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
dimension |
number |
Yes |
Dimensionality of sparse vectors. Defaults to |
|
normalize |
boolean |
Yes |
Normalize vectors to L2 norm of 1.0. Defaults to |
|
field |
string |
|
The name of the field from which to take the value. |
|
source |
string |
|
The store name. |
FeatureExtractorText Object
The feature extractor of type 'text'
. Used for constructing module:qm.FeatureSpace objects.
Example
var qm = require('qminer');
// create a simple base, where each record contains the title of the article and it's content
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: "Articles",
fields: [
{ name: "Title", type: "string" },
{ name: "Text", type: "string" }
]
}]
});
// create a feature spave containing the text (bag of words) extractor, where the values are normalized,
// weighted with 'tfidf' and the tokenizer is of 'simple' type, it uses english stopwords.
var ftr = new qm.FeatureSpace(base, {
type: "text", source: "Articles", field: "Text", normalize: true, weight: "tfidf",
tokenizer: { type: "simple", stopwords: "en"}
});
// add a few records to the store
base.store("Articles").push({ Title: "Article1", Text: "The last time we drew up the league table for bad banks, UBS was on top." });
base.store("Articles").push({ Title: "Article2", Text: "Barclays dropped a bombshell on its investment bankers last week." });
// update the feature space to get the the terms for the bag-of-words text representation
ftr.updateRecords(base.store("Articles").allRecords);
// extract the feature vector for the first article
var vec = ftr.extractSparseVector(base.store("Articles")[0]);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the extractor. Important: It must be equal |
|
normalize |
boolean |
Yes |
Normalize the resulting vector of the extractor to have L2 norm 1.0. Defaults to |
|
weight |
string |
Yes |
Type of weighting used for scoring terms. Possible options are:
Defaults to |
|
hashDimension |
number |
Yes |
A hashing code to set the fixed dimensionality. All values are hashed and divided modulo hashDimension to get the corresponding dimension. |
|
hashTable |
boolean |
Yes |
If true, stores the hash table which is used for for feature naming. Defaults to |
|
field |
string |
|
The name of the field from which to take the value. |
|
tokenizer |
|
The settings for extraction of text. |
|
|
mode |
string |
Yes |
How are multi-record cases combined into single vector. Possible options:
|
|
stream |
string |
Yes |
Details on forgetting old IDFs when running on stream. Possible options:
|
|
source |
string |
|
The store name. |
FeatureTokenizer Object
The settings for extraction of text used in module:qm~FeatureExtractorText.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
Yes |
The type of the encoding text. Possible options:
Defaults to |
|
stopwords |
(string or Array of string) |
Yes |
The stopwords used for extraction. Possible options:
Defaults to |
|
stemmer |
string |
Yes |
The stemmer used for extraction. Possible options:
Defaults to |
|
uppercase |
boolean |
Yes |
Changing all words to uppercase. Defaults to |
PerformanceStat object
The performance statistics used to describe module:qm~PerformanceStatBase and module:qm~PerformanceStatStore.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
alloc_count |
number |
|
\ TODO: Add the description |
|
alloc_size |
number |
|
\ TODO: Add the description |
|
alloc_unused_size |
number |
|
\ TODO: Add the description |
|
avg_get_len |
number |
|
\ TODO: Add the description |
|
avg_put_len |
number |
|
\ TODO: Add the description |
|
avg_get_new_len |
number |
|
\ TODO: Add the description |
|
dels |
number |
|
\ TODO: Add the description |
|
gets |
number |
|
\ TODO: Add the description |
|
puts |
number |
|
\ TODO: Add the description |
|
puts_new |
number |
|
\ TODO: Add the description |
|
released_count |
number |
|
\ TODO: Add the description |
|
released_size |
number |
|
\ TODO: Add the description |
|
size_changes |
number |
|
\ TODO: Add the description |
PerformanceStatBase object
The performance statistics that is returned by module:qm.Base#getStats.
Properties
| Name | Type | Optional | Description | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
stores |
Array of module:qm~PerformanceStatStore |
|
The performance statistics of the stores in base. \ TODO: Check if this is right |
||||||||||||||||||||||||||||
|
gix_stats |
object |
|
The statistics of the base. \ TODO: Check if this is right Values in
|
||||||||||||||||||||||||||||
|
gix_blob |
|
\ TODO: Add the description |
PerformanceStatStore object
The performance statistics of the store found in module:qm~PerformanceStatBase.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
Store name. |
|
blob_storage_memory |
|
\ TODO: Add the description |
|
|
blob_storage_cache |
|
\ TODO: Add the description |
QMinerFlags Object
The object containing the QMiner compile flags.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
buildTime |
string |
|
The module build time. |
|
win |
boolean |
|
True, if the module is compiled for Windows. |
|
linux |
boolean |
|
True, if the module is compiled for Linux. |
|
darwin |
boolean |
|
True, if the module is compiled for Darwin. |
|
x86 |
boolean |
|
True, if the module is compiled for x86 system. |
|
x64 |
boolean |
|
True, if the module is compiled for x64 system. |
|
omp |
boolean |
|
True, if the module is compiled for omp. |
|
debug |
boolean |
|
True, if the module is compiled for debug mode. |
|
gcc |
boolean |
|
True, if the module is compiled for gcc. |
|
clang |
boolean |
|
True, if the module is compiled for clang. |
|
blas |
boolean |
|
True, if the module is compiled with Blas. |
|
blas_intel |
boolean |
|
True, if the module is compiled with Intel Blas. |
|
blas_amd |
boolean |
|
True, if the module is compiled with AMD Blas. |
|
blas_openblas |
boolean |
|
True, if the module is compiled with OpenBLAS. |
|
lapacke |
boolean |
|
True, if the module is compiled with Lapacke. |
QueryObject object
The object used for querying records with module:qm.Base#search. How to construct a query is found on the QMiner Wiki page.
SchemaDef Object
Store schema definition used in module:qm~BaseConstructorParam.
Example
var qm = require('qminer');
// create a simple movies store, where each record contains only the movie title.
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: "Movies",
fields: [{ name: "title", type: "string" }]
}]
});
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The name of the store. Store name can be composed by from English letters, numbers, _ or $ characters. It can only begin with a character. |
|
fields |
Array of module:qm~SchemaFieldDef |
|
The array of field descriptors. |
|
joins |
Array of module:qm~SchemaJoinDef |
Yes |
The array of join descriptors, used for linking records from different stores. Defaults to |
|
keys |
Array of module:qm~SchemaKeyDef |
Yes |
The array of key descriptors. Keys define how records are indexed, which is needed for search using the query language. Defaults to |
|
timeWindow |
Yes |
Time window description. Stores can have a window, which is used by garbage collector to delete records once they fall out of the time window. Window can be defined by number of records or by time. |
SchemaFieldDef Object
Store schema field definition used in module:qm~SchemaDef.
Example
var qm = require('qminer');
var base = new qm.Base({
mode: 'createClean',
schema: [
{ name: 'NewsArticles',
fields: [
{ name: "ID", primary: true, type: "string", shortstring: true },
{ name: "Source", type: "string", codebook: true },
{ name: "DateTime", type: "datetime" },
{ name: "Title", type: "string", store: "cache" },
{ name: "Tokens", type: "string_v", store: "cache", null: true },
{ name: "Vector", type: "num_sp_v", store: "cache", null: true }]
}
]
});
// add a record:
// - we set the date using the ISO string representation
// - we set the string vector Tokens with an array of strings
// - we set the numeric sparse vector Vector with an array of two element arrays
// (index, value), see the sparse vector constructor {@link module:la.SparseVector}
base.store('NewsArticles').push({
ID: 't12344',
Source: 's1234',
DateTime: '2015-01-01T00:05:00',
Title: 'the title',
Tokens: ['token1', 'token2'],
Vector: [[0,1], [1,1]]});
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The name of the field. |
|
type |
string |
|
The type of the field. Possible options:
|
|
primary |
boolean |
Yes |
Field which can be used to identify record. There can be only one primary field in a store. There can be at most one record for each value of the primary field. Currently following fields can be marked as primary: Defaults to |
|
null |
boolean |
Yes |
When set to true, null is a possible value for a field (allow missing values). Defaults to |
|
store |
string |
Yes |
Defines where to store the field. Possible options
Defaults to |
|
default |
Object |
Yes |
Default value for field when not given for a new record. |
|
codebook |
boolean |
Yes |
Useful when many records have only few different values of this field. If set to true, then a separate table of all values is kept, and records only point to this table (replacing variable string field in record serialisation with fixed-length integer). Useful to decrease memory footprint, and faster to update. (STRING FIELD TYPE SPECIFIC). Defaults to |
|
shortstring |
boolean |
Yes |
Useful for string shorter then 127 characters (STRING FIELD TYPE SPECIFIC). Defaults to |
SchemaJoinDef Object
Store schema join definition used in module:qm~SchemaDef.
Example
var qm = require('qminer');
// Create two stores: People which stores only names of persons and Movies, which stores only titles.
// Each person can direct zero or more movies, so we use an index join named 'directed' and
// each movie has a single director, so we use a field join 'director'. The joins are
// inverses of each other. The inverse join simplifies the linking, since only one join needs
// to be specified, and the other direction can be linked automatically (in the example
// below we specify only the 'director' link and the 'directed' join is updated automatically).
//
var base = new qm.Base({
mode: 'createClean',
schema: [
{ name: 'People',
fields: [{ name: 'name', type: 'string', primary: true }],
joins: [{ name: 'directed', 'type': 'index', 'store': 'Movies', 'inverse': 'director' }] },
{ name: 'Movies',
fields: [{ name: 'title', type: 'string', primary: true }],
joins: [{ name: 'director', 'type': 'field', 'store': 'People', 'inverse': 'directed' }] }
]
});
// Adds a movie, automatically adds 'Jim Jarmusch' to People, sets the 'director' join (field join)
// and automatically updates the index join 'directed', since it's an inverse join of 'director'
base.store('Movies').push({ title: 'Broken Flowers', director: { name: 'Jim Jarmusch' } });
// Adds a movie, sets the 'director' join, updates the index join of 'Jim Jarmusch'
base.store('Movies').push({ title: 'Coffee and Cigarettes', director: { name: 'Jim Jarmusch' } });
// Adds movie, automatically adds 'Lars von Trier' to People, sets the 'director' join
// and 'directed' inverse join (automatically)
base.store('Movies').push({ title: 'Dogville', director: { name: 'Lars von Trier' } });
// Adds a person, sets the 'directed' join with multiple movies ('directed' is of type 'index', movies must be given in an array)
base.store('People').push({ name: 'Christopher Nolan', directed: [{ title: 'Inception' }, { title: 'Interstellar' }] });
var movie = base.store('Movies')[0]; // get the first movie (Broken Flowers)
// Each movie has a property corresponding to the join name: 'director'.
// Accessing the property returns a {@link module:qm.Record} from the store People.
var person = movie.director; // get the director
var personName = person.name; // get person's name ('Jim Jarmusch')
// Each person has a property corresponding to the join name: 'directed'.
// Accessing the property returns a {@link module:qm.RecSet} from the store People.
var movies = person.directed; // get all the movies the person directed.
movies.each(function (movie) { var title = movie.title; });
// Gets the following titles:
// 'Broken Flowers'
// 'Coffee and Cigarettes'
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The name of the join. |
|
type |
string |
|
Join types. Possible options:
|
|
store |
string |
|
The store name from which the linked records are. |
SchemaKeyDef Object
Store schema key definition used in module:qm~SchemaDef.
Example
var qm = require('qminer');
// Create a store People which stores only names of persons.
var base = new qm.Base({
mode: 'createClean',
schema: [
{ name: 'People',
fields: [{ name: 'name', type: 'string', primary: true }],
keys: [
{ field: 'name', type: 'value'},
{ field: 'name', name: 'nameText', type: 'text'}
]
}
]
});
base.store('People').push({name : 'John Smith'});
base.store('People').push({name : 'Mary Smith'});
// search based on indexed values
base.search({$from : 'People', name: 'John Smith'}); // Return the record set containing 'John Smith'
// search based on indexed values
base.search({$from : 'People', name: 'Smith'}); // Returns the empty record set
// search based on text indexing
base.search({$from : 'People', nameText: 'Smith'}); // Returns both records
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
field |
string |
|
The name of the field that will be indexed. |
|
type |
string |
|
Key type. Possible options:
|
|
name |
string |
Yes |
Allows using a different name for the key in search queries. This allows for multiple keys to be put against the same field. Default value is the name of the field. |
|
vocabulary |
string |
Yes |
Defines the name of the vocabulary used to store the tokens or values. This can be used indicate to several keys to use the same vocabulary, to save on memory. Supported by |
|
tokenize |
string |
Yes |
Defines the tokenizer that is used for tokenizing the values stored in indexed fields. Tokenizer uses same parameters as in bag-of-words feature extractor. Default is english stopword list and no stemmer. Supported by |
SchemaTimeWindowDef Object
Stores can have a window, which is used by garbage collector to delete records once they
fall out of the time window. Window can be defined by number of records or by time.
Window defined by parameter window, its value being the number of records to be kept. Used in module:qm~SchemaDef.
Important: module:qm.Base#garbageCollect must be called manually to remove records outside time window.
Examples
Define window by number of records
var qm = require('qminer');
// create base
var base = new qm.Base({ mode: 'createClean' });
// create store with window
base.createStore({
"name": "TestStore",
"fields": [
{ "name": "DateTime", "type": "datetime" },
{ "name": "Measurement", "type": "float" }
],
window: 3,
});
// push 5 records into created store
for (var i = 0; i < 5; i++) {
var rec = {
"DateTime": new Date().toISOString(),
"Measurement": i
};
base.store("TestStore").push(rec);
}
// check number of records in store
base.store("TestStore").allRecords.length; // 5
// clean base with garbage collector
base.garbageCollect();
// check number of records in store
base.store("TestStore").allRecords.length; // 3
base.close();
Define window by time
var qm = require('qminer');
// create base
var base = new qm.Base({ mode: 'createClean' });
// create store with window
base.createStore({
"name": "TestStore",
"fields": [
{ "name": "DateTime", "type": "datetime" },
{ "name": "Measurement", "type": "float" }
],
timeWindow: {
duration: 2,
unit: "hour",
field: "DateTime"
}
});
// push 5 records into created store
for (var i = 0; i < 5; i++) {
var rec = {
"DateTime": new Date(new Date().getTime() + i * 60 * 60 * 1001).toISOString(),
"Measurement": i
};
base.store("TestStore").push(rec);
}
// check number of records in store
base.store("TestStore").allRecords.length; // 5
// clean base with garbage collector
base.garbageCollect();
// check number of records in store
base.store("TestStore").allRecords.length; // 2
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
duration |
number |
|
The size of the time window (in number of units). |
|
unit |
string |
|
Defines in which units the window size is specified. Possible options are |
|
field |
string |
Yes |
Name of the datetime field, which defines the time of the record. In case it is not given, the insert time is used in its place. |
StreamAggrAggrResampler module:qm.StreamAggr
This stream aggregate resamples an input time series to a new time seris
of equally spaced measurements. Each new measurement corresponds to an
aggregate (sum,avg,min,max) computed over an interval. The aggregate
exposes the following methods.
1. module:qm.StreamAggr#getFloat returns the last resampled value.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the last resampled value.
3. module:qm.StreamAggr#onStep reads from an input aggregate and tries to resample.
4. module:qm.StreamAggr#onTime updates the current time (no data has arrived, but time has passed) and tries to resample.
5. module:qm.StreamAggr#getParams returns a parameter object.
6. module:qm.StreamAggr#setParams used primarily for setting the out-aggregate.
The stream aggregate exposes its results through getFloat
and getTimestamp
methods (itself represents timeseries).
The resampler has an input time-series aggregate (supports getFloat
and getTimestamp
), from where it reads time series values.
The reading and resampling occourrs wehen resamplers onStep()
or onTime()
methods are called.
When resampling succeeds (all the data needed for the computation becomes available), the resampler
will trigger the onStep()
method of an output stream aggregate that will read the resamplers state through getFloat
and getTime
.
Example
var qm = require('qminer');
// create a base with a simple timeseries store
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: 'default',
fields: [
{ name: 'timestamp', type: 'datetime' },
{ name: 'value', type: 'float' }
]
}]
});
var store = base.store('default');
// the tick aggregate reads from the store (provides time series input to other aggregates)
var raw = store.addStreamAggr({
type: 'timeSeriesTick',
timestamp: 'timestamp',
value: 'value'
});
// will compute sums over 1 second intervals
var resampler = store.addStreamAggr({
type: 'aggrResample',
inAggr: raw.name,
start: '1970-01-01T00:00:00.000',
defaultValue: 0,
aggType: 'sum',
interval: 1000
});
// will print out resampler state on each resample
var resamplerOutput = new qm.StreamAggr(base, new function () {
this.onStep = function () {
console.log('Resampler emitted the sum: ' + resampler.getFloat() +
' for the interval [' + new Date(resampler.getTimestamp()).toISOString() +
' - ' + new Date(resampler.getTimestamp() + resampler.getParams().interval).toISOString() + ')');
}
});
// IMPORTANT. After the output exists, connect it to resampler
resampler.setParams({ outAggr: resamplerOutput.name });
store.push({ timestamp: 1, value: 1 });
store.push({ timestamp: 10, value: 10 });
store.push({ timestamp: 500, value: 100 });
store.push({ timestamp: 2000, value: 1000 }); // triggers two resampling steps (two intervals complete)
// // three measurements for the first interval
// Resampler emitted the sum: 111 for the interval [1970-01-01T00:00:00.000Z - 1970-01-01T00:00:01.000Z)
// // zero measurements for the second interval (does not skip empty)
// Resampler emitted the sum: 0 for the interval [1970-01-01T00:00:01.000Z - 1970-01-01T00:00:02.000Z)
store.push({ timestamp: 2001, value: 10000 });
store.push({ timestamp: 3000, value: 100000 }); // triggers one resampling step (one interval complete)
// // one measurement for the third interval
// Resampler emitted the sum: 11000 for the interval [1970-01-01T00:00:02.000Z - 1970-01-01T00:00:03.000Z)
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
interval |
number |
|
Interval size in milliseconds |
|
aggType |
string |
|
Must be one of the values: |
|
inAggr |
(string or module:qm.StreamAggr) |
|
The name of the input stream aggregate which must implement |
|
start |
(string or number) |
Yes |
Start time (linux timestamp or a web log date string like |
|
roundStart |
string |
Yes |
Must be one of the values: |
|
defaultValue |
number |
Yes |
default value for empty intervals (no data available). Defaults to |
|
skipEmpty |
boolean |
Yes |
If true, the resampler will not call the Defaults to |
|
name |
string |
Yes |
The given name for the stream aggregator. |
|
outAggr |
(string or module:qm.StreamAggr) |
Yes |
The name of the output stream aggregate. Only useful when the |
StreamAggrAnomalyDetectorNN module:qmStreamAggr
This stream aggregator represents the anomaly detector using the Nearest Neighbor algorithm. It calculates the
new incoming point's distance from its nearest neighbor and, depending on the input threshold values, it
classifies the severity of the alarm. It implements the following methods:
1. module:qm.StreamAggr#getInteger returns the severity of the alarm.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the latest alarm.
3. module:qm.StreamAggr#saveJson returns the Json with the description and explanation of the alarm.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store named Cars with 4 fields
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: 'Cars',
fields: [
{ name: 'NumberOfCars', type: 'float' },
{ name: 'Temperature', type: 'float' },
{ name: 'Precipitation', type: 'float' },
{ name: 'Time', type: 'datetime' }
]
}]
});
// create the store
var store = base.store('Cars');
// define a feature space aggregator on the Cars store which needs at least 2 records to be initialized. Use three of the
// four fields of the store to create feature vectors with normalized values.
var aggr = {
name: "ftrSpaceAggr",
type: "featureSpace",
initCount: 2,
update: true, full: false, sparse: true,
featureSpace: [
{ type: "numeric", source: "Cars", field: "NumberOfCars", normalize: "var" },
{ type: "numeric", source: "Cars", field: "Temperature", normalize: "var" },
{ type: "numeric", source: "Cars", field: "Precipitation", normalize: "var" }
]
};
//create the feature space aggregator
var ftrSpaceAggr = base.store('Cars').addStreamAggr(aggr);
// define a new time series tick stream aggregator for the 'Cars' store, that takes the values from the 'NumberOfCars' field
// and the timestamp from the 'Time' field.
var aggr = {
name: "tickAggr",
type: "timeSeriesTick",
store: "Cars",
timestamp: "Time",
value: "NumberOfCars"
};
//create the tick aggregator
var tickAggr = base.store('Cars').addStreamAggr(aggr);
//define an anomaly detection aggregator using nearest neighbor on the cars store that takes as input timestamped features.
// The time stamp is provided by the tick aggregator while the feature vector is provided by the feature space aggregator.
var aggr = {
name: 'AnomalyDetectorAggr',
type: 'nnAnomalyDetector',
inAggrSpV: 'ftrSpaceAggr',
inAggrTm: 'tickAggr',
rate: [0.7, 0.5, 0.15],
windowSize: 2
};
//create the anomaly detection aggregator
var anomaly = base.store('Cars').addStreamAggr(aggr);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. It cannot be connect to the module:qm~StreamAggrTimeSeriesWindow. |
StreamAggregator module:qm.StreamAggr
Stream aggregator types.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
timeSeries |
|
The time series type. |
|
|
timeSeriesBufferVector |
|
The time series buffer vector type. |
|
|
tick |
|
The time series tick type. |
|
|
record-buffer |
|
The record buffer type. |
|
|
ftr-space |
|
The feature space type. |
|
|
sum |
|
The sum type. Calculates the sum of values. |
|
|
min |
|
The minimal type. Saves the minimal value in the buffer. |
|
|
max |
|
The maximal type. Saves the maximal value in the buffer. |
|
|
sparse-vec-sum |
|
The sparse-vector-sum type. |
|
|
ma |
|
The moving average type. Calculates the average within the window. |
|
|
ema |
|
The exponental moving average type. Calculates the exponental average of the values. |
|
|
ema-sp-vec |
|
The exponental moving average for sparse vectors type. |
|
|
var |
|
The moving variance type. Calculates the variance of values within the window. |
|
|
cov |
|
The moving covariance type. Calculates the covariance of values within the window. |
|
|
cor |
|
The moving correlation type. Calculates the correlation of values within the window. |
|
|
res |
|
The resampler type. Resamples the records so that they come in in the same time interval. |
|
|
aggr-res |
|
The aggregating (avg/sum) resampler type. Resamplers the records so that it takes the records in the time window and returns one sample. |
|
|
mer |
|
The merger type. Merges the records from two stream series. |
|
|
hist |
|
The online histogram type. |
|
|
slotted-hist |
|
The online slotted-histogram type. |
|
|
vec-diff |
|
The difference of two vectors (e.g. online histograms) type. |
|
|
lin-reg |
|
The linear regressor type. |
|
|
detector-nn |
|
The anomaly detector type. Detects anomalies using the k nearest neighbour algorithm. |
|
|
treshold |
|
The threshold indicator type. |
|
|
tdigest |
|
The quantile estimator type. It estimates the quantiles of the given data using TDigest. |
|
|
record-switch-aggr |
|
The record switch type. |
|
|
pagehinkley |
|
The Page-Hinkley test for concept drift detection type. |
StreamAggrEMA module:qmStreamAggr
This stream aggregator represents the exponential moving average window buffer. It calculates the weighted moving average
of the values in the connected stream aggregator, where the weights are exponentially decreasing. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the exponentional average of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series tick stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window should be 1 hour.
var timeser = {
name: 'TimeSeriesAggr',
type: 'timeSeriesTick',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius'
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);
// add an exponentional moving average aggregator, that is connected with the 'TimeSeriesAggr' aggregator.
// It should interpolate with the previous value, the decay should be 3 seconds and the initWindow should be 2 seconds.
var ema = {
name: 'emaAggr',
type: 'ema',
store: 'Heat',
inAggr: 'TimeSeriesAggr',
emaType: 'previous',
interval: 3000,
initWindow: 2000
};
var expoMovingAverage = base.store("Heat").addStreamAggr(ema);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. It cannot be connect to the module:qm~StreamAggrTimeSeriesWindow. |
|
emaType |
string |
|
The type of interpolation. Possible options are:
|
|
interval |
number |
|
The time interval defining the decay. It must be greater than |
|
initWindow |
number |
|
The time window of required values for initialization. |
StreamAggrEMASpVec module:qmStreamAggr
This stream aggregator represents the exponential moving average window buffer for sparse vectors. It calculates the weighted moving average
of the values in the connected stream aggregator, where the weights are exponentially decreasing. It implements the following methods:
1. module:qm.StreamAggr#getValueVector returns the exponentional average of the sparse vector values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Data",
fields: [
{ name: "Text", type: "string" },
{ name: "Time", type: "datetime" }
]
}]
});
var store = base.store("Data");
// create a new time series that emits data as sparse vector, based on text
var aggr = {
name: 'featureSpaceWindow',
type: 'timeSeriesWinBufFeatureSpace',
store: store.name,
timestamp: 'Time',
featureSpace: {
type: "categorical",
source: store.name,
field: "Text"
},
winsize: 1 // keep only most recent value in window
};
// attach sum
var sa = store.addStreamAggr(aggr);
var aggr2 = {
name: 'sparseVectorSum',
type: 'winBufSpVecSum',
store: store.name,
inAggr: aggr.name // this means that sum is equal to the most recent data
};
// ok, now attach EMA
var sa2 = store.addStreamAggr(aggr2);
var ema_def = {
name: 'sparseVectorEma',
type: 'emaSpVec',
store: store.name,
inAggr: aggr2.name,
emaType: "next",
interval: 2000,
initWindow: 0
};
var ema = store.addStreamAggr(ema_def);
// add some data
store.push({ Time: 1000, Text: 'a' });
store.push({ Time: 2000, Text: 'b' });
store.push({ Time: 3000, Text: 'c' });
// display EMA data
ema.getValueVector().print();
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. It cannot be connect to the module:qm~StreamAggrTimeSeriesWindow. |
|
emaType |
string |
|
The type of interpolation. Possible options:
|
|
interval |
number |
|
The time interval defining the decay. It must be greater than |
|
initWindow |
number |
Yes |
The time window of required values for initialization. Defaults to |
|
cuttof |
number |
Yes |
Minimal value for any dimension. If value of certain dimension falls bellow this value, the dimension is pruned from average. Defaults to |
StreamAggrFeatureSpace module:qmStreamAggr
This stream aggregator creates the feature space and stores the specified features of the last input. It implements the following methods:
1. module:qm.StreamAggr#getFloatVector returns the dense feature vectors.
2. module:qm.StreamAggr#getFeatureSpace returns the feature space.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store named Cars with 4 fields
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: 'Cars',
fields: [
{ name: 'NumberOfCars', type: 'float' },
{ name: 'Temperature', type: 'float' },
{ name: 'Precipitation', type: 'float' },
{ name: 'Time', type: 'datetime' }
]
}]
});
// create the store
var store = base.store('Cars');
// define a feature space aggregator on the Cars store which needs at least 2 records to be initialized. Use three of the
// four fields of the store to create feature vectors with normalized values.
var aggr = {
name: "ftrSpaceAggr",
type: "featureSpace",
initCount: 2,
update: true, full: false, sparse: true,
featureSpace: [
{ type: "numeric", source: "Cars", field: "NumberOfCars", normalize: "var" },
{ type: "numeric", source: "Cars", field: "Temperature", normalize: "var" },
{ type: "numeric", source: "Cars", field: "Precipitation", normalize: "var" }
]
};
//create the feature space aggregator
var ftrSpaceAggr = base.store('Cars').addStreamAggr(aggr);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
initCount |
number |
|
The number of records needed before it initializes. |
|
update |
boolean |
|
If true, updates the feature space. |
|
full |
boolean |
|
If true, saves the full vector of features. |
|
sparse |
boolean |
|
If true, saves the sparse vector of features. |
|
FeatureSpace |
Array of module:qm~FeatureExtractor |
|
Array of feature extractors. |
StreamAggrHistogram module:qm.StreamAggr
This stream aggregator represents an online histogram. It can connect to a buffered aggregate (such as module:qm~StreamAggrTimeSeriesWindow)
or a time series (such as module:qm~StreamAggregateEMA).
The aggregate defines an ordered set of points p(0), ..., p(n)
that define n bins. Infinites at both ends are allowed.
A new measurement is tested for inclusion in the left-closed right-opened intervals [p(i), p(i+1))
and the corresponding
bin counter is increased for the appropriate bin (or decreased if the point is outgoing from the buffer).
It implements the following methods:
1. module:qm.StreamAggr#getFloatLength returns the number of bins.
2. module:qm.StreamAggr#getFloatAt returns the count for a bin index.
3. module:qm.StreamAggr#getFloatVector returns the vector of counts, the length is equal to the number of bins.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var timeser = {
name: 'TimeSeriesBuffer',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius',
winsize: 86400000 // one day in miliseconds
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);
// add a histogram aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var aggrJson = {
name: 'Histogram',
type: 'onlineHistogram',
store: 'Heat',
inAggr: 'TimeSeriesBuffer',
lowerBound: 0,
upperBound: 10,
bins: 5,
addNegInf: false,
addPosInf: false
};
var hist = base.store("Heat").addStreamAggr(aggrJson);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type for the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. |
|
lowerBound |
number |
|
The lowest non-infinite bin point. |
|
upperBound |
number |
|
The highest non-infinite bin point. |
|
bins |
number |
Yes |
The number of bins bounded by Defaults to |
|
addNegInf |
boolean |
Yes |
Include a bin Defaults to |
|
addPosInf |
boolean |
Yes |
Include a bin Defaults to |
|
autoResize |
boolean |
Yes |
The histogram will be empty at the beginning and double its size on demand (resize only when incrementing counts). The unbounded bins are guaranteed to stay between lowerBound and upperBound and in all cases the bin size equals (upperBound - lowerBound)/bins. Defaults to |
StreamAggrMax module:qm.StreamAggr
This stream aggregator represents the maximum moving window buffer. It monitors the maximal value in the connected stream aggregator.
It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the maximal value of the records in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var timeser = {
name: 'TimeSeriesAggr',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius',
winsize: 86400000 // one day in miliseconds
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);
// add a max aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var max = {
name: 'MaxAggr',
type: 'winBufMax',
store: 'Heat',
inAggr: 'TimeSeriesAggr'
};
var maximal = base.store("Heat").addStreamAggr(max);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type for the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. |
StreamAggrMerger module:qm.StreamAggr
This stream aggregator represents the merger aggregator. It merges records from two or more stores into a new store
depending on the timestamp. No methods are implemented for this aggregator.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Cars",
fields: [
{ name: "NumberOfCars", type: "float" },
{ name: "Time", type: "datetime" }
]
},
{
name: "Temperature",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
},
{
name: "Merged",
fields: [
{ name: "NumberOfCars", type: "float" },
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new merger stream aggregator that mergers the records of the 'Cars' and 'Temperature' stores.
// The records are interpolated linearly and stored in the 'Merged' store.
var mer = {
name: 'MergerAggr',
type: 'merger',
outStore: 'Merged',
createStore: false,
timestamp: 'Time',
fields: [
{ source: 'Cars', inField: 'NumberOfCars', outField: 'NumberOfCars', interpolation: 'linear', timestamp: 'Time' },
{ source: 'Temperature', inField: 'Celsius', outField: 'Celsius', interpolation: 'linear', timestamp: 'Time' }
]
};
var merger = new qm.StreamAggr(base, mer);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
outStore |
string |
|
The name of the store where it saves the merged records. |
|
createStore |
boolean |
|
If the outStore must be created. |
|
timestamp |
string |
|
The store field of outStore, where the timestamp is saved. |
|
fields |
Array of Object |
|
An array of |
StreamAggrMin module:qm.StreamAggr
This stream aggregator represents the minimum moving window buffer. It monitors the minimal value in the connected stream aggregator.
It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the minimal value of the records in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var timeser = {
name: 'TimeSeriesAggr',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius',
winsize: 86400000 // 1 day in miliseconds
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);
// add a min aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var min = {
name: 'MinAggr',
type: 'winBufMin',
store: 'Heat',
inAggr: 'TimeSeriesAggr'
};
var minimal = base.store("Heat").addStreamAggr(min);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. |
StreamAggrMovingAverage module:qmStreamAggr
This stream aggregator represents the moving average window buffer. It calculates the moving average value of the connected stream aggregator values.
It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the average of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window should be 1 day.
var timeser = {
name: 'TimeSeriesAggr',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius',
winsize: 86400000
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);
// add a moving average aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var ma = {
name: 'movingAverageAggr',
type: 'ma',
store: 'Heat',
inAggr: 'TimeSeriesAggr'
};
var movingAverage = base.store("Heat").addStreamAggr(ma);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. |
StreamAggrMovingCorrelation module:qm.StreamAggr
This stream aggregator represents the moving covariance window buffer. It calculates the moving correlation of the three stream aggregators,
that it's connected to. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the correlation of the values in it's buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in it's buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "WaterConsumption", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day
var Celsius = {
name: 'CelsiusAggr',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius',
winsize: 86400000
}; base.store("Heat").addStreamAggr(Celsius);
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'WaterConsumption' field
// and the timestamp from the 'Time' field. The size of the window is 1 day
var water = {
name: 'WaterAggr',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'WaterConsumption',
winsize: 86400000
}; base.store("Heat").addStreamAggr(water);
// add a covariance aggregator, that is connected with the 'CelsiusAggr' and 'WaterAggr' aggregators
var covariance = {
name: 'covarianceAggr',
type: 'covariance',
store: 'Heat',
inAggrX: 'CelsiusAggr',
inAggrY: 'WaterAggr'
}; base.store("Heat").addStreamAggr(covariance);
// add the two variance aggregators, that take from the 'Celsius' and 'WaterConsumption' fields, respectively
var celVar = {
name: 'CelsiusVarAggr',
type: 'variance',
store: 'Heat',
inAggr: 'CelsiusAggr'
}; base.store("Heat").addStreamAggr(celVar);
var waterVar = {
name: 'waterVarAggr',
type: 'variance',
store: 'Heat',
inAggr: 'WaterAggr'
}; base.store("Heat").addStreamAggr(waterVar);
// add a correlation aggregator, that is connected to 'CovarianceAggr', 'CelsiusVarAggr' and 'WaterValAggr' aggregators
var corr = {
name: 'corrAggr',
type: 'correlation',
store: 'Heat',
inAggrCov: 'covarianceAggr',
inAggrVarX: 'CelsiusVarAggr',
inAggrVarY: 'waterVarAggr'
};
var correlation = base.store("Heat").addStreamAggr(corr);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggrCov |
string |
|
The name of the covariance stream aggregator. |
|
inAggrVarX |
string |
|
The name of the first variance stream aggregator. |
|
inAggrVarY |
string |
|
The name of the second variance stream aggregator. |
StreamAggrMovingCovariance module:qm.StreamAggr
This stream aggregator represents the moving covariance window buffer. It calculates the moving covariance of the two stream aggregators, that it's connected to.
It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the covariance of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "WaterConsumption", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var Celsius = {
name: 'CelsiusAggr',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius',
winsize: 86400000
}; base.store("Heat").addStreamAggr(Celsius);
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'WaterConsumption' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var water = {
name: 'WaterAggr',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'WaterConsumption',
winsize: 86400000
}; base.store("Heat").addStreamAggr(water);
// add a covariance aggregator, that is connected with the 'CelsiusAggr' and 'WaterAggr' stream aggregators
var covariance = {
name: 'covAggr',
type: 'covariance',
store: 'Heat',
inAggrX: 'CelsiusAggr',
inAggrY: 'WaterAggr'
};
var covarianceAggr = base.store("Heat").addStreamAggr(covariance);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggrX |
string |
|
The name of the first stream aggregator to which it connects and gets data. |
|
inAggrY |
string |
|
The name of the recond stream aggregator to which it connects and gets data. |
StreamAggrMovingVariance module:qm.StreamAggr
This stream aggregator represents the moving variance window buffer. It calculates the moving variance of the stream aggregator, that it's connected to.
It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the variance of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day
var timeser = {
name: 'TimeSeriesAggr',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius',
winsize: 86400000
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);
// add a variance aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var variance = {
name: 'varAggr',
type: 'variance',
store: 'Heat',
inAggr: 'TimeSeriesAggr'
};
var varianceAggr = base.store("Heat").addStreamAggr(variance);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. |
StreamAggrPageHinkley module:qm.StreamAggr
This stream aggregate enables detecting concept drift based on Page-Hinkley test
on a stream of numeric data. It is based on the book
Gamma, Knowledge Discovery from Data Streams, 2013, pp. 76
1. module:qm.StreamAggr#getInteger takes a string input (either 'drift'
or 'driftOffset'
)
and returns returns the value of the property or null
if it's unknown. 'drift'
is set to 1 if the drift
has been detected in this step, otherise to 0. 'drift
' is always set to 0 in the next step. 'driftOffset'
monitors the offset (number of onStep calls) since last concept drift was detected.
2. module:qm.StreamAggr#getParams returns a parameter object.
3. module:qm.StreamAggr#setParams used for changing Page-Hinkley test parameters
Example
// main library
let qm = require('qminer');
// create a base with a simple store
// the store records results of clustering
base = new qm.Base({
mode: "createClean",
schema: [{
name: "Store",
fields: [
{ name: "Value", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Store' store that takes the recorded cluster id
// and the timestamp from the 'Time' field. The size of the window is 2 hours.
let timeser = {
name: 'seriesTick1',
type: 'timeSeriesTick',
store: 'Store',
timestamp: 'Time',
value: 'Value'
};
let timeSeries1 = base.store("Store").addStreamAggr(timeser);
// add a histogram aggregator, that is connected with the 'TimeSeries1' aggregator
let aggrPHT = {
name: 'PageHinkley',
type: 'pagehinkley',
store: 'Store',
inAggr: 'seriesTick1',
minInstances: 1,
delta: 0.005,
lambda: 50,
alpha: 0.9999
};
store = base.store("Store");
pht = store.addStreamAggr(aggrPHT);
// creating start time
let time = new Date();
let changes = 0;
// simulating concept drift at element 1000 in a time series
for (let i = 0; i < 2000; i++) {
// add one second to the timestamp and create an ISO string
time.setSeconds(time.getSeconds() + 1);
let timeStr = time.toISOString();
// create value
let value = Math.random() * 1;
if (i > 1000) {
value = Math.random() * 2 + 1;
}
// adding values to the signal store
store.push({ Time: timeStr, Value: value });
// counting changes
if (pht.saveJson().drift == 1) {
changes++;
}
}
// checking if drift has been correctly detected
if (changes >= 1) { console.log("Last concept drift was detected " + pht.val.driftOffset + " samples ago."); }
else { console.log("No concept drift was detected!"); }
// clean up
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
Yes |
The given name of the stream aggregator (autogenerated by default). |
|
type |
string |
|
The type for the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store consistent with the records that it processes. |
|
inAggr |
string |
|
The name of the stream aggregate used for input (i.e. module:qm.StreamAggrTimeSeriesTick). |
|
minInstances |
number |
|
Minimal number of instances needed for initialization of the aggregator (when can first concept drift be initialized?). |
|
delta |
number |
|
The delta factor for the Page Hinkley test. |
|
lambda |
number |
|
The change detection threshold. |
|
alpha |
number |
|
The forgetting factor, used to weight the observed value and the mean. |
StreamAggrRecordBuffer module:qm.StreamAggr
This stream aggregator represents record buffer. It stores the values inside a moving window. It implements all the stream aggregate methods except module:qm.StreamAggr#getFloat and module:qm.StreamAggr#getTimestamp.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store. The size of the window is 3 records.
var aggr = {
name: 'Delay',
type: 'recordBuffer',
size: 3
};
base.store("Heat").addStreamAggr(aggr);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
size |
number |
|
The size of the window. |
StreamAggrRecordSwitch module:qm.StreamAggr
This stream aggregate enables switching control flow between stream aggregates based
on string keys. It is based on a hash table from keys (strings read from records) to
stream aggregates. When OnAdd of a record switch is called, the aggregate looks for
an appropriate target aggregate and triggers its onAdd. The aggregate
exposes the following methods.
1. module:qm.StreamAggr#getInteger takes a string input and returns returns 1 if the string is a known key and null
if it's unknown.
2. module:qm.StreamAggr#onAdd reads the appropriate field as a key and triggers the onAdd of a target aggregate if the key is known.
3. module:qm.StreamAggr#getParams returns a parameter object.
4. module:qm.StreamAggr#setParams used primarily for adding (using $add) new targets or seting (using $set) the internal hashmap with new targets.
Example
// main library
var qm = require('qminer');
// create a store
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: 'testStore',
fields: [
{ name: 'switchField', type: 'string' }
]
}]
});
// select store
var store = base.store('testStore');
// first JS aggregate
var outAggr1 = new qm.StreamAggr(base, new function () {
this.onAdd = function (rec) {
console.log('first');
}
});
// second JS aggregate
var outAggr2 = new qm.StreamAggr(base, new function () {
this.onAdd = function (rec) {
console.log('second');
}
});
// switcher aggregate: calls outAggr1 when rec.switchField == 'a' and outAggr2 when rec.switchField == 'b'
var switcher = store.addStreamAggr({
type: 'recordSwitchAggr',
store: 'testStore',
fieldName: 'switchField',
$set: [{ key: 'a', aggrName: outAggr1.name },
{ key: 'b', aggrName: outAggr2.name }],
throwMissing: false
});
store.push({ switchField: 'a' });
// outAggr1 prints `first`
store.push({ switchField: 'b' });
// outAggr2 prints `second`
store.push({ switchField: 'b' });
// outAggr2 prints `second`
store.push({ switchField: 'c' });
// nothing happens
store.push({ switchField: 'a' });
// outAggr1 prints `first`
// clean up
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
Yes |
The given name of the stream aggregator (autogenerated by default). |
|
type |
string |
|
The type for the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store consistent with the records that it processes. |
|
fieldName |
string |
|
The name of the field whose values are used for switching |
|
throwMissing |
boolean |
Yes |
If true, the aggregate will throw an exception when the record's key is not recognized (no appropirate target aggregate exists). Defaults to |
|
$set |
Array of Object |
Yes |
An array with objects like |
StreamAggrResampler module:qm.StreamAggr
This stream aggregator represents the resampler window buffer. It creates new values that are interpolated by using the values from an existing store. No methods are implemented for this aggregator.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
},
{
name: "interpolatedValues",
fields: [
{ name: "Value", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new resampler stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The interpolated values are stored in the 'interpolatedValues' store.
// The interpolation should be linear and the interval should be 2 seconds
var res = {
name: 'resamplerAggr',
type: 'resampler',
store: 'Heat',
outStore: 'interpolatedValues',
timestamp: 'Time',
fields: [{
name: 'Celsius',
interpolator: 'linear'
}],
createStore: false,
interval: 2000
};
var resampler = base.store("Heat").addStreamAggr(res);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
outStore |
string |
|
The store in which the samples are stored. |
|
timestamp |
string |
|
The store field from which it takes the timestamps. |
|
fields |
Array of Object |
|
The array off |
|
createStore |
boolean |
|
If true, |
|
interval |
number |
|
The interval size. The frequency on which it makes the interpolated values. |
StreamAggrSimpleLinearRegression module:qm.StreamAggr
This stream aggregator computes a simple linear regression given two stream aggregates
that support the module:qm.StreamAggr#getFloatVector method and represent variates (input) and covariates (output).
Optionally the aggregate computes quantile bands: for each quantile q
a parallel line to the fitted
line is found, so that q
fraction of (x,y)
datapoints fall below the line. For example, under Gaussian noise,
if Y = a + k X + N(0,sig)
then the 0.95 quantile band will equal a + 2 sig
. This means that 95% of (x,y)
pairs
lie below the line Y = a + 2 sig + k X
.
The results are returned as a JSON by calling module:qm.StreamAggr#saveJson and are of type module:qm~StreamAggrSimpleLinearRegressionResult.
Example
// import the qm module
var qm = require('qminer');
// create a base with X,Y measurements
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: 'Function',
fields: [
{ name: 'Time', type: 'datetime' },
{ name: 'X', type: 'float' },
{ name: 'Y', type: 'float' }
]
}]
});
var store = base.store('Function');
// 1 second buffer for X values
var winX = store.addStreamAggr({
type: 'timeSeriesWinBuf',
timestamp: 'Time',
value: 'X',
winsize: 1000
});
// 1 second buffer for Y values
var winY = store.addStreamAggr({
type: 'timeSeriesWinBuf',
timestamp: 'Time',
value: 'Y',
winsize: 1000
});
// the will find regression line, as well as two parallel quartile lines
var linReg = store.addStreamAggr({
type: 'simpleLinearRegression',
inAggrX: winX.name,
inAggrY: winY.name,
storeX: "Function",
storeY: "Function",
quantiles: [0.25, 0.75]
});
store.push({ Time: '2015-06-10T14:13:32.001', X: 0, Y: -2 });
store.push({ Time: '2015-06-10T14:13:32.002', X: 0, Y: -1 });
store.push({ Time: '2015-06-10T14:13:32.003', X: 0, Y: 1 });
store.push({ Time: '2015-06-10T14:13:32.004', X: 0, Y: 2 });
store.push({ Time: '2015-06-10T14:13:32.005', X: 1, Y: -1 });
store.push({ Time: '2015-06-10T14:13:32.006', X: 1, Y: -0 });
store.push({ Time: '2015-06-10T14:13:32.007', X: 1, Y: 2 });
store.push({ Time: '2015-06-10T14:13:32.008', X: 1, Y: 3 });
var res = linReg.saveJson();
res.bands[0]; // -1.5
res.bands[1]; // 1.5
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type for the stream aggregator. Important: It must be equal to |
|
storeX |
string |
|
The name of the store from which it takes the data for the first vector. |
|
storeY |
string |
|
The name of the store from which it takes the data for the second vector. |
|
inAggrX |
string |
|
The name of the first stream aggregator to which it connects and gets data. |
|
inAggrY |
string |
|
The name of the second stream aggregator to which it connects and gets data. |
|
quantiles |
Array of number |
|
An array of numbers between 0 and 1 for which the quantile bands will be computed. |
StreamAggrSimpleLinearRegressionResult Object
Simple linear regression result JSON returned by module:qm~StreamAggrSimpleLinearRegression.
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
intercept |
number |
|
Regressor intercept. |
|
slope |
number |
|
Regressor slope. |
|
quantiles |
Array of number |
Yes |
Quantiles for which bands will be computed. |
|
bands |
Array of number |
Yes |
Computed band intercepts. |
StreamAggrSlottedHistogram module:qm.StreamAggr
This stream aggregator represents an online slotted histogram. It can connect to a buffered aggregate (such as module:qm~StreamAggrTimeSeriesWindow)
or a time series (such as module:qm~StreamAggregateEMA).
It maps historical values into single period (e.g. into hours of the week).
The aggregate defines an ordered set of points p(0), ..., p(n)
that define n bins. Infinites at both ends are NOT allowed.
A new measurement is tested for inclusion in the left-closed right-opened intervals [p(i), p(i+1))
and the corresponding
bin counter is increased for the appropriate bin (or decreased if the point is outgoing from the buffer).
It implements the following methods:
1. module:qm.StreamAggr#getFloatLength returns the number of bins.
2. module:qm.StreamAggr#getFloatAt returns the count for a bin index.
3. module:qm.StreamAggr#getFloatVector returns the vector of counts, the length is equal to the number of bins.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 4 weeks
var timeser = {
name: 'TimeSeriesBuffer',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius',
winsize:<PHONE_NUMBER> // 4 weeks
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);
// add a slotted-histogram aggregator, that is connected with the 'TimeSeriesAggr' aggregator
// it will present accumulated histogram for the last 2 hours (window) of the week (period) for the last 4 weeks (see aggregate above)
var aggrJson = {
name: 'Histogram',
type: 'onlineSlottedHistogram',
store: 'Heat',
inAggr: 'TimeSeriesBuffer',
period: 604800000, // 1 week
window: 7200000, // 2h
bins: 5, // 5 possible clusters
granularity: 300000 // 5 min
};
var hist = base.store("Heat").addStreamAggr(aggrJson);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type for the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. |
|
period |
number |
|
Cycle length in miliseconds. |
|
window |
number |
|
Window length that is reported when aggregate is queried. |
|
bins |
number |
|
The number of bins - input data is expected to be withing interval |
|
granularity |
number |
|
Storage granularity in miliseconds. History is stored in slots with this length. Number of slots is equal to period/granularity. |
StreamAggrSparseVecSum module:qm.StreamAggr
This stream aggregator represents the sparse-vector-sum moving window buffer. It sums all the sparse-vector values, that are in the connected stream aggregator.
It implements the following methods:
1. module:qm.StreamAggr#getValueVector returns the sum of the values of the records in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
var qm = require('qminer');
var base = new qm.Base({
mode: 'createClean',
schema: [{
name: 'Docs',
fields: [
{ name: 'Time', type: 'datetime' },
{ name: 'Text', type: 'string' }
]
}]
});
var store = base.store('Docs');
var aggr = {
name: 'featureSpaceWindow',
type: 'timeSeriesWinBufFeatureSpace',
store: 'Docs',
timestamp: 'Time',
featureSpace: {
type: "categorical",
source: "Docs",
field: "Text"
},
winsize: 1000
};
var sa = store.addStreamAggr(aggr);
var aggr2 = {
name: 'sparseVectorSum',
type: 'winBufSpVecSum',
store: 'Docs',
inAggr: 'featureSpaceWindow'
};
var sa2 = store.addStreamAggr(aggr2);
store.push({ Time: '2015-06-10T14:13:32.0', Text: 'a' }); // 0
store.push({ Time: '2015-06-10T14:13:33.0', Text: 'b' }); // 1
store.push({ Time: '2015-06-10T14:14:34.0', Text: 'c' }); // 2
store.push({ Time: '2015-06-10T14:15:35.0', Text: 'd' }); // 3
store.push({ Time: '2015-06-10T14:15:36.0', Text: 'e' }); // 4
store.push({ Time: '2015-06-10T14:15:37.0', Text: 'f' }); // 5
var valVec2 = sa2.getValueVector(); // [0, 0, 0, 0, 1, 1] - only vectors 4 and 5 remain in window
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. |
StreamAggrSum module:qm.StreamAggr
This stream aggregator represents the sum moving window buffer. It sums all the values, that are in the connected stream aggregator.
It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the sum of the values of the records in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Income",
fields: [
{ name: "Amount", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Income' store, that takes the values from the 'Amount' field
// and the timestamp from the 'Time' field. The size of the window should 1 week.
var timeser = {
name: 'TimeSeriesAggr',
type: 'timeSeriesWinBuf',
store: 'Income',
timestamp: 'Time',
value: 'Amount',
winsize: 604800000 // 7 days in miliseconds
};
var timeSeries = base.store("Income").addStreamAggr(timeser);
// add a sum aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var sum = {
name: 'SumAggr',
type: 'winBufSum',
store: 'Heat',
inAggr: 'TimeSeriesAggr'
};
var sumAggr = base.store("Income").addStreamAggr(sum);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. |
StreamAggrTDigest module:qm.StreamAggr
This stream aggregator computes the quantile estimators using the TDigest algorithm. The quantile values are returned using module:qm.StreamAggr#getFloatVector.
Example
// import the qm module
var qm = require('qminer');
// create a base with the Time and Value fields
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Processor",
fields: [
{ name: "Value", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
var store = base.store('Processor');
// create a new time series stream aggregator for the 'Processor' store, that takes the value of the processor
// and the timestamp from the 'Time' field. The size of the window is 1 second.
var tick = {
name: 'TickAggr',
type: 'timeSeriesTick',
store: 'Processor',
timestamp: 'Time',
value: 'Value',
winsize: 1000 // one day in miliseconds
};
var timeSeries = store.addStreamAggr(tick);
// create the TDigest stream aggregator
var aggr = {
name: 'TDigest',
type: 'tdigest',
store: 'Processor',
inAggr: 'TickAggr',
quantiles: [0.90, 0.95, 0.99, 0.999],
minCount: 5
};
// add the stream aggregator to the 'Processor' store
var td = store.addStreamAggr(aggr);
store.push({ Time: '2015-12-01T14:20:32.0', Value: 0.9948628368 });
store.push({ Time: '2015-12-01T14:20:33.0', Value: 0.1077458826 });
store.push({ Time: '2015-12-01T14:20:34.0', Value: 0.9855685823 });
store.push({ Time: '2015-12-01T14:20:35.0', Value: 0.7796449082 });
// with this record the aggregator will initialize becuase it is the 5th record
store.push({ Time: '2015-12-01T14:20:36.0', Value: 0.0844943286 });
store.push({ Time: '2015-12-01T14:20:37.0', Value: 0.187490856 });
store.push({ Time: '2015-12-01T14:20:38.0', Value: 0.0779815107 });
store.push({ Time: '2015-12-01T14:20:39.0', Value: 0.8945312691 });
store.push({ Time: '2015-12-01T14:20:40.0', Value: 0.5574567409 });
// get the quantile estimations
var result = td.getFloatVector();
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type for the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. |
|
quantiles |
Array of number |
|
An array of numbers between 0 and 1 for which the quantile bands will be computed. |
|
minCount |
Number |
|
The minimal number of values given before it start to compute the quantiles. |
StreamAggrThreshold module:qmStreamAggr
This stream aggregator represents a threshold indicator. It outputs 1 if the current value in the data streams is
above the threshold and 0 otherwise. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the exponentional average of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series tick stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window should be 1 hour.
var timeser = {
name: 'TimeSeriesTickAggr',
type: 'timeSeriesTick',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius'
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);
// a threshold aggregator, that is connected wit hthe 'TimeSeriesAggr' aggregator.
// It should output 1 when the temperature is over 3 degrees Celsius and 0 otherwise
var thresholdAggr = {
name: 'thresholdAggr1',
type: 'threshold',
store: 'Heat',
inAggr: 'TimeSeriesTickAggr',
threshold: 3
};
var expoMovingAverage = base.store("Heat").addStreamAggr(thresholdAggr);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
inAggr |
string |
|
The name of the stream aggregator to which it connects and gets data. It cannot be connect to the module:qm~StreamAggrTimeSeriesWindow. |
|
threshold |
string |
|
The threshold mentioned above. |
StreamAggrTimeSeriesTick module:qm.StreamAggr
This stream aggregator represents the time series tick window buffer. It exposes the data to other stream aggregators
(similar to module:qm~StreamAggrTimeSeriesWindow). It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the last value added in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Students",
fields: [
{ name: "Id", type: "float" },
{ name: "TimeOfGraduation", type: "datetime" }
]
}]
});
// create a new time series tick stream aggregator for the 'Students' store, that takes the values from the 'Id' field
// and the timestamp from the 'TimeOfGraduation' field.
var tick = {
name: 'TimeSeriesTickAggr',
type: 'timeSeriesTick',
store: 'Students',
timestamp: 'TimeOfGraduation',
value: 'Id',
};
var timeSeriesTick = base.store("Students").addStreamAggr(tick);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name for the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which it takes the data. |
|
value |
string |
|
The name of the store field, from which it takes the values. |
|
timestamp |
string |
|
The name of the store field, from which it takes the timestamp. |
StreamAggrTimeSeriesWindow module:qm.StreamAggr
This stream aggregator represents the time series window buffer. It stores the values inside a moving window. It implements all the stream aggregate methods except module:qm.StreamAggr#getFloat and module:qm.StreamAggr#getTimestamp.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 2 seconds (2000ms).
var aggr = {
name: 'TimeSeriesAggr',
type: 'timeSeriesWinBuf',
store: 'Heat',
timestamp: 'Time',
value: 'Celsius',
winsize: 2000
};
base.store("Heat").addStreamAggr(aggr);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which to takes the data. |
|
timestamp |
string |
|
The field of the store, where it takes the timestamp. |
|
value |
string |
|
The field of the store, where it takes the values. |
|
winsize |
number |
|
The size of the window, in milliseconds. |
|
delay |
number |
|
Delay in milliseconds. |
StreamAggrTimeSeriesWindowVector module:qm.StreamAggr
This stream aggregator represents the values read from a time series window buffer. It implements module:qm.StreamAggr#getFloatVector, module:qm.StreamAggr#getFloatAt and module:qm.StreamAggr#getFloatLength.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
mode: "createClean",
schema: [{
name: "Heat",
fields: [
{ name: "Celsius", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
var store = base.store("Heat");
var tick = store.addStreamAggr({
type: 'timeSeriesTick',
timestamp: 'Time',
value: 'Celsius'
});
var winbufvec = store.addStreamAggr({
type: 'timeSeriesWinBufVector',
inAggr: tick.name,
winsize: 2000
});
store.push({ Time: '2015-06-10T14:13:32.0', Celsius: 1 });
winbufvec.getFloatVector().print(); // prints 1
store.push({ Time: '2015-06-10T14:33:30.0', Celsius: 2 });
winbufvec.getFloatVector().print(); // prints 2
store.push({ Time: '2015-06-10T14:33:31.0', Celsius: 3 });
winbufvec.getFloatVector().print(); // prints 2,3
store.push({ Time: '2015-06-10T14:33:32.0', Celsius: 4 });
winbufvec.getFloatVector().print(); // prints 2,3,4
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type of the stream aggregator. Important: It must be equal to |
|
store |
string |
|
The name of the store from which to takes the data. |
|
inAggr |
string |
|
The name of the window buffer aggregate that represents the input. |
StreamAggrVecDiff module:qm.StreamAggr
This stream aggregator represents difference between two vectors (e.g. online histograms).
It implements the following methods:
1. module:qm.StreamAggr#getFloatLength returns the number of bins.
2. module:qm.StreamAggr#getFloatAt returns the count for a bin index.
3. module:qm.StreamAggr#getFloatVector returns the vector of counts, the length is equal to the number of bins.
Example
// import the qm module
var qm = require('qminer');
// create a base with a simple store
// the store records results of clustering
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "Rpm",
fields: [
{ name: "ClusterId", type: "float" },
{ name: "Time", type: "datetime" }
]
}]
});
var store = base.store('Rpm');
// create a new time series stream aggregator for the 'Rpm' store that takes the recorded cluster id
// and the timestamp from the 'Time' field. The size of the window is 4 weeks.
var timeser1 = {
name: 'TimeSeries1',
type: 'timeSeriesWinBuf',
store: 'Rpm',
timestamp: 'Time',
value: 'ClusterId',
winsize: 7200000 // 2 hours
};
var timeSeries1 = base.store("Rpm").addStreamAggr(timeser1);
// add a histogram aggregator, that is connected with the 'TimeSeries1' aggregator
var aggrJson1 = {
name: 'Histogram1',
type: 'onlineHistogram',
store: 'Rpm',
inAggr: 'TimeSeries1',
lowerBound: 0,
upperBound: 5,
bins: 5,
addNegInf: false,
addPosInf: false
};
var hist1 = base.store("Rpm").addStreamAggr(aggrJson1);
// create a new time series stream aggregator for the 'Rpm' store that takes the recorded cluster id
// and the timestamp from the 'Time' field.
var timeser2 = {
name: 'TimeSeries2',
type: 'timeSeriesWinBuf',
store: 'Rpm',
timestamp: 'Time',
value: 'ClusterId',
winsize: 21600000 // 6 hours
};
var timeSeries2 = base.store("Rpm").addStreamAggr(timeser2);
// add a histogram aggregator, that is connected with the 'TimeSeries1' aggregator
var aggrJson2 = {
name: 'Histogram2',
type: 'onlineHistogram',
store: 'Rpm',
inAggr: 'TimeSeries2',
lowerBound: 0,
upperBound: 5,
bins: 5,
addNegInf: false,
addPosInf: false
};
var hist2 = base.store("Rpm").addStreamAggr(aggrJson2);
// add diff aggregator that subtracts Histogram1 with 2h window from Histogram2 with 6h window
var aggrJson3 = {
name: 'DiffAggr',
type: 'onlineVecDiff',
storeX: 'Rpm',
storeY: 'Rpm',
inAggrX: 'Histogram2',
inAggrY: 'Histogram1'
}
var diff = store.addStreamAggr(aggrJson3);
base.close();
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
The type for the stream aggregator. Important: It must be equal to |
|
storeX |
string |
|
The name of the store from which it takes the data for the first vector. |
|
storeY |
string |
|
The name of the store from which it takes the data for the second vector. |
|
inAggrX |
string |
|
The name of the first stream aggregator to which it connects and gets data. |
|
inAggrY |
string |
|
The name of the second stream aggregator to which it connects and gets data. |
StreamAggrWindowQuantiles module:qm.StreamAggr
This stream aggregate computes approximate quantiles on a sliding time window using the SW-GK algorithm proposed in: http://dl.acm.org/citation.cfm?id=2954329
The quantile values are returned using module:qm.StreamAggr#getFloatVector.
Example
var assert = require('assert');
var qm = require('qminer');
// variables
var batchSize = 1000;
var nbatches = 10;
var dt = 10;
var windowMSec = batchSize*dt;
var quantileEps = 0.01;
var countEps = 0.0001;
var maxRelErr = quantileEps + 2*countEps;
var targetQuants = (function () {
var quants = [];
for (var prob = 0; prob <= 1; prob += 0.001) {
quants.push(prob);
}
return quants;
})();
// create a base with a simple store
// the store records results of throwing two independent fair dices
var base = new qm.Base({
mode: "createClean",
schema: [
{
name: "hugeDie",
fields: [
{ name: "value", type: "float" },
{ name: "time", type: "datetime" }
]
}]
});
var store = base.store('hugeDie');
// create a new time series stream aggregator for the 'Dice' store, that takes the expected values of throwing a dice
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var windowAggr = store.addStreamAggr({
name: 'TimeSeries1',
type: 'timeSeriesWinBuf',
store: store,
timestamp: 'time',
value: 'value',
winsize: windowMSec
});
var gk = store.addStreamAggr({
type: 'windowQuantiles',
inAggr: windowAggr,
quantileEps: quantileEps,
countEps: countEps,
quantiles: targetQuants
})
var vals = [];
for (var i = 0; i < batchSize; i++) {
vals.push(i);
}
for (var batchN = 0; batchN < nbatches; batchN++) {
// shuffle the array
for (var i = 0; i < batchSize; i++) {
var swapN = Math.floor(Math.random()*batchSize);
var temp = vals[i];
vals[i] = vals[swapN];
vals[swapN] = temp;
}
for (var i = 0; i < batchSize; i++) {
var time = (batchN*batchSize + i)*dt;
store.push({ time: time, value: vals[i] })
}
var result = gk.getFloatVector();
for (var i = 0; i < targetQuants.length; i++) {
var pval = targetQuants[i];
var quant_hat = result[i];
assert(Math.floor((pval - maxRelErr)*batchSize) <= quant_hat);
assert(Math.ceil((pval + maxRelErr)*batchSize) >= quant_hat);
}
}
Properties
| Name | Type | Optional | Description |
|---|---|---|---|
|
name |
string |
|
The given name of the stream aggregator. |
|
type |
string |
|
Must use type 'windowQuantiles'. |
|
inAggr |
string |
|
The name of the stream aggregate which defines the time window. |
|
quantiles |
Array of number |
|
An array of p-values for which the algorithm will return quantiles. |
|
quantileEps |
number |
|
Maximal relative error of the quantile estimation procedure. |
|
countEps |
number |
|
Maximal relative error of the count procedure. If the number of items in the window is N, then the algorithms error is bound by N*(quantileEps + 2*countEps + O(countEps^2)). However in practice, the error should be less. |