Called when a socket connections is closed to cleanup all additional resources associated with it
api.checkWebSocketRequest(data, callback)
Called before allowing the WebSocket connection to be authorized
api.handleWebSocketConnect(socket)
Wrap external WeSocket connection into the Express routing, respond on backend command
api.createWebSocketRequest(socket, url, reply)
Wrap WebSocket into HTTP request to be proceses by the Express routes
api.closeWebSocketRequest(socket)
Close all pending requests, this is called on socket close or disconnect
api.prepareRequest(req)
Prepare request options that the API routes will merge with, can be used by pre process hooks, initialize required properties for subsequent use
api.prepareOptions(req)
Parse or re-parse special headers about app version, language and timezone, it is called early to parse headers first and then right after the query parameters are available, query values have higher priority than headers.
api.startMetrics(req, res, next)
This is supposed to be called at the beginning of request processing to start metrics and install the handler which will be called at the end to finalize the metrics and call the cleanup handlers
api.handleMetrics(req, elapsed)
Finish metrics collection about the current rquest
api.handleCleanup(req)
Call registered cleanup hooks and clear the request explicitly
api.checkQuery(req, res, next)
Parse incoming query parameters
api.checkBody(req, res, next)
Parse multipart forms for uploaded files
api.checkRouting(req, name)
Check if the current request must be re-routed to another endpoint
api.checkRedirectPlaceholders(req, pathname)
Replace redirect placeholders
api.checkRedirect(req, options)
Check a request for possible redirection condition based on the configuration, this can be SSL checks or
defined redirect rules. This is used by API servers and proxy servers for early redirections. It returns null
if no redirects or errors happend, otherwise an object with status that is expected by the api.sendStatus
method.
The options is expected to contain the following cached request properties:
api.checkRedirectRules(req, options, name)
Redirect rules, supports regexpobj and regexpmap parameters
api.checkAccountType(account, type)
Return true if the current user belong to the specified type, account type may contain more than one type.
NOTE: after this call the type
property is converted into an array
api.checkRateLimits(req, options, callback)
Perform rate limiting by specified property, if not given no limiting is done.
The following options properties can be used:
type - predefined: ip, path, login, id
, determines by which property to perform rate limiting, when using account properties
the rate limiter should be called after the request signature has been parsed. Any other value is treated as
custom type and used as is.
This property is required.
The predefined types:
-api-rlimits-/api/path-rate=2
ip - to use the specified IP address for type=ip
max - max capacity to be used by default
rate - fill rate to be used by default
interval - interval in ms within which the rate is measured, default 1000 ms
message - more descriptive text to be used in the error message for the type, if not specified a generic error message is used
total - apply this factor to the rate, it is used in case of multiple servers behind a loadbalancer, so for total 3 servers in the cluster the factor will be 3, i.e. each individual server checks for a third of the total request rate
The metrics are kept in the LRU cache in the master process.
When used for accounts, it is possible to override rate limits for each account in the bk_auth
table by setting rlimit_max
and rlimit_rate
columns. To enable account rate limits the global defaults still must be set with the config paramaters -api-rlimit-login-max
and -api-rlimit-login-rate
for example.
Example:
api.checkLimit(req, { type: "ip", rate: 100, interval: 60000 }, function(err) {
if (err) return api.sendReply(err);
...
});
api.registerRateLimits(name, rate, max, interval)
Register access rate limit for a given name, all other rate limit properties will be applied as described in the checkRateLimits
api.registerControlParams(options)
Add special control parameters that will be recognized in the query and placed in the req.options
for every request.
Control params start with underscore and will be converted into the configured type according to the spec.
The options
is an object in the format that is used by lib.toParams
, no default type is allowed, even for string
it needs to be defined as { type: "string" }.
No existing control parameters will be overridden, also care must be taken when defining new control parameters so they do not conflict with the existing ones.
These are default common parameters that can be used by any module:
_count, _page, _tm, _sort, _select, _ext, _start, _token, _session, _format, _total, _encoding, _ops
These are the reserved names that cannot be used for parameters, they are defined by the engine for every request:
path, apath, ip, host, mtime, cleanup, secure, noscan, appName, appVersion, appPlatform, appLocale, appTimezone, apiVersion
NOTE: noscan
is set to 1 in every request to prevent accidental full scans, this means it cannot be enabled via the API but any module
can do it in the code if needed.
Example:
mod.configureMiddleware = function(options, callback) {
api.registerControlParams({ notify: { type: "bool" }, level: { type: "int", min: 1, max: 10 } });
callback();
}
Then if a request arrives for example as `_notify=true&_level=5`, it will be parsed and placed in the `req.options`:
mod.configureWeb = function(options, callback) {
api.app.all("/send", function(req, res) {
if (req.options.notify) { ... }
if (req.options.level > 5) { ... }
});
callback()
}
api.getOptions(req, controls)
Convert query options into internal options, such options are prepended with the underscore to distinguish control parameters from the query parameters.
For security purposes this is the only place that translates special control query parameters into the options properties,
all the supported options are defined in the api.controls
and can be used by the apps freely but with caution. See registerControlParams
.
if controls
is an object it will be used to define additional control parameters or override existing ones for this request only. Same rules as for
registerControlParams
apply.
api.getOptions(req, { count: { min: 5, max: 100 } })
api.getQuery(req, params, controls)
Parse query parameters according to the params
, optionally process control parameters if controls
si specified, this call combines
lib.toParams()
with api.getOptions
. Returns a query object or an error message, on success all controls will be set in the req.options
var query = api.getQuery(req, { q: { required: 1 } }, { _count: { type: "int", min: 10, max: 25 } });
api.getTokenSecret(req)
Return a secret to be used for enrypting tokens, it uses the account property if configured or the global API token
to be used to encrypt data and pass it to the clients. -api-query-token-secret
can be configured and if a column in the bk_auth
with such name exists it is used as a secret, otherwise the value of this property is used as a secret.
api.getResultPage(req, options, rows, info)
Return an object to be returned to the client as a page of result data with possibly next token if present in the info. This result object can be used for pagination responses.
api.getPublicColumns(table, options)
Columns that are allowed to be visible, used in select to limit number of columns to be returned by a query
options may be used to define the following properties:
skip - a regexp with names to be excluded as well
allow - a list of properties which can be checked along with the pub
property for a column to be considered public
disallow - a list of properties which if set will prevent a column to be returned, it is checked before the 'allow' rule
api.getPublicColumns("bk_account", { allow: ["admins"], skip: /device_id|0$/ });
api.checkResultColumns(table, data, options)
Process records and keep only public properties as defined in the table columns. This method is supposed to be used in the post process callbacks after all records have been processes and are ready to be returned to the client, the last step would be to cleanup all non public columns if necessary.
table
can be a single table name or a list of table names which combined public columns need to be kept in the rows. List of request tables
is kept in the req.options.cleanup
which by default is a table name of the API endpoint, for example for /account/get it will contain bk_account, for
/connection/get - bk_connection.
In the options
account object can be present to detect account own records which will not be cleaned and all properties will be returned, by default id
property is used to detect current account but can be specified by the options.account_key
property.
By default primary keys are not kept and must be marked with pub
property in the table definition to be returned.
If any column is marked with secure
property this means never return that column in the result even for the owner of the record
If any column is marked with admin
or admins
property and the current account is an admin this property will be returned as well. The options.admin
can be used to make it an artificial admin.
The options.cleanup_strict
will enforce that all columns not present in the table definition will be skipped as well, by default all
new columns or columns created on the fly are returned to the client.
The options.cleanup_rules
can be an object with property names and the values -1, 0, or 1 which correspond to:
-1 - never return, 0 return only to the owner, 1 always return.
The options.pool
property must match the actual rowset to be applied properly, in case the records have been retrieved for the different
database pool.
api.clearQuery(table, query, options)
Clear request query properties specified in the table definition or in custom schema.
The table
argument can be a table name or an object with properties as columns.
If options.filter
is not specified the query
will only keep existing columns for the given table.
If options.filter
is a list then the query
will delete properties for columns that contain any specified
property from the filter list. This is used for the bk_auth
table to remove properties that supposed to be updated by admins only.
The filter will keep non-existent columns in the query
. To remove such columns when using the filter specify options.force
.
If a name in the filter is prefixed with ! then the logic is reversed, keep all except this property
If options.keep
is a regexp it will be used to keep matched properties by name in the query
regardless of any condition.
If options.clear
is a regexp it will be used to remove matched properties by name in the query
.
Example:
api.clearQuery("bk_account", req.query)
api.clearQuery("bk_auth", req.query, { filter: ["admin"] })
api.clearQuery("bk_auth", req.query, { filter: ["!pub"] })
api.clearQuery("bk_account", req.query, { filter: ["admin","secure"] })
api.clearQuery("bk_account", req.query, { filter: ["admin","!secure"], keep: /^__/ })
api.clearQuery({ name: {}, id: { admin: 1 } }, req.query, { filter: ["admin"] })
api.findHook(type, method, path)
Find registered hooks for given type and path
api.addHook(type, method, path, callback)
Register a hook callback for the type and method and request url, if already exists does nothing.
api.registerAccessCheck(method, path, callback)
Register a handler to check access for any given endpoint, it works the same way as the global accessCheck function and is called before validating the signature or session cookies. No account information is available at this point yet.
Example:
api.registerAccessCheck('', 'account', function(req, cb) { cb({ status: 500, message: "access disabled"}) }))
api.registerAccessCheck('POST', '/account/add', function(req, cb) {
if (!req.query.invitecode) return cb({ status: 400, message: "invitation code is required" });
cb();
});
api.registerPreProcess(method, path, callback)
Similar to registerAccessCheck
but this callback will be called after the signature or session is verified but before
the API route method is called. The req.account
object will always exist at this point but may not contain the user in case of an error.
The purpose of this hook is to perform some preparations or check permissions of a valid user to resources or in case of error perform any other action
like redirection or returning something explaining what to do in case of failure. The callback for this call is different then in checkAccess
hooks.
cb
callbackExample:
api.registerPreProcess('GET', '/account/get', function(req, status, cb) {
if (status.status != 200) status = { status: 302, url: '/error.html' };
cb(status)
});
Example with admin access only:
api.registerPreProcess('POST', '/data/', function(req, status, cb) {
if (req.account.type != "admin") return cb({ status: 401, message: "access denied, admins only" });
cb();
});
api.registerPostProcess(method, path, callback)
Register a callback to be called after successfull API action, status 200 only. To trigger this callback the primary response handler must return
results using api.sendJSON
or api.sendFormatted
methods.
The purpose is to perform some additional actions after the standard API completed or to customize the result
Note: the req.account,req.options,req.query
objects may become empty if any callback decided to do some async action, they are explicitly emptied at the end of the request,
in such cases make a copy of the needed objects if it will needed
Example, just update the rows, it will be sent at the end of processing all post hooks
api.registerPostProcess('', '/data/', function(req, res, rows) {
rows.forEach(function(row) { ...});
});
Example, add data to the rows and return result after it
api.registerPostProcess('', '/data/', function(req, res, row) {
db.get("bk_account", { id: row.id }, function(err, rec) {
row.name = rec.name;
res.json(row);
});
return true;
});
api.registerCleanup(method, path, callback)
Register a cleanup callback that will be called at the end of a request, all registered cleanup callbacks will be called in the order of registration. At this time the result has been sent so connection is not valid anymore but the request and account objects are still available.
Example, do custom logging of all requests
api.registerCleanup('', '/data/', function(req, next) {
db.add("log", req.query, next);
});
api.registerSendStatus(method, path, callback)
Register a status callback that will be called when api.sendReply
or api.sendStatus
is called,
all registered callbacks will be called in the order of registration. At this time the result has NOT been sent yet so connection is
still valid and can be channged.
Example, do custom logging of all requests
api.registerSendStatus('', '/data/', function(req, data) {
logger.info("response", req.path, data);
});
api.registerSignature(method, path, callback)
The purpose of this hook is to manage custom signatures.
Example:
api.registerSignature('', '/', function(req, account, sig, cb) {
if (sig) {
if (invalid) sig = null;
} else {
sig = api.createSignature(.....);
}
cb(sig)
});
api.registerSecret(login, callback)
Register a secret generation method.
api.registerOAuthStrategy(strategy, options, callback)
Given passport strategy setup OAuth callbacks and handle the login process by creating a mapping account for each
OAUTH authenticated account.
The callback if specified will be called as function(req, options, info) with req.user
signifies the successful
login and hold the account properties. If given it is up to the callback to perform any redirects reqauired for
completion of the login process.
The following options properties are accepted:
query._profile
, query._accessToken
, query._refreshToken
will be set for the authenticated profile object from the provider.api.sendJSON(req, err, rows)
Send result back with possibly executing post-process callback, this is used by all API handlers to allow custom post processing in the apps. If err is not null the error message is returned immediately.
api.sendFormatted(req, err, data, options)
Send result back formatting according to the options properties:
api.sendStatus(res, options)
Return reply to the client using the options object, it contains the following properties:
i18n Note:
The API server attaches fake i18n functions req.__
and res.__
which are used automatically for the message
property
before sending the response.
With real i18n module these can/will be replaced performing actual translation without
using i18n.__
method for messages explicitely in the application code for sendStatus
or sendReply
methods.
api.sendReply(res, status, text)
Send formatted JSON reply to an API client, if status is an instance of Error then error message with status 500 is sent back.
If the status is an object it is sent as is.
All Error objects will return a generic error message without exposing the real error message, it will log all error exceptions in the logger subject to log throttling configuration.
api.sendFile(req, file, redirect)
Send file back to the client, res is Express response object
api.subscribe(req, options)
Subscribe for events, this is used by /acount/subscribe
API call but can be used in generic way, if no options
provided by default it will listen on req.account.id, the default API implementation for Connection, Counter, Messages publish
events using account id as a key.
api.unsubscribe(req, options)
Disconnect from subscription service. This forces disconnect even for persistent connections like websockets.
api.publish(key, event, options)
Publish an event for an account, key is account id or other key used for subscription, event is a string or an object
api.sendEvent(req, key, data, next)
Process a message received from subscription server or other event notifier, it is used by api.subscribe
method for delivery events to the clients
api.configureDefaultAPI()
Default API calls that endpoints for health check, authentication and public image access
api.prepareAccountSecret(account, options, callback)
If specified in the options, prepare credentials to be stored in the db, if no error occured return null, otherwise an error object
api.fetchAccount(query, options, callback)
Given a profile data from some other system, check if there is an account or create a new account for the given
profile, returns account record in the callback. req.query contains profile fields converted to bk_auth names
so the whole req.query can be saved as it is. query.login
must exist.
This method is supposed to be called after the user is authenticated and verified, it does not check secrets but only existence of a user by login. On success existing or new account is returned by the callback.
If new account is created, the generated secret will be returned and must be saved by the client for subsequent API calls unless cookie session is established.
if query.icon' is set with the url of the profile image, it will be downloaded and saved as account icon type
0.
options.width`
if specified will be used to resize the image.
In case when a new account was created the account record will have a property _added
set to true.
This is to explicitely distinguish existing and new accounts.
api.addAccount(query, options, callback)
Register new account, return new account record in the callback, when options.admin is true then allow to set all properties otherwise admin properties will not be updated
api.updateAccount(query, options, callback)
Update existing account, if options.admin
is true then allow to update all properties
api.setAccountSecret(query, options, callback)
Change account secret
api.handleSignature(req, res, next)
Perform authorization of the incoming request for access and permissions
api.handleSessionSignature(req, callback)
Setup session cookies or access token for automatic authentication without signing, req must be complete with all required properties after successful authorization.
api.newSignature(req)
Returns a new signature object with all required properties filled form the request object
api.getSignature(req)
Parse incoming request for signature and return all pieces wrapped in an object, this object will be used by verifySignature
function.
If the signature successfully recognized it is saved in the request as req.signature
,
it always returns a signature object, a new one or existing
api.verifySignature(req, sig, account, callback)
Returns true if the signature sig
matches given account secret. account
object must be a bk_auth
record.
api.createSignature(login, secret, method, host, uri, options)
Create secure signature for an HTTP request. Returns an object with HTTP headers to be sent in the response.
The options may contains the following:
api.checkRequestSignature(req, callback)
Verify request signature from the request object, uses properties: .host, .method, .url or .originalUrl, .headers
api.checkAccountSignature(req, sig, account, callback)
Check account record against a incoming request signature
api.setCurrentAccount(req, account)
Assign or clear the current account record for the given request, if account is null the account is cleared
api.checkAccess(req, callback)
Perform URL based access checks, this is called before the signature verification, very early in the request processing step.
Checks access permissions, calls the callback with the following argument:
api.checkAuthorization(req, status, callback)
Perform authorization checks after the account been checked for valid signature, this is called even if the signature verification failed, in case of a custom authentication middlware this must be called at the end and use the status object returned in the callback to return an error or proceed with the request. In any case the result of this function is final.
If a user has valid login by default access to all API endpoints is granted, to restrict access to specific APIs use any combinations of
api-allow
or api-deny
config parameters.
api.checkLogin(req, callback)
Check login and secret from a client
api.fileUrl(file, options)
Returns absolute file url if it is configured with any prefix or S3 bucket, otherwise returns empty string
api.getFile(req, file, options)
Send a file to the client
api.putFile(req, name, options, callback)
Upload file and store in the filesystem or S3, try to find the file in multipart form, in the body or query by the given name
Output file name is built according to the following options properties:
On return the options may have the following properties set:
api.storeFile(tmpfile, outfile, options, callback)
Place the uploaded tmpfile to the destination pointed by outfile
api.delFile(file, options, callback)
Delete file by name from the local filesystem or S3 drive if filesS3 is defined in api or options objects
api.detectFile(file, callback)
Returns detected mime type and ext for a file
api.scaleIcon(infile, options, callback)
Scale image using ImageMagick, return err if failed
The callback takes 4 arguments: function(err, data, info)
where data
will contain a new image data and info
is an object with the info about the new or unmodified image: ext, width, height.
For animated GIF files it will not convert but return it as is unless it is specified in the options to do so, frame and no_animation
must be used if no animation is required, see bk_wand for more info. This means that the resulting image type must be consulted
from the info.format because it may be not the desired format.
api.iconPath(id, options)
Full path to the icon, perform necessary hashing and sharding, id can be a number or any string.
options.type
may contain special placeholders:
api.iconUrl(row, options)
Returns constructed icon url from the icon record
api.checkIcon(req, row)
Verify icon permissions and format for the result, used in setProcessRow for the bk_icon table
api.sendIcon(req, id, options)
Send an icon to the client, only handles files
api.putIcon(req, name, id, options, callback)
Store an icon for account, the options are the same as for the iconPath
method
iconPath
along with the options to build the icon absolute pathapi.saveIcon(file, id, options, callback)
Save the icon data to the destination, if api.imagesS3 or options.imagesS3 specified then plave the image on the S3 drive. Store in the proper location according to the types for given id, this function is used after downloading new image or when moving images from other places. On success the callback will be called with the second argument set to the output file name where the image has been saved. Valid properties in the options:
resizeImage
functionapi.delIcon(id, options, callback)
Delete an icon for account, .type defines icon prefix
api.isIconUrl(url)
Return true if the given file or url point ot an image
api.isIcon(buf)
Returns detected image type if the given buffer contains an image, it checks the header only
api.handleProxyRequest(req, res, callback)
Process incoming proxy request, can be overriden for custom logic with frontend proxy server. If any response is sent or an error returned in the calback then the request will be aborted and will not be forwarded to the web processes
api.createProxyServer()
Create a proxy server to handle incoming requests and distribute them to the workers
api.createProxyWorker()
Create/fork a worker to handle API requests, register a new port for load balancing between web workers
api.getProxyPort()
Return a target port for proxy requests, rotates between all web workers
api.getProxyTarget(req)
Return a target for proxy requests
api.runProxyRequest(req, res, ssl)
Process a proxy request, perform all filtering or redirects
api.createStatisticsTables()
Collected metrics per worker process, basic columns are defined in the table to be collected like
api and db request rates(.rmean), response times(.hmean) and total number of requests(0).
Counters ending with `0are snapshots, i.e. they must be summed up for any given interval.
All other counters are averages. Only subset of all available API endpoints is defined here
for example purposes, for SQL databases all columns must be defined but for NoSQL this is not required,
depending on the database that is used for collection the metrics must be added to the table. All
url_columns
are the API requests, not the DB calls made by the app, the length of URL path to be stored is defined in the API module
by the
api-url-metrics-` config parameter.
Database tables
bk_collect: {
id: { primary: 1 },
mtime: { type: "now", primary: 1 },
app: {},
ip: {},
type: {},
instance: {},
worker: {},
pid: { type: "int" },
latency: { type: "int" },
cpus: { type: "int" },
mem: { type: "bigint" },
rss_hmean: { type: "real" },
heap_hmean: { type: "real" },
avg_hmean: { type: "real" },
free_hmean: { type: "real" },
util_hmean: { type: "real" },
api_req_rmean: { type: "real" },
api_req_hmean: { type: "real" },
api_req_0: { type: "real" },
api_err_0: { type: "real" },
api_bad_0: { type: "real" },
api_400_0: { type: "real" },
api_401_0: { type: "real" },
api_403_0: { type: "real" },
api_417_0: { type: "real" },
api_429_0: { type: "real" },
api_que_rmean: { type: "real" },
api_que_hmean: { type: "real" },
pool_req_rmean: { type: "real" },
pool_req_hmean: { type: "real" },
pool_req_0: { type: "real" },
pool_err_0: { type: "real" },
pool_que_rmean: { type: "real" },
pool_que_hmean: { type: "real" },
ctime: { type: "mtime" },
}
api.initStatistics()
Setup statistics collections
api.getStatistics(options)
Updates metrics with the current values and returns an object ready to be saved in the database, i.e. flattened ito one object where all property names of the complex objects are combined into one name separated by comma.
api.sendStatistics()
Send collected statistics to the collection server, backend-host
must be configured and possibly backend-login
and backend-secret
in case
the system API is secured, the user can be any valid user registered in the bk_auth table.
api.saveStatistics(obj, options, callback)
Save collected statistics in the bk_collect table, this can be called via API or directly by the backend, this wrapper is supposed to be overrriden by the application with additional logic how the statistics is saved. Columns in the bk_collect table must be defined for any metrics to be saved, use api.describeTable with additional columns from the api.metrics object in additional to the default ones.
Example, add pool cache stats to the table
api.describeTable({ bk_collect: { pool_cache_rmean: { type: "real" },
pool_cache_hmean: { type: "real" } });
api.calcStatistics(query, options, callback)
Calculate statistics for a period of time, query and options must confirm to the db.select conventions.
app
This is a skeleton module to be extended by the specific application logic. It provides all callbacks and hooks that are called by the core backend modules during different phases, like initialization, shutting down, etc...
It should be used for custom functions and methods to be defined, the app
module is always available.
All app modules in the modules/ subdirectory use the same prototype, i.e. all hooks are available for custom app modules as well.
app.configure(options, callback)
Called after all config files are loaded and command line args are parsed, home directory is set but before the db is initialized, the primary purpose of this early call is to setup environment before connecting to the database. This is called regardless of the server to be started and intended to initialize the common environment before the database and other subsystems are initialized.
app.configureModule(options, callback)
Called after the core.init has been initialized successfully, this can be redefined in the applications to add additional init steps that all processes require to have. All database pools and other confugration is ready at this point. This hook is called regardless of what kind of server is about to start, it is always called before starting a server or shell.
app.configureMiddleware(options, callback)
This handler is called during the Express server initialization just after the security middleware.
NOTE: api.app
refers to the Express instance.
app.configureWeb(options, callback)
This handler is called after the Express server has been setup and all default API endpoints initialized but the Web server is not ready for incoming requests yet. This handler can setup additional API endpoints, add/modify table descriptions.
NOTE: api.app
refers to the Express instance
app.shutdownWeb(options, callback)
Perform shutdown sequence when a Web process is about to exit
NOTE: api.app
refers to the Express instance
app.configureMaster(options, callback)
This handler is called during the master server startup, this is the process that monitors the worker jobs and performs jobs scheduling
app.configureServer(options, callback)
This handler is called during the Web server startup, this is the master process that creates Web workers for handling Web requests, this process interacts with the Web workers via IPC sockets between processes and relaunches them if any Web worker dies.
app.configureWorker(options, callback)
This handler is called on job worker instance startup after the tables are intialized and it is ready to process the job
app.shutdownWorker(options, callback)
Perform last minute operations inside a worker process before exit, the callback must be called eventually which will exit the process. This method can be overrided to implement custom worker shutdown procedure in order to finish pending tasks like network calls.
app.configureMonitor(options, callback)
This callback is called when the monitor process is ready, there is no any other code is supposed to run inside the monitor, but in case it is needed, this is the hook to be used.
app.configureShell(options, callback)
This callback is called by the shell process to setup additional command or to execute a command which is not supported by the standard shell. Setting options.done to 1 will stop the shell, this is a signal that command has already been processed.
aws.configure(options, callback)
Initialization of metadata
aws.configureServer(options, callback)
Execute on Web server startup
aws.configureMaster(options, callback)
Execute on master server startup
aws.readCredentials(profile, callback)
Read key and secret from the AWS SDK credentials file, if no profile is given in the config or command line only the default peofile will be loaded.
aws.getInstanceMeta(path, callback)
Retrieve instance meta data
aws.getInstanceCredentials(callback)
Retrieve instance credentials using EC2 instance profile and setup for AWS access
aws.getInstanceInfo(callback)
Retrieve instance launch index from the meta data if running on AWS instance
aws.parseXMLResponse(err, params, options, callback)
Parse AWS response and try to extract error code and message, convert XML into an object.
aws.querySign(region, service, host, method, path, body, headers, credentials, options)
Build version 4 signature headers
aws.queryPrepare(action, version, obj, options)
Return a request object ready to be sent to AWS, properly formatted
aws.querySigner()
It is called in the context of a http request
aws.queryAWS(region, service, proto, host, path, obj, options, callback)
Make AWS request, return parsed response as Javascript object or null in case of error
aws.queryEndpoint(service, version, action, obj, options, callback)
AWS generic query interface
aws.copyCredentials(obj, options)
Copy all credentials properties from the options into the obj
aws.querySQS(action, obj, options, callback)
AWS SQS API request
aws.queryIAM(action, obj, options, callback)
AWS AIM API request
aws.querySTS(action, obj, options, callback)
AWS STS API request
aws.querySES(action, obj, options, callback)
AWS SES API request
aws.queryCFN(action, obj, options, callback)
AWS CFN API request
aws.queryCW(action, obj, options, callback)
AWS CloudWatch API request
aws.queryElastiCache(action, obj, options, callback)
AWS Elastic Cache API request
aws.queryAS(action, obj, options, callback)
AWS Autoscaling API request
aws.queryRoute53(method, path, data, options, callback)
Make a request to Route53 service
aws.queryRekognition(action, obj, options, callback)
Make a request to the Rekognition service
aws.querySSM(action, obj, options, callback)
AWS SSM API request
aws.queryCWL(action, obj, options, callback)
AWS CloudWatch Log API request
aws.queryACM(action, obj, options, callback)
AWS ACM API request
aws.getTagValue(obj, key)
Returns a tag value by key, default key is Name
aws.stsAssumeRole(options, callback)
Assume a role and return new credentials that can be used in other API calls
aws.sqsReceiveMessage(url, options, callback)
Receive message(s) from the SQS queue, the callback will receive a list with messages if no error. The following options can be specified:
aws.sqsSendMessage(url, body, options, callback)
Send a message to the SQS queue. The options can specify the following:
aws.sesSendEmail(to, subject, body, options, callback)
Send an email via SES The following options supported:
aws.sesSendRawEmail(body, options, callback)
Send raw email The following options accepted:
aws.cwPutMetricAlarm(options, callback)
Creates or updates an alarm and associates it with the specified Amazon CloudWatch metric. The options specify the following:
CPUUtilization
AWS/EC2
>=
.Average
60
15
90
aws.cwPutMetricData(namespace, data, options, callback)
Publishes metric data points to Amazon CloudWatch. The argumernts specify the following:
AWS
value: Number,
dimension1: name1,
..
},
}, ...
{ metricName: { value: [min, max, sum, sample],
dimension1: ...
},
}, ...The options can specify the following:
aws.cwListMetrics(options, callback)
Return metrics for the given query, the options can be specified:
aws.route53Change(names, options, callback)
Create or update a host in the Route53 database.
names
is a host name to be set with the current IP address or a list with objects in the format
[ { name: "..", value: "<IP_ADDRESS>", type: "A", ttl: 300 } ...]The options
may contain the following:
aws.detectLabels(name, options, callback)
Detect image featires using AWS Rekognition service, the name
can be a Buffer, a local file or an url to the S3 bucket. In the latter case
the url can be just apath to the file inside a bucket if options.bucket
is specified, otherwise it must be a public S3 url with the bucket name
to be the first part of the host name. For CDN/CloudFront cases use the option.bucket
option.
aws._queryDDB(target, service, action, obj, options, callback)
DynamoDB requests
aws.toDynamoDB(value, level)
Convert a Javascript object into DynamoDB object
aws.fromDynamoDB(value, level)
Convert a DynamoDB object into Javascript object
aws.queryExpression(params, obj, options, join)
Build a condition expression for the given object, all properties in the obj are used
aws.ddbListTables(options, callback)
Return list of tables in .TableNames property of the result
Example:
{ TableNames: [ name, ...] }
aws.ddbDescribeTable(name, options, callback)
Return table definition and parameters in the result structure with property of the given table name
Example:
{ name: { AttributeDefinitions: [], KeySchema: [] ...} }
aws.ddbCreateTable(name, attrs, options, callback)
Create a table
Example:
ddbCreateTable('users', { id: 'S', mtime: 'N', name: 'S'},
{ keys: ["id", "name"],
local: { mtime: { mtime: "HASH" } },
global: { name: { name: 'HASH', ProvisionedThroughput: { ReadCapacityUnits: 50 } } },
projections: { mtime: ['gender','age'],
name: ['name','gender'] },
stream: "NEW_IMAGE",
readCapacity: 10,
writeCapacity: 10 });
aws.ddbUpdateTable(options, callback)
Update tables provisioned throughput settings, options is used instead of table name so this call can be used directly in the cron jobs to adjust provisionined throughput on demand. Options must provide the following properties:
Example
aws.ddbUpdateTable({ name: "users", add: { name_id: { name: "S", id: 'N', readCapacity: 20, writeCapacity: 20, projections: ["mtime","email"] } })
aws.ddbUpdateTable({ name: "users", add: { name: { name: "S", readCapacity: 20, writeCapacity: 20, projections: ["mtime","email"] } })
aws.ddbUpdateTable({ name: "users", del: "name" })
aws.ddbUpdateTable({ name: "users", update: { name: { readCapacity: 10, writeCapacity: 10 } })
Example of crontab job in etc/crontab:
[
{ "type": "server", "cron": "0 0 1 * * *", "job": { "aws.ddbUpdateTable": { "name": "bk_account", "readCapacity": 1000, "writeCapacity": 1000 } } },
{ "type": "server", "cron": "0 0 6 * * *", "job": { "aws.ddbUpdateTable": { "name": "bk_account", "readCapacity": 2000, "writeCapacity": 2000 } } }
]
aws.ddbUpdateTimeToLive(options, callback)
Update TTL attribute. The options properties:
aws.ddbDescribeTimeToLive(name, options, callback)
Returns status of Time to live attribute for a table
aws.ddbDeleteTable(name, options, callback)
Remove a table from the database.
By default the callback will ba callled only after the table is deleted, specifying options.nowait
will return immediately
aws.ddbWaitForTable(name, item, options, callback)
Call the callback after specified period of time or when table status become different from the given waiting status. if options.waitTimeout is not specified calls the callback immediately. options.waitStatus is checked if given and keeps waiting while the status is equal to it. options.waitDelay can be specified how often to request new status, default is 250ms.
aws.ddbPutItem(name, item, options, callback)
Put or add an item
any other exact value to be checked against which corresponds to { Exists: true, Value: value }
for ExpressionAttributeValues parameters
for ExpressionAttributeNames parameter
Example:
ddbPutItem("users", { id: 1, name: "john", mtime: 11233434 }, { expected: { name: null } })
aws.ddbUpdateItem(name, keys, item, options, callback)
Update an item
options.ops
the same way as for queries.*
or new
means ALL_NEW, old
means ALL_OLD, `updated` means UPDATED_NEW, `old_updated` means UPDATED_OLD
Example:
ddbUpdateItem("users", { id: 1, name: "john" }, { gender: 'male', icons: '1.png' }, { action: { icons: 'ADD' }, expected: { id: 1 }, ReturnValues: "ALL_NEW" })
ddbUpdateItem("users", { id: 1, name: "john" }, { gender: 'male', icons: '1.png' }, { action: { icons: 'ADD' }, expected: { id: null } })
ddbUpdateItem("users", { id: 1, name: "john" }, { gender: 'male', icons: '1.png', num: 1 }, { action: { num: 'ADD', icons: 'ADD' }, expected: { id: null, num: 0 }, ops: { num: "gt" } })
aws.ddbDeleteItem(name, keys, options, callback)
Delete an item from a table
Example:
ddbDeleteItem("users", { id: 1, name: "john" }, {})
aws.ddbBatchWriteItem(items, options, callback)
Update items from the list at the same time
Example:
{ table: [ { put: { id: 1, name: "tt" } }, { del: { id: 2 } }] }
aws.ddbBatchGetItem(items, options, callback)
Retrieve all items for given list of keys
Example:
{ users: { keys: [{ id: 1, name: "john" },{ id: .., name: .. }], select: ['name','id'], consistent: true }, ... }
aws.ddbGetItem(name, keys, options, callback)
Retrieve one item by primary key
keys - an object with primary key attributes name and value.
select - list of columns to return, otherwise all columns will be returned
options may contain any native property allowed in the request or special properties:
consistent - set consistency level for the request
names - an object with a map to be used for attribute names in condition and update expressions, to be used for ExpressionAttributeNames parameter Example:
ddbGetItem("users", { id: 1, name: "john" }, { select: 'id,name' })
aws.ddbQueryTable(name, condition, options, callback)
Query on a table, return all matching items
put into QueryFilter instead of KeyConditions. If keys are absent, all properties in the condition are treated as primary keys.
Example:
aws.ddbQueryTable("users", { id: 1, name: "john" }, { select: 'id,name', ops: { name: 'gt' } })
aws.ddbQueryTable("users", { id: 1, name: "john", status: "ok" }, { keys: ["id"], select: 'id,name', ops: { name: 'gt' } })
aws.ddbQueryTable("users", { id: 1 }, { expr: "status=:s", values: { s: "status" } })
aws.ddbScanTable(name, condition, options, callback)
Scan a table for all matching items
Example:
aws.ddbScanTable("users", { id: 1, name: 'a' }, { ops: { name: 'gt' }})
aws.ddbScanTable("users", "id=:id AND name=:name", { values: { id: 1, name: 'a' } });
aws.ddbTransactWriteItems(items, options, callback)
Update items from the list at the same time in one transaction, on any failure everything is rolled back
Example:
{ op: "put": table: "table-name", obj: { id: 1, name: "tt" } },
{ op: "del": table: "table-name", obj: { id: 2 } },
{ op: "update": table: "table-name", obj: { id: 1, name: "test" }, options: { expected: { status: "ok" } } },
{ op: "check": table: "table-name", obj: { id: 1 }, options: { expected: { status: "ok" } } }
aws.queryEC2(action, obj, options, callback)
AWS EC2 API request
aws.queryELB(action, obj, options, callback)
AWS ELB API request
aws.ec2RunInstances(options, callback)
Run AWS instances, supports all native EC2 parameters with first capital letter but also accepts simple parameters in the options:
Name:
, any occurences of %i will be replaced with the instance indexThe callback will take 3 arguments: callback(err, rc, info) where info will contain properties that can be used by `aws.ec2PrepareInstance
aws.ec2AfterRunInstances(options, callback)
Perform the final tasks after an instance has been launched like wait for status, assign Elastic IP or tags..
aws.ec2WaitForInstance(instanceId, status, options, callback)
Check an instance status and keep waiting until it is equal what we expect or timeout occured.
The status
can be one of: pending | running | shutting-down | terminated | stopping | stopped
The options can specify the following:
aws.ec2DescribeSecurityGroups(options, callback)
Describe security groups, optionally if options.filter
regexp is provided then limit the result to the matched groups only,
return list of groups to the callback
aws.ec2DescribeInstances(options, callback)
Describe instances according to the query filters, returns a list with instances, the following properties can be used:
aws.ec2ParseInstances(rc)
Pare DescribeInstances results and return a list of instances as a plain list
aws.ec2CreateTags(id, name, options, callback)
Create tags for a resource. The name is a string, an array or an object with tags. The options also may contain tags property which is an object with tag key and value
Example
aws.ec2CreateTags("i-1234","My Instance", { tags: { tag2 : "val2", tag3: "val3" } } )
aws.ec2CreateTags("i-1234", { tag2: "val2", tag3: "val3" })
aws.ec2CreateTags("i-1234", [ "tag2", "val2", "tag3", "val3" ])
aws.ec2AssociateAddress(instanceId, elasticIp, options, callback)
Associate an Elastic IP with an instance. Default behaviour is to reassociate if the EIP is taken. The options can specify the following:
aws.ec2CreateImage(options, callback)
Create an EBS image from the instance given or the current instance running
aws.ec2DeregisterImage(ami_id, options, callback)
Deregister an AMI by id. If options.snapshots
is set, then delete all snapshots for this image as well
aws.elbRegisterInstances(name, instance, options, callback)
Register an instance(s) with ELB, instance can be one id or a list of ids
aws.elbDeregisterInstances(name, instance, options, callback)
Deregister an instance(s) from ELB, instance can be one id or a list of ids
aws.elb2RegisterInstances(target, instance, options, callback)
Register an instance(s) with ELB, instance can be one id or a list of ids
aws.ssmSendCommand(cmds, instances, options, callback)
Run a shell command
aws.ssmWaitForCommand(cmdId, instanceId, options, callback)
Return a command details
aws.signS3(method, bucket, path, body, options)
Sign S3 AWS request, returns url to be send to S3 server, options will have all updated headers to be sent as well
aws.queryS3(bucket, path, options, callback)
S3 requests Options may contain the following properties:
aws.s3List(path, options, callback)
Retrieve a list of files from S3 bucket, only files inside the path will be returned
aws.s3GetFile(path, options, callback)
Retrieve a file from S3 bucket, root of the path is a bucket, path can have a protocol prepended like s3://, it will be ignored
aws.s3PutFile(path, file, options, callback)
Upload a file to S3 bucket, file
can be a Buffer or a file name
aws.s3ParseUrl(link)
Parse an S3 URL and return an object with bucket and path
aws.s3Proxy(res, bucket, path, options, callback)
Proxy a file from S3 bucket into the existing HTTP response res
aws.querySNS(action, obj, options, callback)
AWS SNS API request
aws.snsCreatePlatformEndpoint(token, options, callback)
Creates an endpoint for a device and mobile app on one of the supported push notification services, such as GCM and APNS.
The following properties can be specified in the options:
-sns-app-arn
will be used.All capitalized properties in the options will be pased as is. The callback will be called with an error if any and the endpoint ARN
aws.snsSetEndpointAttributes(arn, options, callback)
Sets the attributes for an endpoint for a device on one of the supported push notification services, such as GCM and APNS.
The following properties can be specified in the options:
aws.snsDeleteEndpoint(arn, options, callback)
Deletes the endpoint from Amazon SNS.
aws.snsPublish(arn, msg, options, callback)
Sends a message to all of a topic's subscribed endpoints or to a mobile endpoint. If msg is an object, then it will be pushed as JSON. The options may take the following properties:
aws.snsCreateTopic(name, options, callback)
Creates a topic to which notifications can be published. The callback returns topic ARN on success.
aws.snsSetTopicAttributes(arn, options, callback)
Updates the topic attributes. The following options can be used:
aws.snsDeleteTopic(arn, options, callback)
Deletes the topic from Amazon SNS.
aws.snsSubscribe(arn, endpoint, options, callback)
Creates a topic to which notifications can be published. The callback returns topic ARN on success, if the topic requires confirmation the arn returned will be null and a token will be sent to the endpoint for confirmation.
aws.snsConfirmSubscription(arn, token, options, callback)
Verifies an endpoint owner's intent to receive messages by validating the token sent to the endpoint by an earlier Subscribe action. If the token is valid, the action creates a new subscription and returns its Amazon Resource Name (ARN) in the callback.
aws.snsSetSubscriptionAttributes(arn, options, callback)
Updates the subscription attributes. The following options can be used:
aws.snsUnsubscribe(arn, options, callback)
Creates a topic to which notifications can be published. The callback returns topic ARN on success.
aws.snsListTopics(options, callback)
Creates a topic to which notifications can be published. The callback returns topic ARN on success.
core.init(options, callback)
Main initialization, must be called prior to perform any actions.
If options are given they may contain the following properties:
core.run(options, callback)
Run any backend function after environment has been initialized, this is to be used in shell scripts, core.init will parse all command line arguments, the simplest case to run from /data directory and it will use default environment or pass -home dir so the script will reuse same config and paths as the server context can be specified for the callback, if no then it run in the core context
core.exit(code, msg)
Exit the process with possible message to be displayed and status code
core.setHome(home)
Switch to new home directory, exit if we cannot, this is important for relative paths to work if used, no need to do this in worker because we already switched to home directory in the master and all child processes inherit current directory Important note: If run with combined server or as a daemon then this MUST be an absolute path, otherwise calling it in the spawned web master will fail due to the fact that we already set the home and relative path will not work after that.
core.parseConfig(data, pass)
Parse config lines for the file or other place
core.parseArgs(argv, pass)
Parse command line arguments
core.processArgs(ctx, argv, pass)
Config parameters defined in a module as a list of parameter names prefixed with module name, a parameters can be a string which defines text parameter or an object with the properties: name, type, value, decimals, min, max, separator type can be bool, number, list, json
core.describeArgs(module, args)
Add custom config parameters to be understood and processed by the config parser
Example:
core.describeArgs("api", [ { name: "num", type: "int", descr: "int param" }, { name: "list", array: 1, descr: "list of words" } ]);
core.describeArgs([ { name: "api-list", array: 1, descr: "list of words" } ]);
core.loadConfig(file, callback)
Parse the config file, configFile can point to a file or can be skipped and the default file will be loaded
core.loadDnsConfig(options, callback)
Load configuration from the DNS TXT records
core.loadLocales(options, callback)
Load configured locales
core.runMethods(name, options, callback)
Run a method for every module, a method must conform to the following signature: function(options, callback)
and
call the callback when finished. The callback second argument will be the options, so it is possible to pass anything
in the options back to the caller.
The following properties can be specified in the options:
lib.objDescr
core.runMethod(module, name, options, callback)
Run a method for the given module
core.addModule()
Adds reference to the objects in the core for further access, specify module name, module reference pairs.
This is used the the core itcore to register all internal modules and makes it available in the shell and in the core.modules
object.
Also this is used when cresting modular backend application by separating the logic into different modules, by registering such modules with the core it makes the module a first class citizen in the backendjs core and exposes all the callbacks and methods.
For example, the module below will register API routes and some methods
var bkjs = require("backendjs");
var mymod = {}
exports.module = mymod;
core.addModule("mymod", mymod);
mymod.configureWeb = function(options, callback) {
bkjs.api.app.all("/mymod", function(req, res) {
res.json({});
});
}
In the main app.js just load it and the rest will be done automatically, i.e. routes will be created ...
var mymod = require("./mymod.js");
Running the shell will make the object mymod
available
./app.sh -shell
> mymod
{}
core.loadModules(dir, options, callback)
Dynamically load services from the specified directory.
The modules are loaded using require
as a normal nodejs module but in addition if the module exports
init
method it is called immediately with options passed as an argument. This is a synchronous function so it is supposed to be
called on startup, not dynamically during a request processing. Only top level .js files are loaded, not subdirectories. core.addModule
is called
automatically.
Modules can be sorted by a priority, if .priority property is defined in the module it will be used to sort the modules, the higher priority the
closer to the top the module will be. The position of a module in the core.modules
will define the order runMethods
will call.
Caution must be taken for module naming, it is possible to override any default bkjs module which will result in unexpected behaviour
The following options properties can be specified:
Example, to load all modules from the local relative directory
core.loadModules("modules")
core.httpGet(uri, params, callback)
Make a HTTP request, see httpGet
module for more details.
core.sendRequest(options, callback)
Make a HTTP request using httpGet
with ability to sign requests.
The POST request is made, if data is an object, it is converted into string.
Returns params as in httpGet
with .json property assigned with an object from parsed JSON response.
When used with API endpoints, the backend-host
parameter must be set in the config or command line to the base URL of the backend,
like http://localhost:8000, this is when uri
is relative URL. Absolute URLs do not need this parameter.
Special parameters for options:
core.processName()
Return unique process name based on the cluster status, worker or master and the role. This is can be reused by other workers within the role thus making it usable for repeating environments or storage solutions.
core.showHelp(options)
Print help about command line arguments and exit
core.sendmail(options, callback)
Send email
core.killBackend(name, signal, callback)
Kill all backend processes that match name and not the current process
core.shutdown()
Shutdown the machine now
core.setTimeout(name, callback, timeout)
Set or reset a timer
core.createServer(options, callback)
Create a Web server with options and request handler, returns a server object.
Options can have the following properties:
core.createRepl(options)
Create REPL interface with all modules available
core.startRepl(port, bind, options)
Start command prompt on TCP socket, context can be an object with properties assigned with additional object to be accessible in the shell
core.watchTmp(dir, options, callback)
Watch temp files and remove files that are older than given number of seconds since now, remove only files that match pattern if given Options properties:
core.watchLogs(options, callback)
Watch log files for errors and report via email or POST url, see config parameters starting with logwatcher-
about how this works
core.cookieGet(domain, callback)
Return cookies that match given domain
core.cookieSave(cookiejar, setcookies, hostname, callback)
Save new cookies arrived in the request, merge with existing cookies from the jar which is a list of cookies before the request
db
The Database API, a thin abstraction layer on top of SQLite, PostgreSQL, DynamoDB and Cassandra. The idea is not to introduce new abstraction layer on top of all databases but to make the API usable for common use cases. On the source code level access to all databases will be possible using this API but any specific usage like SQL queries syntax or data types available only for some databases will not be unified or automatically converted but passed to the database directly. Only conversion between JavaScript types and database types is unified to some degree meaning JavaScript data type will be converted into the corresponding data type supported by any particular database and vice versa.
Basic operations are supported for all database and modelled after NoSQL usage, this means no SQL joins are supported by the API, only single table access. SQL joins can be passed as SQL statements directly to the database using low level db.query API call, all high level operations like add/put/del perform SQL generation for single table on the fly.
The common convention is to pass options object with flags that are common for all drivers along with specific, this options object can be modified with new properties but all driver should try not to modify or delete existing properties, so the same options object can be reused in subsequent operations.
All queries and update operations ignore properties that starts with underscore.
Before the DB functions can be used the core.init
MUST be called first, the typical usage:
var backend = require("backendjs"), core = backend.core, db = backend.db;
core.init(function(err) {
db.add(...
...
});
All database methods can use default db pool or any other available db pool by using pool: name
in the options. If not specified,
then default db pool is used, sqlite is default if no -db-pool config parameter specified in the command line or the config file.
Even if the specified pool does not exist, the default pool will be returned, this allows to pre-confgure the app with different pools
in the code and enable or disable any particular pool at any time.
Example, use PostgreSQL db pool to get a record and update the current pool
db.get("bk_account", { id: "123" }, { pool: "pgsql" }, function(err, row) {
if (row) db.update("bk_account", row);
});
Most database pools can be configured with options min
and max
for number of connections to be maintained, so no overload will happen and keep warm connection for
faster responses. Even for DynamoDB which uses HTTPS this can be configured without hitting provisioned limits which will return an error but
put extra requests into the waiting queue and execute once some requests finished.
Example:
db-pgsql-pool-max = 100
db-dynamodb-pool-max = 100
Also, to spread functionality between different databases it is possible to assign some tables to the specific pools using db-X-pool-tables
parameters
thus redirecting the requests to one or another databases depending on the table, this for example can be useful when using fast but expensive
database like DynamoDB for real-time requests and slower SQL database running on some slow instance for rare requests, reports or statistics processing.
Example, run the backend with default PostgreSQL database but keep all config parametrs in the DynamoDB table for availability:
db-pool = pgsql
db-dynamodb-pool = default
db-dynamodb-pool-tables = bk_config
The following databases are supported with the basic db API methods: Sqlite, PostgreSQL, MySQL, DynamoDB, MongoDB, Elasticsearch, Cassandra, Redis, LMDB, LevelDB, Riak, CouchDB
All these drivers fully support all methods and operations, some natively, some with emulation in the user space except Redis driver cannot perform sorting due to using Hash items for records, sorting can be done in memory but with pagination it is not possible so this part must be mentioned specifically. But the rest of the operations on top of Redis are fully supported which makes it a good candidate to use for in-memory tables like sessions with the same database API, later moving to other database will not require any application code changes.
Multiple connections of the same type can be opened, just add -n suffix to all database config parameters where n is a number, referer to such pools in the code as poolN
.
Example:
db-sqlite1-pool = billing
db-sqlite1-pool-max = 10
db-sqlite1-pool-options-path = /data/db
db-sqlite1-pool-options-journal_mode = OFF
in the Javascript:
db.select("bills", { status: "ok" }, { pool: "sqlite1" }, lib.log)
Database tables
// Configuration store, same parameters as in the commandline or config file, can be placed in separate config groups
// to be used by different backends or workers
bk_config: {
name: { primary: 1 }, // name of the parameter
type: { primary: 1 }, // config type or tag
value: {}, // the value
status: { value: "ok" }, // ok - availaible
ttl: { type: "int" }, // refresh interval in seconds since last read
mtime: { type: "now" }
},
// General purpose properties, can be used to store arbitrary values
bk_property: {
name: { primary: 1 },
value: {},
count: { type: "counter" }, // general purpose counter value
ttl: { type: "int" }, // time to live, seconds since last update
mtime: { type: "now" }
},
createPool:(opts)
None database driver
db.init(options, callback)
Initialize all database pools. the options may containt the following properties:
db.initConfig(options, callback)
Load configuration from the config database, must be configured with db-config-type
pointing to the database pool where bk_config table contains
configuration parameters.
The priority of the paramaters is fixed and goes from the most broad to the most specific, most specific always wins, this allows for very flexible configuration policies defined by the app or place where instances running and separated by the run mode.
The following list of properties will be queried from the config database and the sorting order is very important, the last values
will override values received for the earlier properties, for example, if two properties defined in the bk_config
table with the
types myapp
and prod-myapp
, then the last value will be used only.
The major elements are the following:
-run-mode: production
myapp
-worker
-nat
The modifiers which are appended to each major attributes:
-192.168
us-east-1
-us-east-1a
The top level list is the following:
All modifiers are appended for every item in the list like runMode-network
, runMode-appName-tag-region
,...
The options takes the following properties:
NOTE: The config parameters from the DB always take precedence even over config.local.
On return, the callback second argument will receive all parameters received form the database as a list: -name value ...
db.getConfig(options, callback)
Return all config records for the given instance, the result will be sorted most relevant at the top
db.refreshConfig(options, callback)
Refresh parameters which are configured with a TTL
db.createTables(options, callback)
Create or upgrade the tables for the given pool
db.dropTables(tables, options, callback)
Delete all specified tables from the specific pool or all active pools if options.pool
is empty, tables
can be a list of tables or an
object with table definitions
db.query(req, options, callback)
Execute query using native database driver, the query is passed directly to the driver.
db-pool-tables
.unique
propertycount
, skip all post processing and convertionobj
property, it will include all generated and updated columnsreturning
property does, it only
returns the query record with new columns from memoryExample with SQL driver
db.query({ text: "SELECT a.id,c.type FROM bk_account a,bk_connection c WHERE a.id=c.id and a.id=?", values: ['123'] }, { pool: 'pgsql' }, function(err, rows, info) {
});
db.get(table, query, options, callback)
Retrieve one record from the database by primary key, returns found record or null if not found Options can use the following special properties:
db.select
NOTE: On return the info.cached
will be set to 1 if the record was retrieved from cache or was put in the cache.
Example
db.get("bk_account", { id: '12345' }, function(err, row) {
if (row) console.log(row.name);
});
db.select(table, query, options, callback)
Select objects from the database that match supplied conditions.
>, gt, <, lt, =, !=, <>, >=, ge, <=, le, in, between, regexp, iregexp, begins_with, like%, ilike%
db.get
method)select
property might be used to get all required properties. For Elasticsearch if sort is null then scrolling scan will be used,
if no timeout
or scroll
are given the default is 1m.get
db-cache-keys-table-name
parameterOn return, the callback can check third argument which is an object with some predefined properties along with driver specific state returned by the query:
Example: get by primary key, refer above for default table definitions
db.select("bk_message", { id: '123' }, { count: 2 }, function(err, rows) {
});
Example: get all icons with type greater or equal to 2
db.select("bk_icon", { id: '123', type: '2' }, { select: 'id,type', ops: { type: 'ge' } }, function(err, rows) {
});
Example: get unread msgs sorted by time, recent first
db.select("bk_message", { id: '123', status: 'N:' }, { sort: "status", desc: 1, ops: { status: "begins_with" } }, function(err, rows) {
});
Example: allow all accounts icons to be visible
db.select("bk_account", {}, function(err, rows) {
rows.forEach(function(row) {
row.acl_allow = 'auth';
db.update("bk_icon", row);
});
});
Example: scan accounts with custom filter, not by primary key: all females
db.select("bk_account", { gender: 'f' }, function(err, rows) {
});
Example: select connections using primary key and other filter columns: all likes for the last day
db.select("bk_connection", { id: '123', type: 'like', mtime: Date.now()-86400000 }, { ops: { type: "begins_with", mtime: "gt" } }, function(err, rows) {
});
db.search(table, query, options, callback)
Perform full text search on the given table, the database implementation may ignore table name completely in case of global text index.
Query in general is a text string with the format that is supported by the underlying driver,
the db module DOES NOT PARSE the query at all if the driver supports full text search, otherwise it behaves like select
.
Options make take the same properties as in the select
method.
A special query property q
may be used for generic search in all fields.
Without full text search support in the driver this may return nothing or an error.
Example
db.search("bk_account", { type: "admin", q: "john*" }, { pool: "elasticsearch" }, lib.log);
db.search("bk_account", "john*", { pool: "elasticsearch" }, lib.log);
db.add(table, obj, options, callback)
Insert new object into the database
On return the obj
will contain all new columns generated before adding the record
Note: SQL, DynamoDB, MongoDB, Redis drivers are fully atomic but other drivers may be subject to race conditions
Example
db.add("bk_account", { id: '123', name: 'test', gender: 'm' }, function(err, rows, info) {
});
db.incr(table, obj, options, callback)
Counter operation, increase or decrease column values, similar to update but all specified columns except primary key will be incremented, use negative value to decrease the value.
If no options.updateOps
object specified or no 'incr' operations are provided then
all columns with type 'counter' will be used for the action incr
Note: The record must exist already for SQL databases, for DynamoDB and Cassandra a new record will be created
if does not exist yet. To disable upsert pass noupsert
in the options.
Example
db.incr("bk_counter", { id: '123', like0: 1, invite0: 1 }, function(err, rows, info) {
});
db.put(table, obj, options, callback)
Add/update an object in the database, if object already exists it will be replaced with all new properties from the obj
db.add
methodExample
db.put("bk_account", { id: '123', name: 'test', gender: 'm' }, function(err, rows, info) {
});
db.update(table, obj, options, callback)
Update existing object in the database.
db.add
method with the following additional properties:obj
,
a property named $or or $and will be treated as a sub-expression if it is an object.Note: not all database drivers support atomic update with conditions, all drivers for SQL, DynamoDB, MongoDB, Redis fully atomic, but other drivers perform get before put and so subject to race conditions
Example
db.update("bk_account", { id: '123', gender: 'm' }, function(err, rows, info) {
console.log('updated:', info.affected_rows);
});
db.update("bk_account", { id: '123', gender: 'm', prefix: 'Mr' }, { pool: pgsql' }, function(err, rows, info) {
console.log('updated:', info.affected_rows);
});
db.update("bk_account", { id: '123', gender: 'm', prefix: 'Mr' }, { expected: { gender: "f" } }, function(err, rows, info) {
console.log('updated:', info.affected_rows);
});
db.update("bk_account", { id: '123', gender: 'm', prefix: 'Mr' }, { expected: { "$or": { gender: "f", g1: null }, aliases: { g1: "gender" } }, function(err, rows, info) {
console.log('updated:', info.affected_rows);
});
db.updateAll(table, query, obj, options, callback)
Update all records that match given condition in the query
, one by one, the input is the same as for db.select
and every record
returned will be updated using db.update
call by the primary key, so make sure options.select include the primary key for every row found by the select.
All properties from the obj
will be set in every matched record.
The callback will receive on completion the err and all rows found and updated. This is mostly for non-SQL databases and for very large range it may take a long time to finish due to sequential update every record one by one. Special properties that can be in the options for this call:
op
can be set to put
or add
options.updateProcess(row, options)
. If it returns non-empty value the update will stop and return it as the error.options.updateFilter(row, options, (skip) => {})
If no options.select
is specified only the primary keys will be returned or collected
Example, update birthday format if not null
db.updateAll("bk_account",
{ birthday: 1 },
{ mtime: Date.now() },
{ ops: { birthday: "not null" },
updateProcess: function(r, o) {
r.birthday = lib.strftime(new Date(r.birthday, "%Y-%m-D"));
},
updateFilter: function(r, o, cb) {
cb(r.status == 'ok');
} },
function(err, count) {
console.log(count, "rows updated");
});
db.del(table, obj, options, callback)
Delete an object in the database, no error if the object does not exist
db.update
methodExample
db.del("bk_account", { id: '123' }, function(err, rows, info) {
console.log('updated:', info.affected_rows);
});
db.delAll(table, query, options, callback)
Delete all records that match given condition, one by one, the input is the same as for db.select
and every record
returned will be deleted using db.del
call. The callback will receive on completion the err and all rows found and deleted.
Special properties that can be in the options for this call:
options.delProcess(row, options)
. If it returns non-empty value the scan will stop and return it as the error.options.delFilter(row, options, (skip) => {})
If no options.select
is specified only the primary keys will be returned or collected
If db-skip-drop
matches the table name and there is no query provided it will exit with error
db.replace(table, obj, options, callback)
Add/update the object, check existence by the primary key. This is not equivalent of REPLACE INTO, it does db.get
to check if the object exists in the database and performs db.add
or db.update
depending on the existence.
Example: updates record 123 only if gender is not 'm' or adds new record
db.replace("bk_account", { id: '123', gender: 'm' }, { check_data: true });
Example: updates record 123 only if mtime of the record is less or equal yesterday
db.replace("bk_account", { id: '123', mtime: Date.now() - 86400000 }, { check_mtime: 'mtime' });
db.list(table, query, options, callback)
Convenient helper to retrieve all records by primary key, the obj must be a list with key property or a string with list of primary key column Example
db.list("bk_account", ["id1", "id2"], function(err, rows) { console.log(err, rows) });
db.list("bk_account", "id1,id2", function(err, rows) { console.log(err, rows) });
db.batch(list, options, callback)
Perform a batch of operations at the same time, all operations for the same table will be run together one by one but different tables will be updated in parallel.
list
an array of objects to put/delete from the database in the format:On return the second arg to the callback is a list of records with errors, same input record with added property errstatus
and errmsg
Example:
var ops = [ { op: "add", table: "bk_counter", obj: { id:1, like:1 } },
{ op: "add", table: "bk_auth", obj: { login: "test", id:1, name:"test" }]
db.batch(ops, { factorCapacity: 0.5 }, lib.log);
db.bulk(list, options, callback)
Bulk operations, it will be noop if the driver does not support it.
The input format is the same as for the db.batch
method.
On return the second arg to the callback is a list of records with errors, same input record with added property errstatus
and errmsg
NOTE: DynamoDB only supports add, put, del
Example
var ops = [ { op: "add", table: "bk_counter", obj: { id:1, like:1 } },
{ op: "del", table: "bk_auth", obj: { login: "test1" } },
{ op: "incr", table: "bk_counter", obj: { id:2, like:1 } },
{ op: "add", table: "bk_auth", obj: { login: "test2", id:2, name:"test2" } }]
db.bulk(ops, { pool: "elasticsearch" }, lib.log);
db.scan(table, query, options, rowCallback, endCallback)
Convenient helper for scanning a table for some processing, rows are retrieved in batches and passed to the callback until there are no more
records matching given criteria. The obj is the same as passed to the db.select
method which defined a condition which records to get.
The rowCallback must be present and is called for every row or batch retrieved and second parameter which is the function to be called
once the processing is complete. At the end, the callback will be called just with 1 argument, err, this indicates end of scan operation.
Basically, db.scan is the same as db.select but can be used to retrieve large number of records in batches and allows async processing of such records.
To hint a driver that scanning is in progress the options.scanning
will be set to true.
Parameters:
db.select
db.select
, with the following additions:read
0.9
table
, useful for cases when the row callback performs
writes into that other table and capacity is differentExample:
db.scan("bk_account", {}, { count: 10, pool: "dynamodb" }, function(row, next) {
// Copy all accounts from one db into another
db.add("bk_account", row, { pool: "pgsql" }, next);
}, function(err) { });
db.copy(table, query, options, callback)
Copy records from one table to another between different DB pools or regions
Parameters:
sort
index in desc orderdb.join(table, rows, options, callback)
Join the given list of records with the records from other table by primary key. The properties from the joined table will be merged with the original rows preserving the existing properties
A special case when table is empty db.join
just returns same rows to the callback, this is
for convenience of doing joins on some conditions and trigger it by setting the table name or skip the join completely.
Example:
db.join("bk_account", [{id:"123",key1:1},{id:"234",key1:2}], lib.log)
db.join("bk_account", [{aid:"123",key1:1},{aid:"234",key1:2}], { keysMap: { id: "aid" }}, lib.log)
db.join("bk_account", [{id:"123",state:"NY"},{id:"234",state:"VA"}], { columnsMap: { state: "astate" }}, lib.log)
db.create(table, columns, options, callback)
Create a table using column definitions represented as a list of objects. Each column definition can contain the following properties:
name
- column nametype
- column type: int, bigint, real, string, now, counter or other supported typeprimary
- column is part of the primary keyunique
- column is part of an unique keyindex
- column is part of an indexvalue
- default value for the columnlen
- column lengthpub
- columns is public, this is very important property because it allows anybody to see it when used in the default API functions, i.e. anybody with valid
credentials can retrieve all public columns from all other tables, and if one of the other tables is account table this may expose some personal infoamtion,
so by default only a few columns are marked as public in the bk_account tablesecure
- an opposite for the pub property, if defined this property should never be returned to the client by the API handlersadmin
- if defined this property can only be updated an admin accountadmins
- if defined this property can be visible by the owner and an admin if result is returned by api.sendJSON
hidden
- completely ignored by all update operations but could be used by the public columns cleaning procedure, if it is computed and not stored in the db
it can contain pub property to be returned to the clientreadonly
- only add/put operations will use the value, incr/update will not affect the valuewriteonly
- only incr/update can change this value, add/put will ignore itnoresult
- delete this property from the result, mostly for joined artificial columns which used for indexes onlyrandom
- add a random number between 0 and this value, useful with type: "now"list
- splits the column value into an array, optional separator
property can be used, default separator is ,|
autoexpire
- for supported databases, mark the column to be auto expired, the value must be time in the future as Epoch secondsautoincr
- for counter tables, mark the column to be auto-incremented by the connection API if the connection type has the same name as the column namejoin
- a list with property names that must be joined together before performing a db operation, it will use the given record to produce new property,
this will work both ways, to the db and when reading a record from the db it will split joined property and assign individual
properties the value from the joined value.join_ops
- an array with operations for which perform columns join only, if not specified it applies for all operations,
allowed values: add, put, incr, update, del, get, selectSome properties may be defined multiple times with number suffixes like: unique1, unique2, index1, index2 to create more than one index for the table, same
properties define a composite key in the order of definition or sorted by the property value, for example: { a: { index:2 }, b: { index:1 } }
will create index (b,a)
because of the index:
property value being not the same. If all index properties are set to 1 then a composite index will use the order of the properties.
Special column types:
uuid
- autogenerate the column value with UUID, optional prefix
property will be prepended, { type: "uuid", prefix: "u_" }
now
- defines a column to be automatically filled with the current timestamp, { type: "now" }
counter
- defines a columns that will be automatically incremented by the db.incr
command, on creation it is set with 0uid
- defines a columns to be automatically filled with the current user id, this assumes that account object is passed in the options from the API leveluname
- defines a columns to be automatically filled with the current user name, this assumes that account object is passed in the options from the API levelNOTE: Index creation is not required and all index properties can be omitted, it can be done more effectively using native tools for any specific database,
this format is for simple and common use cases without using any other tools but it does not cover all possible variations for every database. But all indexes and
primary keys created outside of the backend application will be detected properly by db.cacheColumns
and by each pool cacheIndexes
methods.
Each database pool also can support native options that are passed directly to the driver in the options, these properties are defined in the object with the same name as the db driver, all properties are combined, for example to define provisioned throughput for the DynamoDB index:
db.create("test_table", { id: { primary: 1, type: "int", index: 1, dynamodb: { readCapacity: 50, writeCapacity: 50 } },
type: { primary: 1, pub: 1, projections: 1 },
name: { index: 1, pub: 1 } }
});
Create DynamoDB table with global secondary index, the first index property if not the same as primary key hash defines global index, if it is the same then local,
or if the second key column contains global
property then it is a global index as well, below we create global secondary index on property 'name' only,
in the example above it was local secondary index for id and name. Also a local secondary index is created on id,title
.
DynamoDB projection is defined by a projections
property, can be a number/boolean or an array with index numbers:
db.create("test_table", { id: { primary: 1, type: "int", index1: 1 },
type: { primary: 1, projections: [0] },
name: { index: 1, projections: 1 },
title: { index1: 1, projections: [1] } },
descr: { index: 1, projections: [0, 1] },
});
When using real DynamoDB creating a table may take some time, for such cases if options.waitTimeout
is not specified it defaults to 1min,
so the callback is called as soon as the table is active or after the timeout whichever comes first.
Pass MongoDB options directly: db.create("test_table", { id: { primary: 1, type: "int", mongodb: { w: 1, capped: true, max: 100, size: 100 } }, type: { primary: 1, pub: 1 }, name: { index: 1, pub: 1, mongodb: { sparse: true, min: 2, max: 5 } } });
db.upgrade(table, columns, options, callback)
Upgrade a table with missing columns from the definition list, if after the upgrade new columns must be re-read from the database
then info.affected_rows
must be non zero.
db.drop(table, options, callback)
Drop a table
db.convertError(pool, table, op, err, options)
Convert native database error in some generic human readable string
db.describeTables(tables, callback)
Define new tables or extend/customize existing tables. Table definitions are used with every database operation, on startup, the backend read all existing table columns from the database and cache them in the memory but some properties like public columns are only specific to the backend so to mark such columns the table with such properties must be described using this method. Only columns with changed properties need to be specified, other columns will be left as it is.
Example
db.describeTables({ bk_account: { name: { pub: 1 } },
test: { id: { primary: 1, type: "int" },
name: { pub: 1, index: 1 } });
db.refreshColumns(options, callback)
Refresh columns for all polls which need it
db.cacheColumns(options, callback)
Reload all columns into the cache for the pool, options can be a pool name or an object like { pool: name }
.
if tables
property is an arary it asks to refresh only specified tables if that is possible.
db.prepare(op, table, obj, options)
Prepare for execution for the given operation: add, del, put, update,... Returns prepared object to be passed to the driver's .query method. This method is a part of the driver helpers and is not used directly in the applications.
db.prepareRow(pool, req)
Preprocess an object for a given operation, convert types, assign defaults...
db.prepareForUpdate(pool, req)
Keep only columns from the table definition if we have it Go over all properties in the object and makes sure the types of the values correspond to the column definition types, this is for those databases which are very sensitive on the types like DynamoDB.
db.joinColumn(req, obj, name, col, orig)
Join several columns to produce a combined property if configured, given a column description and an object record it replaces the column value with joined value if needed. Empty properties will be still joined as empty strings. It always uses the original value even if one of the properties has been joined already.
Checks for join
and join_ops
properties in the column definition.
The options.skip_join
can be used to restrict joins, it is a list with columns that should not be joined
The col.join_pools
can be an array with pool names which are allowed to do the join, other pools will skip joining this column.
The col.nojoin_pools
can be an array with pool names which are not allowed to do the join, other pools will skip joining this column
The options.join_strict
can be used to perform join only if all columns in the list are not empty, so the join
is for all columns or none
The options.join_hash
can be used to store a hash of the joined column to reduce the sspace and make the result value easier to use
db.unjoinColumns(rows, name, col, options)
Split joined columns for all rows
db.convertRows(pool, req, rows, options)
Convert rows returned by the database into the Javascript format or into the format defined by the table columns. The following special properties in the column definition chnage the format:
type = json - if a column type is json and the value is a string returned will be converted into a Javascript object
dflt property is defined for a json type and record does not have a value it will be set to specified default value
list - split the value into an array, optional .separator property can be specified
unjoin - a true value or a list of names, it produces new properties by splitting the value by a separator and assigning pieces to
separate properties using names from the list, this is the opposite of the join
property and is used separately if
splitting is required, if joined properties already in the record then no need to split it. If not a list
the names are used form the join property.
Example:
db.describeTables([ { user: { id: {}, name: {}, pair: { join: ["left","right"], unjoin: 1 } } ]);
db.put("test", { id: "1", type: "user", name: "Test", left: "123", right: "000" })
db.select("test", {}, lib.log)
db.setProcessColumns(callback)
Add a callback to be called after each cache columns event, it will be called for each pool separately. The callback to be called may take options argument and it is called in the context of the pool.
The primary goal for this hook is to allow management of the existing tables which are not own by the backendjs application. For such tables, because we have not created them, we need to define column properties after the fact and to keep column definitions in the app for such cases is not realistic. This callback will allow to handle such situations and can be used to set necessary propeties to the table columns.
Example, a few public columns, allow an admin to see all the columns
db.setProcessColumns(function() {
var cols = db.getColumns("users", { pool: this.name });
for (var p in cols) {
if (["id","name"].indexOf(p) > -1) cols[p].pub = 1; else cols[p].admin = 1;
}
})
db.getProcessRows(type, table, options)
Returns a list of hooks to be used for processing rows for the given table
db.runProcessRows(type, table, req, rows)
Run registered pre- or post- process callbacks.
type
is one of the pre
or 'post`table
- the table to run the hooks for, usually the same as req.table but can be '*' for global hooksreq
is the original db request object with the following required properties: op, table, obj, options, info
,rows
is the result rows for post callbacks and the same request object for pre callbacks.db.setProcessRow(type, table, options, callback)
Assign a processRow callback for a table, this callback will be called for every row on every result being retrieved from the specified table thus providing an opportunity to customize the result.
type defines at what time the callback will be called:
pre
- making a request to the db on the query recordpost
- after the request finished to be called on the result rowsAll assigned callback to this table will be called in the order of the assignment.
The callback accepts 2 arguments: function(req, row) where:
req
- the original request for a db operation with requiredop
- current db operation, like add, put, ....table
- current table being updatedobj
- the record with datapool
- current request db pool nameoptions
- current request db optionsinfo
- an object returned with special properties like affected_rows, next_token, only passed to the post
callbacksrow
- a row from the resultWhen producing complex properties by combining other properties it needs to be synchronized using both pre and post callbacks to keep the record consistent.
For queries returning rows, if the callback returns true for a row it will be filtered out and not included in the final result set.
Example
db.setProcessRow("post", "bk_account", function(req, row) {
if (row.birthday) row.age = Math.floor((Date.now() - lib.toDate(row.birthday))/(86400000*365));
});
db.setProcessRow("post", "bk_icon", function(req, row) {
if (row.type == "private" && row.id != req.options.account.id) return true;
});
db.existsPool(name)
Returns true if a pool exists
db.existsTable(table, options)
Returns true if a table exists
db.getPool(table, options)
Return database pool by table name or default pool, options can be a pool name or an object with { pool: name } to return
the pool by given name. This call always returns a valid pool object, in case no requested pool found, it returns
the default pool, in case of invalid pool name it returns none
pool.
A special pool none
always returns empty result and no errors.
Pools specific tables will not be returned even if a different pool name is provided, the none pool will be returned.
db.getPoolTables(name, options)
Return all tables know to the given pool, returned tables are in the object with
column information merged from cached columns from the database with description columns
given by the application. If options.names
is 1 then return just table names as a list.
db.getPools()
Return a list of all active database pools, returns list of objects with name: and type: properties
db.getColumns(table, options)
Return columns for a table or null, columns is an object with column names and objects for definition.
db.getColumn(table, name, options)
Return the column definition for a table, for non-existent columns it returns an empty object
db.getCapacity(table, options)
Return an object with capacity property which is the max write capacity for the table, for DynamoDB only.
By default it checks writeCapacity
property of all table columns and picks the max.
The options can specify the capacity explicitely:
write
, read
or a number with max capacity to usedb.checkCapacity(cap, consumed, callback)
Check if number of requests exceeds the capacity per second, delay if necessary, for DynamoDB only but can be used for pacing
requests with any database or can be used generically. The cap
must be initialized with db.getCapacity
call.
db.getSelectedColumns(table, options)
Return list of selected or allowed only columns, empty list if no options.select
is specified or it is equal to *
. By default only allowed or existing
columns will be returned, to pass the list as is to the driver just use options.select_all
instead.
db.getFilteredColumns(table, filter, options)
Return table columns filtered by a property filter, only return columns that contain(or not) any property from the filter list. If the filter is an object then values must match, null means if exists. ! at the beginning of the name means empty or does not exist.
Example:
db.getFilteredColumns("bk_account", "pub")
db.getFilteredColumns("bk_account", "!internal")
db.getFilteredColumns("bk_account", { pub: null, index: 2 })
db.getFilteredColumns("bk_account", { type: "now" }, { list: 1 })
db.checkCustomColumn(req, name)
Returns type for a global cusgtom column if exists otherwise null, all resolved
columns will be saved in req.allow
for further reference as name: type.
For request specific custom columns pass options.custom_columns
object in the format: [ RegExp, type, ...]
db.skipColumn(name, val, options, columns)
Verify column against common options for inclusion/exclusion into the operation, returns 1 if the column must be skipped
options.no_columns=1
options.skip_null=1
options.skip_values
to a regexpoptions.skip_columns=["a","b"]
options.allow_columns=["a","b"]
options.skip_matched=[{ admin: 1 }, { owner: 1 }]
options.allow_matched={ admin: null }
options.allow_pools=["sqlite","mysql"]
options.skip_pools=["sqlite","mysql"]
name: { allow_pools: ["sqlite","mysql"] }
db.filterRows(obj, rows, options)
Given object with data and list of keys perform comparison in memory for all rows, return only rows that match all keys. This method is used
by custom filters in db.select
by the drivers which cannot perform comparisons with non-indexes columns like DynamoDb, Cassandra.
The rows that satisfy primary key conditions are returned and then called this function to eliminate the records that do not satisfy non-indexed column conditions.
Options support the following propertis:
db.getKeys(table, options)
Return primary keys for a table or empty array, if allkeys
is given in the options then return
a list of all properties involed in primary keys including joined columns
db.getIndexes(table, options)
Return indexes for a table or empty object, each item in the object is an array with index columns
db.getIndexColumns(table, options)
Return columns for all indexes as alist
db.getIndexForKeys(table, keys, options)
Return an index name that can be used for searching for the given keys, the index match is performed on the index columns from the left to right and stop on the first missing key, for example for given keys { id: "1", name: "2", state: "VA" } the index ["id", "state"] or ["id","name"] will be returned but the index ["id","city","state"] will not.
db.getSearchKeys(table, options)
Return keys for the table search, if options.keys provided and not empty it will be used otherwise table's primary keys will be returned. This is a wrapper that makes sure that valid keys are used and deals with input errors like empty keys list to be consistent between different databases. This function always returns an Array even if it is empty.
db.getSearchQuery(table, obj, options)
Return query object based on the keys specified in the options or primary keys for the table, only search properties will be returned in the query object
db.getQueryForKeys(keys, obj, options)
Returns an object based on the list of keys, basically returns a subset of properties.
options.keysMap
defines an object to map record properties with the actual names to be returned.
db.getBindValue(table, options, val, info)
Return possibly converted value to be used for inserting/updating values in the database, is used for SQL parameterized statements
Parameters:
db.getColumnValue(table, options, val, info)
Return transformed value for the column value returned by the database, same parameters as for getBindValue
db.getCached(op, table, query, options, callback)
Retrieve cached result or put a record into the cache prefixed with table:key[:key...]
Options accept the same parameters as for the usual get action but it is very important that all the options
be the same for every call, especially select
parameters which tells which columns to retrieve and cache.
Additional options:
Example:
db.getCached("get", "bk_account", { id: req.query.id }, { select: "latitude,longitude" }, function(err, row) {
var distance = lib.geoDistance(req.query.latitude, req.query.longitude, row.latitude, row.longitudde);
});
db.getCache(table, query, options, callback)
Retrieve an object from the cache by key, sets cacheKey
in the options for later use
db.putCache(table, query, options)
Store a record in the cache
db.delCache(table, query, options)
Notify or clear cached record, this is called after del/update operation to clear cached version by primary keys
db.getCacheKey(table, query, options)
Returns concatenated values for the primary keys, this is used for caching records by primary key
db.getCacheOptions(table, options)
Setup common cache properties
db.getCache2Ttl(table, options)
Return TTL for level 2 cache, negative means use js cache
db.getCacheKeys(table, query, name)
Return a list of global cache keys, if a name is given only returns the matching key
db.delCacheKeys(req, result, options, callback)
Delete all global cache keys for the table record
db.parsePoolOptions(val, options)
Callback to be called by the config parser to preprocess a config parameter for generic options, the options
is an object with all the info about the parameter, all values can be modified to change the behavior, if not used
the config parameter will be a string value assigned in the pool's configOptions
.
db.Pool(options)
Create a new database pool with default methods and properties
pooling will be enabled and will result in the unlimited connections, this is default for DynamoDB
The db methods cover most use cases but in case native driver needs to be used this is how to get the client and use it with its native API,
it is required to call pool.release
at the end to return the connection back to the connection pool.
var pool = db.getPool("", { pool: "mongodb" });
pool.get(function(err, client) {
var collection = client.collection('bk_account');
collection.findOne({ id: '123' }, function() {
pool.release(client);
});
});
db.Pool.prototype.configure(options)
Reconfigure properties, only subset of properties are allowed here so it is safe to apply all of them directly, this is called during realtime config update
db.Pool.prototype.open(callback)
Open a connection to the database, default is to return an empty object as a client
db.Pool.prototype.close(client, callback)
Close a connection, default is do nothing
db.Pool.prototype.query(client, req, options, callback)
Query the database, always return an array as a result (i.e. the second argument for the callback)
db.Pool.prototype.cacheColumns(options, callback)
Cache columns for all tables
db.Pool.prototype.cacheIndexes(options, callback)
Cache indexes for all tables
db.Pool.prototype.nextToken(client, req, rows)
Return next token from the client object
db.Pool.prototype.prepareOptions(options)
Update the options with pool config parameters if needed, the options is from the request
db.Pool.prototype.prepareRow(req)
Default prepareRow is to perform pool specific actions for prepared row before passing it to the op specific columns filterting
db.Pool.prototype.prepare(req)
Default prepare is to return all parameters in an object
db.Pool.prototype.bindValue(value, info, options)
Return the value to be used in binding, mostly for SQL drivers, on input value and col info are passed, this callback may convert the value into something different depending on the DB driver requirements, like timestamp as string into milliseconds
db.Pool.prototype.convertError(table, op, err, options)
Converts native DB driver error into other human readable format
db.Pool.prototype.processColumns(pool)
that is called after this pool cached columms from the database, it is called sychnroniously inside the db.cacheColumns
method.
db.Pool.prototype.resolveTable(op, table, obj, options)
Return possible different table at the time of the query, it is called by the db.prepare
method
and if exist it must return the same or new table name for the given query parameters.
db.SqlPool(options)
Create a database pool for SQL like databases
db.SqlPool.prototype.cacheColumns(options, callback)
Call column caching callback with our pool name
db.SqlPool.prototype.prepare(req)
Prepare for execution, return an object with formatted or transformed SQL query for the database driver of this pool
db.SqlPool.prototype.query(client, req, options, callback)
Execute a query or if req.text is an Array then run all queries in sequence
db.SqlPool.prototype.nextToken(client, req, rows)
Support for pagination, for SQL this is the OFFSET for the next request
Pool.prototype.convertError(table, op, err, options)
Convert into recognizable error codes
Pool.prototype.query(client, req, options, callback)
Simulate query as in SQL driver but performing AWS call, text will be a table name and values will be request options
pool
Create a database pool that works with ElasticSearch server, only the hostname and port will be used, by default each table is stored in its own index.
To define shards and replicas per index:
-db-elasticsearch-pool-options-shards-INDEX_NAME=NUM
-db-elasticsearch-pool-options-replicas-INDEX_NAME=NUM
To support multiple seed nodes a parameter -db-elasticsearch-pool-options-servers=<IP_ADDRESS>,<IP_ADDRESS>
can be specified, if the primary node
fails it will switch to other configured nodes. To control the switch retries and timeout there are options:
-db-elasticsearch-pool-options-retry-count=3
-db-elasticsearch-pool-options-retry-timeout=250
On successful connect to any node the driver retrieves full list of nodes in the cluster and switches to a random node, this happens
every discovery-interval
in milliseconds, default is 1h, it can be specified as -db-elasticserch-pool-options-discovery-interval=300000
db.sqlQuery(pool, client, req, options, callback)
Execute one or more SQL statements
db.sqlCacheColumns(pool, options, callback)
Cache columns using the information_schema
db.sqlPrepare(pool, req)
Prepare SQL statement for the given operation
db.sqlQuote(val)
Quote value to be used in SQL expressions
db.sqlValue(value, options)
Return properly quoted value to be used directly in SQL expressions, format according to the type
db.sqlValueIn(list, type)
Return list in format to be used with SQL IN ()
db.sqlExpr(pool, name, value, options)
Build SQL expressions for the column and value options may contain the following properties:
db.sqlTime(d)
Return time formatted for SQL usage as ISO, if no date specified returns current time
db.sqlLimit(pool, req)
Build SQL orderby/limit/offset conditions, config can define defaults for sorting and paging
db.sqlWhere(pool, req, query, keys, join)
Build SQL where condition from the keys and object values, returns SQL statement to be used in WHERE
db.sqlCreate(pool, req)
Create SQL table using table definition
db.sqlUpgrade(pool, req)
Create ALTER TABLE ADD COLUMN statements for missing columns
db.sqlDrop(pool, req)
Create SQL DROP TABLE statement
db.sqlGet(pool, req)
Get one object from the database, options may define the following properties:
db.sqlSelect(pool, req)
Select object from the database, options may define the following properties:
db.sqlInsert(pool, req)
Build SQL insert statement
db.sqlUpdate(pool, req)
Build SQL statement for update
db.sqlDelete(pool, req)
Build SQL statement for delete
Ipc.prototype.handleServerMessages(worker, msg)
To be used in messages processing that came from the clients or other way
Ipc.prototype.sendReplPort(role, worker)
Send REPL port to a worker if needed
Ipc.prototype.newMsg(op, msg, options)
Returns an IPC message object, msg
must be an object if given. To support JSON text messages
it can be represented as a string of JSON array: [op,msg]
Ipc.prototype.emitMsg(op, msg, options)
Wrapper around EventEmitter emit
call to send unified IPC messages in the same format
Ipc.prototype.sendMsg(op, msg, options, callback)
Send a message to the master process via IPC messages, callback is used for commands that return value back
timeout
property can be used to specify a timeout for how long to wait the reply, if not given the default is usedIf called inside the server, it process the message directly, reply is passed in the callback if given.
Examples:
ipc.sendMsg("op1", { data: "data" }, { timeout: 100 })
ipc.sendMsg("op1", { name: "name", value: "data" }, function(data) { console.log(data); })
ipc.sendMsg("op1", { 1: 1, 2: 2 }, { timeout: 100 })
ipc.sendMsg("op1", { 1: 1, 2: 2 }, function(data) { console.log(data); })
ipc.newMsg(JSON.stringify([ "op1", { name: "test" } ]))
Ipc.prototype.broadcast(channel, msg, options, callback)
Send a message to a channel, this is high level routine that uses the corresponding queue, it uses eventually ipc.publish.
Ipc.prototype.initServer()
This function is called by a master server process to setup IPC channels and support for cache and messaging
Ipc.prototype.initWorker()
This function is called by a worker process to setup IPC channels and support for cache and messaging
Ipc.prototype.createClient(url, options)
Return a new client for the given host or null if not supported
Ipc.prototype.getClient(prefix, options)
Return a cache or queue client by name if specified in the options or use default client for the prefix which always exists, supported prefixes are: queue, cache
Ipc.prototype.getQueue(name)
Return queue object by name, if name is wrong the default queue us returned
Ipc.prototype.getCache(name)
Retirn cache object by name, if name is wrong the default cache is returned
Ipc.prototype.initClients(prefix)
Initialize a client for cache or queue purposes, previous client will be closed.
Ipc.prototype.stats(options, callback)
Returns the cache statistics, the format depends on the cache type used
Ipc.prototype.clear(pattern, options, callback)
Clear all or only items that match the given pattern
Ipc.prototype.get(key, options, callback)
Retrieve an item from the cache by key.
options.set
is given and no value exists in the cache it will be set as the initial value, still
nothing will be returned to signify that a new value assigned.options.mapName
defines a map from which the key will be retrieved if the cache supports maps, to get the whole map
the key must be set to *If the key
is an array then it returns an array with values for each key, for non existent keys an empty
string will be returned. For maps only if the key
is * it will return the whole object, otherise only value(s)
are returned.
Example
ipc.get(["my:key1", "my:key2"], function(err, data) { console.log(data) });
ipc.get("my:key", function(err, data) { console.log(data) });
ipc.get("my:counter", { set: 10 }, function(err, data) { console.log(data) });
ipc.get("*", { mapName: "my:map" }, function(err, data) { console.log(data) });
ipc.get("key1", { mapName: "my:map" }, function(err, data) { console.log(data) });
ipc.get(["key1", "key2"], { mapName: "my:map" }, function(err, data) { console.log(data) });
Ipc.prototype.del(key, options, callback)
Delete an item by key(s), if key
is an array all keys will be deleted at once atomically if supported
options.mapName
defines a map from which the counter will be deleted if the cache supports maps, to delete the whole map
the key must be set to *Example:
ipc.del("my:key")
ipc.del("key1", { mapName: "my:map" })
ipc.del("*", { mapName: "my:map" })
Ipc.prototype.put(key, val, options, callback)
Replace or put a new item in the cache.
options.ttl
can be passed in milliseconds if the driver supports itoptions.mapName
defines a map where the counter will be stored if the cache supports maps, to store the whole map in one
operation the key
must be set to * and the val
must be an objectoptions.setmax
if not empty tell the driver to set this new number only if there is no existing
value or it is less that the new number, only works for numeric valuesExample:
ipc.put("my:key", 2)
ipc.put("my:key", 1, { setmax: 1 })
ipc.put("*", { key1: 1, key2: 2 }, { mapName: "my:map" })
Ipc.prototype.incr(key, val, options, callback)
Increase/decrease a counter in the cache by val
, non existent items are treated as 0, if a callback is given an
error and the new value will be returned.
options.ttl
in milliseconds can be used if the driver supports itoptions.mapName
defines a map where the counter will be stored if the cache supports mapsExample:
ipc.incr("my:key", 1)
ipc.incr("key1", 1, { mapName: "my:map" })
Ipc.prototype.subscribe(channel, options, callback)
Subscribe to a publish server for messages for the given channel, the callback will be called only on new message received.
Example:
ipc.subscribe("alerts", function(msg) {
req.res.json(data);
}, req);
Ipc.prototype.unsubscribe(channel, options, callback)
Close a subscription for the given channel, no more messages will be delivered.
Ipc.prototype.publish(channel, msg, options, callback)
Publish an event to the channel to be delivered to all subscribers.
options.queueName
defines the queue, if not specified then it is sent to the default queueIpc.prototype.listen(options, callback)
Listen for messages from the given queue, the callback will be called only on new message received.
The callback accepts 2 arguments, a message and optional next callback, if it is provided it must be called at the end to confirm or reject the message processing. Only errors with code>=500 will result in rejection, not all drivers support the next callback if the underlying queue does not support message acknowledgement.
Depending on the implementation, this can work as fan-out, delivering messages to all subscribed to the same channel or can implement job queue model where only one subscriber receives a message.
For cases when the next
callback is provided this means the queue implementation requires an acknowledgement of successful processing,
returning an error with err.status >= 500
will keep the message in the queue to be processed later. Special code 600
means to keep the job
in the queue and report as warning in the log.
Example:
ipc.listen({ queueName: "jobs" }, function(msg, next) {
req.res.json(data);
if (next) next();
}, req);
Ipc.prototype.unlisten(channel, options, callback)
Stop listening for message, if no callback is provided all listeners for the key will be unsubscribed, otherwise only the specified listener. The callback will not be called. It keeps a count how many subscribe/unsubscribe calls been made and stops any internal listeners once nobody is subscribed. This is specific to a queue which relies on polling.
Ipc.prototype.submit(msg, options, callback)
Submit a message to the queue
options.queueName
defines the queue, if not specified then it is sent to the default queueoptions.stime
defines when the message should be processed, it will be held in the queue until the time comesoptions.etime
defines when the message expires, i.e. will be dropped if not executed before this time.Ipc.prototype.monitor(options)
Queue specific monitor services that must be run in the master process, this is intended to perform queue cleanup or dealing with stuck messages
Ipc.prototype.limiter(options, callback)
Check for rate limit using the default or specific queue, by default TokenBucket using local LRU cache is used unless a queue client provides its own implementation.
The options must have the following properties:
The callback takes 1 argument, delay
which is a number of milliseconds till the bucket can be used again if not consumed, i.e. 0 means consumed.
Ipc.prototype.checkLimiter(options, callback)
Keep checking the limiter until it is clear to proceed with the operation, if there is no available tokens in the bucket it will wait and try again until the bucket is filled.
Ipc.prototype.lock(name, options, callback)
Implementation of a lock with optional ttl, only one instance can lock it, can be for some period of time and will expire after timeout.
A lock must be uniquely named and the ttl period is specified by options.ttl
in milliseconds.
This is intended to be used for background job processing or something similar when
only one instance is needed to run. At the end of the processing ipc.unlock
must be called to enable another instance immediately,
otherwise it will be available after the ttl only.
if options.timeout
is given the function will keep trying to lock for the timeout
milliseconds.
if options.set
is given it will unconditionally set the lock for the specified ttl, this is for cases when
the lock must be active for longer because of the long running task
The callback must be passed which will take an error and a boolean value, if true is returned it means the timer has been locked by the caller, otherwise it is already locked by other instance. In case of an error the lock is not supposed to be locked by the caller.
Example:
ipc.lock("my-lock", { ttl: 60000, timeout: 30000 }, function(err, locked) {
if (locked) {
...
ipc.unlock("my-lock");
}
});
Ipc.prototype.unlock(name, options, callback)
Unconditionally unlock the lock, any client can unlock any lock.
IpcClient.prototype.close()
Close current connection, ports.... not valid after this call
IpcClient.prototype.applyOptions()
Prepare options to be used safely
IpcClient.prototype.stats(options, callback)
Returns the cache statistics to the callback as the forst argument, the object tructure is specific to each cache implementstion
IpcClient.prototype.clear(pattern, callback)
Clear all or only matched keys from the cache
IpcClient.prototype.get(key, options, callback)
Returns an item from the cache by a key, callback is required and it acceptes only the item, on any error null or undefined will be returned
IpcClient.prototype.put(key, val, options, callback)
Store an item in the cache, options.ttl
can be used to specify TTL in milliseconds
IpcClient.prototype.incr(key, val, options, callback)
Add/substract a number from the an item, returns new number in the callback if provided, in case of an error null/indefined should be returned
IpcClient.prototype.del(key, options, callback)
Delete an item from the cache
IpcClient.prototype.poller()
This is called to configure and setup event processing on first subscribe call, if nobody subscribes this will never be called and thus no initialization for event processing either, for clients that only publish this may save lots of processing and memory.
This method must take care how to keep the poller running via interval or timeout as long as the this._polling=1
.
IpcClient.prototype.schedulePoller(timeout)
Schedule next poller iteration immediately or after timeout, check configured polling rate, make sure it polls no more than configured number of times per second
IpcClient.prototype.monitor()
Queue monitor or cleanup service, when poller is involved this will be started and can be used for cleaning up stale messages or other maintainence work the requires.
IpcClient.prototype.isPolling()
Returns 1 if the poller is active
IpcClient.prototype.limiter(options, callback)
Rate limit check, by default it uses the master LRU cache meaning it works within one physical machine only.
The options must have the following properties:
metrics.TokenBucket
rate limiter.The callback argument will be called with an object where the property consumed
set to true if consumed or false otherwise,
and the property delay
with number of milliseconds till the bucket can be used again.
IpcClient.prototype.subscribe(channel, options, callback)
Subscribe to receive notification from the given channel
IpcClient.prototype.unsubscribe(channel, options, callback)
Stop receiving notifications on the given channel
IpcClient.prototype.publish(channel, msg, options, callback)
Publish an event
IpcClient.prototype.listen(options, callback)
Listen for incoming messages
IpcClient.prototype.unlisten(options, callback)
Stop receiving messages
IpcClient.prototype.submit(msg, options, callback)
Submit a message to a queue
IpcClient.prototype.lock(name, options, callback)
By default always lock because of lack of actual implementtion
client
Cache client based on Memcached server using https://github.com/3rd-Eden/node-memcached
To support more than one server use either one:
ipc-cache=memcache://host1?bk-servers=host2,host3
ipc-cache-memcache=memcache://host1 ipc-cache-memcache-options-servers=host1,host2
To pass memcached module specific options:
ipc-cache-options-failures=5 ipc-cache-options-maxValue=1024
jobs
Job queue processor
When launched with jobs-workers
parameter equal or greater than 0, the master spawns a number of workers which subscribe to
configured job queues or the default queue and listen for messsges.
A job message is an object that defines what method from which module to run with the options as the first argument and a callback as the second.
Multiple job queues can be defined and processed at the same time.
jobs.configureMaster(options, callback)
Initialize jobs processing in the master process
jobs.configureWorker(options, callback)
Initialize a worker to be ready for jobs to execute, in instance mode setup timers to exit on no activity.
jobs.shutdownWorker(options, callback)
Perform graceful worker shutdown, to be used for workers restart
jobs.initServer(options, callback)
Initialize a master that will manage jobs workers
jobs.initWorker(options, callback)
Initialize a worker for processing jobs
jobs.exitWorker(options)
Perform graceful worker shutdown and then exit the process
jobs.isCancelled(name, tag)
Returns true if a task with given name must be cancelled, this flag is set from the jobs master and stoppable tasks must check it from time to time to terminate gracefully
jobs.cancelTask(name, options)
Send cancellation request to a worker or all workers, this has to be called from the jobs master.
options.workers
can be a single worker id or a list of worker ids, if not given the request will be sent to all workers for the current process cluster.
options.tag
is an opaque data that will be used to verifying which task should be cancelled, without it all tasks with given name will be cancelled.
jobs.getMaxRuntime()
Find the max runtime allowed in seconds
jobs.checkTimes()
Check how long we run a job and force kill if exceeded, check if total life time is exceeded
jobs._badJob(jobspec)
Make sure the job is valid and has all required fields, returns a normalized job object or an error, the jobspec must be in the following formats:
"module.method"
{ job: "module.method" }
{ job: { "module.method": {}, .... } }
{ job: [ "module.method", { "module.method": {} ... } ...] }
any task in string format "module.method" will be converted into { "module.method: {} } automatically
jobs.submitJob(jobspec, options, callback)
Submit a job for execution, it will be saved in a queue and will be picked up later and executed.
The queue and the way how it will be executed depends on the configured queue. See isJob
for
the format of the job objects.
if jobspec.uniqueTtl
is greater than zero it defines number of milliseconds for this job to stay in the queue or run,
it creates a global lock using the job object as the hash key, no other job can be run until the ttl expires or the job
finished, non unique jobs will be kept in the queue and repeated later according to the visibilityTimeeout
setting.
jobspec.uniqueKey
can define an alternative unique key for this job for cases when different jobs must be run sequentially
if jobspec.uniqueKeep
is true then keep the unique lock after the jobs finished, otherwise it is cleared
if jobspec.uniqueDrop
if true will make non-unique jobs to be silently dropped instead of keeping them in the queue
jobspec.logger
defines the logger level which will be used to log when the job is finished, default is debug
jobspec.maxRuntime
defines max number of seconds this job can run, if not specified then the queue default is used
jobspec.uniqueTag
defines additional tag to be used for job cancelling, for cases when multiple jobs are running with the same method
if jobspec.uniqueOnce
is true than the visibility timeout is not kept alive while the job is running
jobspec.noWait
will run the job and delete it from the queue immediately, not at the end, for one-off jobs
jobspec.startTime
and/or jobspec.endTime
will define the time period during whihc this job is allowed to run, if
outside the period it will be dropped
options.delay
is only supported by SQS currently, it delays the job execution for the specified amount of ms
Special queueName: jobs.selfQueue
is reserved to run the job immediately inside the current process,
it will call the runJob
directly, this is useful in cases when already inside a worker and instead of submitting a new job
just run it directly. Any queue can be configured to run in selfQueue
by setting -ipc-queue[-NAME]-options-self-queue 1
.
jobs.runJob(jobspec, options, callback)
Run all tasks in the job object, all errors will be just logged, but if noerrors
is defined in the top
level job object then the whole job will stop on first error returned by any task.
jobs._runJob(jobspec, options, callback)
Sequentially execute all tasks in the list, run all subtasks in parallel
jobs.runTask(name, jobspec, options, callback)
Execute a task by name, the options
will be passed to the function as the first argument, calls the callback on finish or error
jobs._finishTask(err, name, jobspec, options, callback)
Complete task execution, cleanup and update the status
jobs.scheduleCronjob(jobspec)
Create a new cron job, for remote jobs additional property args can be used in the object to define arguments for the instance backend process, properties must start with -
Example:
{ "cron": "0 */10 * * * *", "job": "server.processQueue" },
{ "cron": "0 */30 * * * *", "job": { "server.processQueue": { name: "queue1" } } },
{ "cron": "0 5 * * * *", "job": [ { "scraper.run": { "url": "host1" } }, { "scraper.run": { "url": "host2" } } ] }
jobs.scheduleCronjobs(type, list)
Schedule a list of cron jobs, types is used to cleanup previous jobs for the same type for cases when
a new list needs to replace the existing jobs. Empty list does nothing, to reset the jobs for the particular type and
empty invalid jobs must be passed, like: [ {} ]
Returns number of cron jobs actually scheduled.
jobs.loadCronjobs()
Load crontab from JSON file as list of job specs:
Example:
[ { cron: "0 0 * * * *", job: "scraper.run" }, ..]
jobs.parseCronjobs(type, data)
Parse a JSON data with cron jobs and schedule for the given type, this can be used to handle configuration properties
lib.tryCall(callback)
Run a callback if a valid function, all arguments after the callback will be passed as is
lib.tryCatch(callback)
Run a callback inside try..catch block, all arguments after the callback will be passed as is, in case of error all arguments will be printed in the log
lib.log()
Print all arguments into the console, for debugging purposes
lib.__()
Simple i18n translation method compatible with other popular modules, supports the following usage:
lib.loadLocale(file, callback)
Load a file with locale translations into memory
lib.getArg(name, dflt)
Return commandline argument value by name
lib.getArgInt(name, dflt)
Return commandline argument value as a number
lib.typeName(v)
Return object type, try to detect any distinguished type
lib.isArg(name)
Returns true of given arg(s) are present in the command line, name can be a string or an array of strings.
lib.toVersion(str)
Returns a floating number from the version string, it assumes common semver format as major.minor.patch, all non-digits will be removed, underscores will be treated as dots. Returns a floating number which can be used in comparing versions.
Example
> lib.toVersion("1.0.3")
1.000003
> lib.toVersion("<IP_ADDRESS>")
1.000003004
> lib.toVersion("<IP_ADDRESS>") > lib.toVersion("1.0.3")
true
> lib.toVersion("<IP_ADDRESS>") > lib.toVersion("1.0.0")
true
> lib.toVersion("<IP_ADDRESS>") > lib.toVersion("1.1.0")
false
lib.encodeURIComponent(str)
Encode with additional symbols, convert these into percent encoded:
! -> %21, * -> %2A, ' -> %27, ( -> %28, ) -> %29
lib.toTitle(name)
Convert text into capitalized words
lib.toCamel(name, chars)
Convert into camelized form, optional chars can define the separators, default is -, _ and .
lib.toUncamel(str, sep)
Convert Camel names into names separated by the given separator or dash if not.
lib.toNumber(val, options)
Safe version, uses 0 instead of NaN, handle booleans, if float specified, returns as float.
Options:
Example:
lib.toNumber("123")
lib.toNumber("1.23", { float: 1, dflt: 0, min: 0, max: 2 })
lib.toClamp(num, min, max)
Return a number clamped between the range
lib.toBool(val, dflt)
Return true if value represents true condition, i.e. non empty value
lib.toDate(val, dflt, invalid)
Return Date object for given text or numeric date representation, for invalid date returns 1969 unless invalid
parameter is given,
in this case invalid date returned as null. If dflt
is NaN, null or 0 returns null as well.
lib.toMtime(val, dflt)
Return milliseconds from the date or date string, only number as dflt is supported, for invalid dates returns 0
lib.toBase62(num, alphabet)
Return base62 representation for a number
lib.toValue(val, type, options)
Convert value to the proper type
lib.toRegexp(str, options)
Safely create a regexp object, if invalid returns undefined, the options can be a string with srandard RegExp flags or an object with the following properties:
lib.toRegexpMap(obj, val, options)
Add a regexp to the list of regexp objects, this is used in the config type regexpmap
.
lib.toRegexpObj(obj, val, options)
Add a regexp to the object that consist of list of patterns and compiled regexp, this is used in the config type regexpobj
lib.toDuration(mtime, options)
Return duration in human format, mtime is msecs
lib.toAge(mtime, options)
Given time in msecs, return how long ago it happened
lib.toParams(query, schema, options)
Process incoming query and convert parameters according to the type definition, the schema contains the definition of the paramaters against which to validate incoming data. It is an object with property names and definitoons that at least must specify the type, all other options are type specific.
The options can define the following properties:
If any of the properties have required:1
and the value will not be resolved then the function returns a string with the errmsg
message
or the default message, this is useful for detection of invalid or missing input data.
Example:
var account = lib.toParams(req.query, { id: { type: "int" },
count: { type: "int", min: 1, max: 10, dflt: 5 },
page: { type: "int", min: 1, max: 10, dflt: NaN, required: 1, errmsg: "Page number between 1 and 10 is required" },
name: { type: "string", max: 32, trunc: 1 },
pair: { type: "map", separator: "|" },
code: { type: "string", regexp: /^[a-z]-[0-9]+$/, errmsg: "Valid code is required" },
start: { type: "token", required: 1 },
email1: { type: "email", required: { email: null } },
data: { type: "json", datatype: "obj" },
mtime: { type: "mtime", name: "timestamp" },
flag: { type: "bool", novalue: false },
descr: { novalue: { name: "name", value: "test" },
email: { type: "list", datatype: "email", novalue: ["a@a"] } },
internal: { ignore: 1 },
tm: { type:" timestamp", optional: 1 },
status: { value: "ready" },
state: { values: ["ok","bad","good"] },
status: { value: [{ name: "state", value: "ok", set: "1" }, { name: "state", value: ["bad","good"], op: "in" }],
obj: { type: "obj", params: { id: { type: "int" }, name: {} } },
arr: { type: "array", params: { id: { type: "int" }, name: {} } },
state: { type: "list", datatype: "string, values: ["VA","DC] } },
ssn: { type: "string", regexp: /^[0-9]{3}-[0-9]{3}-[0-9]{4}$/, errmsg: "Valid SSN is required" } },
phone: { type: "list", datatype: "number } },
{ data: { start: { secret: req.account.secret },
name: { dflt: "test" },
count: { max: 100 },
email: { ignore: req.account.type != "admin" },
'*': { empty: 1, null: 1 },
})
if (typeof account == "string) return api.sendReply(res, 400, account);
lib.toFormat(format, data, options)
Convert a list of records into the specified format, supported formats are: xml, csv, json
.
csv
the default separator is comma but can be specified with options.separator
. To produce columns header specify options.header
.json
format puts each record as a separate JSON object on each line, so to read it back
it will require to read every line and parse it and add to the list.xml
format the name of the row tag is <row>