General purpose backend library. The primary goal is to have a scalable platform for running and managing Node.js servers for Web services implementation.
This project only covers the lower portion of the Web services ecosystem: Node.js processes, HTTP servers, basic API functionality, database access, caching, messaging between processes, metrics and monitoring, a library of tools for developing Node.js servers.
For the UI and presentation layer there are no restrictions what to use as long as it can run on top of the Express server.
Features:
Check out the Documentation for more details.
To install the module with all optional dependencies if they are available in the system
npm install backendjs
To install from the git
npm install git+https://github.com/vseryakov/backendjs.git
or simply
npm install vseryakov/backendjs
Only core required dependencies are installed but there are many modules which require a module to work correctly.
All optional dependencies are listed in the package.json under "modDependencies" so npm cannot use it, only manual install of required modules is supported or it is possible to install all optional dependencies for development purposes.
Here is the list of modules required for each internal feature:
pg
- PostgreSQL database accessargon2
or bcrypt
- for user password hashingmmmagic
- file detection in uploads, only used when allow
is passed to the api.putFile
consolidate
- for API templating, disabled by defaultredis
- for Redis queue and cache driverunix-dgram
- for syslog on Linux to use local syslogbkjs-sqlite
- to use SQLite database driverweb-push
- for Web push notifications@parse/node-apn
- for Apple push notificationsbkjs-wand
- for scaling images in uploads using ImageMagick modulesharp
- scaling images in uploads using VPS imagingnats
- NATS driver for queue and eventsamqplib
- RabbitMQ driver for queue and events (alpha)The command below will show all core and optional dependencies, npm install
will install only the core dependencies
bkjs deps -dry-run -mods
Simplest way of using the backendjs, it will start the server listening on port 8000
$ node
> const bkjs = require('backendjs')
> bkjs.server.start()
Access is allowed only with valid signature except urls that are explicitly allowed without it (see api-allow
config parameter below)
Same but using the helper tool, by default no database driver are enablked so here we use embedded SQLite database and listen on port 8000.
bkjs web -db-pool sqlite -db-sqlite-pool default
or to the PostgreSQL server, database backend (if not running local server can be started with bkjs init-pgsql
if postgresql is installed)
bkjs web -db-pool pg -db-pg-pool postgresql://postgres@localhost/backend
If running on EC2 instance with IAM profile no need to specify AWS credentials:
bkjs web -db-pool dynamodb -db-dynamodb-pool default
To start the server and connect to the DynamoDB (command line parameters can be saved in the etc/config file
, see below about config files)
bkjs web -db-pool dynamodb -db-dynamodb-pool default -aws-key XXXX -aws-secret XXXX
or to the ElasticSearch server, database backend
bkjs web -db-pool elasticsearch -db-elasticsearch-pool http://<IP_ADDRESS>:9200
All commands above will behave exactly the same
Tables are not created by default, in order to initialize the database, run the server or the shell with -db-create-tables
flag,
it is called only inside a master process, a worker never creates tables on start
prepare the tables in the shell
bksh -db-pool dynamodb -db-dynamodb-pool default -db-create-tables
run the server and create tables on start, run Elasticsearch locally first on the local machine
bkjs get-elasticsearch
bkjs run-elasticsearch
bkjs web -db-pool elasticsearch -db-elasticsearch-pool http://<IP_ADDRESS>:9200 -db-create-tables
While the local backendjs is runnning, the documentation is always available at http://localhost:8000/doc.html (or whatever port is the server using)
To add users from the command line
bksh -user-add login test secret test name TestUser email<EMAIL_ADDRESS>To start Node.js shell with backendjs loaded and initialized, all command line parameters apply to the shell as well
bkjs shell
To access the database while in the shell using callbacks
> db.select("bk_user", {}, lib.log);
> db.add("bk_user", { id: 'test2', login: 'test2', secret: 'test2', name' Test 2 name' }, lib.log);
> db.select("bk_user", { id: 'test2' }, lib.log);
> db.select("bk_user", { id: ['test1','test2'] }, { ops: { id: "in" } }, lib.log);
or the same using async/await, same methods with a
prepended to the name
> await db.aselect("bk_user", {});
> await db.aadd("bk_user", { id: 'test2', login: 'test2', secret: 'test2', name' Test 2 name' });
> await db.aselect("bk_user", { id: 'test2' });
To search using Elasticsearch (assuming it runs on EC2 and it is synced with DynamoDB using streams)
> await db.select("bk_user", { q: 'test' }, { pool: "elasticsearch" });
The library is packaged with copies of Bootstrap, jQuery, Knockout.js for quick Web development in web/js and web/css directories, all scripts are available from the browser with /js or /css paths. To use all at once as a bundle run the following command:
cd node_modules/backendjs && npm run devbuild
Go to examples/api
directory:
Run the application, it will start the Web server on port 8000:
./app.sh
Now log in with the new account,
Go to http://localhost:8000/api.html and click on Login at the top-right corner, then enter 'test' as login and 'test' as secret in the login popup dialog.
To see your account details run the command in the console /account/get
To see current metrics run the command in the console /system/stats/get
When the web server is started with -watch
parameter or as bkjs watch
then any change in the source files will make the server restart automatically
letting you focus on the source code and not server management, this mode is only enabled by default in development mode,
check app.sh
for parameters before running it in production.
Almost everything in the backend is configurable using config files, a config database or DNS. The whole principle behind it is that once deployed in production, even quick restarts are impossible to do so there should be a way to push config changes to the processes without restarting.
Every module defines a set of config parameters that defines the behavior of the code, due to the single threaded nature of the Node.js. It is simple to update any config parameter to a new value so the code can operate differently. To achieve this the code must be written in a special way, like driven by configuration which can be changed at any time.
All configuration goes through the configuration process that checks all inputs and produces valid output which is applied to the module variables. Config file or database table with configuration can be loaded on demand or periodically, for example all local config files are watched for modification and reloaded automatically, the config database is loaded periodically which is defined by another config parameter.
When the backendjs server starts it spawns several processes that perform different tasks.
There are 2 major tasks of the backend that can be run at the same time or in any combination:
These features can be run standalone or under the guard of the monitor which tracks all running processes and restarted any failed ones.
This is the typical output from the ps command on Linux server:
ec2-user 891 0.0 0.6 1071632 49504 ? Ssl 14:33 0:01 bkjs: monitor
ec2-user 899 0.0 0.6 1073844 52892 ? Sl 14:33 0:01 bkjs: master
ec2-user 908 0.0 0.8 1081020 68780 ? Sl 14:33 0:02 bkjs: server
ec2-user 917 0.0 0.7 1072820 59008 ? Sl 14:33 0:01 bkjs: web
ec2-user 919 0.0 0.7 1072820 60792 ? Sl 14:33 0:02 bkjs: web
ec2-user 921 0.0 0.7 1072120 40721 ? Sl 14:33 0:02 bkjs: worker
To enable any task a command line parameter must be provided, it cannot be specified in the config file. The bkjs
utility supports several
commands that simplify running the backend in different modes.
bkjs watch
- runs the master and Web server in wather mode checking all source files for changes, this is the common command to be used
in development, it passes the command line switches: -watch -master
bkjs start
- this command is supposed to be run at the server startup as a service, it runs in the background and the monitors all tasks,
the env variable BKJS_SERVER
can be set in the profile to one of the master or monitor
to define which run mode to use, default mode is monitorbkjs monitor
- this command is supposed to be run at the server startup, it runs in the background and the monitors all processes,
the command line parameters are: -daemon -monitor -master -syslog
bkjs master
- this command is supposed to be run at the server startup, it runs in the background and the monitors all processes,
the command line parameters are: -daemon -monitor -master -syslog
, web server and workers are started by defaultbkjs web
- this command runs just web server process with child processes as web workersbkjs run
- this command runs without other parameters, all additional parameters can be added in the command line, this command
is a barebone helper to be used with any other custom settings.bkjs run -api
- this command runs a single process as web server, sutable for Dockerbkjs run -worker
- this command runs a single process worker, suatable for Dockerbkjs shell
or bksh
- start backendjs shell, no API or Web server is initialized, only the database poolsThe main purpose of the backendjs is to provide API to access the data, the data can be stored in the database or some other way but the access to that data will be over HTTP and returned back as JSON. This is default functionality but any custom application may return data in whatever format is required.
Basically the backendjs is a Web server with ability to perform data processing using local or remote jobs which can be scheduled similar to Unix cron.
The principle behind the system is that nowadays the API services just return data which Web apps or mobiles apps can render to the user without the backend involved. It does not mean this is simple gateway between the database, in many cases it is but if special processing of the data is needed before sending it to the user, it is possible to do and backendjs provides many convenient helpers and tools for it.
When the API layer is initialized, the api module contains app
object which is an Express server.
Special module/namespace app
is designated to be used for application development/extension. This module is available in the same way as api
and core
which makes it easy to refer and extend with additional methods and structures.
The typical structure of a single file backendjs application is the following:
const bkjs = require('backendjs');
const api = bkjs.api;
const app = bkjs.app;
const db = bkjs.db;
app.listArg = [];
// Define the module config parameters
core.describeArgs('app', [
{ name: "list-arg", array: 1, type: "list", descr: "List of words" },
{ name: "int-arg", type: "int", descr: "An integer parameter" },
]);
// Describe the tables or data models, all DB pools will use it, the master or shell
// process only creates new tables, workers just use the existing tables
db.describeTables({
...
});
// Optionally customize the Express environment, setup MVC routes or else, `api.app` is the Express server
app.configureMiddleware = function(options, callback)
{
...
callback()
}
// Register API endpoints, i.e. url callbacks
app.configureWeb = function(options, callback)
{
api.app.get('/some/api/endpoint', (req, res) => {
// to return an error, the message will be translated with internal i18n module if locales
// are loaded and the request requires it
api.sendReply(res, err);
// or with custom status and message, explicitely translated
api.sendReply(res, 404, res.__({ phrase: "not found", locale: "fr" }));
// with config check
if (app.intArg > 5) ...
if (app.listArg.indexOf(req.query.name) > -1) ...
// to send data back with optional postprocessing hooks
api.sendJSON(req, err, data);
// or simply
res.json(data);
});
...
callback();
}
// Optionally register post processing of the returned data from the default calls
api.registerPostProcess('', /^\/account\/([a-z\/]+)$/, (req, res, rows) => { ... });
...
// Optionally register access permissions callbacks
api.registerAccessCheck('', /^\/test\/list$/, (req, status, callback) => { ... });
api.registerPreProcess('', /^\/test\/list$/, (req, status, callback) => { ... });
...
bkjs.server.start();
Another probably easier way to create single file apps is to use your namespace instead of app
:
const bkjs = require("backendjs");
const api = bkjs.api;
const db = bkjs.db;
const mymod = {
name: "mymod",
args: [
{ name: "types", type: "list", descr: "Types allowed" },
{ name: "size", type: "int", descr: "Records in one page" },
],
tables: {
mytable: {
id: { type: "int", primary: 1 },
name: { primary: 2 },
type: { type: "list" },
descr: {}
}
}
};
exports.module = mymod;
bkjs.core.addModule(mymod);
mymod.configureWeb = function(options, callback)
{
api.app.all("/mymod", async function(req, res) {
if (!req.query.id) return api.sendReply(res, 400, "id is required");
req.query.type = mod.types;
const rows = await db.aselect("mymod", req.query, { ops: { type: "in" }, count: mod.size });
api.sendJSON(req, null, rows);
});
}
bkjs.server.start();
Except the app.configureWeb
and server.start()
all other functions are optional, they are here for the sake of completeness of the example. Also
because running the backend involves more than just running web server many things can be setup using the configuration options like common access permissions,
configuration of the cron jobs so the amount of code to be written to have fully functioning production API server is not that much, basically only
request endpoint callbacks must be provided in the application.
As with any Node.js application, node modules are the way to build and extend the functionality, backendjs does not restrict how the application is structured.
By default no system modules are loaded except bk_user
, it must be configured by the -preload-modules
config parameter to
preload modules from the backendjs/modules/.
Another way to add functionality to the backend is via external modules specific to the backend, these modules are loaded on startup from the backend
home subdirectory modules/
. The format is the same as for regular Node.js modules and only top level .js files are loaded on the backend startup.
Once loaded they have the same access to the backend as the rest of the code, the only difference is that they reside in the backend home and
can be shipped regardless of the npm, node modules and other env setup. These modules are exposed in the core.modules
the same way as all other core submodules
methods.
Let's assume the modules/
contains file facebook.js which implements custom FB logic:
const bkjs = require("backendjs");
const core = bkjs.core;
const mod = {
name: "facebook",
args: [
{ name: "token", descr: "API token" },
]
}
module.exports = mod;
mod.configureWeb = function(options, callback) {
...
}
mod.makeRequest = function(options, callback) {
core.sendRequest({ url: options.path, query: { access_token: fb.token } }, callback);
}
This is the main app code:
const bkjs = require("backendjs");
const core = bkjs.core;
// Using facebook module in the main app
api.app.get("/me", (req, res) => {
core.modules.facebook.makeRequest({ path: "/me" }, (err, data) => {
bkjs.api.sendJSON(req, err, data);
});
});
bkj.server.start();
In case different modules is better keep separately for maintenance or development purposes they can be split into
separate NPM packages, the structure is the same, modules must be in the modules/ folder and the package must be loadable
via require as usual. In most cases just empty index.js is enough. Such modules will not be loaded via require though but
by the backendjs core.loadModule
machinery, the NPM packages are just keep different module directories separate from each other.
The config parameter allow-packages
can be used to specify NPM package names to be loaded separated by comma, as with the default
application structure all subfolders inside each NPM package will be added to the core:
If there is a config file present as etc/config
it will be loaded as well, this way each package can maintain its default config parameters if necessary
without touching other or global configuration. Although such config files will not be reloaded on changes, when NPM installs or updates packages it
moves files around so watching the old config is no point because the updated config file will be different.
The backend support multiple databases and provides the same db layer for access. Common operations are supported and all other specific usage can be achieved by
using SQL directly or other query language supported by any particular database.
The database operations supported in the unified way provide simple actions like db.get, db.put, db.update, db.del, db.select
. The db.query
method provides generic
access to the database driver and executes given query directly by the db driver, it can be SQL or other driver specific query request.
Before the tables can be queried the schema must be defined and created, the backend db layer provides simple functions to do it:
db.describeTables({
album: {
id: { primary: 1 }, // Primary key for an album
name: { pub: 1 }, // Album name, public column
mtime: { type: "now" }, // Modification timestamp
},
photo: {
album_id: { primary: 1 }, // Combined primary key
id: { primary: 1 }, // consisting of album and photo id
name: { pub: 1, index: 1 }, // Photo name or description, public column with the index for faster search
mtime: { type: "now" }
}
});
Each database may restrict how the schema is defined and used, the db layer does not provide an artificial layer hiding all specifics, it just provides the same API and syntax, for example, DynamoDB tables must have only hash primary key or combined hash and range key, so when creating table to be used with DynamoDB, only one or two columns can be marked with primary property while for SQL databases the composite primary key can consist of more than 2 columns.
The backendjs always creates several tables in the configured database pools by default, these tables are required to support default API functionality and some
are required for backend operations. Refer below for the JavaScript modules documentation that described which tables are created by default. In the custom applications
the db.describeTables
method can modify columns in the default table and add more columns if needed.
For example, to make age and some other columns in the accounts table public and visible by other users with additional columns the following can be
done in the api.initApplication
method. It will extend the bk_user table and the application can use new columns the same way as the already existing columns.
Using the birthday column we make 'age' property automatically calculated and visible in the result, this is done by the internal method api.processAccountRow
which
is registered as post process callback for the bk_user table. The computed property age
will be returned because it is not present in the table definition
and all properties not defined and configured are passed as is.
The cleanup of the public columns is done by the api.sendJSON
which is used by all API routes when ready to send data back to the client. If any post-process
hooks are registered and return data itself then it is the hook responsibility to cleanup non-public columns.
db.describeTables({
bk_user: {
birthday: {},
ssn: {},
salary: { type: "int" },
occupation: {},
home_phone: {},
work_phone: {},
});
app.configureWeb = function(options, callback)
{
db.setProcessRow("post", "bk_user", this.processAccountRow);
...
callback();
}
app.processAccountRow = function(req, row, options)
{
if (row.birthday) row.age = Math.floor((Date.now() - core.toDate(row.birthday))/(86400000*365));
}
To define tables inside a module just provide a tables
property in the module object, it will be picked up by database initialization automatically.
const mod = {
name: "billing",
tables: {
invoices: {
id: { type: "int", primary: 1 },
name: {},
price: { type: "real" },
mtime: { type: "now" }
}
}
}
module.exports = mod;
// Run db setup once all the DB pools are configured, for example produce dynamic icon property
// for each record retrieved
mod.configureModule = function(options, callback)
{
db.setProcessRows("post", "invoices", function(req, row, opts) {
if (row.id) row.icon = "/images/" + row.id + ".png";
});
callback();
}
This is useful for easier naming conventions or switching to a different table name on the fly without changinbf the code, access to the table by it is real name is always available.
For example:
bksh -db-aliases-bk_user users
> await db.aget("bk_user", { login: "u1" })
> { login: "u1", name: "user", .... }
> await db.aget("users", { login: "u1" })
> { login: "u1", name: "user", .... }
All methods will put input parameters in the req.query
, GET or POST.
One way to verify input values is to use lib.toParams
, only specified parameters will be returned and converted according to
the type or ignored.
Example:
var params = {
test1: { id: { type: "text" },
count: { type: "int" },
email: { regexp: /^[^@]+@[^@]+$/ }
}
};
api.app.all("/endpoint/test1", function(req, res) {
const query = lib.toParams(req.query, params.test1);
if (typeof query == "string") return api.sendReply(res, 400, query);
...
});
Here is an example how to create simple TODO application using any database supported by the backend. It supports basic operations like add/update/delete a record, show all records.
Create a file named app.js
with the code below.
const bkjs = require('backendjs');
const api = bkjs.api;
const lib = bkjs.lib;
const app = bkjs.app;
const db = bkjs.db;
// Describe the table to store todo records
db.describeTables({
todo: {
id: { type: "uuid", primary: 1 }, // Store unique task id
due: {}, // Due date
name: {}, // Short task name
descr: {}, // Full description
mtime: { type: "now" } // Last update time in ms
}
});
// API routes
app.configureWeb = function(options, callback)
{
api.app.get(/^\/todo\/([a-z]+)$/, async function(req, res) {
var options = api.getOptions(req);
switch (req.params[0]) {
case "get":
if (!req.query.id) return api.sendReply(res, 400, "id is required");
const rows = await db.aget("todo", { id: req.query.id }, options);
api.sendJSON(req, null, rows);
break;
case "select":
options.noscan = 0; // Allow empty scan of the whole table if no query is given, disabled by default
const rows = await db.aselect("todo", req.query, options);
api.sendJSON(req, null, rows);
break;
case "add":
if (!req.query.name) return api.sendReply(res, 400, "name is required");
// By default due date is tomorrow
if (req.query.due) req.query.due = lib.toDate(req.query.due, Date.now() + 86400000).toISOString();
db.add("todo", req.query, options, (err, rows) => {
api.sendJSON(req, err, rows);
});
break;
case "update":
if (!req.query.id) return api.sendReply(res, 400, "id is required");
const rows = await db.aupdate("todo", req.query, options);
api.sendJSON(req, null, rows);
break;
case "del":
if (!req.query.id) return api.sendReply(res, 400, "id is required");
db.del("todo", { id: req.query.id }, options, (err, rows) => {
api.sendJSON(req, err, rows);
});
break;
}
});
callback();
}
bkjs.server.start();
Now run it with an option to allow API access without an account:
node app.js -log debug -web -api-allow-path /todo -db-create-tables
To use a different database, for example PostgresSQL(running localy) or DynamoDB(assuming EC2 instance), all config parametetrs can be stored in the etc/config as well
node app.js -log debug -web -api-allow-path /todo -db-pool dynamodb -db-dynamodb-pool default -db-create-tables
node app.js -log debug -web -api-allow-path /todo -db-pool pg -db-pg-pool default -db-create-tables
API commands can be executed in the browser or using curl
:
curl 'http://localhost:8000/todo?name=TestTask1&descr=Descr1&due=2015-01-01`
curl 'http://localhost:8000/todo/select'
When the backend server starts and no -home argument passed in the command line the backend makes its home environment in the ~/.bkjs
directory.
It is also possible to set the default home using BKJS_HOME environment variable.
The backend directory structure is the following:
etc
- configuration directory, all config files are there
etc/profile
- shell script loaded by the bkjs utility to customize env variables
etc/config
- config parameters, same as specified in the command line but without leading -, each config parameter per line:
Example:
debug=1
db-pool=dynamodb
db-dynamodb-pool=http://localhost:9000
<EMAIL_ADDRESS>To specify other config file: bkjs shell -config-file file
etc/config.local
- same as the config but for the cases when local environment is different than the production or for dev specific parameters
on startup the following local config files will be loaded if present: etc/config.runMode
and etc/config.instance.tag
. These will be loaded after the main config but before config.local. The runMode is set to development by default and can be changed with -run-mode
config parameter, the instance tag is set with -instance-tag
config parameter.
etc/crontab
- jobs to be run with intervals, JSON file with a list of cron jobs objects:
Example:
Create file in ~/.backend/etc/crontab with the following contents:
[ { "cron": "0 1 1 * * 1,3", "job": { "app.cleanSessions": { "interval": 3600000 } } } ]
Define the function that the cron will call with the options specified, callback must be called at the end, create this app.js file
var bkjs = require("backendjs");
bkjs.app.cleanSessions = function(options, callback) {
bkjs.db.delAll("session", { mtime: options.interval + Date.now() }, { ops: "le" }, callback);
}
bkjs.server.start()
Start the jobs queue and the web server at once
bkjs master -jobs-workers 1 -jobs-cron
etc/crontab.local - additional local crontab that is read after the main one, for local or dev environment
run-mode
and db-pool
config parameters can be configured in DNS as TXT records, the backend on startup will try to resolve such records and use the value if not empty.
All params that marked with DNS TXT can be configured in the DNS server for the domain where the backend is running, the config parameter name is
concatenated with the domain and queried for the TXT record, for example: run-mode
parameter will be queried for run-mode.domain.name for TXT record type.
modules
- loadable modules with specific functionality
images
- all images to be served by the API server, every subfolder represent naming space with lots of subfolders for images
var
- database files created by the server
tmp
- temporary files
web
- Web pages served by the static Express middleware
Database layer support caching of the responses using db.getCached
call, it retrieves exactly one record from the configured cache, if no record exists it
will pull it from the database and on success will store it in the cache before returning to the client. When dealing with cached records, there is a special option
that must be passed to all put/update/del database methods in order to clear local cache, so next time the record will be retrieved with new changes from the database
and refresh the cache, that is { cached: true }
can be passed in the options parameter for the db methods that may modify records with cached contents. In any case
it is required to clear cache manually there is db.clearCache
method for that.
Also there is a configuration option -db-caching
to make any table automatically cached for all requests.
If no cache is configured the local driver is used, it keeps the cache on the master process in the LRU pool and any worker or Web process
communicate with it via internal messaging provided by the cluster
module. This works only for a single server.
Set ipc-client=redis://HOST[:PORT]
that points to the server running Redis server.
The config option max_attempts
defines maximum number of times to reconnect before giving up. Any other node-redis
module parameter can be passed as well in
the options or url, the system supports special parameters that start with bk-
, it will extract them into options automatically.
For example:
ipc-client=redis://host1?bk-max_attempts=3
ipc-client-backup=redis://host2
ipc-client-backup-options-max_attempts=3
If configured all processes subscribe to it and listen for system messages, it must support PUB/SUB and does not need to be reliable. Websockets in the API server also use the system bus to send broadcasts between multiple api instances.
ipc-client-system=redis://
ipc-system-queue=system
To configure the backend to use Redis for job processing set ipc-queue=redis://HOST
where HOST is IP address or hostname of the single Redis server.
This driver implements reliable Redis queue, with visibilityTimeout
config option works similar to AWS SQS.
Once configured, then all calls to jobs.submitJob
will push jobs to be executed to the Redis queue, starting somewhere a backend master
process with -jobs-workers 2
will launch 2 worker processes which will start pulling jobs from the queue and execute.
The naming convention is that any function defined as function(options, callback)
can be used as a job to be executed in one of the worker processes.
An example of how to perform jobs in the API routes:
core.describeArgs('app', [
{ name: "queue", descr: "Queue for jobs" },
]);
app.queue = "somequeue";
app.processAccounts = function(options, callback) {
db.select("bk_user", { type: options.type || "user" }, (err, rows) => {
...
callback();
});
}
api.all("/process/accounts", function(req, res) {
jobs.submitJob({ job: { "app.processAccounts": { type: req.query.type } } }, { queueName: app.queue }, (err) => {
api.sendReply(res, err);
});
});
To use AWS SQS for job processing set ipc-queue=https://sqs.amazonaws.com....
, this queue system will poll SQS for new messages on a worker
and after successful execution will delete the message. For long running jobs it will automatically extend visibility timeout if it is configured.
The local queue is implemented on the master process as a list, communication is done via local sockets between the master and workers. This is intended for a single server development purposes only.
To use NATS (https://nats.io) configure a queue like ipc-queue-nats=nats://HOST:PORT, it supports broadcasts and job queues only, visibility timeout is supported as well.
To configure the backend to use RabbitMQ for messaging set ipc-queue=amqp://HOST
and optionally amqp-options=JSON
with options to the amqp module.
Additional objects from the config JSON are used for specific AMQP functions: { queueParams: {}, subscribeParams: {}, publishParams: {} }. These
will be passed to the corresponding AMQP methods: amqp.queue, amqp.queue.subcribe, amqp.publish
. See AMQP Node.js module for more info.
This is default setup of the backend when all API requests except must provide valid signature and all HTML, JavaScript, CSS and image files
are available to everyone. This mode assumes that Web development will be based on 'single-page' design when only data is requested from the Web server and all
rendering is done using JavaScript. This is how the examples/api/api.html
developers console is implemented, using JQuery-UI and Knockout.js.
To see current default config parameters run any of the following commands:
bkjs bkhelp | grep api-allow
node -e 'require("backendjs").core.showHelp()'
This is a mode when the whole Web site is secure by default, even access to the HTML files must be authenticated. In this mode the pages must defined 'Backend.session = true' during the initialization on every html page, it will enable Web sessions for the site and then no need to sign every API request.
The typical client JavaScript verification for the html page may look like this, it will redirect to login page if needed, this assumes the default path '/public' still allowed without the signature:
<link href="/css/bkjs.bundle.css" rel="stylesheet">
<script src="/js/bkjs.bundle.js" type="text/javascript"></script>
<script>
$(function () {
bkjs.session = true;
$(bkjs).on("bkjs.nologin", function() { window.location='/public/index.html'; });
bkjs.koInit();
});
</script>
On the backend side in your application app.js it needs more secure settings defined i.e. no html except /public will be accessible and
in case of error will be redirected to the login page by the server. Note, in the login page bkjs.session
must be set to true for all
html pages to work after login without singing every API request.
app.configureMiddleware = function(options, callback) {
this.allow.splice(this.allow.indexOf('^/$'), 1);
this.allow.splice(this.allow.indexOf('\\.html$'), 1);
callback();
}
api.registerPreProcess('', /^\/$|\.html$/, function(req, status, callback) {
if (status.status != 200) {
status.status = 302;
status.url = '/public/index.html';
}
callback(status);
});
The simplest way is to configure ws-port
to the same value as the HTTP port. This will run WebSockets server along the regular Web server.
In the browser the connection config is stored in the bkjs.wsconf
and by default it connects to the local server on port 8000.
There are two ways to send messages via Websockets to the server from a browser:
as urls, eg. bkjs.wsSend('/project/update?id=1&name=Test2')
In this case the url will be parsed and checked for access and authorization before letting it pass via Express routes. This method allows to share the same route handlers between HTTP and Websockets requests, the handlers will use the same code and all responses will be sent back, only in the Websockets case the response will arrived in the message listener (see an example below)
bkjs.wsConnect({ path: "/project/ws?id=1" });
$(bkjs).on("bkjs.ws.message", (msg) => {
switch (msg.op) {
case "/account/update":
bkjs.wsSend("/account/ws/account");
break;
case "/project/update":
for (const p in msg.project) app.project[p] = msg.project[p];
break;
case "/message/new":
bkjs.showAlert("info", `New message: ${msg.msg}`);
break;
}
});
as JSON objects, eg. bkjs.wsSend({ op: "/project/update", project: { id: 1, name: "Test2" } })
In this case the server still have to check for access so it treats all JSON messages as coming from the path which was used during the connect,
i.e. the one stored in the bkjs.wsconf.path
. The Express route handler for this path will receive all messages from Websocket clients, the response will be
received in the event listener the same way as for the first use case.
// Notify all clients who is using the project being updated
api.app.all("/project/ws", (req, res) => {
switch (req.query.op) {
case "/project/update":
// some code ....
api.wsNotify({ query: { id: req.query.project.id } }, { op: "/project/update", project: req.query.project });
break;
}
res.send("");
});
In any case all Websocket messages sent from the server will arrive in the event handler and must be formatted properly in order to distinguish what is what, this is
the application logic. If the server needs to send a message to all or some specific clients for example due to some updates in the DB, it must use the
api.wsNotify
function.
// Received a new message for a user from external API service, notify all websocket clients by account id
api.app.post("/api/message", (req, res) => {
....
... processing logic
....
api.wsNotify({ account_id: req.query.uid }, { op: "/message/new", msg: req.query.msg });
});
There is no ready to use support for different versions of API because there is no just one solution that satisfies all applications. But there are tools ready to use that will allow to implement such versioning system in the backend. Some examples are provided below:
Fixed versions
This is similar to AWS version system when versions are fixed and changed not very often. A client can specify the core version
using bk-version
header. When a request is parsed and the version is provided it will be set in the request options object as apiVersion
.
All API routes are defined using Express middleware and one of the possible ways of dealing with different versions can look like this, by appending version to the command it is very simple to call only changed API code.
api.all(/\/domain\/(get|put|del)/, function(req, res) {
var options = api.getOptions(req);
var cmd = req.params[0];
if (options.apiVersion) cmd += "/" + options.apiVersion;
switch (cmd) {
case "get":
break;
case "get/2015-01-01":
break;
case "put":
break;
case "put/2015-02-01":
break;
case "del"
break;
}
});
Application semver support For cases when applications support Semver kind of versioning and it may be too many releases the method above still can be used while the number of versions is small, once too many different versions with different minor/patch numbers, it is easier to support greater/less comparisons.
The application version bk-app
can be supplied in the query or as a header or in the user-agent HTTP header which is the easiest case for mobile apps.
In the middlware, the code can look like this:
var options = api.getOptions(req);
var version = lib.toVersion(options.appVersion);
switch (req.params[0]) {
case "get":
if (version < lib.toVersion("1.2.5")) {
res.json({ id: 1, name: "name", description: "descr" });
break;
}
if (version < lib.toVersion("1.1")) {
res.json([id, name]);
break;
}
res.json({ id: 1, name: "name", descr: "descr" });
break;
}
The actual implementation can be modularized, split into functions, controllers.... there are no restrictions how to build the working backend code, the backend just provides all necessary information for the middleware modules.
The purpose of the bkjs
shell script is to act as a helper tool in configuring and managing the backend environment
and as well to be used in operations on production systems. It is not required for the backend operations and provided as a convenience tool
which is used in the backend development and can be useful for others running or testing the backend.
Running without arguments will bring help screen with description of all available commands.
The tool is multi-command utility where the first argument is the command to be executed with optional additional arguments if needed. On Linux, when started the bkjs tries to load and source the following config files:
/etc/default/bkjs
/etc/sysconfig/bkjs
$BKJS_HOME/etc/profile
Any of the following config files can redefine any environment variable thus pointing to the correct backend environment directory or customize the running environment, these should be regular shell scripts using bash syntax.
Most common used commands are:
bkjs watch - run the backend or the app for development purposes, uses local app.js if exists otherwise runs generic server
bkjs shell - start REPL shell with the backend module loaded and available for use, all submodules are available in the shell as well like core, db, api
bkjs sync [-path path] [-host host] [-user user] - sync sources of the app with the remote site, this is for development version of the backend only
bkjs init-server [-home path] [-user user] [-host name] [-domain name] - initialize Linux instance(Amazon) for backend use, optional -home can be specified where the backend home will be instead of ~/.bkjs, optional -user tells to use existing user instead of the current user and not root.
This command will create /etc/sysconfig/bkjs
file with BKJS_HOME set to the home of the
backendjs app which was passed in the command line. This makes the bkjs or bksh run globally regardless of the current directory.
Then run the dev build script to produce web/js/bkjs.bundle.js and web/css/bkjs.bundle.css
cd node_modules/backendjs && npm run devbuild
Now instead of including a bunch of .js or css files in the html pages it only needs /js/bkjs.bundle.js and /css/bkjs.bundle.css. The configuration is in the package.json file.
The list of files to be used in bundles is in the package.json under config.bundles
.
To enable auto bundler in your project just add to the local config ~/.bkjs/etc/config.local
a list of directories to be
watched for changes. For example adding these lines to the local config will enable the watcher and bundle support
watch-web=web/js,web/css,$HOME/src/js,$HOME/src/css
watch-ignore=.bundle.(js|css)$
build-web=bkjs web-bundle -dev
The simple script below allows to build the bundle and refresh Chrome tab automatically, saves several clicks:
#!/bin/bash
bkjs web-bundle -dev -file $2
[ "$?" != "0" ] && exit
osascript -e "tell application \"Google Chrome\" to reload (tabs of window 1 whose URL contains \"$1\")"
To use it call this script instead in the config.local:
build-web=web-bundle.sh /website
NOTE: Because the rebuild happens while the watcher is running there are cases like the server is restarting or pulling a large update from the repository when the bundle build may not be called or called too early. To force rebuild run the command:
bkjs web-bundle -dev -all -force
start new AWS instance via AWS console, use Amazon Linux
login as ec2-user
install commands
git clone https://github.com/vseryakov/backendjs.git
sudo backendjs/bkjs install-ec2 -tools $(pwd)/backendjs/tools
run ps agx
, it should show several backend processes running after monit started the service
try to access the instance via HTTP port for the API console or documentation
NOTE: if running behind a Load balancer and actual IP address is needed set Express option in the command line -api-express-options {"trust%20proxy":1}
. In the config file
replacing spaces with %20 is not required.
Note: on OS X laptop the -aws-sdk-profile uc
when AWS credentials are in the ~/.aws/credentials.
On the running machine which will be used for an image:
bksh -aws-create-image -no-reboot
Use an instance by tag for an image:
bksh -aws-create-image -no-reboot -instance-id `bkjs ec2-show -tag api -fmt id | head -1`
When launching from an EC2 instance no need to specify any AWS credentials.
admin (EC2)
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type t2.small -subnet-name api -name admin -elb-name Admin -alarm-name alarms -public-ip 1 -dry-run
api (EC2)
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type m3.large -subnet-name api -name api -elb-name api -alarm-name alarms -public-ip 1 -dry-run
jobs (EC2)
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type t2.small -subnet-name internal -name sync -alarm-name alarms -dry-run
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type t2.small -subnet-name internal -name sync -zone 1c -alarm-name alarms -dry-run
Elasticsearch
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type m3.large -subnet-name internal -name elasticsearch -bkjs-cmd stop-service -bkjs-cmd "init-elasticsearch-service -memsize 50" -alarm-name alarms -public-ip 1 -dry-run
Redis
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type m3.large -subnet-name internal -name redis -bkjs-cmd stop-service -bkjs-cmd "init-redis-service -memsize 70" -alarm-name alarms -public-ip 1 -dry-run
bksh -aws-create-launch-template-version -name jobs -aws-sdk-profile uc -dry-run
bksh -aws-create-launch-template-version -name api -aws-sdk-profile uc -dry-run
bksh -aws-set-route53 -name elasticsearch.ec-internal -filter elasticsearch
By default the Web proceses spawned by the server are load balanced using default cluster module which relies on the OS to do scheduling. On Linux with node 0.10 this is proven not to work properly due to the kernel keeping the context switches to a minimum thus resulting in one process to be very busy while the others idle. Node versions 4 and above perform round-robin by default.
For such case the Backendjs implements the proxy mode by setting proxy-port
config parameter to any number above 1000, this will be the initial
port for the web processes to listen for incoming requests, for example if use -proxy-port 3000
and launch 2 web processes they will listen on ports
3000 and 3001. The main server process will start internal HTTP proxy and will perform round-robin load balancing the incoming requests between the web processes by forwarding
them to the web processes over TCP and then returning the responses back to the clients.
The first thing when deploying the backend into production is to change API HTTP port, by default is is 8000, but we would want port 80 so regardless how the environment is setup it is ultimately 2 ways to specify the port for HTTP server to use:
config file
The config file is always located in the etc/ folder in the backend home directory, how the home is specified depends on the system but basically it can be
defined via command line arguments as -home
or via environment variables when using bkjs. See bkjs documentation but on AWS instances created with bkjs
init-server
command, for non-standard home use /etc/sysconfig/bkjs
profile, specify BKJS_HOME=/home/backend
there and the rest will be taken care of
command line arguments
When running node scripts which use the backend, just specify -home
command line argument with the directory where your backend should be and the backend will use it
Example:
node app.js -home $HOME -port 80
config database
If -db-config
is specified in the command line or db-config=
in the local config file, this will trigger loading additional
config parameters from the specified database pool, it will load all records from the bk_config
table on that db pool. Using the database to store
configuration make it easier to maintain dynamic environment for example in case of auto scaling or launching on demand, this way
a new instance will query current config from the database and this eliminates supporting text files and distributing them to all instances.
The config database is refreshed from time to time acording to the db-config-interval
parameter, also all records with ttl
property in the bk_config
will be pulled every ttl interval and updated in place.
DNS records
Some config options may be kept in the DNS TXT records and every time a instance is started it will query the local DNS for such parameters. Only a small subset of
all config parameters support DNS store. To see which parameters can be stored in the DNS run bkjs show-help
and look for 'DNS TXT configurable'.
git clone https://github.com/vseryakov/backendjs.git
or git clone git@github.com:vseryakov/backendjs.git
cd backendjs
if Node.js is already installed skip to the next section
to install binary release run the command, it will install it into /opt/local on Darwin
bkjs install-node
bkjs install-node -prefix /usr/local/node
Important: Add NODE_PATH=$BKJS_PREFIX/lib/node_modules to your environment in .profile or .bash_profile so node can find global modules, replace $BKJS_PREFIX with the actual path unless this variable is also set in the .profile
to install all dependencies and make backendjs module and bkjs globally available:
npm link backendjs
to run local server on port 8000 run command:
bkjs web
to start the backend in command line mode, the backend environment is prepared and initialized including all database pools. This command line access allows you to test and run all functions from all modules of the backend without running full server similar to Node.js REPL functionality. All modules are accessible from the command line.
$ ./bkjs shell
> core.version
'0.70.0'
> logger.setLevel('info')
Included a simple testing tool, it is used for internal bkjs testing but can be used for other applications as well.
The convention is to create a test file in the tests/ folder, each test file can define one or more test
functions named in the form tests.test_NAME
where NAME is any custom name for the test, for example:
File tests/example.js
:
tests.test_example = function(callback)
{
expect(1 == 2, "expect 1 eq 2")
}
Then to run all tests
bkjs run-tests
More details are in the documentation or doc.html
While creating Backendjs there were many questions and issues to be considered, some I was able to implement, some still not. Below are the thoughts that might be useful when designing, developing or choosing the API platform:
All API endpoints are optional and can be disabled or replaced easily. By default the naming convention is:
/namespace/command[/subname[/subcommand]]
Any HTTP methods can be used because its the command in the URL that defines the operation. The payload can be url-encoded query parameters or JSON or any other format supported by any particular endpoint. This makes the backend universal and usable with any environment, not just a Web browser. Request signature can be passed in the query so it does not require HTTP headers at all.
All requests to the API server must be signed with account login/secret pair.
The resulting signature is sent as HTTP header bk-signature
or in the header specified by the api-signature-name
config parameter.
For JSON content type, the method must be POST and no query parameters specified, instead everything should be inside the JSON object which is placed in the body of the request. For additional safety, SHA1 checksum of the JSON payload can be calculated and passed in the signature, this is the only way to ensure the body is not modified when not using query parameters.
See web/js/bkjs.js function bkjs.createSignature
or
api.js function api.createSignature
for the JavaScript implementations.
There is also native iOS implementation Bkjs.m.
/auth
This API request returns the current user record from the bk_user
table if the request is verified and the signature provided
is valid. If no signature or it is invalid the result will be an error with the corresponding error code and message.
By default this endpoint is secured, i.e. requires a valid signature.
Parameters:
_session=1
- if the call is authenticated a cookie with the session signature is returned, from now on
all requests with such cookie will be authenticated, the primary use for this is Web apps/login
Same as the /auth but it uses secret for user authentication, this request does not need a signature, just simple login and secret query parameters to be sent to the backend. This must be sent over SSL.
Parameters:
login
- account loginsecret
- account secret_session=1
- same as in /auth requestOn successful login, the result contains full account record including the secret, this is the only time when the secret is returned back
Example:
$.ajax({ url: "/login?login=test123&secret=test123&_session=1",
success: function(json, status, xhr) { console.log(json) }
});
> { id: "XXXX...", name: "Test User", login: "test123", ...}
/logout
Logout the current user, clear session cookies if exist. For pure API access with the signature this will not do anything on the backend side.
The accounts API manages accounts and authentication, it provides basic user account features with common fields like email, name, address.
/account/get
Returns information about the current account, all account columns are returned except the secret and other table columns with the property priv
Response:
{ "id": "57d07a4e28fc4f33bdca9f6c8e04d6c3",
"name": "Test User",
"mtime":<PHONE_NUMBER>,
"login": "testuser",
"type": ["user"],
}
How to make an account as admin
# Run backend shell
bkjs shell
# Update record by login
> db.update("bk_user", { login: 'login@name', type: 'admin' });
/account/update
Update current account with new values, the parameters are columns of the table bk_user
, only columns with non empty values will be updated.
Example:
/account/update?name=New%2BName
When running with AWS load balancer there should be a url that a load balancer polls all the time and this must be very quick and lightweight request. For this
purpose there is an API endpoint /ping
that just responds with status 200. It is open by default in the default api-allow-path
config parameter.
The data API is a generic way to access any table in the database with common operations, as oppose to the any specific APIs above this API only deals with one table and one record without maintaining any other features like auto counters, cache...
Because it exposes the whole database to anybody who has a login it is a good idea to disable this endpoint in the production or provide access callback that verifies who can access it.
To disable this endpoint completely in the config: deny-modules=bk_data
To allow admins to access it only in the config: api-allow-admin=^/data
To allow admins to access it only:
api.registerPreProcess('GET', '/data', function(req, status, cb) { if (req.account.type != "admin") return cb({ status: 401, message: 'access denied' }; cb(status)); });
This is implemented by the data
module from the core.
/data/columns
/data/columns/TABLE
Return columns for all tables or the specific TABLE
/data/keys/TABLE
Return primary keys for the given TABLE
/data/(select|search|list|get|add|put|update|del|incr|replace)/TABLE
Perform database operation on the given TABLE, all options for the db
functiobns are passed as query parametrrs prepended with underscore,
regular parameters are the table columns.
By default the API does not allow table scans without a condition to avoid expensive and long queries, to enable a scan pass _noscan=0
.
For this to work the Data API must be configured as unsecure in the config file using the parameter api-unsecure=data
.
Some tables like messages and connections perform data convertion before returning the results, mostly splitting combined columns like type into
separate fields. To return raw data pass the parameter _noprocessrows=1
.
Example:
/data/get/bk_user?login=12345
/data/update/bk_user?login=12345&name=Admin
/data/select/bk_user?name=john&_ops=name,gt&_select=name,email
/data/select/bk_user?_noscan=0&_noprocessrows=1
The system API returns information about the backend statistics, allows provisioning and configuration commands and other internal maintenance functions. By default is is open for access to all users but same security considerations apply here as for the Data API.
This is implemented by the system
module from the core. To enable this functionality specify -preload-modules=bk_system
.
/system/restart
Perform restart of the Web processes, this will be done gracefully, only one Web worker process will be restarting while the other processes will keep
serving requests. The intention is to allow code updates on live systems without service interruption.
/system/cache/(init|stats|keys|get|set|put|incr|del|clear)
Access to the caching functions
/system/config/(init)
Access to the config functions
/system/msg/(init|send)
Access to the messaging functions
/system/jobs/(send)
Access to the jobs functions
/system/queue/(init|publish)
Access to the queue functions
/system/params/get
Return all config parameters applied from the config file(s) or remote database.
/system/stats/get
Database pool statistics and other diagnostics
Individual sub-objects:
Vlad Seryakov
Check out the Documentation for more details.
HTTP API to the server from the clients, this module implements the basic HTTP(S) API functionality with some common features.  How many processes to spawn can be configured via -server-max-workers
config parameter.
When an HTTP request arrives it goes over Express middleware, but before processing any registered routes there are several steps performed:
req
object which is by convention is a Request object, assigned with common backend properties to be used later:bk_user
table will be setapi.sendJSON
method, registered post process callbacks will be called for such responseConfig parameters
api-images-url
, descr: "URL where images are stored, for cases of central image server(s), must be full URL with optional path"api-images-s3
, descr: "S3 bucket name where to store and retrieve images"api-images-raw
, type: "bool", descr: "Return raw urls for the images, requires images-url to be configured. The path will reflect the actual 2 level structure and account id in the image name"api-images-s3-options
, type: "json", logger: "warn", descr: "S3 options to sign images urls, may have expires:, key:, secret: properties"api-images-ext
, descr: "Default image extension to use when saving images"api-images-mod
, descr: "Images scaling module, sharp or default bkjs-wand"api-files-raw
, type: "bool", descr: "Return raw urls for the files, requires files-url to be configured. The path will reflect the actual 2 level structure and account id in the file name"api-files-url
, descr: "URL where files are stored, for cases of central file server(s), must be full URL with optional path"api-files-s3
, descr: "S3 bucket name where to store files uploaded with the File API"api-max-request-queue
, type: "number", min: 0, descr: "Max number of requests in the processing queue, if exceeds this value server returns too busy error"api-no-access-log
, type: "bool", descr: "Disable access logging in both file or syslog"api-access-log-file
, descr: "File for access logging"api-access-log-level
, type: "int", descr: "Syslog level priority, default is local5.info, 21 * 8 + 6"api-access-log-fields
, array: 1, type: "list", descr: "Additional fields from the request or account to put in the access log, prefix defines where the field is lcoated: q: - query, h: - headers, a: - account otherwise from the request, Example: -api-log-fields h:Referer,a:name,q:action"api-salt
, descr: "Salt to be used for scrambling credentials or other hashing activities"api-qs-options-(.+)
, autotype: 1, obj: "qsOptions", strip: "qs-options-", nocamel: 1, descr: "Options to pass to qs.parse: depth, arrayLimit, allowDots, comma, plainObjects, allowPrototypes, parseArrays"api-no-static
, type: "bool", descr: "Disable static files from /web folder, no .js or .html files will be served by the server"api-static-options-(.+)
, autotype: 1, obj: "staticOptions", strip: "static-options-", nocamel: 1, descr: "Options to pass to serve-static module: maxAge, dotfiles, etag, redirect, fallthrough, extensions, index, lastModified"api-vhost-path-([^/]+)
, type: "regexp", obj: "vhostPath", nocamel: 1, strip: "vhost-path-", regexp: "i", descr: "Define a virtual host regexp to be matched against the hostname header to serve static content from a different root, a vhost path must be inside the web directory, if the regexp starts with !, that means negative match, example: api-vhost-path-test_dir=test.com$"api-no-vhost-path
, type: "regexpobj", descr: "Add to the list of URL paths that should be served for all virtual hosts"api-templating
, descr: "Templating engine to use, see consolidate.js for supported engines, the 'consolidate' package must be installed to use this"api-no-session
, type: "bool", descr: "Disable cookie session support, all requests must be signed for Web clients"api-session-age
, type: "int", min: 0, descr: "Session age in milliseconds, for cookie based authentication"api-session-domain-(.+)
, type: "regexp", obj: "session-domain", nocamel: 1, regexp: "i", descr: "Cookie domain by Host: header, if not matched session is bound to the exact host only, example: -api-session-domain-site.com=site.com$"api-session-same-site
, descr: "Session SameSite option, for cookie based authentication"api-session-cache
, descr: "Cache name for session control"api-query-token-secret
, descr: "Name of the property to be used for encrypting tokens for pagination or other sensitive data, any property from bk_user can be used, if empty no secret is used, if not a valid property then it is used as the secret"api-app-header-name
, descr: "Name for the app name/version query parameter or header, it is can be used to tell the server about the application version"api-version-header-name
, descr: "Name for the access version query parameter or header, this is the core protocol version that can be sent to specify which core functionality a client expects"api-no-cache-files
, type: "regexpobj", descr: "Set cache-control=no-cache header for matching static files",api-tz-header-name
, descr: "Name for the timezone offset header a client can send for time sensitive requests, the backend decides how to treat this offset"api-signature-header-name
, descr: "Name for the access signature query parameter, header and session cookie"api-lang-header-name
, descr: "Name for the language query parameter, header and session cookie, primary language for a client"api-signature-age
, type: "int", descr: "Max age for request signature in milliseconds, how old the API signature can be to be considered valid, the 'expires' field in the signature must be less than current time plus this age, this is to support time drifts"api-access-time-interval
, type: "int", min: 0, descr: "Intervals to refresh last access time for accounts, only updates the cache if bk_user
is configured to be cached"api-access-token-secret
, descr: "A generic secret to be used for API access or signatures"api-allow-authenticated
, type: "regexpobj", descr: "Add URLs which can be accessed by any authenticated user account, can be partial urls or Regexp, it is checked before any other account types, if matched then no account specific paths will be checked anymore(any of the allow-account-...)"api-allow-acl-authenticated
, type: "list", descr: "Combine regexps from the specified acls for the check explained by -api-allow-authenticated
parameter"api-allow-admin
, type: "regexpobj", descr: "Add URLs which can be accessed by admin accounts only, can be partial urls or Regexp"api-allow-acl-admin
, type: "list", descr: "Combine regexps from the specified acls for the check explained by -api-allow-admin
parameter"api-allow-account-([a-z0-9_]+)
, type: "regexpobj", obj: "allow-account", descr: "Add URLs which can be accessed by specific account type, can be partial urls or Regexp"api-allow-acl-([a-z0-9_]+)
, type: "rlist", obj: "allow-acl", descr: "Combine regexps from the specified acls for allow checks for the specified account type"api-only-account-([a-z0-9_,]+)
, type: "regexpobj", obj: "only-account", descr: "Add URLs which can be accessed by specific account type only, can be partial urls or Regexp"api-only-acl-([a-z0-9_,]+)
, type: "rlist", obj: "only-acl", descr: "Combine regexps from the specified acls allowed for the specified account type only"api-deny-authenticated
, type: "regexpobj", descr: "Add URLs which CAN NOT be accessed by any authenticated user account, can be partial urls or Regexp, it is checked before any other account types, if matched then no account specific paths will be checked anymore(any of the deny-account-...)"api-deny-acl-authenticated
, type: "list", descr: "Combine regexps from the specified acls for the check explained by -api-deny-authenticated
parameter"api-deny-account-([a-z0-9_]+)
, type: "regexpobj", obj: "deny-account", descr: "Add URLs which CAN NOT be accessed by specific account type, can be partial urls or Regexp, this is checked before any allow parameters"api-deny-acl-([a-z0-9_]+)
, type: "list", obj: "deny-acl", descr: "Combine regexps from the specified acls for deny checks for the specified account type"api-acl-([a-z0-9_]+)
, type: "regexpobj", obj: "acl", descr: "Add URLs to the named ACL which can be used in allow/deny rules per account"api-allow
, type: "regexpobj", set: 1, descr: "Regexp for URLs that dont need credentials, replaces the whole access list"api-allow-path
, type: "regexpobj", key: "allow", descr: "Add to the list of allowed URL paths without authentication, adds to the -api-allow
parameter"api-allow-acl
, type: "list", descr: "Combine regexps from the specified acls for the check explained by -api-allow
parameter"api-deny
, type: "regexpobj", set: 1, descr: "Regexp for URLs that will be denied access, replaces the whole access list"api-deny-path
, type: "regexpobj", key: "deny", descr: "Add to the list of URL paths to be denied without authentication, adds to the -api-deny
parameter"api-deny-acl
, type: "list", descr: "Combine regexps from the specified acls for the check explained by -api-deny
parameter"api-allow-anonymous
, type: "regexpobj", descr: "Add to the list of allowed URL paths that can be served with or without valid account, the difference with -api-allow-path
is that it will check for signature and an account but will continue if no login is provided, return error in case of wrong account or not account found"api-allow-acl-anonymous
, type: "list", descr: "Combine regexps from the specified acls for the check explained by -allow-anonymous
parameter"api-allow-empty
, type: "regexpobj", descr: "Regexp for URLs that should return empty responses if not found, for example return nothing for non-existent javascript files or css files"api-ignore-allow
, type: "regexpobj", descr: "Regexp for URLs that should be ignored by the allow rules, the processing will continue"api-ignore-allow-path
, type: "regexpobj", key: "ignore-allow", descr: "Add to the list of URL paths which should be ignored by the allow rules, in order to keep allow/deny rules simple, for example to keep some js files from open to all: -allow-path \.js -ignore-allow-path /secure/"api-ignore-allow-acl
, type: "list", descr: "Combine regexps from the specified acls for the check explained by -ignore-allow-path
parameter"api-allow-ip
, type: "regexpobj", descr: "Add to the list of regexps for IPs that only allowed access from. It is checked before endpoint access list"api-deny-ip
, type: "regexpobj", descr: "Add to the list of regexps for IPs that will be denied access. It is checked before endpoint access list."api-allow-ssl
, type: "regexpobj", descr: "Add to the list of allowed locations using HTTPs only, plain HTTP requests to these urls will be refused"api-ignore-ssl
, type: "regexpobj", descr: "Allow plain HTTP from matched IP addresss or locations"api-redirect-ssl
, type: "regexpobj", descr: "Add to the list of the locations to be redirected to the same path but using HTTPS protocol, for proxy mode the proxy server will perform redirects"api-express-options
, type: "json", logger: "warn", descr: "Set Express config options during initialization,example: -api-express-options { \"trust proxy\": 1, \"strict routing\": true }
"api-mime-body
, type: "regexpobj", descr: "Collect full request body in the req.body property for the given MIME type in addition to json and form posts, this is for custom body processing"api-mime-ignore
, type: "regexpobj", descr: "Ignore the body for the following MIME content types, request body will not be parsed at all"api-mime-map-(.+)
, obj: "mime-map", descr: "File extension to MIME content type mapping, this is used by static-serve, example: -api-mime-map-mobileconfig application/x-apple-aspen-config"api-ignore-content-type
, type: "regexpobj", descr: "Ignore the content type for the following endpoint paths, keep the body unparsed"api-platform-match
, type: "regexpmap", regexp: "i", descr: "An JSON object with list of regexps to match user-agent header for platform detection, example: { 'ios|iphone|ipad': 'ios', 'android': 'android' }"api-cors-origin
, descr: "Origin header for CORS requests"api-cors-allow
, type: "regexpobj", descr: "Enable CORS requests if a request host/path matches the given regexp"api-server-header
, descr: "Custom Server: header to return for all requests"api-error-message
, descr: "Default error message to return in case of exceptions"api-restart
, descr: "On address in use error condition restart the specified servers, this assumes an external monitor like monit to handle restarts"api-allow-error-code
, type: "regexpobj", descr: "Error codes in exceptions to return in the response to the user, if not matched the error-message will be returned"api-rlimits-(rate|max|interval|ttl|ip)-(.+)
, type: "int", obj: "rlimitsMap.$2", make: "$1", descr: "Rate limiter parameters for Token Bucket algorithm. ttl
is to expire cache entries, ip
is to limit by IP address as well, ex. -api-rlimits-ip-ip=10, -api-rlimits-rate-/path=1"api-rlimits-map-(.+)
, type: "map", obj: "rlimitsMap", make: "$1", maptype: "int", descr: "Rate limiter parameters for Token Bucket algorithm. set all at once, ex. -api-rlimits-map-/url=rate:1,interval:2000"api-rlimits-queue-(.+)
, obj: "rlimitsMap.$1", make: "queue", descr: "Queue to use for the given path"api-rlimits-(interval|ttl)
, type: "int", obj: "rlimits", make: "$1", descr: "Default rate limiter parameters, default interval is 1s, ttl
is to expire old cache entries"api-rlimits-message
, obj: "rlimits", descr: "Message to show when any limits reached"api-exit-on-error
, type: "bool", descr: "Exit on uncaught exception in the route handler"api-timeout
, type: "number", min: 0, max: 3600000, descr: "HTTP request idle timeout for servers in ms, how long to keep the connection socket open, this does not affect Long Poll requests"api-keep-alive-timeout
, type: "int", descr: "Number of milliseconds to keep the HTTP conection alive"api-request-timeout
, type: "int", min: 0, descr: "Number of milliseconds to receive the entire request from the client"api-max-requests-per-socket
, type: "int", min: 0, descr: "The maximum number of requests a socket can handle before closing keep alive connection"api-upload-limit
, type: "number", descr: "Max size for uploads, bytes"api-query-limit
, type: "number", descr: "Max size for posts and queries, bytes"api-limiter-queue
, descr: "Name of an ipc queue for API rate limiting"api-errlog-limiter-max
, type: "int", descr: "How many error messages to put in the log before throttling kicks in"api-errlog-limiter-interval
, type: "int", descr: "Interval for error log limiter, max errors per this interval"api-errlog-limiter-ignore
, type: "regexpobj", descr: "Do not show errors that match the regexp"api-proxy-reverse
, type: "url", descr: "A Web server where to proxy requests not macthed by the url patterns or host header, in the form: http://host[:port]"api-proxy-url-(.+)
, type: "regexpobj", reverse: 1, nocamel: 1, obj: 'proxy-url', descr: "URL regexp to be passed to other web server running behind, each parameter defines an url regexp and the destination in the value in the form http://host[:port], example: -api-proxy-url-^/api http://<IP_ADDRESS>:8080"api-proxy-host-(.+)
, type: "regexpobj", reverse: 1, obj: 'proxy-host', lower: /.+/, descr: "Virtual host mapping, to match any Host: header, each parameter defines a host name and the destination in the value in the form http://host[:port], example: -api-proxy-host-www.myhost.com http://<IP_ADDRESS>:8080"api-routing-(.+)
, type: "regexpobj", reverse: 1, nocamel: 1, obj: 'routing', descr: "Locations to be re-routed to other path, this is done inside the server at the beginning, only the path is replaced, same format and placeholders as in redirect-url, use ! in front of regexp to remove particular redirect from the list, example: -api-routing-^/account/get /acount/read"api-ignore-routing
, type: "regexpobj", descr: "Ignore locations from the routing"api-auth-routing-(.+)
, type: "regexpobj", reverse: 1, nocamel: 1, obj: 'auth-routing', descr: "URL path to be re-routed to other path after the authentication is successful, this is done inside the server, only the path is replaced, same format and placeholders as in redirect-url, example: -api-routing-auth-^/account/get /acount/read"api-redirect-url
, type: "regexpmap", descr: "Add to the list a JSON object with property name defining a location regexp to be matched early against in order to redirect using the value of the property, if the regexp starts with !, that means it must be removed from the list, variables can be used for substitution: @HOST@, @PATH@, @URL@, @BASE@, @DIR@, @QUERY@, status code can be prepended to the location, example: { '^[^/]+/path/$': '/path2/index.html', '.+/$': '301<EMAIL_ADDRESS>} "api-login-redirect-(.+)
, type: "regexpobj", reverse: 1, nocamel: 1, obj: "login-redirect", descr: "Define a location where to redirect if no login is provided, same format and placeholders as in redirect-url, example: api-login-redirect-^/admin/=/login.html"api-auth-status
, type: "int", descr: "Default authenticated status, if no auth rules matched but valid signature this is the status returned"api-auth-message:
, descr: "Default authenticated message to be returned with default auth status"api-reset-acl
, type: "callback", callback: function(v) { if (v) this.resetAcl() descr: "Reset all ACL related rules and permissions"api-response-headers
, type: "regexpmap", json: 1, descr: "An JSON object with list of regexps to match against the location and set response headers defined as a ist of pairs name, value..., -api-response-headers={ "^/": ["x-frame-options","sameorigin","x-xss-protection","1; mode=block"] }"api-cleanup-rules-(.+)
, obj: "cleanupRules.$1", type: "map", datatype: "auto", merge: 1, nocamel: 1, descr: "Rules for the cleanupResult per table, ex. api-cleanup-rules-bk_user=email:0,phone:1"api-cleanup-strict
, type: "bool", descr: "Default mode for cleanup results"api-request-cleanup
, type: "list", array: 1, descr: "List of fields to explicitely cleanup on request end"api-query-defaults-([a-z0-9_]+)-(.+)
, obj: "queryDefaults.$2", make: "$1", autotype: 1, descr: "Global query defaults for getQuery, can be path specific, ex. -api-query-defaults-max-name 128 -api-query-defaults-max-/endpoint-name 255"api-csrf-set-path
, type: "regexpobj", descr: "Regexp for URLs to set CSRF token for all methods"api-csrf-check-path
, type: "regexpobj", descr: "Regexp for URLs to set CSRF token for skip methods and verify for others"api-csrf-skip-method
, type: "regexp", descr: "Do not check for CSRF token for specified methods"api-csrf-skip-status
, type: "regexp", descr: "Do not return CSRF token for specified status codes"api-csrf-header-name
, descr: "Name for the CSRF header"api-csrf-age
, type: "int", min: 0, descr: "CSRF token age in milliseconds"api-err-(.+)
, descr: "Error messages for various cases"api-compressed-([^/]+)
, type: "regexp", obj: "compressed", nocamel: 1, strip: "compressed-", reverse: 1, regexp: "i", descr: "Match static paths to be returned compressed, files must exist and be pre-compressed with the given extention , example: -api-compress-bundle.js gz"api.init(options, callback)
Initialize API layer, this must be called before the api
module can be used but it is called by the server module automatically so api.init
is
rearely need to called directly, only for new server implementation or if using in the shell for testing.
During the init sequence, this function calls api.initMiddleware
and api.initApplication
methods which by default are empty but can be redefined in the user aplications.
The bkjs.js uses its own request parser that places query parameters into req.query
or req.body
depending on the method.
For GET method, req.query
contains all url-encoded parameters, for POST method req.body
contains url-encoded parameters or parsed JSON payload or multipart payload.
The reason not to do this by default is that this may not be the alwayse wanted case and distinguishing data coming in the request or in the body may be desirable,
also, this will needed only for Express handlers .all
, when registering handler by method like .get
or .post
then the handler needs to deal with only either source of the request data.
api.shutdown(callback)
Gracefully close all connections, call the callback after that
api.shutdownWeb(options, callback)
Gracefully close all database pools when the shutdown is initiated by a Web process
api.configureStatic()
Templating and static paths
api.configureAccessLog()
Setup access log stream
api.handleServerRequest(req, res)
Start Express middleware processing wrapped in the node domain
api.prepareRequest(req)
Prepare request options that the API routes will merge with, can be used by pre process hooks, initialize required properties for subsequent use
api.prepareOptions(req)
Parse or re-parse special headers about app version, language and timezone, it is called early to parse headers first and then right after the query parameters are available, query values have higher priority than headers.
api.startMetrics(req, res, next)
This is supposed to be called at the beginning of request processing to start metrics and install the handler which will be called at the end to finalize the metrics and call the cleanup handlers
api.handleMetrics(req, elapsed)
Finish metrics collection about the current rquest
api.handleCleanup(req)
Call registered cleanup hooks and clear the request explicitly
api.checkQuery(req, res, next)
Parse incoming query parameters
api.checkBody(req, res, next)
Parse multipart forms for uploaded files
api.checkRouting(req, name, ignore)
Check if the current request must be re-routed to another endpoint
api.checkRedirectPlaceholders(req, pathname)
Replace redirect placeholders
api.checkRedirect(req, options)
Check a request for possible redirection condition based on the configuration, this can be SSL checks or
defined redirect rules. This is used by API servers and proxy servers for early redirections. It returns null
if no redirects or errors happend, otherwise an object with status that is expected by the api.sendStatus
method.
The options is expected to contain the following cached request properties:
api.checkRedirectRules(req, options, name)
Redirect rules, supports regexpobj and regexpmap parameters
api.checkRateLimits(req, options, callback)
Perform rate limiting by specified property, if not given no limiting is done.
The following options properties can be used:
type - predefined: ip, path, opath
, determines by which property to perform rate limiting, when using account properties
the rate limiter should be called after the request signature has been parsed. Any other value is treated as
custom type and used as is. If it is an array all items will be checked sequentially.
This property is required.
The predefined types checked for every request:
ip - check every IP address
opath - same as path but uses original path before routing
path - limit number of requests for an API path by IP address, * can be used at the end to match only the beginning
-api-rlimits-rate-ip=100 -api-rlimits-rate-/api/path=2 -api-rlimits-ip-/api/path=1 -api-rlimits-rate-/api/path/*=1
ip - to use the specified IP address
max - max capacity to be used by default
rate - fill rate to be used by default
interval - interval in ms within which the rate is measured, default 1000 ms
message - more descriptive text to be used in the error message for the type, if not specified a generic error message is used
queue - which queue to use instead of the default, some limits is more useful with global queues like Redis instead of the default
The metrics are kept in the LRU cache in the master process by default.
Example:
api.checkRateLimits(req, { type: "ip", rate: 100, interval: 60000 }, (err, info) => {
if (err) return api.sendReply(err);
...
});
api.sendJSON(req, err, data)
Send result back with possibly executing post-process callback, this is used by all API handlers to allow custom post processing in the apps. If err is not null the error message is returned immediately.
api.sendFormatted(req, err, data, options)
Send result back formatting according to the options properties:
api.sendStatus(res, options)
Return reply to the client using the options object, it contains the following properties:
i18n Note:
The API server attaches fake i18n functions req.__
and res.__
which are used automatically for the message
property
before sending the response.
With real i18n module these can/will be replaced performing actual translation without
using i18n.__
method for messages explicitely in the application code for sendStatus
or sendReply
methods.
api.sendReply(res, status, text)
Send formatted JSON reply to an API client, if status is an instance of Error then error message with status 500 is sent back.
If the status is an object it is sent as is.
All Error objects will return a generic error message without exposing the real error message, it will log all error exceptions in the logger subject to log throttling configuration.
api.sendFile(req, file, redirect)
Send file back to the client, res is Express response object
api.handleSignature(req, res, callback)
Perform authorization of the incoming request for access and permissions
api.newSignature(req)
Returns a new signature object with all required properties filled form the request object
api.getSignature(req)
Parse incoming request for signature and return all pieces wrapped in an object, this object will be used by verifySignature
function.
If the signature successfully recognized it is saved in the request as req.signature
,
it always returns a signature object, a new one or existing
api.verifySignature(req, sig, account, callback)
Returns true if the signature sig
matches given account secret. account
object must be a bk_user
record.
api.createSignature(login, secret, method, host, uri, options)
Create secure signature for an HTTP request. Returns an object with HTTP headers to be sent in the response.
The options may contains the following:
api.checkRequestSignature(req, callback)
Verify request signature from the request object, uses properties: .host, .method, .url or .originalUrl, .headers
api.checkAccess(req, callback)
Perform URL based access checks, this is called before the signature verification, very early in the request processing step.
Checks access permissions, calls the callback with the following argument:
api.checkAuthorization(req, status, callback)
Perform authorization checks after the account been checked for valid signature, this is called even if the signature verification failed, in case of a custom authentication middlware this must be called at the end and use the status object returned in the callback to return an error or proceed with the request. In any case the result of this function is final.
If a user has valid login by default access to all API endpoints is granted, to restrict access to specific APIs use any combinations of
api-allow
or api-deny
config parameters.
api.getCsrfToken(req)
Return existing or create a new token
api.checkCsrfToken(req)
For configured endpoints check for a token and fail if not present or invalid
api.fileUrl(file, options)
Returns absolute file url if it is configured with any prefix or S3 bucket, otherwise returns empty string
api.getFile(req, file, options)
Send a file to the client
api.readFile(file, options, callback)
Returns contents of a file, all specific parameters are passed as is, the contents of the file is returned to the callback,
see lib.readFile
or aws.s3GetFile
for specific options.
api.copyFile(source, dest, options, callback)
Copy a file from one location to another, can deal with local and S3 files if starts with s3:// prefix
api.listFile(options, callback)
Returns a list of file names inside the given folder, options.filter
can be a regexp to restrict which files to return
api.putFile(req, name, options, callback)
Upload file and store in the filesystem or S3, try to find the file in multipart form, in the body or query by the given name
Output file name is built according to the following options properties:
On return the options may have the following properties set:
api.storeFile(tmpfile, outfile, options, callback)
Place the uploaded tmpfile to the destination pointed by outfile
api.delFile(file, options, callback)
Delete file by name from the local filesystem or S3 drive if filesS3 is defined in api or options objects
api.detectFile(file, callback)
Returns detected mime type and ext for a file, requires mmmagic
package
api.findHook(type, method, path)
Find registered hooks for given type and path
api.addHook(type, method, path, callback)
Register a hook callback for the type and method and request url, if already exists does nothing.
api.registerRateLimits(name, rate, max, interval, queue)
Register access rate limit for a given name, all other rate limit properties will be applied as described in the checkRateLimits
api.registerControlParams(options)
Add special control parameters that will be recognized in the query and placed in the req.options
for every request.
Control params start with underscore and will be converted into the configured type according to the spec.
The options
is an object in the format that is used by lib.toParams
, no default type is allowed, even for string
it needs to be defined as { type: "string" }.
No existing control parameters will be overridden, also care must be taken when defining new control parameters so they do not conflict with the existing ones.
These are default common parameters that can be used by any module:
_count, _page, _tm, _sort, _select, _ext, _start, _token, _session, _format, _total, _encoding, _ops
These are the reserved names that cannot be used for parameters, they are defined by the engine for every request:
path, apath, ip, host, mtime, cleanup, secure, noscan, appName, appVersion, appLocale, appTimezone, apiVersion
NOTE: noscan
is set to 1 in every request to prevent accidental full scans, this means it cannot be enabled via the API but any module
can do it in the code if needed.
Example:
mod.configureMiddleware = function(options, callback) {
api.registerControlParams({ notify: { type: "bool" }, level: { type: "int", min: 1, max: 10 } });
callback();
}
Then if a request arrives for example as `_notify=true&_level=5`, it will be parsed and placed in the `req.options`:
mod.configureWeb = function(options, callback) {
api.app.all("/send", function(req, res) {
if (req.options.notify) { ... }
if (req.options.level > 5) { ... }
});
callback()
}
api.registerAccessCheck(method, path, callback)
Register a handler to check access for any given endpoint, it works the same way as the global accessCheck function and is called before validating the signature or session cookies. No account information is available at this point yet.
Example:
api.registerAccessCheck('', 'account', function(req, cb) { cb({ status: 500, message: "access disabled"}) }))
api.registerAccessCheck('POST', '/account/add', function(req, cb) {
if (!req.query.invitecode) return cb({ status: 400, message: "invitation code is required" });
cb();
});
api.registerAuthCheck(method, path, callback)
This callback will be called after the signature or session is verified but before
the ACL authorizaton is called. The req.account
object will always exist at this point but may not contain the user in case of an error.
The purpose of this hook is to perform alternative authentication like API access with keys. Because it is called before the authorization it is also possible to customize user roles.
To just continue to next hopok or step return nothing in the cb
,
any returned status will be final, an error status will be immediately returned in the response,
status 200 will continue to the authorization step
Example:
api.registerPreProcess(method, path, callback)
Similar to registerAuthCheck
, this callback will be called after the signature or session is verified and ACL authorization performed but before
the API route method is called. The req.account
object will always exist at this point but may not contain the user in case of an error.
The purpose of this hook is to perform some preparations or check permissions of a valid user to resources or in case of error perform any other action like redirection or returning something explaining what to do in case of failure.
cb
callbackExample:
api.registerPreProcess('GET', '/account/get', function(req, status, cb) {
if (status.status != 200) status = { status: 302, url: '/error.html' };
cb(status)
});
Example with admin access only:
api.registerPreProcess('POST', '/data/', function(req, status, cb) {
if (req.account.type != "admin") return cb({ status: 401, message: "access denied, admins only" });
cb();
});
api.registerPostProcess(method, path, callback)
Register a callback to be called after successfull API action, status 200 only. To trigger this callback the primary response handler must return
results using api.sendJSON
or api.sendFormatted
methods.
The purpose is to perform some additional actions after the standard API completed or to customize the result
Note: the req.account,req.options,req.query
objects may become empty if any callback decided to do some async action, they are explicitly emptied at the end of the request,
in such cases make a copy of the needed objects if it will needed
Example, just update the rows, it will be sent at the end of processing all post hooks
api.registerPostProcess('', '/data/', function(req, res, rows) {
rows.forEach(function(row) { ...});
});
Example, add data to the rows and return result after it
api.registerPostProcess('', '/data/', function(req, res, row) {
db.get("bk_user", { id: row.id }, function(err, rec) {
row.name = rec.name;
res.json(row);
});
return true;
});
api.registerCleanup(method, path, callback)
Register a cleanup callback that will be called at the end of a request, all registered cleanup callbacks will be called in the order of registration. At this time the result has been sent so connection is not valid anymore but the request and account objects are still available.
Example, do custom logging of all requests
api.registerCleanup('', '/data/', function(req, next) {
db.add("log", req.query, next);
});
api.registerSendStatus(method, path, callback)
Register a status callback that will be called when api.sendReply
or api.sendStatus
is called,
all registered callbacks will be called in the order of registration. At this time the result has NOT been sent yet so connection is
still valid and can be changed. The callback behavior is similar to the api.registerPostProcess
.
To indicate that this hook will send the result eventually it must return true, otherwise the result will be sent afer all hooks are called
Example, do custom logging of all requests
api.registerSendStatus('', '/data/', function(req, res, data) {
logger.info("response", req.path, data);
});
api.registerSignature(method, path, callback)
The purpose of this hook is to manage custom signatures.
Example:
api.registerSignature('', '/', function(req, account, sig, cb) {
if (sig) {
if (invalid) sig = null;
} else {
sig = api.createSignature(.....);
}
cb(sig)
});
api.registerSecret(login, callback)
Register a secret generation method.
api.registerPreHeaders(req, callback)
Register a callback to be called just before HTTP headers are flushed, the callback may update response headers
api.scaleIcon(infile, options, callback)
Scale image using Sharp or Imagemagick, return err if failed
The callback takes 3 arguments: function(err, data, info)
where data
will contain a new image data and info
is an object with the info about the new or unmodified image: ext, width, height.
api.iconPath(id, options)
Full path to the icon, perform necessary hashing and sharding, id can be a number or any string.
options.type
may contain special placeholders:
api.iconUrl(file, options)
Returns constructed icon url from the icon record
api.sendIcon(req, id, options)
Send an icon to the client, only handles files
api.putIcon(req, name, id, options, callback)
Store an icon for account, the options are the same as for the iconPath
method
iconPath
along with the options to build the icon absolute pathapi.saveIcon(file, id, options, callback)
Save the icon data to the destination, if api.imagesS3 or options.imagesS3 specified then plave the image on the S3 drive. Store in the proper location according to the types for given id, this function is used after downloading new image or when moving images from other places. On success the callback will be called with the second argument set to the output file name where the image has been saved. Valid properties in the options:
scaleIcon
functionapi.delIcon(id, options, callback)
Delete an icon for account, .type defines icon prefix
api.isIconUrl(url)
Return true if the given file or url point ot an image
api.isIcon(buf)
Returns detected image type if the given buffer contains an image, it checks the header only
api.handleProxyRequest(req, res, callback)
Process incoming proxy request, can be overriden for custom logic with frontend proxy server. If any response is sent or an error returned in the calback then the request will be aborted and will not be forwarded to the web processes
api.createProxyServer()
Create a proxy server to handle incoming requests and distribute them to the workers
api.createProxyWorker()
Create/fork a worker to handle API requests, register a new port for load balancing between web workers
api.getProxyPort()
Return a target port for proxy requests, rotates between all web workers
api.getProxyTarget(req)
Return a target for proxy requests
api.runProxyRequest(req, res, ssl)
Process a proxy request, perform all filtering or redirects
api.getSessionCookie(req, name)
Return named encrypted cookie
api.setSessionCookie(req, name, value)
Set a cookie by name and domain, the value is always encrypted
api.handleSessionSignature(req, callback)
Setup session cookies or access token for automatic authentication without signing, req must be complete with all required properties after successful authorization.
api.checkAccountType(account, type)
Return true if the current user belong to the specified type, account type may contain more than one type.
NOTE: after this call the type
property is converted into an array
api.setCurrentAccount(req, account)
Assign or clear the current account record for the given request, if account is null the account is cleared.
All columns in the auth table marked with the auth
property will be also set in the req.options.account
which is used
for permissions when the full account record is not available and only the options are passed.
api.getOptions(req, controls)
Convert query options into internal options, such options are prepended with the underscore to distinguish control parameters from the query parameters.
For security purposes this is the only place that translates special control query parameters into the options properties,
all the supported options are defined in the api.controls
and can be used by the apps freely but with caution. See registerControlParams
.
if controls
is an object it will be used to define additional control parameters or override existing ones for this request only. Same rules as for
registerControlParams
apply.
api.getOptions(req, { count: { min: 5, max: 100 } })
api.getQuery(req, params, options)
Parse query parameters according to the params
. Uses global api defaults, if provided in options as well defaults are all merged.
Returns a query object or an error message or null
var query = api.getQuery(req, { q: { required: 1 } }, { null: 1 });
api.getTokenSecret(req)
Return a secret to be used for enrypting tokens, it uses the account property if configured or the global API token
to be used to encrypt data and pass it to the clients. -api-query-token-secret
can be configured and if a column in the bk_user
with such name exists it is used as a secret, otherwise the value of this property is used as a secret.
api.getResultPage(req, options, rows, info)
Return an object to be returned to the client as a page of result data with possibly next token if present in the info. This result object can be used for pagination responses.
api.getPublicColumns(table, options)
Columns that are allowed to be visible, used in select to limit number of columns to be returned by a query
options may be used to define the following properties:
skip - a regexp with names to be excluded as well
allow - a list of properties which can be checked along with the pub
property for a column to be considered public
disallow - a list of properties which if set will prevent a column to be returned, it is checked before the 'allow' rule
api.getPublicColumns("bk_user", { allow: ["admins"], skip: /device_id|0$/ });
api.cleanupResult(table, data, options)
Process records and keep only public properties as defined in the table columns. This method is supposed to be used in the post process callbacks after all records have been processes and are ready to be returned to the client, the last step would be to cleanup all non public columns if necessary.
table
can be a single table name or a list of table names which combined public columns need to be kept in the rows. List of request tables
is kept in the req.options.cleanup
which by default is empty.
By default primary keys are not kept and must be marked with pub
property in the table definition to be returned.
If any column is marked with priv
property this means never return that column in the result even for the owner of the record
The options.isInternal
allows to return everything except secure columns
Columns with the pub_admin
property will be returned only if the options contains isAdmin
, same with the pub_staff
property, it requires options.isStaff
.
To return data based on the current account roles a special property in the format pub_types
must be set as a string or an array
with roles to be present in the current account type`` field. This is checked only if the column is allowed, this is an additional restriction, i.e. a column must be allowed by the
pub` property or other way.
To retstrict by role define priv_types
in a column with a list of roles which should be denied access to the field.
The options.pool
property must match the actual rowset to be applied properly, in case the records
have been retrieved for the different database pool.
The options.cleanup_strict
will enforce that all columns not present in the table definition will be skipped as well, by default all
new columns or columns created on the fly are returned to the client. api.cleanupStrict
can be configured globbly.
The options.cleanup_rules
can be an object with property names and the values 0, or 1 for pub
or 2
for admin`` or
3for
staff``
The options.cleanup_copy
means to return a copy of every modified record, the original data is preserved
api.clearQuery(table, query, options)
Clear request query properties specified in the table definition or in custom schema.
The table
argument can be a table name or an object with properties as columns.
If options.filter
is not specified the query
will only keep existing columns for the given table.
If options.filter
is a list then the query
will delete properties for columns that contain any specified
property from the filter list. This is used for the bk_user
table to remove properties that supposed to be updated by admins only.
The filter will keep non-existent columns in the query
. To remove such columns when using the filter specify options.force
.
If a name in the filter is prefixed with ! then the logic is reversed, keep all except this property
If options.keep
is a regexp it will be used to keep matched properties by name in the query
regardless of any condition.
If options.clear
is a regexp it will be used to remove matched properties by name in the query
.
Example:
api.clearQuery("bk_user", req.query)
api.clearQuery("bk_user", req.query, "internal")
api.clearQuery("bk_user", req.query, { filter: "internal" })
api.clearQuery("bk_user", req.query, { filter: ["internal"] })
api.clearQuery("bk_user", req.query, { filter: ["!pub"] })
api.clearQuery("bk_user", req.query, { filter: ["internal","priv"] })
api.clearQuery("bk_user", req.query, { filter: ["internal","!priv"], keep: /^__/ })
api.clearQuery({ name: {}, id: { admin: 1 } }, req.query, { filter: ["internal"] })
api.handleWebSocketUpgrade(req, socket, head)
Check if the request is allowed to upgrade to Websocket
api.handleWebSocketConnect(ws, req)
Wrap external WebSocket connection into the Express routing, respond on backend command
api.handleWebSocketRequest(ws, data)
Wrap WebSocket into HTTP request to be proceses by the Express routes
api.wsSet(type, req, value)
Update a Websocket connection properties:
api.wsSend(wsid, msg)
Send to a websocket inside an api server directly
api.wsNotify(options, msg, callback)
Broadcast a message according to the options, if no websocket queue is defined send directly using wsBroadcast
api.wsBroadcast(options, msg)
Send a message to all websockets inside an api process that match the criteria from the options:
lib.isMatched
is used for comparisonlib.isMatched
is used for comparisonapi.cleanupResult
, if it is an array then
the first item is a table and the second item is the property name inside the msg
to be cleanup only, eg. cleanup: ["bk_user","user"].
All properties starting with is`` or
cleanup_`` will be passed to the cleanupResult.module.method
to run the same way as the preprocess
function, this is a more
reliable way to be use preprocess with wsNotify
Author: Vlad Seryakov<EMAIL_ADDRESS>backendjs 2018
This is a skeleton module to be extended by the specific application logic. It provides all callbacks and hooks that are called by the core backend modules during different phases, like initialization, shutting down, etc...
It should be used for custom functions and methods to be defined, the app
module is always available.
All app modules in the modules/ subdirectory use the same prototype, i.e. all hooks are available for custom app modules as well.
app.configure(options, callback)
Called after all config files are loaded and command line args are parsed, home directory is set but before the db is initialized, the primary purpose of this early call is to setup environment before connecting to the database. This is called regardless of the server to be started and intended to initialize the common environment before the database and other subsystems are initialized.
app.configureModule(options, callback)
Called after the core.init has been initialized successfully, this can be redefined in the applications to add additional init steps that all processes require to have. All database pools and other confugration is ready at this point. This hook is called regardless of what kind of server is about to start, it is always called before starting a server or shell.
app.configureMiddleware(options, callback)
This handler is called during the Express server initialization just after the security middleware.
NOTE: api.app
refers to the Express instance.
app.configureWeb(options, callback)
This handler is called after the Express server has been setup and all default API endpoints initialized but the Web server is not ready for incoming requests yet. This handler can setup additional API endpoints, add/modify table descriptions.
NOTE: api.app
refers to the Express instance
app.shutdownWeb(options, callback)
Perform shutdown sequence when a Web process is about to exit
NOTE: api.app
refers to the Express instance
app.configureMaster(options, callback)
This handler is called during the master server startup, this is the process that monitors the worker jobs and performs jobs scheduling
app.configureServer(options, callback)
This handler is called during the Web server startup, this is the master process that creates Web workers for handling Web requests, this process interacts with the Web workers via IPC sockets between processes and relaunches them if any Web worker dies.
app.configureWorker(options, callback)
This handler is called on job worker instance startup after the tables are intialized and it is ready to process the job
app.shutdownWorker(options, callback)
Perform last minute operations inside a worker process before exit, the callback must be called eventually which will exit the process. This method can be overrided to implement custom worker shutdown procedure in order to finish pending tasks like network calls.
app.configureMonitor(options, callback)
This callback is called when the monitor process is ready, there is no any other code is supposed to run inside the monitor, but in case it is needed, this is the hook to be used.
app.configureShell(options, callback)
This callback is called by the shell process to setup additional command or to execute a command which is not supported by the standard shell. Setting options.done to 1 will stop the shell, this is a signal that command has already been processed.
Config parameters
auth-table
, descr: "Table to use for user accounts"auth-err-(.+)
, descr: "Error messages for various cases"auth-admin-roles
, type: "list", descr: "List of special super admin roles"auth-sigversion
, type: "int", descr: "Signature version for secrets"auth-hash
, descr: "Hashing method to use by default: bcrypt, argon2, none"auth-bcrypt
, type: "int", min: 12, descr: "Number of iterations for bcrypt"auth-argon2
, type: "map", datatype: "auto", nocamel: 1, descr: "Argon2 parameteres, ex: type:2,memoryCost:1,hashLength:32"auth-max-length
, type: "int", descr: "Max login and name length"auth-users
, type: "json", logger: "error", descr: "An object with users"auth-users-file
, descr: "A JSON file with a list of users"auth.loadUsers(callback)
Load users from a JSON file, only add or update records
auth.prepareSecret(query, options, callback)
If specified in the options, prepare credentials to be stored in the db, if no error occurred return null, otherwise an error object
auth.checkSecret(user, password, callback)
Verify an existing user record with given password,
auth.get(query, options, callback)
Returns an account record by login or id, to make use of a cache add to the config db-cache-keys-bk_user-id=id
auth.add(query, options, callback)
Registers a new account, returns new record in the callback, when options.isInternal
is true then allow to set all properties
otherwise internal properties will not be added
auth.update(query, options, callback)
Updates an existing account by login or id, if options.isInternal
is true then allow to update all properties, returns a new record in the callback
auth.del(query, options, callback)
Deletes an existing account by login or id, no admin checks, returns the old record in the callback
AWS Cloud API interface
Config parameters
aws-key
, descr: "AWS access key"aws-secret
, descr: "AWS access secret"aws-token
, descr: "AWS security token"aws-region
, descr: "AWS region", pass: 1aws-zone
, descr: "AWS availability zone"aws-meta
, type: "bool", descr: "Retrieve instance metadata, 0 to disable"aws-sdk-profile
, descr: "AWS SDK profile to use when reading credentials file"aws-sns-app-arn
, descr: "SNS Platform application ARN to be used for push notifications"aws-key-name
, descr: "AWS instance keypair name for remote job instances or other AWS commands"aws-elb-name
, descr: "AWS ELB name to be registered with on start up or other AWS commands"aws-target-group
, descr: "AWS ELB target group to be registered with on start up or other AWS commands"aws-elastic-ip
, descr: "AWS Elastic IP to be associated on start"aws-host-name
, type: "list", descr: "List of hosts to update in Route54 zone with the current private IP address, hosts must be in FQDN format, supports @..@ core.instance placeholders"aws-iam-profile
, descr: "IAM instance profile name for instances or commands"aws-image-id
, descr: "AWS image id to be used for instances or commands"aws-subnet-id
, descr: "AWS subnet id to be used for instances or commands"aws-vpc-id
, descr: "AWS VPC id to be used for instances or commands"aws-group-id
, array: 1, descr: "AWS security group(s) to be used for instances or commands"aws-instance-type
, descr: "AWS instance type to launch on demand"aws-account-id
, descr: "AWS account id if not running on an instance"aws-eni-id
, type: "list", descr: "AWS Elastic Network Interfaces to attach on start, format is: eni[:index],eni..."aws-config-parameters
, descr: "Prefix for AWS Config Parameters Store to load and parse as config before initializing the database pools, example: /bkjs/config/"aws-set-parameters
, type: "list", descr: "AWS Config Parameters Store to set on start, supports @..@ core.instance placeholders: format is: path:value,...."aws-conf-file
, descr: "S3 url for config file to download on start"aws-conf-file-interval
, type: "int", descr: "Load S3 config file every specified interval in minites"aws.configure(options, callback)
Initialization of metadata
aws.configureServer(options, callback)
Execute on Web server startup
aws.configureMaster(options, callback)
Execute on master server startup
aws.readCredentials(profile, callback)
Read key and secret from the AWS SDK credentials file, if no profile is given in the config or command line only the default peofile will be loaded.
aws.readConfig(callback)
Read and apply config from S3 bucket
aws.getInstanceMeta(path, callback)
Retrieve instance meta data
aws.getInstanceCredentials(path, callback)
Retrieve instance credentials using EC2 instance profile and setup for AWS access
aws.getInstanceInfo(options, callback)
Retrieve instance launch index from the meta data if running on AWS instance
aws.parseXMLResponse(err, params, options, callback)
Parse AWS response and try to extract error code and message, convert XML into an object.
aws.querySign(region, service, host, method, path, body, headers, credentials, options)
Build version 4 signature headers
aws.queryPrepare(action, version, obj, options)
Return a request object ready to be sent to AWS, properly formatted
aws.querySigner()
It is called in the context of a http request
aws.queryAWS(region, service, proto, host, path, obj, options, callback)
Make AWS request, return parsed response as Javascript object or null in case of error
aws.queryEndpoint(service, version, action, obj, options, callback)
AWS generic query interface
aws.getServiceRegion(service, region)
Check for supported regions per service, return the first one if the given region is not supported
aws.copyCredentials(obj, options)
Copy all credentials properties from the options into the obj
aws.querySQS(action, obj, options, callback)
AWS SQS API request
aws.queryIAM(action, obj, options, callback)
AWS AIM API request
aws.querySTS(action, obj, options, callback)
AWS STS API request
aws.querySES(action, obj, options, callback)
AWS SES API request
aws.queryCFN(action, obj, options, callback)
AWS CFN API request
aws.queryElastiCache(action, obj, options, callback)
AWS Elastic Cache API request
aws.queryAS(action, obj, options, callback)
AWS Autoscaling API request
aws.queryRekognition(action, obj, options, callback)
Make a request to the Rekognition service
aws.querySSM(action, obj, options, callback)
AWS SSM API request
aws.queryACM(action, obj, options, callback)
AWS ACM API request
aws.queryComprehend(action, obj, options, callback)
AWS Comprehend API request
aws.queryTranscribe(action, obj, options, callback)
AWS Transcribe API request
aws.getTagValue(obj, key)
Returns a tag value by key, default key is Name
aws.stsAssumeRole(options, callback)
Assume a role and return new credentials that can be used in other API calls
aws.sqsReceiveMessage(url, options, callback)
Receive message(s) from the SQS queue, the callback will receive a list with messages if no error. The following options can be specified:
aws.sqsSendMessage(url, body, options, callback)
Send a message to the SQS queue. The options can specify the following:
aws.sesSendEmail(to, subject, body, options, callback)
Send an email via SES The following options supported:
aws.sesSendRawEmail(body, options, callback)
Send raw email The following options accepted:
aws.detectLabels(name, options, callback)
Detect image featires using AWS Rekognition service, the name
can be a Buffer, a local file or an url to the S3 bucket. In the latter case
the url can be just apath to the file inside a bucket if options.bucket
is specified, otherwise it must be a public S3 url with the bucket name
to be the first part of the host name. For CDN/CloudFront cases use the option.bucket
option.
aws.queryCW(action, obj, options, callback)
AWS CloudWatch API request
aws.queryCWL(action, obj, options, callback)
AWS CloudWatch Log API request
aws.cwPutMetricAlarm(options, callback)
Creates or updates an alarm and associates it with the specified Amazon CloudWatch metric. The options specify the following:
CPUUtilization
AWS/EC2
>=
.Average
60
15
90
aws.cwPutMetricData(namespace, data, options, callback)
Publishes metric data points to Amazon CloudWatch. The argumernts specify the following:
AWS
The options can specify the following:
aws.cwListMetrics(options, callback)
Return metrics for the given query, the options can be specified:
aws.cwGetMetricData(options, callback)
Return collected metric statistics
Options:
Example:
aws.cwGetMetricData({ age: 5, metrics: [{ name: "NetworkOut", label: "Traffic", stat: "Average", dimensions: { InstanceId: "i-1234567" } } ] }, lib.log)
aws.cwlFilterLogEvents(options, callback)
Lists log events from the specified log group. You can list all the log events or filter the results using a filter pattern, a time range, and the name of the log stream. Options:
aws._queryDDB(target, service, action, obj, options, callback)
DynamoDB requests
aws.toDynamoDB(value, level)
Convert a Javascript object into DynamoDB object
aws.fromDynamoDB(value, level)
Convert a DynamoDB object into Javascript object
aws.queryExpression(params, obj, options, join)
Build a condition expression for the given object, all properties in the obj are used
aws.ddbListTables(options, callback)
Return list of tables in .TableNames property of the result
Example:
{ TableNames: [ name, ...] }
aws.ddbDescribeTable(name, options, callback)
Return table definition and parameters in the result structure with property of the given table name
Example:
{ name: { AttributeDefinitions: [], KeySchema: [] ...} }
aws.ddbCreateTable(name, attrs, options, callback)
Create a table
Example:
ddbCreateTable('users', { id: 'S', mtime: 'N', name: 'S'},
{ keys: ["id", "name"],
local: { mtime: { mtime: "HASH" } },
global: { name: { name: 'HASH', ProvisionedThroughput: { ReadCapacityUnits: 50 } } },
projections: { mtime: ['gender','age'],
name: ['name','gender'] },
stream: "NEW_IMAGE",
readCapacity: 10,
writeCapacity: 10 });
aws.ddbUpdateTable(options, callback)
Update tables provisioned throughput settings, options is used instead of table name so this call can be used directly in the cron jobs to adjust provisionined throughput on demand. Options must provide the following properties:
Example
aws.ddbUpdateTable({ name: "users", add: { name_id: { name: "S", id: 'N', readCapacity: 20, writeCapacity: 20, projections: ["mtime","email"] } })
aws.ddbUpdateTable({ name: "users", add: { name: { name: "S", readCapacity: 20, writeCapacity: 20, projections: ["mtime","email"] } })
aws.ddbUpdateTable({ name: "users", del: "name" })
aws.ddbUpdateTable({ name: "users", update: { name: { readCapacity: 10, writeCapacity: 10 } })
Example of crontab job in etc/crontab:
[
{ "type": "server", "cron": "0 0 1 * * *", "job": { "aws.ddbUpdateTable": { "name": "bk_user", "readCapacity": 1000, "writeCapacity": 1000 } } },
{ "type": "server", "cron": "0 0 6 * * *", "job": { "aws.ddbUpdateTable": { "name": "bk_user", "readCapacity": 2000, "writeCapacity": 2000 } } }
]
aws.ddbUpdateTimeToLive(options, callback)
Update TTL attribute. The options properties:
aws.ddbDescribeTimeToLive(name, options, callback)
Returns status of Time to live attribute for a table
aws.ddbDeleteTable(name, options, callback)
Remove a table from the database.
By default the callback will ba callled only after the table is deleted, specifying options.nowait
will return immediately
aws.ddbWaitForTable(name, item, options, callback)
Call the callback after specified period of time or when table status become different from the given waiting status. if options.waitTimeout is not specified calls the callback immediately. options.waitStatus is checked if given and keeps waiting while the status is equal to it. options.waitDelay can be specified how often to request new status, default is 250ms.
aws.ddbPutItem(name, item, options, callback)
Put or add an item
Example:
ddbPutItem("users", { id: 1, name: "john", mtime: 11233434 }, { expected: { name: null } })
aws.ddbUpdateItem(name, keys, item, options, callback)
Update an item
options.ops
the same way as for queries.*
or new
means ALL_NEW, old
means ALL_OLD,
updated
means UPDATED_NEW, old_updated
means UPDATED_OLDExample:
ddbUpdateItem("users", { id: 1, name: "john" }, { gender: 'male', icons: '1.png' }, { action: { icons: 'add' }, expected: { id: 1 }, returning: "*" })
ddbUpdateItem("users", { id: 1, name: "john" }, { gender: 'male', icons: '1.png' }, { action: { icons: 'incr' }, expected: { id: null } })
ddbUpdateItem("users", { id: 1, name: "john" }, { gender: 'male', icons: '1.png', num: 1 }, { action: { num: 'add', icons: 'add' }, expected: { id: null, num: 0 }, ops: { num: "gt" } })
aws.ddbDeleteItem(name, keys, options, callback)
Delete an item from a table
Example:
ddbDeleteItem("users", { id: 1, name: "john" }, {})
aws.ddbBatchWriteItem(items, options, callback)
Update items from the list at the same time
Example:
{ table: [ { put: { id: 1, name: "tt" } }, { del: { id: 2 } }] }
aws.ddbBatchGetItem(items, options, callback)
Retrieve all items for given list of keys
Example:
{ users: { keys: [{ id: 1, name: "john" },{ id: .., name: .. }], select: ['name','id'], consistent: true }, ... }
aws.ddbGetItem(name, keys, options, callback)
Retrieve one item by primary key
consistent - set consistency level for the request
names - an object with a map to be used for attribute names in condition and update expressions, to be used for ExpressionAttributeNames parameter Example:
ddbGetItem("users", { id: 1, name: "john" }, { select: 'id,name' })
aws.ddbQueryTable(name, condition, options, callback)
Query on a table, return all matching items
Example:
aws.ddbQueryTable("users", { id: 1, name: "john" }, { select: 'id,name', ops: { name: 'gt' } })
aws.ddbQueryTable("users", { id: 1, name: "john", status: "ok" }, { keys: ["id"], select: 'id,name', ops: { name: 'gt' } })
aws.ddbQueryTable("users", { id: 1 }, { expr: "status=:s", values: { s: "status" } })
aws.ddbScanTable(name, condition, options, callback)
Scan a table for all matching items
Example:
aws.ddbScanTable("users", { id: 1, name: 'a' }, { ops: { name: 'gt' }})
aws.ddbScanTable("users", "id=:id AND name=:name", { values: { id: 1, name: 'a' } });
aws.ddbTransactWriteItems(items, options, callback)
Update items from the list at the same time in one transaction, on any failure everything is rolled back
Example:
{ op: "put": table: "table-name", obj: { id: 1, name: "tt" } },
{ op: "del": table: "table-name", obj: { id: 2 } },
{ op: "update": table: "table-name", obj: { id: 1, name: "test" }, options: { expected: { status: "ok" } } },
{ op: "check": table: "table-name", obj: { id: 1 }, options: { expected: { status: "ok" } } }
aws.queryEC2(action, obj, options, callback)
AWS EC2 API request
aws.queryELB(action, obj, options, callback)
AWS ELB API request
aws.ec2RunInstances(options, callback)
Run AWS instances, supports all native EC2 parameters with first capital letter but also accepts simple parameters in the options:
Name:
, any occurences of %i will be replaced with the instance indexThe callback will take 3 arguments: callback(err, rc, info) where info will contain properties that can be used by `aws.ec2PrepareInstance
aws.ec2AfterRunInstances(options, callback)
Perform the final tasks after an instance has been launched like wait for status, assign Elastic IP or tags..
aws.ec2WaitForInstance(instanceId, status, options, callback)
Check an instance status and keep waiting until it is equal what we expect or timeout occurred.
The status
can be one of: pending | running | shutting-down | terminated | stopping | stopped
The options can specify the following:
aws.ec2DescribeSecurityGroups(options, callback)
Describe security groups, optionally if options.filter
regexp is provided then limit the result to the matched groups only,
return list of groups to the callback
aws.ec2DescribeInstances(options, callback)
Describe instances according to the query filters, returns a list with instances, the following properties can be used:
aws.ec2ParseInstances(rc)
Pare DescribeInstances results and return a list of instances as a plain list
aws.ec2CreateTags(id, name, options, callback)
Create tags for a resource. The name is a string, an array or an object with tags. The options also may contain tags property which is an object with tag key and value
Example
aws.ec2CreateTags("i-1234","My Instance", { tags: { tag2 : "val2", tag3: "val3" } } )
aws.ec2CreateTags("i-1234", { tag2: "val2", tag3: "val3" })
aws.ec2CreateTags("i-1234", [ "tag2", "val2", "tag3", "val3" ])
aws.ec2AssociateAddress(instanceId, elasticIp, options, callback)
Associate an Elastic IP with an instance. Default behaviour is to reassociate if the EIP is taken. The options can specify the following:
aws.ec2CreateImage(options, callback)
Create an EBS image from the instance given or the current instance running
aws.ec2DeregisterImage(ami_id, options, callback)
Deregister an AMI by id. If options.snapshots
is set, then delete all snapshots for this image as well
aws.elbRegisterInstances(name, instance, options, callback)
Register an instance(s) with ELB, instance can be one id or a list of ids
aws.elbDeregisterInstances(name, instance, options, callback)
Deregister an instance(s) from ELB, instance can be one id or a list of ids
aws.elb2RegisterInstances(target, instance, options, callback)
Register an instance(s) with ELB, instance can be one id or a list of ids
aws.ssmSendCommand(cmds, instances, options, callback)
Run a shell command
aws.ssmWaitForCommand(cmdId, instanceId, options, callback)
Return a command details
aws.queryRoute53(method, path, data, options, callback)
Make a request to Route53 service
aws.route53List(options, callback)
List all zones
aws.route53Get(options, callback)
Return a zone by domain or id
aws.route53Change(names, options, callback)
Create or update a host in the Route53 database.
names
is a host name to be set with the current IP address or a list with objects in the format
[ { name: "..", value: "<IP_ADDRESS>", type: "A", ttl: 300 } ...]The options
may contain the following:
aws.signS3(method, bucket, path, body, options)
Sign S3 AWS request, returns url to be send to S3 server, options will have all updated headers to be sent as well
aws.queryS3(bucket, path, options, callback)
S3 requests Options may contain the following properties:
aws.s3List(path, options, callback)
Retrieve a list of files from S3 bucket, only files inside the path will be returned
aws.s3GetFile(path, options, callback)
Retrieve a file from S3 bucket, root of the path is a bucket, path can have a protocol prepended like s3://, it will be ignored
aws.s3PutFile(path, file, options, callback)
Upload a file to S3 bucket, file
can be a Buffer or a file name
aws.s3CopyFile(path, source, options, callback)
Copy existing S3 file, source must be in the format bucket/path
aws.s3ParseUrl(link)
Parse an S3 URL and return an object with bucket and path
aws.s3Proxy(res, bucket, file, options, callback)
Proxy a file from S3 bucket into the existing HTTP response res
aws.querySNS(action, obj, options, callback)
AWS SNS API request
aws.snsCreatePlatformEndpoint(token, options, callback)
Creates an endpoint for a device and mobile app on one of the supported push notification services, such as GCM and APNS.
The following properties can be specified in the options:
-sns-app-arn
will be used.All capitalized properties in the options will be pased as is. The callback will be called with an error if any and the endpoint ARN
aws.snsSetEndpointAttributes(arn, options, callback)
Sets the attributes for an endpoint for a device on one of the supported push notification services, such as GCM and APNS.
The following properties can be specified in the options:
aws.snsDeleteEndpoint(arn, options, callback)
Deletes the endpoint from Amazon SNS.
aws.snsPublish(arn, msg, options, callback)
Sends a message to all of a topic's subscribed endpoints or to a mobile endpoint. If msg is an object, then it will be pushed as JSON. The options may take the following properties:
aws.snsCreateTopic(name, options, callback)
Creates a topic to which notifications can be published. The callback returns topic ARN on success.
aws.snsSetTopicAttributes(arn, options, callback)
Updates the topic attributes. The following options can be used:
aws.snsDeleteTopic(arn, options, callback)
Deletes the topic from Amazon SNS.
aws.snsSubscribe(arn, endpoint, options, callback)
Creates a topic to which notifications can be published. The callback returns topic ARN on success, if the topic requires confirmation the arn returned will be null and a token will be sent to the endpoint for confirmation.
aws.snsConfirmSubscription(arn, token, options, callback)
Verifies an endpoint owner's intent to receive messages by validating the token sent to the endpoint by an earlier Subscribe action. If the token is valid, the action creates a new subscription and returns its Amazon Resource Name (ARN) in the callback.
aws.snsSetSubscriptionAttributes(arn, options, callback)
Updates the subscription attributes. The following options can be used:
aws.snsUnsubscribe(arn, options, callback)
Creates a topic to which notifications can be published. The callback returns topic ARN on success.
aws.snsListTopics(options, callback)
Creates a topic to which notifications can be published. The callback returns topic ARN on success.
The primary object containing all config options and common functions
Config parameters
help
, type: "callback", callback: function() { this.showHelp() descr: "Print help and exit"log
, type: "callback", callback: function(v) { logger.setLevel(v) descr: "Set debugging level to any of " + Object.keys(logger.levels), pass: 2log-file
, type: "callback", callback: function(v) { if (v) this.logFile=v;logger.setFile(this.logFile, this) descr: "Log to a file, if not specified used default logfile, disables syslog", pass: 1log-ignore
, type: "regexp", obj: "logInspect", strip: /log-/, nocamel: 1, descr: "Regexp with property names which must not be exposed in the log when using custom logger inspector"log-inspect
, type: "callback", callback: function(v) { this.setLogInspect(v) descr: "Install custom secure logger inspection instead of util.inspect"log-filter
, type: "callback", callback: function(v) { if (v) logger.setDebugFilter(v) descr: "Enable debug filters, format is: label,... to enable, and !label,... to disable. Only first argument is used for label in logger.debug", pass: 1no-log-filter
, type: "bool", onupdate: function(v) { if (v) logger.filters={} descr: "Clear all log filters", pass: 1syslog
, type: "callback", callback: function(v) { logger.setSyslog(v || 1, this.name) descr: "Log messages to syslog, pass 0 to disable, 1 or url (tcp|udp|unix):[//host:port]/path?[facility=F][&tag=T][&retryCount=N]...", pass: 1console
, type: "callback", callback: function() { logger.setFile(null) descr: "All logging goes to the console resetting all previous log related settings, this is used in the development mode mostly", pass: 1home
, type: "callback", callback: "setHome", descr: "Specify home directory for the server, the server will try to chdir there or exit if it is not possible, the directory must exist", pass: 2conf-file
, descr: "Name of the config file to be loaded instead of the default etc/config, can be relative or absolute path", pass: 1err-file
, type: "path", descr: "Path to the error log file where daemon will put app errors and crash stacks", pass: 1etc-dir
, type: "path", obj: "path", strip: /Dir/, descr: "Path where to keep config files", pass: 1tmp-dir
, type: "path", obj: "path", strip: /Dir/, descr: "Path where to keep temp files"spool-dir
, type: "path", obj: "path", strip: /Dir/, descr: "Path where to keep modifiable files"log-dir