##Table of contents
General purpose backend library. The primary goal is to have a scalable platform for running and managing node.js servers for Web services implementation.
This project only covers the lower portion of the Web services ecosystem: node.js processes, HTTP servers, basic API functinality, database access, caching, messaging between processes, metrics and monitoring, a library of tools for developing node.js servers.
For the UI and presentation layer there are no restrictions what to use as long as it can run on top of the Express server.
Features:
Check out the Documentation for more details.
To install the module with all optional dependencies if they are available in the system
npm install backendjs
This may take some time because of downloading and compiling required dependencies like ImageMagick. They are not required in all applications but still part of the core of the system to be available once needed.
To install from the git
npm install git+https://github.com/vseryakov/backendjs.git
or simply
npm install vseryakov/backendjs
Simplest way of using the backendjs, it will start the server listening on port 8000
$ node
> var bkjs = require('backendjs')
> bkjs.server.start()
Access is allowed only with valid signature except urls that are exlicitely allowed without it (see api-allow
config parameter below)
Same but using the helper tool, by default it will use embedded Sqlite database and listen on port 8000.
bkjs web
To start the server and connect to the DynamoDB (command line parameters can be saved in the etc/config file
, see below about config files)
bkjs web -db-pool dynamodb -db-dynamodb-pool default -aws-key XXXX -aws-secret XXXX
If running on EC2 instance with IAM profile then no need to specify AWS credentials:
bkjs web -db-pool dynamodb -db-dynamodb-pool default
or to the PostgreSQL server, database backend
bkjs web -db-pool pgsql -db-pgsql-pool<EMAIL_ADDRESS>All commands above will behave exactly the same
Tables are not created by default, in order to initialize the database, run the server or the shell with -db-create-tables
flag,
it is called only inside a master process, a worker never creates tables on start
prepare the tables in the shell
bksh -db-pool pgsql -db-pgsql-pool<EMAIL_ADDRESS>-db-create-tables
run the server and create tables on start
bkjs web -db-pool pgsql -db-pgsql-pool<EMAIL_ADDRESS>-db-create-tables
While the local backendjs is runnning, the documentation is always available at http://localhost:8000/doc.html (or whatever port is the server using)
To add users from the command line
bksh -account-add login test secret test name TestUser email<EMAIL_ADDRESS>By default no external modules are loaded so it needs the accounts module with a
parameter -allow-modules PATTERN
, this will load all modules that match the pattern, default modules start with bk_
:
bkjs web -allow-modules bk_
To start node.js shell with backendjs loaded and initialized, all command line parameters apply to the shell as well
./app.sh -shell
To access the database while in the shell
> db.select("bk_account", {}, function(err, rows, info) { console.log(err, rows) });
> db.select("bk_account", {}, lib.log);
> db.add("bk_account", { login: 'test2', secret: 'test2', name' Test 2 name', gender: 'f' }, lib.log);
> db.select("bk_account", { gender: 'm' }, lib.log);
Go to examples/api
directory:
Run the application, it will start the Web server on port 8000:
./app.sh
Now login with the new account,
Go to http://localhost:8000/api.html and click on Login at the top-right corner, then enter 'test' as login and 'test' as secret in the login popup dialog.
To see your account details run the command in the console /account/get
To see current metrics run the command in the console /system/stats/get
To see charts about accumulated metrics go to http://localhost:8000/metrics.html
When the web server is starte with -watch
parameters any change in the source files will make the server restart automatically
letting you focus on the source code and not server management, this mode is only enabled by default in development mode,
check app.sh
for parameters before running it in the production.
Almost everything in the backend is configurable using a config files, config database or DNS. The whole principle behind it that once deployed in production, even quick restart are impossible to do so there should be a way to push config changes to the processes without restarting.
Every module defines a set of config parameters that defines the behavior of the code, due to single threaded nature of the node.js, it is simple to update any config parameter to a new value so the code can operate differently. To achieve this the code must be written in a special way, like driven by configuration which can be changed at any time.
All configuration goes through the configuration process that checks all inputs and produces valid output which is applied to the module variables. Config file or database table with configuration can be loaded on demand or periodically, for example all local config files are watched for modification and reloaded automaticlaly, the config database is loaded periodically which is defined by another config parameter.
When the backendjs server starts it spawns several processes that perform different tasks.
There are 2 major tasks of the backend that can be run at the same time or in any combination:
These features can be run standalone or under the guard of the monitor which tracks all running processes and restarted any failed ones.
This is the typical output from the ps command on Linux server:
ec2-user 891 0.0 0.6 1071632 49504 ? Ssl 14:33 0:01 bkjs: monitor
ec2-user 899 0.0 0.6 1073844 52892 ? Sl 14:33 0:01 bkjs: master
ec2-user 908 0.0 0.8 1081020 68780 ? Sl 14:33 0:02 bkjs: server
ec2-user 917 0.0 0.7 1072820 59008 ? Sl 14:33 0:01 bkjs: web
ec2-user 919 0.0 0.7 1072820 60792 ? Sl 14:33 0:02 bkjs: web
ec2-user 921 0.0 0.7 1072120 40721 ? Sl 14:33 0:02 bkjs: worker
To enable any task a command line parameter must be provided, it cannot be specified in the config file. The bkjs
utility supports several
commands that simplify running the backend in different modes.
bkjs start
- this command is supposed to be run at the server startup as a service, it runs in the backgroud and the monitors all tasks,
the env variable BKJS_SERVER
can be set in the profile to one of the master or monitor
to define which run mode to use, default mode is monitorbkjs monitor
- this command is supposed to be run at the server startup, it runs in the backgroud and the monitors all processes,
the command line parameters are: -daemon -monitor -master -syslog
bkjs master
- this command is supposed to be run at the server startup, it runs in the backgroud and the monitors all processes,
the command line parameters are: -daemon -monitor -master -syslog
bkjs watch
- runs the master and Web server in wather mode checking all source files for changes, this is the common command to be used
in development, it passes the command line switches: -watch -master
bkjs web
- this command runs just web server process.bkjs run
- this command runs without other parameters, all aditional parameters can be added in the command line, this command
is a barebone helper to be used with any other custom settings.bkjs shell
or bksh
- start backendjs shell, no API or Web server is initialized, only the database poolsThe main puspose of the backendjs is to provide API to access the data, the data can be stored in the database or some other way but the access to that data will be over HTTP and returned back as JSON. This is default functionality but any custom application may return data in whatever format is required.
Basically the backendjs is a Web server with ability to perform data processing using local or remote jobs which can be scheduled similar to Unix cron.
The principle behind the system is that nowadays the API services just return data which Web apps or mobiles apps can render to the user without the backend involved. It does not mean this is simple gateway between the database, in many cases it is but if special processing of the data is needed before sending it to the user, it is possible to do and backendjs provides many convenient helpers and tools for it.
When the API layer is initialized, the api module contains app
object which is an Express server.
Special module/namespace app
is designated to be used for application development/extension. This module is available the same way as the api
or core
which makes it easy to refer and extend with additional methods and structures.
The typical structure of a backendjs application is the following:
var bkjs = require('backendjs');
var api = bkjs.api;
var app = bkjs.app;
var db = bkjs.db;
app.listArg = [];
// Define the module config parameters
core.describeArgs('app', [
{ name: "list-arg", array: 1, type: "list", descr: "List of words" },
{ name: "int-arg", type: "int", descr: "An integer parameter" },
]);
// Describe the tables or data models, all DB pools will use it, the master or shell
// process only creates new tables, workers just use the existing tables
db.describeTables({
...
});
// Optionally customize the Express environment, setup MVC routes or else, `api.app` is the Express server
app.configureMiddleware = function(options, callback)
{
...
callback()
}
// Register API endpoints, i.e. url callbacks
app.configureWeb = function(options, callback)
{
api.app.get('/some/api/endpoint', function(req, res) {
// to return an error, the message will be translated with internal i18n module if locales
// are loaded and the request requires it
api.sendReply(res, err);
// or with custom status and message, explicitely translated
api.sendReply(res, 404, res.__("not found"));
// with config check
if (app.intArg > 5) ...
if (app.listArg.indexOf(req.query.name) > -1) ...
// to send data back with optional postprocessing hooks
api.sendJSON(req, err, data);
// or simply
res.json(data);
});
...
callback();
}
// Optionally register post processing of the returned data from the default calls
api.registerPostProcess('', /^\/account\/([a-z\/]+)$/, function(req, res, rows) { ... });
...
// Optionally register access permissions callbacks
api.registerAccessCheck('', /^\/test\/list$/, function(req, status, callback) { ... });
api.registerPreProcess('', /^\/test\/list$/, function(req, status, callback) { ... });
...
bkjs.server.start();
Except the app.configureWeb
and server.start()
all other functions are optional, they are here for the sake of completness of the example. Also
because running the backend involves more than just running web server many things can be setup using the configuration options like common access permissions,
configuration of the cron jobs so the amount of code to be written to have fully functionaning production API server is not that much, basically only
request endpoint callbacks must be provided in the application.
As with any node.js application, node modules are the way to build and extend the functionality, backendjs does not restrict how the application is structured.
Another way to add functionality to the backend is via external modules specific to the backend, these modules are loaded on startup from the backend
home subdirectory modules/
and from the backendjs package directory for core modules. The format is the same as for regular node.js modules and
only top level .js files are loaded on the backend startup.
By default no modules are loaded except bk_accounts|bk_icons
, it must be configured by the -allow-modules
config parameter.
The modules are managed per process role, by default server
and master
processes do not load any modules at all to keep them
small and because they monitor workers the less code they have the better.
The shell process loads all modules, it is configured with .+
.
To enable any module to be loaded in any process it can be configured by using a role in the config parameter:
// Global modules except server and master
-allow-modules '.+'
// Master modules
-allow-modules-master 'bk_accounts|bk_debug'
Once loaded they have the same access to the backend as the rest of the code, the only difference is that they reside in the backend home and
can be shipped regardless of the npm, node modules and other env setup. These modules are exposed in the core.modules
the same way as all other core submodules
methods.
Let's assume the modules/ contains file facebook.js which implements custom FB logic:
var bkjs = require("backendjs");
var fb = {
}
module.exports = fb;
fb.configureWeb = function(options, callback) {
...
}
fb.makeRequest = function(options, callback) {
...
}
This is the main app code:
var bkjs = require("backendjs");
var core = bkjs.core;
// Using facebook module in the main app
api.app.get("some url", function(req, res) {
core.modules.facebook.makeRequest({}, function(err, data) {
...
});
});
bkj.server.start()
In case different modules is better keep separately for maintenance or development purposes they canbe split into
separate NPM packages, the structure is the same, modules must be in the modules/ folder and the package must be loadable
via require as usual. In most cases just empty index.js is enough. Such modules will not be loaded via rrquire though but
by the backendjs core.loadModule
machinery, the NPM packages are just keep different module directories separate from each other.
The config parameter allow-packages
can be used to specify NPM package names to be loaded separated by comma, as with the default
application structure all subfolders inside each NPM package will be added to the core:
If there is a config file present as etc/config
it will be loaded as well, this way each package can maintain its default config parameters if necessary
without touching other or global configuration. Although such config files will not be reloaded on changes, when NPM installs or updtes packages it
moves files around so watching the old config is no point because the updated config file will be different.
The backend support multiple databases and provides the same db layer for access. Common operations are supported and all other specific usage can be achieved by
using SQL directly or other query language supported by any particular database.
The database operations supported in the unified way provide simple actions like db.get, db.put, db.update, db.del, db.select
. The db.query
method provides generic
access to the database driver and executes given query directly by the db driver, it can be SQL or other driver specific query request.
Before the tables can be queried the schema must be defined and created, the backend db layer provides simple functions to do it:
db.describeTables({
album: {
id: { primary: 1 }, // Primary key for an album
name: { pub: 1 }, // Album name, public column
mtime: { type: "now" }, // Modification timestamp
},
photo: {
album_id: { primary: 1 }, // Combined primary key
id: { primary: 1 }, // consiting of album and photo id
name: { pub: 1, index: 1 }, // Photo name or description, public column with the index for faster search
mtime: { type: "now" }
}
});
Each database may restrict how the schema is defined and used, the db layer does not provide an artificial layer hiding all specifics, it just provides the same API and syntax, for example, DynamoDB tables must have only hash primary key or combined hash and range key, so when creating table to be used with DynamoDB, only one or two columns can be marked with primary property while for SQL databases the composite primary key can consist of more than 2 columns.
The backendjs always creates several tables in the configured database pools by default, these tables are required to support default API functionality and some
are required for backend opertions. Refer below for the Javascript modules documenttion that described which tables are created by default. In the custom applications
the db.describeTables
method can modify columns in the default table and add more columns if needed.
For example, to make age and some other columns in the accounts table public and visible by other users with additional columns the following can be
done in the api.initApplication
method. It will extend the bk_account table and the application can use new columns the same way as the already existing columns.
Using the birthday column we make 'age' property automatically calculated and visible in the result, this is done by the internal method api.processAccountRow
which
is registered as post process callback for the bk_account table. The computed property age
will be returned because it is not present in the table definition
and all properties not defined and configured are passed as is.
The cleanup of the public columns is done by the api.sendJSON
which is used by all API routes when ready to send data back to the client. If any postprocess
hooks are registered and return data itself then it is the hook responsibility to cleanup non-public columns.
db.describeTables({
bk_account: {
gender: { pub: 1 },
birthday: {},
ssn: {},
salary: { type: "int" },
occupation: {},
home_phone: {},
work_phone: {},
});
app.configureWeb = function(options, callback)
{
db.setProcessRow("post", "bk_account", this.processAccountRow);
...
callback();
}
app.processAccountRow = function(req, row, options)
{
if (row.birthday) row.age = Math.floor((Date.now() - core.toDate(row.birthday))/(86400000*365));
}
To define tables inside a module just provide a tables
property in the module object, it will be picked up by database initialization automatically.
var mod = {
name: "billing",
tables: {
invoices: {
id: { type: "int", primary: 1 },
name: {},
price: { type: "real" },
mtime: { type: "now" }
}
}
}
module.exports = mod;
// Run db setup once all the DB pools are configured, for example produce dynamic icon property
// for each record retrieved
mod.configureModule = function(options, callback)
{
db.setProcessRows("post", "invoices", function(req, row, opts) {
if (row.id) row.icon = "/images/" + row.id + ".png";
});
callback();
}
All methods will put input parameters in the req.query
, GET or POST.
One way to verify input values is to use lib.toParams
, only specified parameters will be returned and converted according to
the type or ignored.
Example:
var params = {
test1: { id: { type: "text" },
count: { type: "int" },
email: { regexp: /^[^@]+@[^@]+$/ }
}
};
api.app.all("/endpoint/test1", function(req, res) {
var query = lib.toParams(req.query, params.test1);
...
});
Here is an example how to create simple TODO application using any database supported by the backend. It supports basic operations like add/update/delete a record, show all records.
Create a file named app.js
with the code below.
var bkjs = require('backendjs');
var api = bkjs.api;
var lib = bkjs.lib;
var app = bkjs.app;
var db = bkjs.db;
// Describe the table to store todo records
db.describeTables({
todo: {
id: { type: "uuid", primary: 1 }, // Store unique task id
due: {}, // Due date
name: {}, // Short task name
descr: {}, // Full description
mtime: { type: "now" } // Last update time in ms
}
});
// API routes
app.configureWeb = function(options, callback)
{
api.app.get(/^\/todo\/([a-z]+)$/, function(req, res) {
var options = api.getOptions(req);
switch (req.params[0]) {
case "get":
if (!req.query.id) return api.sendReply(res, 400, "id is required");
db.get("todo", { id: req.query.id }, options, function(err, rows) { api.sendJSON(req, err, rows); });
break;
case "select":
options.noscan = 0; // Allow empty scan of the whole table if no query is given, disabled by default
db.select("todo", req.query, options, function(err, rows) { api.sendJSON(req, err, rows); });
break;
case "add":
if (!req.query.name) return api.sendReply(res, 400, "name is required");
// By default due date is tomorrow
if (req.query.due) req.query.due = lib.toDate(req.query.due, Date.now() + 86400000).toISOString();
db.add("todo", req.query, options, function(err, rows) { api.sendJSON(req, err, rows); });
break;
case "update":
if (!req.query.id) return api.sendReply(res, 400, "id is required");
db.update("todo", req.query, options, function(err, rows) { api.sendJSON(req, err, rows); });
break;
case "del":
if (!req.query.id) return api.sendReply(res, 400, "id is required");
db.del("todo", { id: req.query.id }, options, function(err, rows) { api.sendJSON(req, err, rows); });
break;
}
});
callback();
}
bkjs.server.start();
Now run it with an option to allow API access without an account:
node app.js -log debug -web -api-allow-path /todo -db-create-tables
To use a different database, for example PostgresSQL(running localy) or DynamoDB(assuming EC2 instance), all config parametetrs can be stored in the etc/config as well
node app.js -log debug -web -api-allow-path /todo -db-pool dynamodb -db-dynamodb-pool default -db-create-tables
node app.js -log debug -web -api-allow-path /todo -db-pool pgsql -db-pgsql-pool default -db-create-tables
API commands can be executed in the browser or using curl
:
curl 'http://localhost:8000/todo?name=TestTask1&descr=Descr1&due=2015-01-01`
curl 'http://localhost:8000/todo/select'
When the backend server starts and no -home argument passed in the command line the backend makes its home environment in the ~/.bkjs
directory.
It is also possible to set the default home using BKJS_HOME environment variable.
The backend directory structure is the following:
etc
- configuration directory, all config files are there
etc/profile
- shell script loaded by the bkjs utility to customize env variables
etc/config
- config parameters, same as specified in the command line but without leading -, each config parameter per line:
Example:
debug=1
db-pool=dynamodb
db-dynamodb-pool=http://localhost:9000
<EMAIL_ADDRESS>To specify other config file: bkjs shell -config-file file
etc/config.local - same as the config but for the cases when local environment is different than the production or for dev specific parameters
some config parameters can be condigured in DNS as TXT records, the backend on startup will try to resolve such records and use the value if not empty.
All params that marked with DNS TXT can be configured in the DNS server for the domain where the backend is running, the config parameter name is
concatenated with the domain and queried for the TXT record, for example: cache-host
parameter will be queried for cache-host.domain.name for TXT record type.
etc/crontab
- jobs to be run with intervals, JSON file with a list of cron jobs objects:
Example:
Create file in ~/.backend/etc/crontab with the following contents:
[ { "cron": "0 1 1 * * 1,3", "job": { "app.cleanSessions": { "interval": 3600000 } } } ]
Define the function that the cron will call with the options specified, callback must be called at the end, create this app.js file
var bkjs = require("backendjs");
bkjs.app.cleanSessions = function(options, callback) {
bkjs.db.delAll("session", { mtime: options.interval + Date.now() }, { ops: "le" }, callback);
}
bkjs.server.start()
Start the jobs queue and the web server at once
bkjs master -jobs-workers 1 -jobs-cron
etc/crontab.local - additional local crontab that is read after the main one, for local or dev environment
modules
- loadable modules with specific functionality
images
- all images to be served by the API server, every subfolder represent naming space with lots of subfolders for images
var
- database files created by the server
tmp
- temporary files
web
- Web pages served by the static Express middleware
Database layer support caching of the responses using db.getCached
call, it retrieves exactly one record from the configured cache, if no record exists it
will pull it from the database and on success will store it in the cache before returning to the client. When dealing with cached records, there is a special option
that must be passed to all put/update/del database methods in order to clear local cache, so next time the record will be retrieved with new changes from the database
and refresh the cache, that is { cached: true }
can be passed in the options parameter for the db methods that may modify records with cached contents. In any case
it is required to clear cache manually there is db.clearCache
method for that.
Also there is a configuration option -db-caching
to make any table automatically cached for all requests.
If no cache is configured the local driver is used, it keeps the cache on the master process in the LRU pool and any wroker or Web process
communicate with it via internal messaging provided by the cluster
module. This works only for a single server.
Set ipc-cache=memcache://HOST[:PORT]
that points to the host running memcached. To support multiple servrs add the option
ipc-cache-options-servers=<IP_ADDRESS>,<IP_ADDRESS>:5000
.
Set ipc-cache=redis://HOST[:PORT]
that points to the server running Redis server.
To support more than one master Redis server in the client add additional servers in the servers parameter,
ipc-cache-options-servers=<IP_ADDRESS>,<IP_ADDRESS>:5000
, the client will reconnect automatically on every
disconnect. To support quick failover it needs a parameter for the node-redis
module (which is used by the driver) max_attempts
to be a
number how many attempts to reconnect before switching to another server like ipc-cache-options-max_attempts=3
. If there is only one
server then it will keep reconnecting until total reconnect time exceeds the connect_timeout
ms.
Any other node-redis
module parameter can be passed as well.
Cache configurations also can be passed in the url, the system supports special parameters that start with bk-
, it will extract them into options automatically.
For example:
ipc-cache=redis://host1?bk-servers=host2,host3&bk-max_attempts=3
ipc-cache-backup=redis://host2
ipc-cache-backup-options-max_attempts=3
To enable Redis Sentinel pass in the option -sentinel-servers
: ipc-cache=redis://host1?bk-sentinel-servers=host1,host2
.
The system will connect to the sentinel, get the master cache server and connect the cache driver to it, also it will listen constantly on
sentinel events and failover to a new master autimatically. Sentinel use the regular redis module and supports all the same
parameters, to pass options to the sentinel driver prefix them with sentinel-
:
ipc-cache=redis://host1?bk-servers=host2,host3&bk-max_attempts=3&bk-sentinel-servers=host1,host2,host3
ipc-cache-backup=redis://host2
ipc-cache-backup-options-sentinel-servers=host1,host2
ipc-cache-backup-options-sentinel-max_attempts=5
Publish/subscribe functionality allows clients to receive notifications without constantly polling for new events. A client can be anything but
the backend provides some partially implemented subscription notifications for Web clients using the Long Poll.
The Account API call /account/subscribe
can use any pub/sub mode.
The flow of the pub/sub operations is the following:
/account/subscribe
API request, the connection is made and is kept open indefenitely or as long as configured using api-subscribe-timeout
.api.subscribe
method with the key being the account id, this will subscribe to the events for the current
account and registers a callback to be called if any events occured. The HTTP connection is kept open.ipc.publish
after the DB operation succedes. If the messaging is configured, it publishes the message for the account, the
message being a JSON object with the request API path and mtime, other properties depend on the call made./account/subscribe
receives an eventTo configure the backend to use Redis for PUB/SUB messaging set ipc-queue=redis://HOST
where HOST is IP address or hostname of the single Redis server.
This will use native PUB/SUB Redis feature.
To configure the backend to use Redis for job processing set ipc-queue=redisq://HOST
where HOST is IP address or hostname of the single Redis server.
This driver implements reliable Redis queue, with visibilityTimeout
config option works similar to AWS SQS.
Once configured, then all calls to jobs.submitJob
will push jobs to be executed to the Redis queue, starting somewhere a backend master
process with -jobs-workers 2
will launch 2 worker processes which will start pulling jobs from the queue and execute.
An example of how to perform jobs in the API routes:
app.processAccounts = function(options, callback) {
db.select("bk_account", { type: options.type || "user" }, function(err, rows) {
...
callback();
});
}
api.all("/process/accounts", function(req, res) {
jobs.submitJob({ job: { "app.processAccounts": { type: req.query.type } } }, function(err) {
api.sendReply(res, err);
});
});
To configure the backend to use RabbitMQ for messaging set ipc-queue=amqp://HOST
and optionally amqp-options=JSON
with options to the amqp module.
Additional objects from the config JSON are used for specific AMQP functions: { queueParams: {}, subscribeParams: {}, publishParams: {} }. These
will be passed to the corresponding AMQP methods: amqp.queue, amqp.queue.sibcribe, amqp.publish
. See AMQP node.js module for more info.
This is a simple queue implementation using the atomic UPDATE, it polls for new jobs in the table and updates the status, only who succeeds with the update takes the job and executes it. It is not effective but can be used for simple and not busy systems for more or less long jobs. The advantage is that it uses the same database and does not require additional servers.
To use AWS SQS for job processing set ipc-queue=https://sqs.amazonaws.com....
, this queue system will poll SQS for new messeges on a worker
and after succsesful execution will delete the message. For long running jobs it will automatically extend visibility timeout if it is configured.
The local queue is implemented on the master process as a list, communication is done via local sockets between the master and workers. This is intended for a single server development pusposes only.
This is default setup of the backend when all API requests except must provide valid signature and all HTML, Javascript, CSS and image files
are available to everyone. This mode assumes that Web development will be based on 'single-page' design when only data is requested from the Web server and all
rendering is done using Javascript. This is how the examples/api/api.html
develpers console is implemented, using JQuery-UI and Knockout.js.
To see current default config parameters run any of the following commands:
bkjs bkhelp | grep api-allow
node -e 'require("backendjs").core.showHelp()'
To enable open registration in this mode just add config parameter api-allow-path=^/account/add$
.
This is a mode when the whole Web site is secure by default, even access to the HTML files must be authenticated. In this mode the pages must defined 'Backend.session = true' during the initialization on every html page, it will enable Web sessions for the site and then no need to sign every API reauest.
The typical client Javascript verification for the html page may look like this, it will redirect to login page if needed, this assumes the default path '/public' still allowed without the signature:
<script src="/js/jquery.js"></script>
<link href="/css/bootstrap.css" rel="stylesheet">
<script src="/js/bootstrap.js"></script>
<script src="/js/knockout.js" type="text/javascript"></script>
<script src="/js/crypto.js" type="text/javascript"></script>
<script src="/js/bkjs.js" type="text/javascript"></script>
<script src="/js/bkjs-bootstrap.js" type="text/javascript"></script>
<script src="/js/bkjs-ko.js" type="text/javascript"></script>
<script>
$(function () {
Bkjs.session = true;
$(Bkjs).on("bkjs.nologin", function() { window.location='/public/index.html'; });
Bkjs.koInit();
});
</script>
On the backend side in your application app.js it needs more secure settings defined i.e. no html except /public will be accessible and
in case of error will be redirected to the login page by the server. Note, in the login page Bkjs.session
must be set to true for all
html pages to work after login without singing every API request.
app.configureMiddleware = function(options, callback) {
this.allow.splice(this.allow.indexOf('^/$'), 1);
this.allow.splice(this.allow.indexOf('\\.html$'), 1);
callback();
}
api.registerPreProcess('', /^\/$|\.html$/, function(req, status, callback) {
if (status.status != 200) {
status.status = 302;
status.url = '/public/index.html';
}
callback(status);
});
The simplest way is to configure ws-port
to the same value as the HTTP port. This will run WebSockets server along the regular Web server.
All requests must be properly signed with all parameters encoded as for GET requests.
Example:
wscat --connect ws://localhost:8000
connected (press CTRL+C to quit)
> /account/get
< {
"status": 400,
"message": "Invalid request: no host provided"
}
>
There is no ready to use support for different versions of API at the same because there is no just one solution that satifies all applications. But there are tools ready to use that will allow to implement such versioning system in the backend. Some examples are provided below:
Fixed versions
This is similar to AWS version system when versions are fixed and changed not very often. For such cases the backend exposes core.bkVersion
which is
supposed to be a core backend version. This version is returned with every backend reponse in the Server: header. A client also can specify the core version
using bk-version
header. When a request is parsed and the version is provided it will be set in the request options object as apiVersion
.
All API routes are defined using Express middleware and one of the possible ways of dealing with different versions can look like this, by appending version to the command it is very simple to call only changed API code.
api.all(/\/domain\/(get|put|del)/, function(req, res) {
var options = api.getOptions(req);
var cmd = req.params[0];
if (options.apiVersion) cmd += "/" + options.apiVersion;
switch (cmd) {
case "get":
break;
case "get/2015-01-01":
break;
case "put":
break;
case "put/2015-02-01":
break;
case "del"
break;
}
});
Application semver support For cases when applications support Semver kind of versioning and it may be too many releases the method above still can be used while the number of versions is small, once too many different versions with different minor/patch numbers, it is easier to support greater/less comparisons.
The application version bk-app
can be supplied in the query or as a header or in the user-agent HTTP header which is the easiest case for mobile apps.
In the middlware, the code can look like this:
var options = api.getOptions(req);
var version = lib.toVersion(options.appVersion);
switch (req.params[0]) {
case "get":
if (version < lib.toVersion("1.2.5")) {
res.json({ id: 1, name: "name", description: "descr" });
break;
}
if (version < lib.toVersion("1.1")) {
res.json([id, name]);
break;
}
res.json({ id: 1, name: "name", descr: "descr" });
break;
}
The actual implementation can be modularized, split into functions, controllers.... there are no restrictions how to build the working backend code, the backend just provides all necessary information for the middleware modules.
The purpose of the bkjs
shell script is to act as a helper tool in configuring and managing the backend environment
and as well to be used in operations on production systems. It is not required for the backend operations and provided as a convenience tool
which is used in the backend development and can be useful for others running or testing the backend.
Running without arguments will bring help screen with description of all available commands.
The tool is multi-command utility where the first argument is the command to be executed with optional additional arguments if needed. On Linux, when started the bkjs tries to load and source the following config files:
/etc/sysconfig/bkjs
$BKJS_HOME/etc/profile
Any of the following config files can redefine any environmnt variable thus pointing to the correct backend environment directory or customize the running environment, these should be regular shell scripts using bash syntax.
Most common used commands are:
bkjs watch - run the backend or the app for development purposes, uses local app.js if exists otherwise runs generic server
bkjs shell - start REPL shell with the backend module loaded and available for use, all submodules are available in the shell as well like core, db, api
bkjs sync [-path path] [-host host] [-user user] - sync sources of the app with the remote site, this is for developent version of the backend only
bkjs init-server [-home path] [-user user] [-host name] [-domain name] - initialize Linux instance(Amazon,CentOS) for backend use, optional -home can be specified where the backend home will be instead of ~/.bkjs, optional -user tells to use existing user instead of the current user and not root.
This command will create /etc/sysconfig/bkjs
file with BKJS_HOME set to the home of the
backendjs app which was pased in the command line. This makes the bkjs or bksh run globally regardless of the current directory.
The server supports simple web bundling using uglify-js utility. To enable it just add tyo the local config a list of directories to be watched for changes. For example adding these lines to the local config will enable the watcher and bundle support
watch-web=web/js,web/css,$HOME/src/js,$HOME/src/css
watch-ignore=.bundle.(js|css)$
build-web=bkjs web-bundle -dev
Now instead of incding a bunch of .js or css files in the html pages it only needs /js/bkjs.bundle.js and /css/bkjs.bundle.css. The configuration is in the package.json file.
The simple script below allows to build the bundle and refresh Chrome tab automatically, saves several clicks:
#!/bin/bash
bkjs web-bundle -dev -file $2
[ "$?" != "0" ] && exit
osascript -e "tell application \"Google Chrome\" to reload (tabs of window 1 whose URL contains \"$1\")"
#osascript -e 'tell application "Google Chrome" to tell the active tab of its first window to reload'
To use it call this script instead in the config.local:
build-web=web-bundle.sh /website
Here is the example how to setup new custom AWS server, it is not required and completely optional but bkjs provies some helpful commands that may simplify new image configuration.
start new AWS instance via AWS console, use Amazon Linux
login as ec2-user
install commands
yum-config-manager --enable epel
sudo yum install npm
npm install backendjs
sudo bkjs init-service
bkjs restart
try to access the instance via HTTP port 8000 for the API console or documentation
after reboot the server will be started automatically
To make an API appliance by using the backendjs on the AWS instance as user ec2-user with the backend in the user home
start new AWS instance via AWS console, use Amazon Linux or CentOS 6
login as ec2-user
install commands
git clone https://github.com/vseryakov/backendjs.git
sudo backendjs/bkjs install-ec2 -tools $(pwd)/backendjs/tools
bkjs restart
run ps agx
, it should show several backend processes running
try to access the instance via HTTP port for the API console or documentation
NOTE: if running behind a Load balancer and actual IP address is needed set Express option in the command line -api-express-options {"trust%20proxy":1}
. In the config file
replacing spaces with %20 is not required.
As with any node.js module, the backendjs app can be packaged into zip file according to AWS docs and deployed the same way as any other node.js app. Inside the app package etc/config file can be setup for any external connections.
Note: on OS X laptop the -aws-sdk-profile uc
when AWS credentials are in the ~/.aws/credentials.
On the running machine which will be used for an image:
bksh -aws-create-image -no-reboot
Use an instance by tag for an image:
bksh -aws-create-image -no-reboot -instance-id `bkjs ec2-show -tag api -fmt id | head -1`
When launching from an EC2 instance no need to specify any AWS credentials.
admin (EC2)
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type t2.small -subnet-name api -name admin -elb-name Admin -alarm-name alarms -public-ip 1 -dry-run
api (EC2)
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type m3.large -subnet-name api -name api -elb-name api -alarm-name alarms -public-ip 1 -dry-run
jobs (EC2)
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type t2.small -subnet-name internal -name sync -alarm-name alarms -dry-run bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type t2.small -subnet-name internal -name sync -zone 1c -alarm-name alarms -dry-run
Elasticsearch
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type m3.large -subnet-name internal -name elasticsearch -bkjs-cmd stop-service -bkjs-cmd "init-elasticsearch-service -memsize 50" -alarm-name alarms -public-ip 1 -dry-run
Redis
bksh -aws-sdk-profile uc -aws-launch-instances -aws-instance-type m3.large -subnet-name internal -name redis -bkjs-cmd stop-service -bkjs-cmd "init-redis-service -memsize 70" -alarm-name alarms -public-ip 1 -dry-run
bksh -aws-create-launch-config -config-name elasticsearch -aws-sdk-profile uc -instance-type m3.large -update-groups -bkjs-cmd stop-service -bkjs-cmd init-logwatcher -bkjs-cmd "init-elasticsearch-service -memsize 50" -device /dev/xvda:gp2:16 -dry-run
bksh -aws-create-launch-config -config-name jobs -aws-sdk-profile uc -update-groups -dry-run
bksh -aws-create-launch-config -config-name api -aws-sdk-profile uc -update-groups -dry-run
bksh -aws-set-route53 -name elasticsearch.ec-internal -filter elasticsearch
By default the Web proceses spawned by the server are load balanced using default cluster module which relies on the OS to do scheduling. On Linux with node 0.10 this is proven not to work properly due to the kernel keeping the context switches to a minimum thus resulting in one process to be very busy while the others idle. Node versions 4 and above perform round-robin by default.
For such case the Backendjs implements the proxy mode by setting proxy-port
config paremeter to any number above 1000, this will be the initial
port for the web processes to listen for incoming requests, for example if use -proxy-port 3000
and launch 2 web processes they will listen on ports
3000 and 3001. The main server process will start internal HTTP proxy and will perform round-robin load balancing the incoming requests between the web proceses by forwarding
them to the web processes over TCP and then returning the responses back to the clients.
The first thing when deploying the backend into production is to change API HTTP port, by default is is 8000, but we would want port 80 so regardless how the environment is setup it is ultimatley 2 ways to specify the port for HTTP server to use:
config file
The config file is always located in the etc/ folder in the backend home directory, how the home is specified depends on the system but basically it can be
defined via command line arguments as -home
or via environment variables when using bkjs. See bkjs documentation but on AWS instances created with bkjs
init-server
command, for non-standard home use /etc/sysconfig/bkjs
profile, specify BKJS_HOME=/home/backend
there and the rest will be taken care of
command line arguments
When running node scripts which use the backend, just specify -home
command line argument with the directory where yor backend should be and the backend will use it
Example:
node app.js -home $HOME -port 80
config database
If -db-config
is specified in the command line or db-config=
in the local config file, this will trigger loading additional
config parameters from the specified database pool, it will load all records from the bk_config
table on that db pool. Using the database to store
configuration make it easier to maintain dynamic environment for example in case of auto scaling or lanching on demand, this way
a new instance will query current config from the database and this eliminates supporting text files and distributing them to all instances.
The config database is refreshed from time to time acording to the db-config-interval
parameter, also all records with ttl
property in the bk_config
will be pulled every ttl interval and updated in place.
DNS records
Some config options may be kept in the DNS TXT records and every time a instance is started it will query the local DNS for such parameters. Only a small subset of
all config parameters support DNS store. To see which parmeteres can be stored in the DNS run bkjs show-help
and look for 'DNS TXT configurable'.
for DB drivers and ImageMagick to work propely it needs some dependencies to be installed:
port install libpng jpeg tiff lcms2 mysql56 postgresql93
make sure there is no openjpeg15 installed, it will conflict with ImageMagick jp2 codec
git clone https://github.com/vseryakov/backendjs.git
or git clone git@github.com:vseryakov/backendjs.git
cd backendjs
if node.js is already installed skip to the next section
to install binary release run the command, it will install it into /opt/local on Darwin
bkjs install-node
# To install into different path
bkjs install-node -prefix /usr/local/node
Important: Add NODE_PATH=$BKJS_PREFIX/lib/node_modules to your environment in .profile or .bash_profile so node can find global modules, replace $BKJS_PREFIX with the actual path unless this variable is also set in the .profile
to install all dependencies and make backendjs module and bkjs globally available:
npm link backendjs
to run local server on port 8000 run command:
bkjs web
to start the backend in command line mode, the backend environment is prepared and initialized including all database pools. This command line access allows you to test and run all functions from all modules of the backend without running full server similar to node.js REPL functionality. All modules are accessible from the command line.
$ ./bkjs shell
> core.version
'0.70.0'
> logger.setLevel('info')
While creating Backendjs there were many questions and issues to be considered, some i was able to implement, some still not. Below are the thoughts that might be useful when desining, developing or choosing the API platform:
All API endpoints are optional and can be disabled or replaced easily. By default the naming convention is:
/namespace/command[/subname[/subcommand]]
Any HTTP methods can be used because its the command in the URL that defines the operation. The payload can be urlencoded query parameters or JSON or any other format supported by any particular endpoint. This makes the backend universal and usable with any environment, not just a Web browser. Request signature can be passed in the query so it does not require HTTP headers at all.
All requests to the API server must be signed with account login/secret pair.
The resulting signature is sent as HTTP header bk-signature
or in the header specified by the api-signature-name
config parameter.
For JSON content type, the method must be POST and no query parameters specified, instead everything should be inside the JSON object which is placed in the body of the request. For additional safety, SHA1 checksum of the JSON paylod can be calculated and passed in the signature, this is the only way to ensure the body is not modified when not using query parameters.
See web/js/bkjs.js function Bkjs.createSignature
or
api.js function api.createSignature
for the Javascript implementations.
There is also native iOS implementation Bkjs.m.
/auth
This API request returns the current user record from the bk_auth
table if the request is verified and the signature provided
is valid. If no signature or it is invalid the result will be an error with the corresponding error code and message.
By default this endpoint is secured, i.e. requires a valid signature.
Parameters:
_session=1
- if the call is authenticated a cookie with the session signature is returned, from now on
all requests with such cookie will be authenticated, the primary use for this is Web apps
_accesstoken=1
- returns new access token to be used for subsequent requests without a signature for the current account,
the token is short lived with expiration date returned as well. This access token can be used instead of a signature and
is passed in the query as bk-access-token=TOKEN
.
Example:
/auth?_accesstoken=1
> { id: "XXXX...", name: "Test User", "bk-access-token": "XXXXX....", "bk-access-token-age": 604800000 }
/account/get?bk-access-token=XXXXXX...
> { id: "XXXX...", name: "Test User", ... }
/login
Same as the /auth but it uses secret for user authentication, this request does not need a signature, just simple login and secret query parameters to be sent to the backend. This must be sent over SSL.
The intened usage is for Web sessions which use sessions cookies when sent with _session=1
or to be used with access tokens when
sent with _accesstoken=1
.
Parameters:
login
- account login
zecret
- account secret
_session=1
- same as in /auth request
_accesstoken=1
- same as in /auth reuest
On successful login, the result contains full account record including the secret, this is the only time when the secret is returned back
Example:
$.ajax({ url: "/login?login=test123&secret=test123&_session=1",
success: function(json, status, xhr) { console.log(json) }
});
> { id: "XXXX...", name: "Test User", login: "test123", ...}
/logout
Logout the current user, clear session cookies if exist. For pure API access with the signature this will not do anything on the backend side.
The accounts API manages accounts and authentication, it provides basic user account features with common fields like email, name, address.
This is implemented by the accounts
module from the core. To enable accounts functionality specify -allow-modules=bk_accounts
.
/account/get
Returns information about current account or other accounts, all account columns are returned for the current account and only public columns returned for other accounts. This ensures that no private fields ever be exposed to other API clients. This call also can used to login into the service or verifying if the given login and secret are valid, there is no special login API call because each call must be signed and all calls are stateless and independent.
Parameters:
bk-access-token
Note: When retrieving current account, all properties will be present including the location, for other accounts only the properties marked as pub
in the
bk_account
table will be returned.
Response:
{ "id": "57d07a4e28fc4f33bdca9f6c8e04d6c3",
"name": "Test User",
"name": "Real Name",
"mtime":<PHONE_NUMBER>,
"latitude": 34,
"longitude": -118,
"geohash": "9qh1",
"login": "testuser",
}
/account/add
Add new account, all parameters are the columns from the bk_account
table, required columns are: name, secret, login.
To enable open registration add api-allow-path=^/account/add$
to the config file or specify in the command line.
More complex ways to perform registration will require adding pre and.or post callbacks to handle account registration
for example with invitation codes....
In the table bk_auth
, the column type
is used to distinguish between account roles, by default only account with type admin
can
add other accounts with this type specified, this column can also be used in account permissions implementations. Because it is in the bk_auth table,
all columns of this table are available as req.account
object after the successful authentication where req
is an Express request object used in the middleware
parameters.
Secret and login can be anything, the backend does not require any specific formats and does not process the contents of the login/secret fields.
There are several ways to create authenticated account:
web/js/bkjs.js
, if Bkjs.scramble
is set to 1 then the secret is replaced by the BASE64_HMAC_SHA256(secret, login) automatically,
no actual secret is ever saved or sent, only used in the login form. This is intended for Web apps not to store the actual secret anywhere in the memory or localStorage,
for the backend this is still just a secret.Example:
<EMAIL_ADDRESS>How to make an account as admin
# Run backend shell
bkjs shell
# Update record by login
> db.update("bk_auth", { login: 'login@name', type: 'admin' });
/account/select
Return list of accounts by the given condition, calls db.select
for bk_account table. Parameters are the column values to be matched and
all parameters starting with underscore are control parameters that goes into options of the db.select
call with underscore removed. This will work for SQL
databases only because DynamoDB or Cassandra will not search by non primary keys. In the DynamoDB case this will run ScanTable action which will be very expensive for
large tables. Supports special query parameters _select,_ops
, see docs about db.select
for more info.
Example:
/account/search?email=test&_ops=email,begins_with
/account/search?name=test
Response:
{ "data": [{
"id": "57d07a4e28fc4f33bdca9f6c8e04d6c3",
"name": "Test User1",
"mtime":<PHONE_NUMBER>,
"login": "test1",
},
{
"id": "57d07a4e2824fc43bd669f6c8e04d6c3",
"name": "Test User2",
"mtime":<PHONE_NUMBER>,
"login": "test2",
}],
"next_token": ""
}
/account/del
Delete current account, after this call no more requests will be authenticated with the current credentials
/account/update
Update current account with new values, the parameters are columns of the table bk_account
, only columns with non empty values will be updated.
Example:
/account/update?name=New%2BName&gender=m
/account/put/secret
Change account secret for the current account, no columns except the secret will be updated and expected.
Parameters:
Example:
/account/put/secret?secret=blahblahblah
/account/subcribe
Subscribe to account events delivered via HTTP Long Poll, a client makes the connection and waits for events to come, whenever
somebody updates the account's counter or send a message or creates a connection to this account the event about it will be sent to this HTTP
connection and delivered as JSON object. This is not a persistent queue so if not listening, all events will just be ignored, only events published
since the connect will be delivered. To specify what kind of events needs to be delivered, match
query parameter can be specified which is a
RegExp of the whole event body string.
Note: On the server side there is a config parameter api-subscribe-interval
which defines how often to deliver notifications, by default it is 5 seconds which means
only every 5 seconds new events will be delivered to the Web client, if more than one event happened, they all accumulate and will be sent as a JSON list.
Example:
/account/subscribe
/account/subscribe?match=connection/add.*type:*like
// To run in the browser:
(function poll() {
Bkjs.send({ url: "/account/subscribe", complete: poll }, function(data) {
console.log("received event:", data);
});
})();
Response:
[ { "path": "/message/add", "mtime:"<PHONE_NUMBER>, "type": "1" },
{ "path": "/counter/incr", "mtime:"<PHONE_NUMBER>, "type": "like,invite" } },
{ "path": "/connection/add", "mtime":<PHONE_NUMBER>, "type": "like" } ]
/account/select/icon
Return a list of available account icons, icons that have been uploaded previously with /account/put/icon calls. The url
property is an URL to retrieve this particular icon.
Parameters:
Example:
/account/select/icon?id=12345
Response:
[ { id: '12345', type: '1', url: '/account/get/icon?id=12345&type=1' },
{ id: '12345', type: '2', url: '/account/get/icon?id=12345&type=2' } ]
/account/get/icon
Return an account icon, the icon is returned in the body as binary BLOB, if no icon with specified type exists, i.e. never been uploaded then 404 is returned.
Parameters:
Example:
/account/get/icon?type=2
/account/put/icon
Upload an account icon, once uploaded, the next /account/get
call will return propertis in the format iconN
wheer N is any of the
type query parameters specified here, for example if we uploaded an icon with type 5, then /account/get will return property icon5 with the URL
to retrieve this icon.
By default all icons uploaded only accessible for the account which uploaded them.
Parameters:
application/json
and data should be sent with POST requestExample:
/account/put/icon?type=1&icon=iVBORw0KGgoAAAANSUhEUgAAAAcAAAAJCAYAAAD+WDajAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAOwgAADs....
/account/del/icon
Delete account icon
Parameters:
Example:
/account/icon/del?type=1
When running with AWS load balancer there should be a url that a load balancer polls all the time and this must be very quick and lightweight request. For this
purpose there is an API endpoint /ping
that just responds with status 200. It is not open by default, the allow-path
or other way to allow non-authenticted access
needs to be configured. This is to be able to control how pinging can be perform in the apps in cae it is not simple open access.
This endpoint can server any icon uploaded to the server for any account, it is supposed to be a non-secure method, i.e. no authentication will be performed and no signagture
will be needed once it is confgiured which prefix can be public using api-allow
or api-allow-path
config parameters.
The format of the endpoint is:
/image/prefix/id/type[.png|.jpg]
Example:
# Configure accounts icons to be public in the etc/config
api-allow-path=/image/account/
# Or pass in the command line
./app.sh -api-allow-path /image/account/
# Make requests
/image/account/12345/0
/image/account/12345/1
/image/account/12345/1.jpg
#Return icons for account 12345 for types 0 and 1
The icons API provides ability for an account to store icons of different types. Each account keeps its own icons separate form other
accounts, within the account icons can be separated by prefix
which is just a namespace assigned to the icons set, for example to keep messages
icons separate from albums, or use prefix for each separate album. Within the prefix icons can be assigned with unique type which can be any string.
Prefix and type can consist from alphabetical characters and numbers, dots, underscores and dashes: [a-z0-9._-]. This means, they are identificators, not real titles or names, a special mapping between prefix/type and album titles for example needs to be created separately.
The supposed usage for type is to concatenate common identifiers first with more specific to form unique icon type which later can be queried by prefix or exactly by icon type. For example album id can be prefixed first, then sequential con number like album1:icon1, album1:icon2.... then retrieving all icons for an album would be only query with album1: prefix.
The is implemented by the icons
module from the core. To enable this functionality specify -allow-modules=bk_icons
.
/icon/get
Return icon for the current account in the given prefix, icons are kept on the local disk in the directory
configured by -api-images-dir
parameter(default is images/ in the backend directory). Current account id is used to keep icons
separate from other accounts. Icon presense is checked in the bk_icon table before returning it and if any permissions are set in
the acl_allow
column it will be checked if this icon can be returned.
The following parameters can be used:
prefix
- must be specified, this defines the icons namespacetype
is used to specify unique icon created with such type which can be any string._ext
- image extension, like png or jpg if it was saved with it previously/icon/put
Upload new icon for the given account in the folder prefix, if type is specified it creates an icons for this type to separate
multiple icons for the same prefix. type
can be any string consisting from alpha and digits characters. It creates a record in the bk_icon
table with all the paramaters passed.
The following parameters can be used:
/account/put/icon
for the format and usage/icon/upload
Upload a new image and store on the server, no record is created in bk_icon table, just simple image upload, but all the same query parameters as for /icon/put are accepted. Returns an JSON object with url property being the full path to the uploaded image.
/icon/del
Delete the default icon for the current account in the folder prefix or by type
/icon/select
Return list of available icons for the given prefix adn type, all icons starting with prefix/type will be returned,
the url
property will provide full URL to retrieve the icon contents
Example:
/icon/select?prefix=album&type=me
/icon/select?prefix=album&type=12345
Responses:
[ { id: 'b3dcfd1e63394e769658973f0deaa81a', type: 'me-1', icon: '/icon/get?prefix=album&type=me1' },
{ id: 'b3dcfd1e63394e769658973f0deaa81a', type: 'me-2', icon: '/icon/get?prefix=album&type=me2' } ]
[ { id: 'b3dcfd1e63394e769658973f0deaa81a', type: '12345-f0deaa81a', icon: '/icon/get?prefix=album&type=12345-f0deaa81a' } ]
The file API provides ability to store and retrieve files. The operations are similar to the Icon API.
This is implemented by the files
module from the core. To enable this functionality specify -allow-modules=bk_files
.
/file/get
Return a file with given prefix and name, the contents are returned in the response body.
The following parameters can be used:
prefix
- must be provided, defines the namescape where the file is storedname
- name of the file, required/file/put
Store a file on the backend, the file can be sent using form multipart upload or as JSON
The following parameters can be used:
prefix
- must be provided, defines the namescape where the file is storedname
- name of the file, required_name
- name of the property that contains the file contents, for use with JSON or defines the name of the file attribute for multipart upload_tm
- append the current timestamp to the file name_ext
- extention to be assign to the file, otherwise the actual extension from the file name is used/file/del
Delete file, prefix and name must be given
The connections API maintains two tables bk_connection
and bk_reference
for links between accounts of any type. bk_connection table maintains my
links, i.e. when i make explicit connection to other account, and bk_reference table is automatically updated with reference for that other account that i made
a connection with it. No direct operations on bk_reference is allowed.
This is implemented by the connections
module from the core. To enable this functionality specify -allow-modules=bk_connections
.
/connection/add
/connection/put
Create or replace a connection between two accounts, required parameters are:
peer
- id of account to connect totype
- type of connection, like,dislike,....This call automatically creates a record in the bk_reference table which is reversed connection for easy access to information like ''who is connected to me''.
Example:
/connection/add?peer=12345&type=invite&state=sent
/connection/update
/connection/incr
Update other properties of the existing connection, for connections that may take more than i step or if a connection has other data associated with it beside
the type of the connection.
Example:
/connection/update?peer=12345&type=invite&state=accepted
/connection/del
Delete existing connection(s), id
and/or type
may be be specified, if not all existing connections will be deleted.
Example:
/connection/del?type=invite&peer=12345
/connection/get
Return a single connection for given id
Parameters:
Example:
/connection/get?peer=12345&type=like
Response:
{ "id": "1111",
"type: "like",
"peer": "12345",
"mtime": "2434343543543" }
/reference/get
Return a single reference record for given account id, works the same way as /connection/get
/connection/select
Receive all my connections of the given type, i.e. connection(s) i made, if id
is given only one record for the specified connection will be returned. Supports special
query parameters _select,_ops,_desc
, see docs about db.select
for more info.
Example:
# Return all accounts who i invited
/connection/select?type=invite
# Return connection for specific type and account id
/connection/select?type=invite&peer=12345
# Return accounts who i invited me after specified mtime
/connection/select?type=invite&_ops=mtime,gt&mtime=12334312543
# Return accounts who i invited before specified mtime
/connection/select?type=invite&_ops=mtime,le&_desc=1&mtime=12334312543
Response:
{ "data": [ { "id": "111",
"type": "invite",
"peer": "12345",
"status": "",
"mtime": "12334312543"
}],
"next_token": ""
}
/reference/select
Receive all references that connected with my account, i.e. connections made by somebody else with me, works the same way as for connection query call
Example:
# Return all accounts who invited me
/reference/select?type=invite
# Return accounts who invited me after specified mtime
/reference/select?type=invite&_ops=mtime,gt&mtime=12334312543
Response:
{ "data": [ { "id": "111",
"type": "invite",
"peer": "12345",
"status": "",
"mtime": "12334312543"
}],
"next_token": ""
}
The location API maintains a table bk_location
with geolocation coordinates for accounts and allows searching it by distance. The configuration parameter
min-distance
defines the radius for the smallest bounding box in km containing single location, radius searches will combine neighboring boxes of
this size to cover the whole area with the given distance request, also this affects the length of geohash keys stored in the bk_location table. By default min-distance is 5 km
which means all geohashes in bk_location table will have geohash of size 4. Once min-distance is set it cannot be changed without rebuilding the bk_location table with new geohash size.
The location search is implemented by using geohash as a primary key in the bk_location table with the account id as the second part of the primary key, for DynamoDB this is the range key. When request comes for all matches for the location for example 37.7, -122.4, the search that is executed looks like this:
9q8y
This is implemented by the locations
module from the core. To enable this functionality specify allow-modules=bk_locations
.
/location/put
Store currenct location for current account, latitude and longitude parameters must be given, this call will update the bk_account table as well with
these coordinates
Example:
/location/put?latitude=-188.23232&longitude=23.4545454
/location/get
Return matched accounts within the distance(radius) specified by distance=
parameter in kilometers and current position specified by latitude/longitude paraemeters. This
call returns results in chunks and requires navigation through all pages to receive all matched records. Records returned will start with the closest to the current
point. If there are more matched records than specified by the _count
, the next_token
property is set with the token to be used in the subsequent call,
it must be passed as is as _token=
parameter with all original query parameters.
Note: The current account will not be present in the results even if it is within the range, to know my own location use /account/get
call.
Example:
/location/get?distance=10&latitude=-118.23434&longitude=23.45665656&_count=25
/location/get?distance=10&latitude=-118.23434&longitude=23.45665656&_count=25&_token=FGTHTRHRTHRTHTTR.....
Response:
{ "data": [ { "id": "12345",
"distance": 5,
"latitude": -118.123,
"longitude": 23.45
"mtime": "12334312543"
},
{ "id": "45678",
"distance": 5,
"latitude": -118.133,
"longitude": 23.5
"mtime": "12334312543"
}],
"next_token": ""
}
The messaging API allows sending and receiving messages between accounts, it supports text and images. All new messages arrive into the bk_messsage table, the inbox. The client may keep messages there as new, delete or archive them. Archiving means transfering messages into the bk_archive table. All sent messages are kept in the bk_sent table.
This is implemented by the messages
module from the core. To enable this functionality specify -allow-modules=bk_messages
.
/message/get/unread
Return how many unread messages in the inbox, this is just a flag to signal about new messages, the actual number may not be up to date,
it is cleared on messages read.
Example:
/message/get/unread
Response:
{ count: 1 }
/message/get
Read all messages from the inbox.
Parameters:
_archive
- if set to 1, all returned messages will be archived automatically, so no individual /message/read call needed_trash
- if set to 1, all returned messages will be deleted, not archived_total
- if set to 1 then return how many messages in the inbox
the unread flag with the actual number of unread messages.Example:
# Get all new messages
/message/get
# Get all new messages and archive them
/message/get?_archive=1
# Get all new messages from the specific sender
/message/get?sender=12345
# How many new messages
/message/get?_total=1
/message/get/archive
Receive archived messages. The images are not returned, only link to the image in icon
property of reach record,
the actual image data must be retrieved separately.
Parameters:
mtime
- if specified then only messages received since that time will be retirned, it must be in milliseconds since midnight GMT on January 1, 1970, this is what
Date.now() return in Javascript.sender
- if specified then all messages from the given sender will be returned.NOTE: The mtime
is when the backend server received the message, if client and the server clocks are off this may return wrong data or not return anything at all,
also because the arrival order of the messages cannot be guaranteed, sending fast multiple messages may be received in different order by the backend and this will
result in mtimes that do not correspond to actual times when the message has been sent.
Example:
# Get all messages
/message/get/archive
# Get all messages received after given mtime
/message/get/archive?mtime=123475658690
# Get all messages received before given mtime
/message/get/archive?mtime=123475658690&_ops=mtime,lt
# Get all messages with custom filter: if msg text contains Hi
/message/get/archive?_ops=msg,iregexp&msg=Hi
# Get all messages from the specific sender
/message/get/archive?sender=12345
Response:
{ "data": [ { "sender": "12345",
"msg": "Hi, how r u?",
"mtime": "12334312543"
},
{ "sender": "45678",
"msg": "check this out!",
"icon": "/message/image?sender=45678&mtime=12334312543",
"mtime": "12334312543"
}],
"next_token": ""
}
/message/get/sent
Return all messages i sent out. All the same query rules apply as for the archived messages API call.
Parameters:
recipient
- id of the recipient where i have sent messagesmtime
- time before or after messages sent, defined by _ops parametrsExample:
/message/get/sent?recipient=123
/message/get/sent?recipient=123&mtime=123475658690&_ops=mtime,le
/message/add
Send a message to an account, the following parameters must be specified:
id
- recipient account idmsg
- text of the message, can be empty if icon
property existsicon
- icon of the message, it can be base64 encoded image in the query or JSON string if the whole message is posted as JSON or
can be a multipart file upload if submitted via browser, can be omitted if msg/connection/get?type=invite&id=12345
property exists.Example:
/message/add?id=12345&msg=Hello
/message/add?id=12345&msg=this%2Bis%2Bthe%2Bpic&icon=KHFHTDDKH7676758JFGHFDRDEDET....TGJNK%2D
/message/read
Mark a message as read
Example:
/message/read?sender=12345&mtime=124345656567676
/message/archive
Move a new message to the archive. The required query parameters are sender
and mtime
.
Example:
/message/read?sender=12345&mtime=12366676434
/message/update
Update a message, can be used to keep track of read/unread status, etc...
Example:
/message/update?sender=12345&mtime=124345656567676&read=1
/message/update/archive
Update a message in the archive.
/message/del
Delete new message(s) by sender
and/or mtime
which must be passed as query parameters. If no mtime is given, all messages from the given sender will be deleted.
Example:
/message/del?sender=12345&mtime=124345656567676
/message/del/archive
Delete archived message(s) by sender
and/or mtime
which must be passed as query parameters. If no mtime is given, all messages from the given sender will be deleted.
Example:
/message/del/archive?sender=12345&mtime=124345656567676
/message/del/sent
Delete the message(s) by recipient
and/or mtime
which must be passed as query parameters. If no mtime is given, all messages to the given recipient will be deleted.
Example:
/message/del/sent?recipient=12345&mtime=124345656567676
/message/image
Return the image data for the given message, the required parameters are:
/message/get
reply results for every messageThe counters API maintains realtime counters for every account records, the counters record may contain many different counter columns for different purposes and is always cached with whatever cache service is used, by default it is cached by the Web server process on every machine. Web worker processes ask the master Web server process for the cached records thus only one copy of the cache per machine even in the case of multiple CPU cores.
This is implemented by the counters
module from the core. To enable this functionality specify -allow-modules=bk_counters|bk_accounts
.
/counter/get
Return counter record for current account with all available columns of if id
is given return public columns for given account, it works with bk_counter
table
which by default defines some common columns:
NOTE: The columns with suffixes 0 and 1 are special columns that support the Connections API, every time a new connection is created, the type of new connection
is checked against any columns in the bk_counter table, if a property type0 exists and marked in the table descriptnio as autoincr
then the corresponding
counter property is increased, this is how every time new connectio like/dislike/invite/follow is added, the counters in the bk_counter table are increased.
/counter/put
Replace my counters record, all values if not specified will be set to 0
/counter/incr
Increase one or more counter fields, each column can provide a numeric value and it will be added to the existing value, negative values will be substracted.
if id
parameter is specified, only public columns will be increased for other account.
Example:
/counter/incr?msg_read=5&
/counter/incr?id=12345&ping=1
The data API is a generic way to access any table in the database with common operations, as oppose to the any specific APIs above this API only deals with one table and one record without maintaining any other features like auto counters, cache...
Because it exposes the whole database to anybody who has a login it is a good idea to disable this endpoint in the production or provide access callback that verifies who can access it.
To disable this endpoint completely in the config: deny-modules=data
To allow admins to access it only in the config: api-allow-admin=^/data
To allow admins to access it only:
api.registerPreProcess('GET', '/data', function(req, status, cb) { if (req.account.type != "admin") return cb({ status: 401, message: 'access denied' }; cb(status)); });
This is implemented by the data
module from the core.
/data/columns
/data/columns/TABLE
Return columns for all tables or the specific TABLE
/data/keys/TABLE
Return primary keys for the given TABLE
/data/(select|search|list|get|add|put|update|del|incr|replace)/TABLE
Perform database operation on the given TABLE, all options for the db
functiobns are passed as query parametrrs prepended with underscore,
regular parameters are the table columns.
By default the API does not allow table scans without a condition to avoid expensive and long queries, to enable a scan pass _noscan=0
.
For this to work the Data API must be configured as unsecure in the config file using the parameter api-unsecure=data
.
Some tables like messages and connections perform data convertion before returning the results, mostly splitting combined columns like type into
separate fields. To return raw data pass the parameter _noprocessrows=1
.
Example:
/data/get/bk_account?id=12345
/data/put/bk_counter?id=12345&like0=1
/data/select/bk_account?name=john&_ops=name,gt&_select=name,email
/data/select/bk_connection?_noscan=0&_noprocessrows=1
The pages API provides a simple Wiki like system with Markdown formatting. It keeps all pages in the database table bk_pages
and
exposes an API to manage and render pages.
The pages support public mode, all pages with pub
set to true will be returning without an account, this must be enabled with api-allow-path=^/pages/(get|select|show)
to work.
All .md files will be rendered into html automatically if there is not _raw=1 query parameter and pages view exists (api-pages-view=pages.html by default).
This is implemented by the pages
module from the core. To enable this functionality specify -allow-modules=bk_accounts
.
/pages/get/ID
Return a page with given id or the main page if id is empty. If the query parameter _render=1
is given, the content will be rendered into html from markdown, otherwie
returns all data as is.
/pages/select
Return all pages or only ones which match the query criteria. This potentially scans the whole table to return all pages and
is used to show pages index.
/pages/put
Replace or add a new page.
/pages/del
Delete a page from the database
/pages/show/ID
Render a page with given id, markdown is converted into html using marked
. A view must be condfigured in order to render to work, by default pages.html view
is provided to simply wrap the markdown in the page layout.
The system API returns information about the backend statistics, allows provisioning and configuration commands and other internal maintenance functions. By default is is open for access to all users but same security considerations apply here as for the Data API.
This is implemented by the system
module from the core. To enable this functionality specify -allow-modules=accounts
.
/system/restart
Perform restart of the Web processes, this will be done gracefully, only one Web worker process will be restarting while the other processes will keep
serving requests. The intention is to allow code updates on live systems without service interruption.
/system/cache/(init|stats|keys|get|set|put|incr|del|clear)
Access to the caching functions
/system/config/(init)
Access to the config functions
/system/msg/(init|send)
Access to the messaging functions
/system/jobs/(send)
Access to the jobs functions
/system/queue/(init|publish)
Access to the queue functions
/system/params/get
Return all config parameters applied from the config file(s) or remote database.
/system/stats/get
Database pool statistics and other diagnostics
Individual sub-objects:
Response:
{
"id": "<IP_ADDRESS>-25170",
"ip": "<IP_ADDRESS>",
"mtime":<PHONE_NUMBER>321,
"ctime":<PHONE_NUMBER>760,
"type": "",
"host": "",
"pid": 25170,
"instance": "i-d4c89eff",
"worker": 27,
"latency": 0,
"cpus": 4,
"mem":<PHONE_NUMBER>4,
"rss_hmin": 66879488,
"rss_hmax": 151891968,
"rss_hsum":<PHONE_NUMBER>104,
"rss_hvar":<PHONE_NUMBER>10902.66,
"rss_hmean": 118895507.98312236,
"rss_hdev": 15962833.92793719,
"rss_hcnt": 20619,
"rss_hmed": 147644416,
"rss_h75p": 149262336,
"rss_h95p": 150834585.6,
"rss_h99p": 151550033.92000002,
"rss_h999p": 151886266.368,
"heap_hmin": 25790920,
"heap_hmax": 72316184,
"heap_hsum":<PHONE_NUMBER>504,
"heap_hvar":<PHONE_NUMBER>7311.65,
"heap_hmean": 49948587.68630874,
"heap_hdev": 7373895.648658967,
"heap_hcnt": 20619,
"heap_hmed": 57480704,
"heap_h75p": 61934254,
"heap_h95p": 67752391.2,
"heap_h99p": 70544797.92,
"heap_h999p": 72315029.104,
"avg_hmin": 0.04541015625,
"avg_hmax": 0.06005859375,
"avg_hsum": 938.234375,
"avg_hvar": 4.491222722966496e-7,
"avg_hmean": 0.04550338886463941,
"avg_hdev": 0.0006701658543201448,
"avg_hcnt": 20619,
"avg_hmed": 0.04541015625,
"avg_h75p": 0.04541015625,
"avg_h95p": 0.04541015625,
"avg_h99p": 0.05078125,
"avg_h999p": 0.05997363281250001,
"free_hmin":<PHONE_NUMBER>0,
"free_hmax":<PHONE_NUMBER>0,
"free_hsum":<PHONE_NUMBER>05952,
"free_hvar":<PHONE_NUMBER>606286,
"free_hmean":<PHONE_NUMBER>4.889277,
"free_hdev": 76417229.43555522,
"free_hcnt": 20619,
"free_hmed":<PHONE_NUMBER>0,
"free_h75p":<PHONE_NUMBER>6,
"free_h95p":<PHONE_NUMBER>0.4,
"free_h99p":<PHONE_NUMBER>8,
"free_h999p":<PHONE_NUMBER>3.968,
"util_hmin": 0.05905642141342145,
"util_hmax": 0.0607655708794173,
"util_hsum": 1230.6298386264643,
"util_hvar": 2.1530671850148948e-7,
"util_hmean": 0.059684263961708346,
"util_hdev": 0.0004640115499656118,
"util_hcnt": 20619,
"util_hmed": 0.05920415878947068,
"util_h75p": 0.059217278415661254,
"util_h95p": 0.05934395790869296,
"util_h99p": 0.059361851867105964,
"util_h999p": 0.0593659827984017,
"pool_name": "dynamodb",
"pool_que_rate": 0,
"pool_que_rcnt": 1989,
"pool_que_rmean": 0.0035627883554577716,
"pool_que_r1m": 0,
"pool_que_r5m": 0,
"pool_que_r15m": 0,
"pool_que_hmin": 0,
"pool_que_hmax": 230,
"pool_que_hsum": 45843,
"pool_que_hvar":<PHONE_NUMBER>2909315,
"pool_que_hmean": 23.048265460030166,
"pool_que_hdev": 19.15374319889178,
"pool_que_hcnt": 1989,
"pool_que_hmed": 21,
"pool_que_h75p": 23,
"pool_que_h95p": 33,
"pool_que_h99p":<PHONE_NUMBER>0000007,
"pool_que_h999p": 225.971,
"pool_req_hmin": 1,
"pool_req_hmax": 2,
"pool_req_hsum": 1991,
"pool_req_hvar": 0.001005024617286425,
"pool_req_hmean": 1.0010055304172951,
"pool_req_hdev": 0.03170212322994195,
"pool_req_hcnt": 1989,
"pool_req_hmed": 1,
"pool_req_h75p": 1,
"pool_req_h95p": 1,
"pool_req_h99p": 1,
"pool_req_h999p": 1.9710000000000036,
"pool_count": 0,
"pool_req_0": 2,
"pool_cache_rate": 0.1303780964797914,
"pool_cache_rcnt": 284,
"pool_cache_rmean": 0.0005087436344326025,
"pool_cache_r1m": 0,
"pool_cache_r5m": 0,
"pool_cache_r15m": 0,
"pool_cache_hmin": 0,
"pool_cache_hmax": 2,
"pool_cache_hsum": 70,
"pool_cache_hvar": 0.19345045538247163,
"pool_cache_hmean": 0.24647887323943662,
"pool_cache_hdev": 0.4398300301053483,
"pool_cache_hcnt": 284,
"pool_cache_hmed": 0,
"pool_cache_h75p": 0,
"pool_cache_h95p": 1,
"pool_cache_h99p": 1,
"pool_cache_h999p": 2,
"pool_hits": 239,
"pool_misses": 45,
"cache_inserted": 484,
"cache_deleted": 310,
"cache_cleanups": 0,
"cache_hits": 7642,
"cache_misses": 1411,
"cache_max": 1000000,
"cache_size": 61586,
"cache_count": 174,
"api_que_hmin": 1,
"api_que_hmax": 6,
"api_que_hsum": 13237,
"api_que_hvar": 0.005674280465987009,
"api_que_hmean": 1.0024992426537414,
"api_que_hdev": 0.07532782000022972,
"api_que_hcnt": 13204,
"api_que_hmed": 1,
"api_que_h75p": 1,
"api_que_h95p": 1,
"api_que_h99p": 1,
"api_que_h999p": 2,
"api_nreq": 1,
"api_req_rate": 0,
"api_req_rcnt": 13203,
"api_req_rmean": 0.02365120609256502,
"api_req_r1m": 0,
"api_req_r5m": 0,
"api_req_r15m": 0,
"api_req_hmin": 0,
"api_req_hmax": 536,
"api_req_hsum": 20115,
"api_req_hvar": 89.12554520926801,
"api_req_hmean": 1.5235173824130879,
"api_req_hdev": 9.440632669968046,
"api_req_hcnt": 13203,
"api_req_hmed": 1,
"api_req_h75p": 1,
"api_req_h95p": 1,
"api_req_h99p": 33.13000000000011,
"api_req_h999p": 99.36200000000008,
"url_message_get_rate": 0,
"url_message_get_rcnt": 24,
"url_message_get_rmean": 0.00004299242196761214,
"url_message_get_r1m": 0,
"url_message_get_r5m": 0,
"url_message_get_r15m": 0,
"url_message_get_hmin": 16,
"url_message_get_hmax": 71,
"url_message_get_hsum": 792,
"url_message_get_hvar":<PHONE_NUMBER>8695653,
"url_message_get_hmean": 33,
"url_message_get_hdev": 14.434258764722092,
"url_message_get_hcnt": 24,
"url_message_get_hmed": 30.5,
"url_message_get_h75p": 40.75,
"url_message_get_h95p": 68,
"url_message_get_h99p": 71,
"url_message_get_h999p": 71,
"url_message_get_0": 0,
"api_req_0": 20,
"url_ping_rate": 0,
"url_ping_rcnt": 12407,
"url_ping_rmean": 0.022226981327796796,
"url_ping_r1m": 0,
"url_ping_r5m": 0,
"url_ping_r15m": 0,
"url_ping_hmin": 0,
"url_ping_hmax": 4,
"url_ping_hsum": 6915,
"url_ping_hvar": 0.25785489698686204,
"url_ping_hmean": 0.5573466591440316,
"url_ping_hdev": 0.5077941482400737,
"url_ping_hcnt": 12407,
"url_ping_hmed": 1,
"url_ping_h75p": 1,
"url_ping_h95p": 1,
"url_ping_h99p": 1,
"url_ping_h999p": 2,
"url_ping_0": 5,
"url_image_account_rate": 0,
"url_image_account_rcnt": 95,
"url_image_account_rmean": 0.00017084907295404685,
"url_image_account_r1m": 0,
"url_image_account_r5m": 0,
"url_image_account_r15m": 0,
"url_image_account_hmin": 17,
"url_image_account_hmax": 121,
"url_image_account_hsum": 4295,
"url_image_account_hvar":<PHONE_NUMBER>7323637,
"url_image_account_hmean": 45.21052631578947,
"url_image_account_hdev": 19.29827174317007,
"url_image_account_hcnt": 95,
"url_image_account_hmed": 42,
"url_image_account_h75p": 51,
"url_image_account_h95p": 89.59999999999991,
"url_image_account_h99p": 121,
"url_image_account_h999p": 121,
"url_image_account_0": 0,
"incr_follow_0": 0,
"api_bad_0": 3,
"url_account_update_rate": 0,
"url_account_update_rcnt": 6,
"url_account_update_rmean": 0.000010813705805470248,
"url_account_update_r1m": 0,
"url_account_update_r5m": 0,
"url_account_update_r15m": 0,
"url_account_update_hmin": 53,
"url_account_update_hmax": 182,
"url_account_update_hsum": 573,
"url_account_update_hvar": 2041.5,
"url_account_update_hmean": 95.5,
"url_account_update_hdev": 45.18296139032943,
"url_account_update_hcnt": 6,
"url_account_update_hmed": 82,
"url_account_update_h75p": 120.5,
"url_account_update_h95p": 182,
"url_account_update_h99p": 182,
"url_account_update_h999p": 182,
"url_account_update_0": 0,
"auth_add_0": 0,
"url_account_get_rate": 0,
"url_account_get_rcnt": 9,
"url_account_get_rmean": 0.0001993511695335063,
"url_account_get_r1m": 0,
"url_account_get_r5m": 0,
"url_account_get_r15m": 0,
"url_account_get_hmin": 2,
"url_account_get_hmax": 100,
"url_account_get_hsum": 435,
"url_account_get_hvar":<PHONE_NUMBER>000001,
"url_account_get_hmean": 48.333333333333336,
"url_account_get_hdev": 29.051678092667903,
"url_account_get_hcnt": 9,
"url_account_get_hmed": 46,
"url_account_get_h75p": 67,
"url_account_get_h95p": 100,
"url_account_get_h99p": 100,
"url_account_get_h999p": 100,
"url_account_get_0": 1,
"url_system_stats_rate": 0,
"url_system_stats_rcnt": 1,
"url_system_stats_rmean": 0.04501665616278023,
"url_system_stats_r1m": 0,
"url_system_stats_r5m": 0,
"url_system_stats_r15m": 0,
"url_system_stats_hmin": 3,
"url_system_stats_hmax": 3,
"url_system_stats_hsum": 3,
"url_system_stats_hmean": 3,
"url_system_stats_hdev": 0,
"url_system_stats_hcnt": 1,
"url_system_stats_hmed": 3,
"url_system_stats_h75p": 3,
"url_system_stats_h95p": 3,
"url_system_stats_h99p": 3,
"url_system_stats_h999p": 3,
"url_system_stats_0": 2
}
Vlad Seryakov
Check out the Documentation for more details.
help
- Print help and exitlog
- Set debugging level to any of test,dev,debug,info,notice,log,warn,error,nonelog-filter
- Enable debug filters, format is: label,... to enable, and !label,... to disable. Only first argument is used for label in logger.debuglog-file
- Log to a file, if not specified used default logfile, disables sysloglog-ignore
- Regexp with property names which must not be exposed in the log when using custom logger inspectorlog-inspect
- Install custom secure logger inspection instead of util.inspectsyslog
- Write all logging messages to syslog, connect to the local syslog server over Unix domain socketconsole
- All logging goes to the console resetting all previous log related settings, this is used in the development mode mostlyhome
- Specify home directory for the server, the server will try to chdir there or exit if it is not possible, the directory must existconf-file
- Name of the config file to be loaded instead of the default etc/config, can be relative or absolute patherr-file
- Path to the error log file where daemon will put app errors and crash stacksetc-dir
- Path where to keep config filestmp-dir
- Path where to keep temp filesspool-dir
- Path where to keep modifiable fileslog-dir
- Path where to keep other log files, log-file and err-file are not affected by thisfiles-dir
- Path where to keep uploaded filesimages-dir
- Path where to keep imagesweb-path
- Path where to keep web pages and other static files to be served by the web serversviews-path
- Path where to keep virtual hosts web pages, every subdirectory name is a host name to match with Host: header, www. is always stripped before matching vhost directorymodules-path
- Directory from where to load modules, these are the backendjs modules but in the same format and same conventions as regular node.js modules, the format of the files is NAME_{web,worker,shell}.js. The modules can load any other files or directories, this is just an entry pointlocales-path
- Path where to keep locale translationsallow-packages
- NPM packages to load on startup, the modules, locales, viewes, web subfolders from the package will be added automatically to the system paths, modules will be loaded if present, the config file in etc subfolder will be parsed if presentrole
- Override servers roles, this may have very strange side effects and should only be used for testing purposesuid
- User id or name to switch after startup if running as root, used by Web servers and job workersgid
- Group id or name to switch after startup if running to rootforce-uid
- Drop privileges if running as root by all processes as early as possibly, this reqiures uid being set to non-root user. A convenient switch to start the backend without using any other tools like su or sudo.umask
- Permissions mask for new files, calls system umask on startup, if not specified the current umask is usedport
- port to listen for the HTTP server, this is global defaultbind
- Bind to this address only, if not specified listen on all interfacesbacklog
- The maximum length of the queue of pending connections, used by HTTP server in listen.ws-port
- port to listen for WebSocket server, it can be the same as HTTP/S ports to co-exist on existing web serversws-bind
- Bind to this address only for WebSocket, if not specified listen on all interfaces, only when the port is different from existing web portsssl-port
- port to listen for HTTPS server, this is global default, be advised that proxy-port takes precedencessl-bind
- Bind to this address only for HTTPS server, if not specified listen on all interfacesssl-key
- Path to SSL prvate keyssl-cert
- Path to SSL certificatessl-pfx
- A string or Buffer containing the private key, certificate and CA certs of the server in PFX or PKCS12 format. (Mutually exclusive with the key, cert and ca options.)ssl-ca
- An array of strings or Buffers of trusted certificates in PEM format. If this is omitted several well known root CAs will be used, like VeriSign. These are used to authorize connections.ssl-passphrase
- A string of passphrase for the private key or pfxssl-crl
- Either a string or list of strings of PEM encoded CRLs (Certificate Revocation List)ssl-ciphers
- A string describing the ciphers to use or exclude. Consult http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT for details on the formatssl-request-cert
- If true the server will request a certificate from clients that connect and attempt to verify that certificate. concurrency
- How many simultaneous tasks to run at the same time inside one process, this is used by async module only to perform several tasks at once, this is not multithreading but and only makes sense for I/O related taskstimeout
- HTTP request idle timeout for servers in ms, how long to keep the connection socket open, this does not affect Long Poll requestsdaemon
- Daemonize the process, go to the background, can be specified only in the command lineshell
- Run command line shell, load the backend into the memory and prompt for the commands, can be specified only in the command linemonitor
- For production use, monitors the master and Web server processes and restarts if crashed or exited, can be specified only in the command linemaster
- Start the master server, can be specified only in the command line, this process handles job schedules and starts Web server, keeps track of failed processes and restarts themweb
- Start Web server processes, spawn workers that listen on the same port, for use without master process which starts Web servers automaticallyproxy-port
- Start the HTTP reverse proxy server, all Web workers will listen on different ports and will be load-balanced by the proxy, the proxy server will listen on global HTTP port and all workers will listen on ports starting with the proxy-portproxy-ssl
- Start HTTPS reverse proxy to accept incoming SSL requests, ssl-key/cert must be definedsalt
- Set random or specific salt value to be used for consistent suuid generationapp-name
- Set appName and version explicitely an skip reading it from package.json, it can be just a name or name-versionapp-package
- NPM package containing the application package.json, it will be added to the list of package.json files for app name and version discovery. The package must be included in the -allow-packages list.instance-tag
- Set instance tag explicitely, skip all meta data checks for itinstance-region
- Set instance region explicitely, skip all meta data checks for itinstance-zone
- Set instance zone explicitely, skip all meta data checks for itinstance-job
- Enables remote job mode, it means the backendjs is running in the cloud to execute a job or other task and can be terminated during the idle timeoutrun-mode
- Running mode for the app, used to separate different running environment and configurations. DNS TXT configurable.no-monitor
- Disable monitor process, for cases when the master will be monitored by other tool like monit...no-master
- Do not start the master processno-watch
- Disable source code watcherno-web
- Disable Web server processes, without this flag Web servers start by defaultno-db
- Do not initialize DB driversno-dns
- Do not use DNS configuration during the initializationno-configure
- Do not run configure hooks during the initializationrepl-port-([a-z]+)$
- Base REPL port for process role (server, master, web, worker), if specified it initializes REPL in the processes, for workers the port is computed by adding a worker id to the base port, for example if specified -repl-port-web 2090
then a web worker will use any available 2091,2092...repl-bind
- Listen only on specified address for REPL server in the master processrepl-file
- User specified file for REPL historyworker
- Set this process as a worker even it is actually a master, this skips some initializationsallow-modules-?(.+)?
- A regexp with modules name to be loaded on startup, only matched modules will be loaded, basename of the file is matched only, no path or extension, it is per role or global if no role is provideddeny-modules-?(.+)?
- A regexp with modules names that will never be loaded even if allowed, this is for blacklisted modules, can be per role or global for all processeslogwatcher-from
- Email address to send logwatcher notifications from, for cases with strict mail servers accepting only from known addresseslogwatcher-interval
- How often to check for errors in the log files in minuteslogwatcher-any-range
- Number of lines for matched channel any
to be attached to the previous matched channel, if more than this number use the channel any
on its ownlogwatcher-match-[a-z]+
- Regexp patterns that match conditions for logwatcher notifications, this is in addition to default backend logger patterns, suffix defines the log channel to use, like error, warning.... Special channel any
is reserved to send matched lines to the previously matched channel if within configured range. Example: -logwatcher-match-error=^failed:
-logwatcher-match-any=line:[0-9]+
logwatcher-send-[a-z]+
- Email address or other supported transport for the logwatcher notifications, the monitor process scans system and backend log files for errors and sends them to this email address, if not specified no log watching will happen, each channel must define a transport separately, one of error, warning, info, all. Supported transports: table://TABLE, http://URL, sns://ARN, ses://EMAIL, email@addr. Example<EMAIL_ADDRESS>logwatcher-ignore-[a-z]+
- Regexp with patterns that need to be ignored by the logwatcher process, it is added to the list of existing patterns for each specified channel separatelylogwatcher-once-[a-z0-9]+
- Regexp with patterns that need to be included only once by the logwatcher process, it is added to the list of existng patterns by tag to keep track each pattern separately, example: -logwatcher-once-restart 'restarting.+' -logwatcher-once-recon 'reconnecting:.+'logwatcher-file(-[a-z]+)?
- Add a file to be watched by the logwatcher, it will use all configured match patternsuser-agent
- Add HTTP user-agent header to be used in HTTP requests, for scrapers or other HTTP requests that need to be pretended coming from Web browsersbackend-host
- Host of the master backend, can be used for backend nodes communications using core.sendRequest function calls with relative URLs, also used in tests.backend-login
- Credentials login for the master backend access when using core.sendRequestbackend-secret
- Credentials secret for the master backend access when using core.sendRequesthost-name
- Hostname/domain to use for communications, default is current domain of the host machineconfig-domain
- Domain to query for configuration TXT records, must be specified to enable DNS configurationwatch
- Watch sources directory for file changes to restart the server, for development only, the backend module files will be added to the watch list automatically, so only app specific directores should be added. In the production -monitor must be used.watch-ignore
- Files to be ignored by the watcherwatch-match
- Files to be watched, .js and .css is the defaultwatch-web
- List of directories to be watched for file modifications and execute a buildWeb
command to produce bundles, apps, etc... Relative paths will be applied to all packages, example: web/js,web/cssbuild-web
- Command to run on web files modifications, to be used with tools like minify/uglifylocales
- A list of locales to load from the locales/ directory, only language name must be specified, example: en,es. It enables internal support for res.__
and req.__
methods that can be used for translations, for each request the internal language header will be honored forst, then HTTP Accept-Languageno-locales
- Do not load locales on startemail-from
- Email address to be used when sending emails from the backendemail-transport
- Send emails via supported transports: ses:, sendgrid://?key=SG, if not set default SMTP settings are usedsmtp-(.+)
- SMTP server parameters, user, password, host, ssl, tls...see emailjs for detailstmp-watcher-(.+)
- How long to keep files per subdirectory in secondsstop-on-error
- Exit the process on any error when loading modules, for dev purposesaws-key
- AWS access keyaws-secret
- AWS access secretaws-token
- AWS secuiry tokenaws-region
- AWS regionaws-zone
- AWS availability zoneaws-sdk-profile
- AWS SDK profile to use when reading credentials fileaws-sns-app-arn
- SNS Platform application ARN to be used for push notificationsaws-key-name
- AWS instance keypair name for remote job instances or other AWS commandsaws-elb-name
- AWS ELB name to be registered with on start up or other AWS commandsaws-target-group
- AWS ELB target group to be registered with on start up or other AWS commandsaws-elastic-ip
- AWS Elastic IP to be associated on startaws-host-name
- List of hosts to update in Route54 zone with the current private IP address, hosts must be in FQDN format, supports @..@ core.instance placeholdersaws-iam-profile
- IAM instance profile name for instances or commandsaws-image-id
- AWS image id to be used for instances or commandsaws-subnet-id
- AWS subnet id to be used for instances or commandsaws-vpc-id
- AWS VPC id to be used for instances or commandsaws-group-id
- AWS security group(s) to be used for instances or commandsaws-instance-type
- AWS instance type to launch on demandaws-account-id
- AWS account id if not running on an instanceaws-eni-id
- AWS Elastic Network Interfaces to attach on start, format is: eni[:index],eni...aws-config-parameters
- Prefix for AWS Config Parameters Store to load and parse as config before initializing the database pools, example: /bkjs/config/aws-set-parameters
- AWS Config Parameters Store to set on start, supports @..@ core.instance placeholders: format is: path:value,....aws-ddb-read-capacity
- Default DynamoDB read capacity for all tablesaws-ddb-write-capacity
- Default DynamoDB write capacity for all tablesaws-ddb-shard-concurrency
- DynamoDB Streams number of shards to process in parallelaws-ddb-shard-records-count
- DynamoDB Streams records count in a batchaws-ddb-shard-retry-count
- DynamoDB Streams number of times to read records from an open shard with no recordsaws-ddb-shard-retry-closed-count
- DynamoDB Streams number of times to read records from a closed shardaws-ddb-shard-retry-timeout
- DynamoDB Streams min timeout in ms between retries on empty records, exponential backoff is usedaws-ddb-shard-retry-max-timeout
- DynamoDB Streams max timeout in ms on empty records, once reached the timeout is reset back to the min and exponential backoff starts againipc-ping-interval
- Interval for a worker keep-alive pings, if not received within this period it will be killedipc-system-queue
- System queue name to subscribe for control messages, this is a PUB/SUB queue to process system messages like restart, re-init config,...ipc-lru-max
- Max number of items in the LRU cache, this cache is managed by the master Web server process and available to all Web processes maintaining only one copy per machine, Web proceses communicate with LRU cache via IPC mechanism between node processesipc-lru-max-(.+)
- Max number of items in the local LRU cache by process role, works the same way as the global one but only is set for the specified process roleipc-cache-?([a-z0-9]+)?
- An URL that points to the cache server in the format PROTO://HOST[:PORT]?PARAMS
, to use for caching in the API/DB requests, default is local LRU cache, multiple caches can be defined with unique names, all params starting with bk-
will be copied into the options without the prefix and removed from the url, the rest of params will be left in the urlipc-queue-?([a-z0-9]+)?
- An URL that points to the queue server in the format PROTO://HOST[:PORT]?PARAMS
, to use for PUB/SUB or job queues, default is no local queue, multiple queues can be defined with unique names, params are handled the same way as for cacheipc-cache(-([a-z0-9]+)?-?options)-(.+)$
- Additional parameters for cache clients, specific to each implementation, Example: -ipc-cache-options-ttl 3000
ipc-queue(-([a-z0-9]+)?-?options)-(.+)$
- Additional parameters for queue clients, specific to each implementation, Example: -ipc-queue-redis-options-max_attempts 3
db-pool
- Default pool to be used for db access without explicit pool specified. DNS TXT configurable.db-name
- Default database name to be used for default connections in cases when no db is specified in the connection urldb-create-tables
- Create tables in the database or perform table upgrades for new columns in all pools, only master processes can perform this operation, never workersdb-cache-tables
- List of tables that can be cached: bk_auth, bk_counter. This list defines which DB calls will cache data with currently configured cache. This is global for all db pools.db-cache-pools
- List of pools which trigger cache flushes on update.db-cache-sync
- List of tables that perform synchronized cache updates before returning from a DB call, by default cache updates are done in the backgrounddb-cache-keys-([a-z0-9_]+)-(.+)
- List of columns to be used for the table cache, all update operations will flush the cache if the cache key can be created from the record columns. This is for ad-hoc and caches to be used for custom selects which specified the cache key.db-describe-tables
- A JSON object with table descriptions to be merged with the existing definitionsdb-cache-ttl
- Default global TTL for cached tablesdb-cache-ttl-(.+)
- TTL in milliseconds for each individual table being cacheddb-cache-name-(.+)
- Cache client name to use for each table instead of the default in order to split cache usage for different tables, it can be just a table name or pool.table
db-cache2-(.+)
- Tables with TTL for level2 cache, i.e. in the local process LRU memory. It works before the primary cache and keeps records in the local LRU cache for the given amount of time, the TTL is in ms and must be greater than zero for level 2 cache to work. Make sure ipc-lru-max-
is properly configured for each process roledb-jscache2-(.+)
- Same as cache2 but keeps references to the actual objects in the heapdb-jscache2-max
- Max number of items to keep in the Javascript Level 2 cachedb-custom-column-([a-z0-9_]+)-(.+)
- A column that is allowed to be used in any table, the name is a regexp with the value to be a typedb-local
- Local database pool for properties, cookies and other local instance only specific stuffdb-config
- Configuration database pool to be used to retrieve config parameters from the database, must be defined to use remote db for config parameters, set to default
to use current default pooldb-config-interval
- Interval between loading configuration from the database configured with -db-config, in minutes, 0 disables refreshing config from the dbdb-config-roles
- Roles to assume when pulling config parameters from the config tabledb-cache-columns-interval
- How often in minutes to refresh tables columns from the database, it calls cacheColumns for each pool which supports itdb-match-tables-(.+)
- Table columns to be returned by matching the regexp, for ad-hoc tablesdb-skip-drop
- A pattern of table names which will skipped in db.drop operations to prevent accidental table deletiondb-([a-z0-9]+)-pool$
- A database pool name, depending on the driver it can be an URL, name or pathname, examples of db pools: -db-pgsql-pool, -db-dynamodb-pool
, examples of urls: postgresql://[user:password@]hostname[:port]/db, mysql://[user:password@]hostname/db, mongodb://hostname[:port]/dbname, cql://[user:password@]hostname[:port]/dbname
db-([a-z0-9]+)-pool-(disabled)$
- Disable the specified pool but keep the configurationdb-([a-z0-9]+)-pool-(max)$
- Max number of open connections for a pool, default is Infinitydb-([a-z0-9]+)-pool-(min)$
- Min number of open connections for a pooldb-([a-z0-9]+)-pool-(idle)$
- Number of ms for a db pool connection to be idle before being destroyeddb-([a-z0-9]+)-pool-tables$
- A DB pool tables, pattern of tables that belong to this pool onlydb-([a-z0-9]+)-pool-(connect)$
- Connect options for a DB pool driver for new connection, driver specificdb-([a-z0-9]+)-pool-options-([a-zA-Z0-9_.-]+)$
- General options for a DB pooldb-([a-z0-9]+)-pool-(cache-columns)$
- Enable caching table columns for this pool if it supports itdb-([a-z0-9]+)-pool-(create-tables)$
- Create tables for this pool on startupdb-([a-z0-9]+)-pool-cache2-(.+)
- Level 2 cache TTL for the specified pool and table, data is JSON strings in the LRU cachedb-([a-z0-9]+)-pool-jscache2-(.+)
- Level 2 cache TTL for the specified pool and table, data is Js objects in the heapapi-images-url
- URL where images are stored, for cases of central image server(s), must be full URL with optional pathapi-images-s3
- S3 bucket name where to store and retrieve imagesapi-images-raw
- Return raw urls for the images, requires images-url to be configured. The path will reflect the actual 2 level structure and account id in the image nameapi-images-s3-options
- S3 options to sign images urls, may have expires:, key:, secret: propertiesapi-images-ext
- Default image extension to use when saving imagesapi-files-raw
- Return raw urls for the files, requires files-url to be configured. The path will reflect the actual 2 level structure and account id in the file nameapi-files-url
- URL where files are stored, for cases of central file server(s), must be full URL with optional pathapi-files-s3
- S3 bucket name where to store files uploaded with the File APIapi-max-latency
- Max time in ms for a request to wait in the queue, if exceeds this value server returns too busy errorapi-max-cpu-util
- Max CPU utilization allowed, if exceeds this value server returns too busy errorapi-max-memory-heap
- Max number of bytes of V8 heap allowed, if exceeds this value server returns too busy errorapi-max-memory-rss
- Max number of bytes in RSS memory allowed, if exceeds this value server returns too busy errorapi-max-request-queue
- Max number of requests in the processing queue, if exceeds this value server returns too busy errorapi-no-access-log
- Disable access logging in both file or syslogapi-access-log-file
- File for access loggingapi-access-log-fields
- Additional fields from the request or account to put in the access log, prefix defines where the field is lcoated: q: - query, h: - headers, a: - account otherwise from the request, Example: -api-log-fields h:Referer,a:name,q:actionapi-salt
- Salt to be used for scrambling credentials or other hashing activitiesapi-no-static
- Disable static files from /web folder, no .js or .html files will be served by the serverapi-static-options-(.+)
- Options to pass to serve-static module: maxAge, dotfiles, etag, redirect, fallthrough, extensions, index, lastModifiedapi-vhost-path-([^/]+)
- Define a virtual host regexp to be matched against the hostname header to serve static content from a different root, a vhost path must be inside the web directory, if the regexp starts with !, that means negative match, example: api-vhost-path-test_dir=test.com$api-no-vhost-path
- Add to the list of URL paths that should be served for all virtual hostsapi-no-templating
- Disable templating engine completelyapi-templating
- Templating engne to use, see consolidate.js for supported enginesapi-no-session
- Disable cookie session support, all requests must be signed for Web clientsapi-session-age
- Session age in milliseconds, for cookie based authenticationapi-session-secret
- Secret for session cookies, session support enabled only if it is not emptyapi-query-token-secret
- Name of the property to be used for encrypting tokens for pagination or other sensitive data, any property from bk_auth can be used, if empty no secret is used, if not a valid property then it is used as the secretapi-app-header-name
- Name for the app name/version query parameter or header, it is can be used to tell the server about the application versionapi-version-header-name
- Name for the access version query parameter or header, this is the core protocol version that can be sent to specify which core functionality a client expectsapi-no-signature
- Disable signature verification for requestsapi-no-cache-files
- Set cache-control=no-cache header for matching static filesapi-tz-header-name
- Name for the timezone offset header a client can send for time sensitive requests, the backend decides how to treat this offsetapi-signature-header-name
- Name for the access signature query parameter, header and session cookieapi-lang-header-name
- Name for the language query parameter, header and session cookie, primary language for a clientapi-signature-age
- Max age for request signature in milliseconds, how old the API signature can be to be considered valid, the 'expires' field in the signature must be less than current time plus this age, this is to support time driftsapi-no-access-token
- Disable access tokens supportapi-access-time-interval
- Intervals to refresh last access time for accounts, only updates the cache if bk_auth
is configured to be cachedapi-access-token-name
- Name for the access token query parameter or headerapi-access-token-secret
- A secret to be used for access token signatures, additional enryption on top of the signature to use for API access without signing requests, it is required for access tokens to be usedapi-access-token-age
- Access tokens age in milliseconds, for API requests with access tokens onlyapi-disable-session
- Disable access to API endpoints for Web sessions, must be signed properlyapi-allow-authenticated
- Add URLs which can be accessed by any authenticated user account, can be partial urls or Regexp, it is checked before any other account types, if matched then no account specific paths will be checked anymore(any of the allow-account-...)api-allow-acl-authenticated
- Include rules from the named -allow-account-
params to the allow ccheks of the authenticated usersapi-allow-admin
- Add URLs which can be accessed by admin accounts only, can be partial urls or Regexpapi-allow-account-([a-z0-9_]+)
- Add URLs which can be accessed by specific account type only, can be partial urls or Regexpapi-allow-acl-([a-z0-9_]+)
- Combine multiple allow-account regexps for the specified role, this is for reusing same regexp rules for multiple rolesapi-deny-authenticated
- Add URLs which CAN NOT be accessed by any authenticated user account, can be partial urls or Regexp, it is checked before any other account types, if matched then no account specific paths will be checked anymore(any of the deny-account-...)api-deny-acl-authenticated
- Include rules from the named deny-account-
params to the deny checks of the authenticated usersapi-deny-account-([a-z0-9_]+)
- Add URLs which CAN NOT be accessed by specific account type, can be partial urls or Regexp, this is checked before any allow parametersapi-deny-acl-([a-z0-9_]+)
- Combine multiple deny-account regexps for the specified role, this is for reusing same regexp rules for multiple rolesapi-acl-([a-z0-9_]+)
- Add URLs to the named ACL which can be used in allow/deny rules per accountapi-express-options
- Set Express config options during initialization,example: -api-express-options { "trust proxy": 1, "strict routing": true }
api-allow-ip
- Set regexp for IPs that dont need credentials, replaces the whole access list. It is checked before endpoint access listapi-deny-ip
- Set regexp for IPs that will be denied access, replaces the whole access list. It is checked before endpoint access list.api-allow
- Regexp for URLs that dont need credentials, replaces the whole access listapi-allow-path
- Add to the list of allowed URL paths without authentication, return result before even checking for the signatureapi-allow-empty
- Regexp for URLs that should return empty responses if not found, for example return nothing for non-existent javascript filesapi-ignore-allow
- Regexp for URLs that should be ignored by the allow rules, the processing will continueapi-ignore-allow-path
- Add to the list of URL paths which should e ignored by the allow rules, in order to keep allow/deny rules simple, for example to keep some js files from open to all: -allow-path .js -disallow-path /secure/api-allow-anonymous
- Add to the list of allowed URL paths that can be served with or without valid account, the difference with allow-path
is that it will check for signature and an account but will continue if no login is provided, return error in case of wrong account or not account foundapi-allow-ssl
- Add to the list of allowed URL paths using HTTPs only, plain HTTP requests to these urls will be refusedapi-ignore-ssl
- Allow plain HTTP from matched IP addresss or pathsapi-redirect-ssl
- Add to the list of the URL paths to be redirected to the same path but using HTTPS protocol, for proxy mode the proxy server will perform redirectsapi-redirect-url
- Add to the list a JSON object with property name defining a host/path regexp to be matched early against in order to redirect using the value of the property, if the regexp starts with !, that means negative match, 2 variables can be used for substitution: @HOST@, @PATH@, @URL@, @BASE@, @DIR@, @QUERY@, example: { '^[^/]+/path/$': '/path2/index.html', '.+/$': '<EMAIL_ADDRESS>} api-deny
- Regexp for URLs that will be denied access, replaces the whole access listapi-deny-path
- Add to the list of URL paths to be denied without authenticationapi-subscribe-timeout
- Timeout for Long POLL subscribe listener, how long to wait for events before closing the connection, millisecondsapi-subscribe-interval
- Interval between delivering events to subscribed clients, millisecondsapi-mime-body
- Collect full request body in the req.body property for the given MIME type in addition to json and form posts, this is for custom body processingapi-mime-ignore
- Ignore the body for the following MIME content types, request body will not be parsed at allapi-mime-map-(.+)
- File extension to MIME content type mapping, this is used by static-serve, example: -api-mime-map-mobileconfig application/x-apple-aspen-configapi-collect-host
- The backend URL where all collected statistics should be sent over, if set to pool
then each web worker will save metrics directly into the statistics database poolapi-collect-pool
- Database pool where to save collected statistics, make sure to call -api-collect-create before using thisapi-collect-create-tables
- Create statistics tables, it is not created by default with -db-create-tablesapi-collect-interval
- How often to collect statistics and metrics in secondsapi-collect-send-interval
- How often to send collected statistics to the master server in secondsapi-platform-match
- An JSON object with list of regexps to match user-agent header for platform detection, example: { 'ios|iphone|ipad': 'ios', 'android': 'android' }api-cors-origin
- Origin header for CORS requestsapi-no-cors
- Disable CORS requestsapi-server-header
- Custom Server: header to return for all requestsapi-error-message
- Default error message to return in case of exceptionsapi-allow-error-code
- Error codes in exceptions to return in the response to the user, if not matched the error-message will be returnedapi-url-metrics-ignore
- Ignore matched urls from the metrics collectionapi-url-metrics-max
- Max number of path related metrics objects to keep, the most recent or the most popular urls will be kept onlyapi-url-metrics-path-(.+)
- Defines the length of an API request path to be stored in the statistics by matching the beginning of the request url, the smallest path wins, example: -api-url-metrics-/img 2 -api-url-metrics-/account/get 3api-rlimits-max-(.+)
- Set max/burst rate limit by the given property, it is used by the request rate limiter using Token Bucket algorithm. Predefined types: ip, path, id, loginapi-rlimits-rate-(.+)
- Set fill/normal rate limit by the given property, it is used by the request rate limiter using Token Bucket algorithm. Predefined types: ip, path, id, loginapi-rlimits-interval-(.+)
- Set rate interval in ms by the given property, it is used by the request rate limiter using Token Bucket algorithm. Predefined types: ip, path, id, loginapi-rlimits-total
- Total number of servers used in the rate limiter behind a load balancer, rates will be divided by this number so each server handles only a portion of the total rate limitapi-rlimits-interval
- Interval in ms for all rate limiters, defines the time unit, default is 1000 msapi-exit-on-error
- Exit on uncaught exception in the route handlerapi-upload-limit
- Max size for uploads, bytesapi-limiter-queue
- Name of an ipc queue for API rate limitingapi-errlog-limiter-max
- How many error messages to put in the log before throttling kicks inapi-errlog-limiter-interval
- Interval for error log limiter, max errors per this intervalapi-errlog-limiter-ignore
- Do not show errors that match the regexpapi-proxy-reverse
- A Web server where to proxy requests not macthed by the url patterns or host header, in the form: http://host[:port]api-proxy-url-(.+)
- URL regexp to be passed to other web server running behind, each parameter defines an url regexp and the destination in the value in the form http://host[:port], example: -api-proxy-url-^/api http://<IP_ADDRESS>:8080api-proxy-host-(.+)
- Virtual host mapping, to match any Host: header, each parameter defines a host name and the destination in the value in the form http://host[:port], example: -api-proxy-host-www.myhost.com http://<IP_ADDRESS>:8080api-routing-(.+)
- URL path to be re-routed to other path, this is done inside the server at the beginning, only the path is replaced, same format and placeholders as in redirect-url, example: -api-routing-^/account/get /acount/readapi-auth-routing-(.+)
- URL path to be re-routed to other path after the authentication is successful, this is done inside the server, only the path is replaced, same format and placeholders as in redirect-url, example: -api-routing-auth-^/account/get /acount/readapi-login-redirect-(.+)
- Define a location where to redirect if no login is provided, same format and placeholders as in redirect-url, example: api-login-redirect-^/admin/=/login.htmlapi-auth-status
- Default authenticated status, if no auth rules matched but valid signature this is the status returnedapi-auth-message:
- Default authenticated message to be returned the default auth statusmsg-([^-]+)-(cert)(@.+)?
- A certificate for APN or similar services in pfx format, can be a file name with .p12 extension or a string with certificate contents encoded with base64, if the suffix is specified in the config parameter name will be used as the app name, otherwise it is globalmsg-([^-]+)-authkey-([^-]+)-(.+)
- A auth key for APN in p8 format, can be a file name with .p8 extension or a string with the key contents encoded with base64msg-([^-]+)-(key)(@.+)?
- API key for GCM or similar services, if the suffix is specified in the config parameter will be used as the app name, without the suffix it is globalmsg-([^-]+)-(sandbox)(@.+)?
- Enable sandbox for a service, default is production modemsg-([^-]+)-options-([^@]+)(@.+)?
- A config property to the specified agent, driver specificmsg-shutdown-timeout
- How long to wait for messages draining out in ms on shutdown before exitingmsg-app-default
- Default app id(app bundle id) to be used when no app_id is specifiedmsg-app-dependency@(.+)
- List of other apps that are considered in the same app family, sending to the primary app will also send to all dependent appsmsg-app-team-(.+)
- Regexp that identifies all app bundles for a teammsg-app-team
- Default team to use for apps that do not match any specific teamsjobs-workers
- How many worker processes to launch to process the job queue, -1 disables jobs, 0 means launch as many as the CPUs availablejobs-worker-cpu-factor
- A number to multiply the number of CPUs available to make the total number of workers to launch, only used if workers
is 0jobs-worker-args
- Node arguments for workers, for passing v8 jobspec, see process
jobs-worker-env
- Environment to be passed to the worker via fork, see cluster.fork
jobs-worker-delay
- Delay in milliseconds for a worker before it will start accepting jobs, for cases when other dependencies may take some time to startjobs-max-runtime
- Max number of seconds a job can run before being killedjobs-max-lifetime
- Max number of seconds a worker can live, after that amount of time it will exit once all the jobs are finished, 0 means indefinitelyjobs-shutdown-timeout
- Max number of milliseconds to wait for the graceful shutdown sequence to finish, after this timeout the process just exitsjobs-worker-queue
- Queue(s) to subscribe for workers, multiple queues can be processes at the same time, i.e. more than one job can run from different queuesjobs-cron-queue
- Default queue to use for cron jobsjobs-global-queue
- Default queue for all jobs, the queueName is ignoredjobs-unique-queue
- Default queue name to use for keeping track of unique jobsjobs-cron
- Allow cron jobs to be executed from the local etc/crontab file or via config parameterjobs-schedule
- Cron jobs to be scheduled, the JSON must be in the same format as crontab fileserver-max-processes
- Max number of processes to launch for Web servers, 0 means NumberOfCPUs-1
, < 0 means NumberOfCPUs*abs(N)
server-crash-delay
- Delay between respawing the crashed processserver-restart-delay
- Delay between respawning the server after changesserver-no-restart
- Do not restart any processes terminated, for debugging crashes onlyserver-log-errors
- If true, log crash errors from child processes by the logger, otherwise write to the daemon err-file. The reason for this is that the logger puts everything into one line thus breaking formatting for stack traces.server-process-name
- Path to the command to spawn by the monitor instead of node, for external processes guarded by this monitorserver-process-args
- Arguments for spawned processes, for passing v8 options or other flags in case of external processesserver-worker-args
- Node arguments for workers, job and web processes, for passing v8 optionsapi
HTTP API to the server from the clients, this module implements the basic HTTP(S) API functionality with some common features.  How many processes to spawn can be configured via -server-max-workers
config parameter.
When an HTTP request arrives it goes over Express middleware, but before processing any registered routes there are several steps performed:
req
object which is by convention is a Request object, assigned with common backend properties to be used later:bk_auth
table will be setapi.sendJSON
method, registered post process callbacks will be called for such responseDatabase tables
// Authentication by login, only keeps id and secret to check the siganture
bk_auth: {
login: { primary: 1 }, // Account login
id: { type: "uuid", prefix: "u_" }, // Auto generated UUID to be linked with other records
name: { type: "text" }, // Account name
status: { type: "text" }, // Status of the account
type: { type: "text", list: 1, admin: 1 }, // Account roles: admin, ....
flags: { type: "list", list: 1, admin: 1 }, // Admin assigned flags about the account
secret: { secure: 1 }, // Signature secret or scrambled password
auth_secret: { admin: 1, secure: 1 }, // Code for 2-factor authentication
session_secret: { admin: 1, secure: 1 }, // Secret for session signatures
acl_deny: { admin: 1, secure: 1 }, // Deny access to matched url, a regexp
acl_allow: { admin: 1, secure: 1 }, // Only grant access if path matches this regexp
query_deny: { admin: 1, secure: 1 }, // Ignore these query params, a regexp
rlimits_max: { type: "int" }, // Burst/max reqs/sec rate allowed for this account, 0 to disable
rlimits_rate: { type: "int" }, // Fill/normal reqs/sec rate for this account, 0 to disable
expires: { type: "bigint", admin: 1, secure: 1 }, // Deny access to the account if this value is before current date, milliseconds
ctime: { type: "now", readonly: 1 }, // Create time
mtime: { type: "now" }
},
api.init(options, callback)
Initialize API layer, this must be called before the api
module can be used but it is called by the server module automatically so api.init
is
rearely need to called directly, only for new server implementation or if using in the shell for testing.
During the init sequence, this function calls api.initMiddleware
and api.initApplication
methods which by default are empty but can be redefined in the user aplications.
The bkjs.js uses its own request parser that places query parameters into req.query
or req.body
depending on the method.
For GET method, req.query
contains all url-encoded parameters, for POST method req.body
contains url-encoded parameters or parsed JSON payload or multipart payload.
The reason not to do this by default is that this may not be the alwayse wanted case and distinguishing data coming in the request or in the body may be desirable,
also, this will needed only for Express handlers .all
, when registering handler by method like .get
or .post
then the handler needs to deal with only either source of the request data.
api.shutdown(callback)
Gracefully close all connections, call the callback after that
api.shutdownWeb(options, callback)
Gracefully close all database pools when the shutdown is initiated by a Web process
api.configureAccessLog()
Setup access log stream
api.handleServerRequest(req, res)
Start Express middleware processing wrapped in the node domain
api.setupSocketConnection(socket)
Called on new socket connection, supports all type of sockets
api.cleanupSocketConnection(socket)