Here you will find the complete MAGE API documentation. If you are just getting started, you might want to have a look at the User Guide first.
Modules
mage.useModules([
'session'
]);
In addition to the MAGE API, the following core modules are also available for use.
In each case, you will need to make sure to add the module name to your mage.useModules()
call.
MAGE
The first line of code in your project is probably the following.
var mage = require('mage');
You’re going to write that a lot, as its the entry point into everything in the framework, including the modules you write and register. You can require “mage” from anywhere in your project.
Properties
Object mage.task
The name and options for the task that the command line interface has decided MAGE should execute.
This defaults to the “serve” task, which means to serve applications to clients. All
tasks available to MAGE are implemented through the CLI. You can run your application
with --help
to get an idea of which tasks you can activate. You can manually change the task
through the setTask
API documented under “Methods”.
Object mage.core
Contains all the subsystems that MAGE has access to. These are integrated into the system mode deeply than modules are, which you use to add application logic to the MAGE app.
The subsystems in “core” are documented in the API documentation.
string mage.version
The version of MAGE.
Object mage.magePackage
Contains information from MAGE’s package.json
file:
- name: “mage”
- version: The version of MAGE.
- path: The path to MAGE on disk.
- package: The parsed contents of MAGE’s
package.json
file.
Object mage.rootPackage
Contains information from your project’s package.json
file:
- name: The “name” field of package.json, or the folder name of your project.
- version: The version of your project (or “no-version”).
- path: The path to your project on disk.
- package: The parsed contents of your project’s
package.json
file.
Module mage.cli
The cli
property is a library that lets you extend and boot up the command-line argument parser.
For more information, please read the CLI documentation.
Methods
Module mage.require(moduleName)
Calls Node.js’ require
function from the MAGE context. This can be useful when you want to
directly require one of MAGE’s own dependencies.
string mage.getRunState()
Returns the lifecycle phase that MAGE is currently in. This can be any of the following.
- init: before starting up
- setup: while starting up
- running: while the project is running and is ready to serve requests
- quitting: while shutting down
Every time the run-state changes, the “runState” event is emitted, with the current run-state as the only argument.
mage.quit(number exitCode)
Shuts down MAGE and exits the process with the given exit code, or 0 if none has been passed.
Please note that if you run this in a worker process, it only shuts down the worker, not the entire project.
Mage mage.addModulesPath(string path)
Adds this path on the disk as a lookup path for modules when using useModules()
.
Returns the mage object to allow API chaining.
Mage mage.useModules(string moduleName, …)
Takes any number of arguments that each represent a module’s name. Arguments may also be an array
with module names. The modules you refer to are instantly imported from the known module paths (see:
addModulesPath
, or from the MAGE built-in modules, and exposed as mage.ModuleName
as well as
mage.core.modules.ModuleName
. Returns the mage object to allow API chaining.
string mage.getModulePath(string moduleName)
If a name by this name has been registered through useModules()
, this method will return the path
on disk where that module was loaded from.
string[] mage.listModules()
Returns all registered module names, in order of registration.
Mage mage.setup(Function callback)
Sets up the task that MAGE has been assigned. By default this is to serve the application to
clients. This function considers all errors fatal and will shut down the process if that happens.
Once setup has completed, this functions calls back to callback(error, appMap)
Please note: while there are no errors returned, the Node.js convention of “errors first” is still applied. You may safely ignore this argument.
The second argument returned in the callback is a map, by name, of application objects that are now available to you. For example:
appMap = {
game: { ... },
dev: { ... },
cms: { ... },
support: { ... }
};
The setup
function returns the mage object to allow API chaining.
Mage mage.start(Function callback)
This will start the application and expose it to the world. That means that from this moment on,
the HTTP server will be accepting connections and serve your applications. When these final
operations have completed, the callback will be called. In the case of error, MAGE will not return
the error to your callback. It will shut down instead. The start
function returns the mage object
to allow API chaining.
boolean mage.isDevelopmentMode([string feature])
Returns true
if MAGE is running in development mode, false
otherwise. If you pass a particular
feature by name, the boolean will reflect on that feature being in development mode or not. For more
information on these features, please read the documentation on
Configuration.
State
The state library exposes a constructor that constructs objects which form an interface between an actor, a session and the archivist. Virtually any command that involves reading and modification of data should be managed by a state object.
The state constructor is exposed on mage.core.State
. When you’re done using the state class,
always make sure to clean it up by calling close()
on it. MAGE’s module and command center systems
that bridge the communication between client and server use the State object for API responses and
server-sent events. We explain more about that and the concepts behind the State class in the
walkthrough.
Methods
State(string actorId, string session)
The constructor. Pass an actorId
to bind the state to the actor. That way events that are emitted
to this actor will batch up inside this state object, waiting to pulled out for delivery. This is
what MAGE’s command center does when returning a response following a user command execution.
Please note: you probably do not want to bind the state object to an actorId.
When you do not pass an actorId, emitting events to it will asynchronously be delivered via MAGE’s
message stream system. If you pass a session
object, it can be used for access level and user
language settings.
state.registerSession(Object session)
Will register the actorId and session. This is called from the constructor if a session was passed.
Please note: you probably do not want to bind the state object to a session.
state.setTimeout(number msec)
If the state object is not closed within the given time, it will automatically close itself.
state.clearTimeout()
If a timeout has been set, this will remove it.
boolean state.canAccess(string accessLevel)
Returns true
if the registered session is authorised at the given access level. Returns false
otherwise.
state.setDescription(string desc)
Tags the state with a description that is used whenever an error is logged from the state object.
string state.getDescription()
Returns the registered description.
string state.language()
Returns the language of the registered session. Returns en
if none is known.
state.emit(string actorId|string actorIds[], string path, data, string language, boolean isJson)
Emits an event to the given actorId’s client.
- actorId: The actorId or actorIds (array) to emit to.
- path: The event name (or dot separated path).
- data: Any data you want to send with this event.
- language: A language code that may be given if the data is bound to a single language only.
- isJson: If the data is a pre-serialized JSON string, pass
true
.
state.broadcast(string path, data, boolean isJson)
Broadcast an event to all the actors.
- path: The event name (or dot separated path).
- data: Any data you want to send with this event.
- isJson: If the data is a pre-serialized JSON string, pass
true
.
state.findActors(string actorIds[], Function callback)
This looks up all actors’ sessions, to see which actors are online and which are not. This can be useful when managing
a pool of users in a room for example. The callback function receives an error argument (in case of database failure),
and a found
argument which is the following object:
var found = {
online: ['someActorId', 'someActorId3'],
offline: ['someActorId2']
};
These lists will contain all actorIds you have passed into the function, but divided into an online
and an offline
group.
state.error(string code, string logDetails, Function callback)
Marks the state as in-error. No archivist mutations will be distributed, and the registered actor
will receive the error code
as the first argument in the client-side callback. Pass null
as a
code for the default “server” error code. Pass logDetails
to write it to the logger’s
error-channel. If you want to call a callback function immediately after, pass it as the third
argument.
state.respond(data)
This is the response that will be sent to the actor’s client-side callback as the second argument. This only has meaning if the state originated in the command center.
state.close(Function callback)
Call this when you’re done with the state object. All archivist mutations will now be distributed to their data stores and events will be sent to the client. If an error has been registered, all archivist changes, all events, and the response will be discarded. Error or no error, always make sure to close the state object when you’re done with it.
Usage example
var State = mage.core.State;
var state = new State('abc');
state.archivist.get('player', { id: state.actorId }, function (error, player) {
if (!error) {
state.emit(state.actorId, 'myevent', { hello: 'world', youAre: player });
console.log('Player:', player);
}
state.close();
});
Read more
Archivist
The archivist rules your data. Its purpose is to help you manage your data, all through a simple unified API, regardless of which data stores you use.
Advantages
Schema Migration
Schema Migration between versions is well supported out of the box. Please read the documentation.
Redundant storage
You can configure multiple data stores of the same type, in order to split your data into as many MySQL databases, couchbase clusters and file systems as you want.
This allows you to write to many stores at once (defined by write-order). It also allows you to configure a read-order, by which data gets loaded. This is very useful in scenarios like “try memcached first, else try MySQL”.
Friendly default integration logic with a bunch of database systems
For all built-in data stores, all serialization and access logic is built-in with friendly default behaviors. That means that you can store into a key/value store like memcached and into an SQL data store – through a single API – without having to program or configure anything specific.
A solution for static content
The archivist API works equally well for SQL databases, key/value stores and file storage. That means that it’s an out-of-the-box solution for your static content, using an API that is consistent with how you manage all your other data.
Highly customizable database integration if needed
Whenever you do want full control over how data gets stored, you have the ability to do so.
Integration with tomes
Integration with tomes is built in. This has a number of specific advantages.
- No need to tell archivist to store changes, since they are automatically detected.
- Tomes can be transported transparently to and from the browser.
Terminology
Topics and Indexes
Each document you store is identified by a topic and an index. Topic is always a string, and index is always a key/value object. Some typical examples of topic/index pairs could be:
topic: "player", index: { actorId: 123 }
topic: "inventory", index: { actorId: "456" }
topic: "cards", index: { actorId: 123, type: "deck" }
In SQL terms: consider topic your table, and index your primary key.
Vaults
Vaults represent the data stores where your data is stored. Each vault type has its own API for
talking to the underlying service, but also exposes a separate Archive API
that is used internally
to store documents. You generally won’t access vaults directly. You can leave that to the archivist.
You will however have to configure them, so that each vault knows where to store data.
See the Vaults section of this documentation to learn more about available vaults and their capabilities.
Archivist
The archivist directs your documents to-and-from the vaults. It’s your primary point of access,
and provides a simple API for reading and writing data. In MAGE, you can always access it through
state.archivist
.
Topic API
Topic APIs are APIs that, uniquely per vault, implement how values are stored and read. A lot of this logic is driven around “topics” and “indexes”, that these APIs can translate into logic that fits the vault in question.
For example, the topic weapons
with index { actorId: 123 }
can be translated into the following
memcached key: weapons/actorId:123
, or into the following MySQL structure:
{ "table": "weapons", "pk": { "actorId": 123 } }
Each vault has friendly defaults, but those can always be overridden with custom logic. For more information on how to do this, please read “Writing your own Topic API”.
MediaTypes
Each document that is stored, can be stored along with its media type. Think of image/jpeg
,
text/plain
, application/octet-stream
, application/json
, but also application/x-tome
.
Media types can be useful in order to recreate a living version of binary- or string-serialized
data. Archivist comes with built-in knowledge of media types and has the ability to convert between
them.
If you want to create a fresh new Tome, you must conjure it, as described in the documentation of
node-tomes. You may also store other types of data. Tomes are
simply supported out-of-the-box and you are encouraged to use them. You can access the Tome
class
by requiring it from MAGE by calling:
var Tome = mage.require('tomes').Tome;
Quick start guide
To start using archivist in your game, you will have to execute the following steps.
Configure your vaults
The archivist configuration sits on the root of your config file under the label archivist
. It
contains 3 child labels:
vaults
describes where you store all your data. The keys are the names you give to your vaults.listOrder
is an array with vault names, describing the order in which we list indexes.readOrder
is an array with vault names, describing the order in which we read data.writeOrder
is an array with vault names, describing the order in which we write data.
The vaults entry is a key/value map, where the key is the unique name of the vault. It’s up to you
to decide on these names. Perhaps often, the name of the vault will match the type of the vault, but
this is absolutely not required. Choose whatever makes sense for your project. The only name that is
reserved is client
, which is named that way by the MAGE command center. You will want to make
sure that the client
vault is represented in your writeOrder
.
It’s important to note that the listOrder
, readOrder
and writeOrder
are system-wide. It’s
likely that not every topic will be stored on every vault. Whenever we read or write a given topic,
the configured order is traversed, and vaults not linked to the topic are ignored. You cannot change
the ordering for individual topics.
Each vault entry in the configuration has 2 properties: type
and config
. The type property is a
fixed ID that is unique for each type of vault. Read the vault documentation referred to in the
Vaults section to see these IDs and how to configure vaults of that type.
Example configuration:
{
"archivist": {
"vaults": {
"static": {
"type": "file",
"config": { "path": "/tmp" }
},
"memcached": {
"type": "memcached",
"config": { "servers": ["localhost:11211"], "prefix": "bob/" }
},
"mysql": {
"type": "mysql",
"config": { "url": "mysql://bob:secret@localhost/bob_game" }
}
},
"listOrder": ["mysql", "static"],
"readOrder": ["memcached", "mysql", "static"],
"writeOrder": ["client", "memcached", "mysql", "static"]
}
}
Configure your topics
In your game’s lib
folder, please create a new folder called archivist
. This folder will be
require
d by MAGE’s archivist, in order to receive your topic configuration per vault-name.
Consider doing the whole configuration in one file: lib/archivist/index.js
.
The format is as follows:
exports.myTopicName = {
mediaType: 'application/json',
readOptions: {
},
index: ['propName', 'propName'],
vaults: {
myVaultName: myTopicAPI
},
afterLoad: function (state, value, cb) {
cb();
}
};
Where you do this for each topic you want to store in your vaults. The index
array must be
provided if your topic depends on an index. This array is the signature of the indexes you will
provide when referring to data.
If mediaType
is provided, this topic’s media type will default to the specified value.
The readOptions
object may be supplied to overwrite default options
that are used when reading
from your archivist. The following defaults are defined, and they can be individually replaced:
{
"mediaTypes": ["application/x-tome", "application/octet-stream"],
"encodings": ["live"],
"optional": false
}
The myTopicAPI
object that you provide per vault-name, may be replaced with true
in order to get all default
behaviors for that vault type. Read about “Advanced usage” to see how you can set up these topic APIs with custom
behaviors.
Finally, you may optionally provide an afterLoad
function that takes 3 arguments: state
, value
and cb
.
When you provide this function, each database load will call this function before returning to userland code. That gives
you (for example) the opportunity to perform document migration. The document can be found in value.data
, but you will
have to make sure yourself that it’s deserialized before you read from it. You can do this by calling
value.setEncoding('live');
. After this, value.data
will contain the unserialized document. You could store a
document schema version number in this document and when you notice it falls behind, migrate the document to the latest
schema. When you’re done handling the afterLoad
logic, you must call cb
to continue. If you pass an error to
cb
, the get()
or mget()
call that the load originated from will receive that error. It is up to you to log it.
In order to keep your configuration maintainable, it makes a lot of sense to categorize your topics. Imagine for example the following configuration:
function dynamicTopic(index) {
return { index: index, vaults: { mysql: true, memcached: true } };
}
function staticTopic() {
return { vaults: { file: true } };
}
exports.player = dynamicTopic(['id']);
exports.inventory = dynamicTopic(['playerId']);
exports.cards = dynamicTopic(['playerId']);
exports.cardDefinitions = staticTopic();
exports.itemDefinitions = staticTopic();
Using the Server API
You can always access the archivist through state.archivist
. If you really need to make your own
instance, you can use the following:
var archivist = new mage.core.archivist.Archivist();
The following API documentation should tell you how to store, read, delete data and how to set their
expiration time. Keep in mind that there could be vault types that do not support particular
operations. A typical one would be touch
, which is generally not well supported. But even other
operations may trigger an error. For example, when trying to write
to a read-only vault, or
opposite.
Adding new data
archivist.add(topic, index, data, mediaType, encoding, expirationTime);
Marks the data
you pass as something you want to store in your vaults, applying the given topic
and index
. If no mediaType
is given, archivist will try to detect one. If no encoding
is
given, archivist will try to detect one. If you want to store this data for a limited time, you can
pass an expirationTime
(unix timestamp in seconds). If a value already existed, you should expect
this call to fail.
This call is very similar to a combination of archivist.exists
followed by archivist.set
when
the exists
value is false
.
Getting data
archivist.get(topic, index, options, function (error, data) { });
Reads data from all vaults configured for this topic, returning the first successful read. Read errors are considered fatal, and you should abort your operations. However, if a vault is responsive but simply doesn’t hold the value you requested, the next vault in line may still be able to deliver. If a value has already been read or written to before in this archivist instance, that value has been cached and will be returned.
The following options are available to you:
optional
: (boolean, default: false) Indicates whether it’s considered an error if data is not found in any of the vaults.mediaTypes
: (array, default:['application/x-tome', 'application/octet-stream']
) Indicates that you only accept these media types, in the given order of priority. If data of another media type is read, a conversion attempt will be made (eg: JSON to Tome).encodings
: (array, default:['live']
) Indicates that you only accept these encodings, in the given order of priority. If data of another encoding is read, a conversion attempt will be made (eg: JavaScript object to utf8 JSON).
This options object is not required, and your callback may be passed as the third argument.
Multi-get
archivist.mget(queries, options, function (error, multiData) { });
For multi-get operations, please use mget
. The options are identical to and just as optional as in
the get
method. There are two supported queries
formats: the array and the map. In both cases,
the result will map to the input.
Array style queries
queries
[
{ "topic": "players", "index": { "id": "abc" } },
{ "topic": "players", "index": { "id": "def" } },
{ "topic": "players", "index": { "id": "hij" } }
]
multiData
The result is an array where the output order matches the input order:
[
{ "name": "Bob" },
undefined,
{ "name": "Harry" }
]
Object map style queries
queries
{
"a": { "topic": "players", "index": { "id": "abc" } },
"b": { "topic": "players", "index": { "id": "def" } },
"c": { "topic": "players", "index": { "id": "hij" } }
}
multiData
The result is an object map where the keys match the input keys:
{
"a": { "name": "Bob" },
"b": undefined,
"c": { "name": "Harry" }
}
Testing existence of data
archivist.exists(topic, index, function (error, exists) { });
This checks if the value exists in a vault or not. The exists
boolean will reflect the result of
this test.
Overwriting data
archivist.set(topic, index, data, mediaType, encoding, expirationTime);
Marks the data
you pass as something you want to write to all your vaults, applying the given
topic
and index
. If no mediaType
is given, archivist will apply the one it already knows
about this value (if a get
or add
happened before), else it will try to detect one. If no
encoding
is given, archivist will try to detect one. If you want to store this data for a limited
time, you can pass an expirationTime
(unix timestamp).
If a vault allows for diff-logic to occur, and the data passed allows diffs to be read, this will be used.
For certain types of data, like Tomes, you do not have to call this function. Whenever you change
a Tome’s contents, it will call set
automatically for you.
Deleting data
archivist.del(topic, index);
Marks data pointed to by topic
and index
as something you want to delete. A subsequent get
will not yield any data.
Setting an expiration time
archivist.touch(topic, index, expirationTime);
Marks data with a new expiration time (unix timestamp in seconds).
Finding data
archivist.list(topic, partialIndex, options, function (error, arrayOfIndexes) { });
Returns an array of indexes on the given topic matching the partial index you provide. A partial index is the same as a normal index object, except you can leave out properties. The options object is not required, and your callback may be passed as the third argument.
You can, for example, query for all players in the game by using an empty partial index:
// A full index for the "players" topic contains only the "id" property.
archivist.list('player', {}, function (error, indexes) {
/* indexes is now [{ id: 5 }, { id: 20 }, ...] */
});
Or for all players that are in a guild:
// A full index for the "guildPlayers" topic consists of "guildId" and "userId"
archivist.list('guildPlayers', { guildId: 'abc' }, function (error, indexes) {
/* indexes is now [{ guildId: 'abc', userId: 5 }, { guildId: 'abc', userId: 20 }, ...] */
});
You may pass the following options:
sort
An array of sorting rules. Each rule has the format:
json
{ "name": "fieldName in the index", "direction": "asc or desc" }
You may give multiple of these in order of importance. Use direction
to specify ascending or
descending sort-order.
chunk
An array of the format [start, length]
where both values are integers and length
is optional.
This will limit the sorted result to the indexes starting at start
(counts from 0) until
start + length
. This allows for paginating your results.
Options example (sorted by id (descending), page 3 with 10 results per page):
{
"sort": [{ "name": "id", "direction": "desc" }],
"chunk": [20, 10]
}
Distributing changes to all vaults
archivist.distribute(function (distributeError, operationErrors) { });
This takes all the queued up operations (add, set, del, touch) and executes them on each of
the relevant vaults. This distribution is automatically done by the state
object in MAGE when it
closes without errors, so you should never have to call this yourself.
Manually adding data to the archivist memory cache
archivist.addToCache(topic, index, data, mediaType, encoding);
This manually adds a value to the archivist cache based on its topic and index, so that next
calls to archivist.get
won’t hit the database but pull the entry from memory. This can be used
to cache a large amount of entries retrieved from the database that may be more efficient to retrieve
in a single query instead of many archivist.get
calls.
For example:
mysql.query('SELECT * FROM topicTable WHERE type = ?', type, function (err, rows) {
if (err) {
// error
}
for (var i = 0; i < rows.length; i++) {
var row = rows[i];
// cache the row
state.archivist.addToCache('topic', { index: row.index }, row.data,
row.mediaType, row.encoding);
}
// now run some heavy computation that uses archivist.get on that data
heavyComputation(cb);
});
Client API
The archivist is exposed on the browser through a MAGE module called “archivist”. You can use it like any other built-in module:
mage.useModules(request, 'archivist');
You can now read from and write to the vaults by calling using the APIs described below. It is
important to understand that the Archivist client keeps a cache in memory. When you do a get or mget
operation, and the value has already been read or set before, you will receive the version from the
cache (see the maxAge
option below to see how you can control this behavior).
The read methods available are identical to the server variation:
archivist.get(topic, index, options, function (error, data) { });
archivist.mget(queries, options, function (error, data) { });
archivist.list(topic, partialIndex, options, function (error, indexes) { });
For get
and mget
the client enables one extra read option that the server doesn’t have:
- maxAge: A number in seconds. If a value is available in cache, but was set longer ago than this
many seconds, it will not be used. That means that setting maxAge to
0
will always bypass the cache.
These are the data mutation operations, again, identical to the server variation:
archivist.add(topic, index, data, mediaType, encoding, expirationTime);
archivist.set(topic, index, data, mediaType, encoding, expirationTime);
archivist.touch(topic, index, expirationTime);
archivist.del(topic, index);
archivist.distribute(function (error) { });
Unlike on the server, where distribute()
is called by State
objects, on the client-side, you
have to do it yourself. Consider the user experience, and call distribute()
when you have made a
batch of changes and want to send them.
Advanced vault usage
Direct access to a vault’s native API
If you want to access a vault directly, you can ask the archivist for the instance. Often a vault
will expose its underlying library on a .client
property. Depending on if you want to list, read
or write data, you can call:
archivist.getListVault(vaultName);
archivist.getReadVault(vaultName);
archivist.getWriteVault(vaultName);
For more information on the APIs exposed by each vault, please refer to their documentation.
Writing your own Topic APIs
Topic APIs are a collection of APIs that enable a vault to get data to and from its underlying data store. The total set of APIs is limited, and each vault type has its own required subset. For more information on the specifics per vault type, please refer to their documentation.
You can integrate these in the way explained in the “Configure your topics” paragraph. Keep in mind that whenever you choose to implement one of the APIs for a topic, the non-implemented ones will still exist in their default implementations.
The following APIs can be implemented.
Serializing data
The serialize method receives a VaultValue
instance which can contain data, in an encoding
,
tagged with a MediaType
and aware of its topic
and index
. When preparing data to be stored
into a vault, the serialize method may have to change the encoding to better fit the requirements of
the vault, and even return completely different/altered data (imagine prepending header/meta
information to the real data). Finally, the returned data is used by the vault.
Example:
function serialize(value) {
return value.setEncoding(['utf8', 'buffer']).data;
}
Deserializing data
The deserialize method receives the data as it was returned by the vault. It has the duty to
initialize the passed VaultValue
instance with that data, in the right encoding
and MediaType
.
If encoding and/or MediaType are omitted, they will be guessed by the underlying system. This can be
acceptable when the data is returned in deserialized form by the vault.
Example:
function deserialize(data, value) {
value.initWithData(null, data, null);
}
Generating a key
Every vault needs a key function to access data. Generally, the key function will take the topic
and index
and turn those into something that is appropriate for the vault. This can be a string
(eg. in the case of memcached), but also a rich object (eg. in the case of MySQL). Think of the key
as the minimal information required to find a piece of data in a vault.
Example (typical SQL):
function key(topic, index) {
return {
table: topic, // topic is used as the table name
pk: index // { columnName: value }
};
}
Selecting a shard
The shard method is similar to the key method, except it doesn’t pinpoint the exact location of data, but a general location, in order to facilitate sharding. A good example is the Client vault, which needs to emit data changes to different users based on certain very specific information. Incidentally, this is currently the only Topic API method you have to implement yourself.
Example (Client):
function shard(value) {
// the Client shard is one or more actor IDs
return value.index.actorId;
}
Example (Client, multiple actors):
function shard(value) {
// the Client shard is one or more actor IDs
value.setEncoding('live');
return [value.index.actorId].concat(value.data.friendIds);
}
Example (Client, static data for all actors):
function shard(value) {
return true;
}
Vaults
Vaults are used by Archvist to store data.
File vault
For static content, it often makes a lot of sense to store your files on disk, in your repository. The “file” vault makes this possible.
Configuration
type: file
config:
path: ./filevault
disableExpiration: true # optional (default: false)
If your application does not need the file vault to support expirationTime and the touch
command,
you may turn it off by adding disableExpiration: true
to your configuration. This can be useful,
because when your filevault holds many files, file expiration tests can take multiple seconds during
the booting of your app. Use with caution.
Bootstrapping a file vault
This is supported through the ./game archivist-create
CLI command. This will create your empty
folder. Running ./game archivist-drop
will destroy it.
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ✔ | fs.readdir(config.path); |
| get | ✔ | fs.readFile('myfile.filevault' and 'myfile.json'); |
| add | ✔ | fs.writeFile('myfile.filevault' and 'myfile.json'); |
| set | ✔ | fs.writeFile('myfile.filevault' and 'myfile.json'); |
| touch | ✔ | fs.readFile('myfile.filevault'); fs.writeFile('myfile.filevault'); |
| del | ✔ | fs.readFile('myfile.filevault'); fs.unlink('myfile.filevault' and 'myfile.json'); |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createKey(topic, index) |
topic#prop=value&prop=value&... (url encoded) |
|
parseKey(path) |
from key to { topic: '', index: { prop: value } } |
|
serialize(value) |
{ meta: { mediaType: “, expirationTime: 0, ext: ‘.json’ }, content: Buffer } | |
deserialize(data, value) |
from serialized to VaultValue with mediaType and expirationTime |
Memory vault
For static content, it often makes a lot of sense to keep your files in memory for quick access. The “memory” vault makes this possible. The Memory vault keeps data serialized in memory.
A word of warning
Because the memory vault only lives in JavaScript memory, it does not transcend into other processes. That means that you can not depend on it to store data for you in one worker, and expect it to be accessible by another. That is obviously also true between multiple servers.
Configuration
{
"type": "memory"
}
The memory vault doesn’t use any special configuration.
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ✔ | for (var trueName in cache) { } |
| get | ✔ | deserialize(cache[trueName(fullIndex, topic)]) |
| add | ✔ | cache[trueName(fullIndex, topic)] = serialize(data) |
| set | ✔ | cache[trueName(fullIndex, topic)] = serialize(data) |
| touch | ✔ | setTimeout() |
| del | ✔ | delete cache[trueName(fullIndex, topic)] |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createKey(topic, index) |
rumplestiltskin trueName | |
serialize(value) |
VaultValue to internal MemoryData object | |
deserialize(data, value) |
from MemoryData to VaultValue |
Redis vault
The node-redis module is supported through the built-in “redis” vault type.
Configuration
{
"type": "redis",
"config": {
"port": 6379,
"host": "<IP_ADDRESS>",
"options": { "options to pass": "to node-redis" },
"prefix": "prefix for all your keys"
}
}
The options
object is described in the node-redis readme.
Both options
and prefix
are optional. The option return_buffers
is turned on by default by the
Archivist, because the default serialization will prepend values with meta data (in order to
preserve mediaType awareness).
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ||
| get | ✔ | redis.get() |
| add | ✔ | redis.set('NX') |
| set | ✔ | redis.set() |
| touch | ✔ | redis.expire() |
| del | ✔ | redis.del() |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createKey(topic, index) |
“topic/indexProp:indexValue/indexProp:indexValue/…” (index is sorted) | |
serialize(value) |
a header containing mediaType followed by a Buffer of the data |
|
deserialize(data, value) |
parses data from the default serialize function |
MySQL vault
The node-mysql module is supported through the built-in “mysql” vault type.
Configuration
URL based config
mysql:
type: mysql
config:
url: "mysql://user:password@host/db?extraConfig=extraValue"
This URL format is documented in the node-mysql readme.
Object based config
mysql:
type: mysql
config:
options:
host: "myhost"
user: "myuser"
password: "mypassword"
database: "mydb"
The available connection options are documented in the node-mysql readme. For pool options please look at Pool options.
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ✔ | SELECT FROM table WHERE partialIndex |
| get | ✔ | SELECT FROM table WHERE fullIndex |
| add | ✔ | INSERT INTO table SET ? |
| set | ✔ | INSERT INTO table SET ? ON DUPLICATE KEY UPDATE ? |
| touch | ||
| del | ✔ | DELETE FROM table WHERE fullIndex |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createKey(topic, index) |
{ table: topic, pk: index } |
|
parseKey(mysqlKey) |
{ topic: key.table, index: key.pk } |
|
serialize(value) |
{ value: utf8orBufferFromValue, mediaType: value.mediaType } |
|
deserialize(data, value) |
parses row.value and row.mediaType into Value |
Bootstrapping a database
This is supported through the ./game archivist-create
CLI command. This will create your empty
database. Tables must be created through migration scripts. Running ./game archivist-drop
will
drop the entire database.
Schema migrations
Archivist allows for migrations, and the MySQL vault supports this.
Helper functions
Currently we have the following helper functions for MySQL database management:
* createDatabase(cb)
: This is used by archivist-create
to create configured databases. You rarely need to call this
yourself.
* dropDatabase(cb)
: This is used by archivist-drop
to destroy configured databases. You rarely need to call this
yourself.
* createTable(tableName, columns, cb)
: This can be used to create tables inside migration scripts as outlined in the
migrations document.
* dropTable(tableName, cb)
: This can be used to drop tables inside migration scripts as outlined in the
migrations document.
How to set up your MySQL tables
Queries against your database are done through a combination of the generated keys and serialized values. A generated key must yield a table name and a primary key. A serialized value must yield a number of column names with their respective values.
The default topic API that comes with this vault behaves as can seen in the table above. This means
that for example, given a topic people
and index { personId: 1 }
, the following table should
exist:
CREATE TABLE people (
personId INT UNSIGNED NOT NULL,
value TEXT NOT NULL,
mediaType VARCHAR(255) NOT NULL,
PRIMARY KEY (personId)
) ENGINE=InnoDB;
Please note that you should probably change column types, as this is really just an example. Also keep in mind character sets (utf8_bin is usually a good choice) and data type ranges. For more information on data types, please refer to the MySQL documentation.
If you want to change how this information is stored, by adding columns, etc, you can overload the
serializer method to do so. For example, consider the following example if you want to add a
timestamp to a lastChanged INT UNSIGNED NOT NULL
column.
exports.people.vaults.mysql.serialize = function (value) {
return {
value: value.setEncoding(['utf8', 'buffer']).data,
mediaType: value.mediaType,
lastChanged: parseInt(Date.now() / 1000)
};
};
Memcached vault
The node-memcached module is supported through the built-in “memcached” vault type.
Using the Memcached vault for Couchbase clusters
Please note that the Memcached vault can be used to interact with Couchbase clusters and buckets (it does not require you to change the bucket type to “memcached”). Alternatively, the Couchbase vault may be chosen for that, but at this time the library it is based on (node-couchbase) is a less mature and less tested library. That should change over time.
Configuration
myvault:
type: memcached
config:
servers:
- "<IP_ADDRESS>:11211"
- "<IP_ADDRESS>:11411"
options:
foo: bar
prefix: "prefix for all your keys"
The usage of the servers
and options
properties are described in the
node-memcached readme. Both options
and prefix
are
optional.
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ||
| get | ✔ | memcached.get() |
| add | ✔ | memcached.add() |
| set | ✔ | memcached.set() |
| touch | ✔ | memcached.touch() |
| del | ✔ | memcached.del() |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createKey(topic, index) |
“topic/indexProp:indexValue/indexProp:indexValue/…” (index is sorted) | |
serialize(value) |
forced to live encoding, node-memcached will serialize and flag type |
|
deserialize(data, value) |
received in live encoding |
Couchbase vault
The node-couchbase module is supported through the built-in
“couchbase” vault type. It supports sharding by creating hashes on strings that your shard
function may provide.
Alternatively, you can also use the Memcached vault which is based on node-memcached.
Read more on Couchbase at couchbase.com.
Sharding
The Couchbase vault supports sharding. That is a great way to cluster user data onto a single vbucket. You can implement this by giving your topic API a shard function, and returning a string or number to shard on (called our hash-key). In many cases, this will be a user ID. This hash key may not come from the data you are writing, because we need to be able to generate this value on-read as well, before we have data in our hands.
Example:
exports.user = {
index: ['userId'],
vaults: {
couchbase: {
shard: function (value) {
return value.index.userId;
}
}
}
};
Flags
Couchbase allows one to set 4 bytes of meta information (flags) with each document that will be returned when you retrieve it. People generally use these flags to store the data type. The Couchbase vault does this too. Two flag styles have been implemented, and through configuration you can switch between them.
The default flag style turns the file extension that goes with the media type into a binary 4-byte
representation. For example: txt
, json
, tome
, bin
.
There is an alternative flag style called node-memcached
. Use this if you want to be compatible
with the Memcached vault.
Switching between Memcached and Couchbase vaults
You can do this, but there are two limitations you must be aware of. Please ask your system operators for advice. The two issues to take into account are the following.
Flag style
If you roll out this vault and you want to be able to switch to a memcached vault some time in the
future, you need to make sure you set flagStyle
to node-memcached
. If you do not do this, the
file types of your documents will not match up once you make the switch! The reason for this is that
we cannot control what the flags look like when using node-memcached
, as it’s built-in.
Sharding
The Couchbase vault supports sharding. That is a great way to cluster user data onto a single
vbucket. Please note however, that node-memcached
using Moxi will not be able to apply the same
sharding logic. Once you start sharding, you cannot switch back to a Memcached vault.
Configuration
type: couchbase
config:
options:
user: Administrator # optional, usually not required
password: "password" # optional, usually not required
hosts: [ "localhost:8091" ] # optional
bucket: default # optional
prefix: "prefix for all keys" # eg: "bob/"
flagStyle: default # "default" or "node-memcached"
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ||
| get | ✔ | couchbase.get() |
| add | ✔ | couchbase.add() |
| set | ✔ | couchbase.set() |
| touch | ✔ | couchbase.touch() |
| del | ✔ | couchbase.remove() |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createKey(topic, index) |
“topic/indexProp:indexValue/indexProp:indexValue/…” (index is sorted) | |
shard(value) |
undefined | |
createFlags(mediaType, flagStyle) |
either default or node-memcached compatible style flag creation based on mediaType |
|
parseFlags(flags, flagStyle) |
will turn both default and node-memcached style flags into a media type |
|
serialize(value) |
returns a utf8 representation of value.data |
|
deserialize(data, mediaType, value) |
writes data and mediaType into value |
Views Migration
Archivist allows for schema migrations, and the Couchbase vault supports this. This should however only be used to migrate document views.
Also as couchbase only allows for 20MB of data per key and as this feature stores all the version data into a single key, please write short concise reports to avoid breaking that limit.
DynamoDB vault
Excerpt from Amazon:
Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. If you are a developer, you can use Amazon DynamoDB to create a database table that can store and retrieve any amount of data, and serve any level of request traffic. Amazon DynamoDB automatically spreads the data and traffic for the table over a sufficient number of servers to handle the request capacity specified by the customer and the amount of data stored, while maintaining consistent and fast performance. All data items are stored on Solid State Disks (SSDs) and are automatically replicated across multiple Availability Zones in a Region to provide built-in high availability and data durability.
Configuration
For production, use a config similar to this:
dynamodb:
type: "dynamodb"
config:
accessKeyId: "The access ID provided by Amazon"
secretAccessKey: "The secret ID provided by Amazon"
region: "A valid region. Refer to the Amazon doc or ask your sysadmin. Asia is ap-northeast-1"
For a development environment running it’s own DynamoDB, use this:
dynamodb:
type: "dynamodb"
config:
# accessKeyId and region are used to generate the database name, secret will be ignored
accessKeyId: "Your name is a good idea here"
secretAccessKey: "Any value, will be ignored"
region: "And here the game/project name is a good idea"
endpoint: "hostname:port"
sslEnabled: false
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ||
| get | ✔ | DynamoDB.getItem |
| add | ✔ | DynamoDB.putItem with Expect.exists = false set to the index keys |
| set | ✔ | DynamoDB.putItem |
| touch | ||
| del | ✔ | DynamoDB.deleteItem |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createKey(index) |
{ index1: { 'S': 'value1' }, index2: ... } |
|
serialize(value) |
{ data: { 'S': utf8FromValue }, mediaType: { 'S': value.mediaType } } |
|
deserialize(data, value) |
parses row.data and row.mediaType into Value | |
transformError(value, error) |
if (error.code === knownError) return new Error('Comprehensive message') |
|
consistent |
A boolean, default to true |
Reads consistency
By default reads made directly through the DynamoDB vault are eventually consistent, but for calls made through archivist the consistency is set to strongly consistent by default in the API. Strongly consistent calls are more expensive and slower than eventually consistent ones, so if you feel that your table doesn’t need to be strongly consistent, feel free to change the “consistent” boolean in the topic DynamoDB API to false.
Schema migrations
Archivist allows for schema migrations, and the DynamoDB vault supports this.
When writing migration scripts, please refer to
Class: AWS.DynamoDB for more
details on how to use the DynamoDB class exposed by the vault as vault.dynamodb
. For most tables
you will only need to create the indexes (at least 1 Hash index, 1 optional Range index and up to 5
secondary indexes), all other columns are created dynamically at insert time by DynamoDB.
See Amazon DynamoDB Data Model for more details on how you should choose your Hash and Range indexes.
For the mandatory ProvisionedThroughput
map in the createTable object, values for production
should be 100 ReadCapacityUnits
and 20 WriteCapacityUnits
, then please ask the game’s sysadmins
to take care of updating those values manually to more appropriate numbers. This has a direct impact
on the price Amazon charges for the service.
How to set up your DynamoDB tables
The default driver expects a table with a basic HashKey and optional RangeKey both set as strings,
it will then store data serialized in 2 columns named data
and mediaType
.
If you need to explicitly unpack your data in multiple columns, you will need to override serialize and deserialize like in the following example:
exports.people.vaults.dynamodb.serialize = function (value) {
// serialize the primary keys as usual
var item = this.createKey(value.index);
// store manually in each column
item.fullName = { 'S': value.data.fullName };
item.email = { 'S': value.data.email };
// AWS requires everything to be strings, even numbers, when sent through their API
item.age = { 'N': value.data.age.toString() };
item.interests = { 'SS': value.data.interests };
return item;
};
// make sure to override deserialize too!
exports.people.vaules.dynamodb.deserialize = function (data, value) {
// just read data from DynamoDB, you may need to do some type conversions manually
value.setData(null, {
fullName: data.fullName.S,
email: data.email.S,
age: parseInt(data.age.N, 10),
interests: data.interests.SS
});
}
Elasticsearch vault
Elasticsearch is a full-text search engine based on Lucene. It is powered by the elasticsearch
module and supports sharding, either automatic (Elasticsearch will shard everything based on the key) or manual through the
shard
function.
Configuration
elasticsearch:
type: elasticsearch
config:
# this is the default index used for storage, you can override this in your code if necessary
index: testgame
# here is your server configuration
server:
hostname: '<IP_ADDRESS>'
port: 9200
secure: false
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ||
| get | ✔ | elasticsearch.get |
| add | ✔ | elasticsearch.index with op_type set to create |
| set | ✔ | elasticsearch.index |
| touch | ||
| del | ✔ | elasticsearch.del |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createTarget(topic, index) |
use the topic as a type and urlencode the index as the id | |
serialize(value) |
uses live encoding but throws on Buffers | |
deserialize(data, value) |
parse live encoding | |
shard(value) |
auto sharding by Elasticsearch |
TTL Support
Elasticsearch supports automatic expiration of documents, but not updating a documents TTL (you need to repost the whole document) so the touch function is not available. Also, if using TTL, you will need to manually update your type’s mapping to enable ttl as is explained here: Ttl Field Reference.
Binary data storage
By default, the serialize function will fail on buffers. It is because Elasticsearch is a full-text document search engine, as such storing binary data doesn’t make much sense. If you really really do need to store binary data, I recommend either embedding it within an object in base64 format, like:
// retrieve our binary buffer
var picture = uploadedUserPhotoAsBuffer();
// then store it for searching stuff
var data = {
name: 'foobar.jpg',
tags: ['cats', 'kawaii'],
data: picture.toString('base64');
};
state.archivist.set('photos', { userId: userId, pictureId: pictureId }, data);
Then manually decode it when getting it back from Elasticsearch.
Another (more recommended) way to do that is to use Elasticsearch to only store the metadata for search purposes and store the binary file itself in a more appropriate vault (such as file).
Searching for data
Archivist doesn’t provide any way to search for data in a complex manner, if you need to search for data you will need to call the Elasticsearch client yourself, here is an example of how to find the picture stored in the example above:
// retrieve client
var esClient = archivist.getReadVault('elasticsearch').client;
// the index is automatically filled in from the configuration but if need you can just override _index here
var options = { _type: 'photos' };
// build query, for example every photos tagged with cats
var query = {
query: {
term: {
tags: 'cats'
}
}
}
// then run your search query
esClient.search(options, query, function (err, data) {
// then parse the data array
});
For more details about the Elasticsearch client, please read this documentation, then for the Elasticsearch query language please read the Query and Query DSL documentation on the Elasticsearch website.
Notes about manual sharding
While it is possible to override the sharding by making the api.shard
return a string, special attention is to be made
so that sharding always yield the same result for records, elasticsearch will not throw any error if the same id with
different data exists in 2 different shards and both values will be retrieved by the search function.
Manta vault
The node-manta module is supported through the built-in “manta” vault type.
About Joyent Manta
From Joyent Manta documentation:
To use Joyent Manta Storage Service, you need a Joyent Cloud account. If you don’t already have an account, you can create one at https://my.joyentcloud.com/signup. You will not be charged for Joyent Manta Storage Service until you use it.
Once you have signed up, you will need to add an SSH public key to your account. Joyent recommends
using RSA keys, as the node-manta
CLI programs will work with RSA keys both locally, and with the
ssh agent
. DSA keys will only work if the private key is on the same system as the CLI, and not
password-protected.
Configuration
Assuming you have set up a public key with Joyent, you can configure the Manta Vault as follows:
type: manta
config:
# url is optional
url: "https://us-east.manta.joyent.com"
user: bob
sign:
keyId: "a3:81:a2:2c:8f:c0:18:43:8a:1e:cd:12:40:fa:65:2a"
# key may be replaced with "keyPath" (path to a private key file), or omitted to fallback to
# "~/.ssh/id_rsa"
key: |
-----BEGIN RSA PRIVATE KEY-----
..etc..
-----END RSA PRIVATE KEY-----
Where keyId
is the fingerprint of your public key, which can be retrieved by running:
ssh-keygen -l -f $HOME/.ssh/id_rsa.pub | awk '{print $2}'
And where key
is your private key. As an alternative to the key
field, you may provide a
keyPath
field which points to the file containing your private key.
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ✔ | manta.ls() |
| get | ✔ | manta.get() |
| add | ||
| set | ✔ | manta.put() |
| touch | ||
| del | ✔ | manta.unlink() |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createFolder(topic) |
urlencode(topic) | |
createFileName(index) |
URL encoded query string or “void”, followed by .bin file extension | |
parseFileName(fileName) |
Parses query string back into index | |
serialize(value) |
{ data: Buffer, mediaType: value.mediaType } |
|
deserialize(data, value) |
From serialized to VaultValue with mediaType |
Client vault
This vault is used to send updates to the player, so that their data is always synchronized in real
time. This vault is always created when an archivist is instantiated by a State
object, using a
name identical to the type: client
.
Configuration
This vault type requires no configuration.
Supported operations
| operation | supported | implementation |
|---|---|---|
| list | ||
| get | ||
| add | ✔ | state.emitToActors('archivist:set') |
| set | ✔ | state.emitToActors('archivist:set' or 'archivist:applyDiff') |
| touch | ✔ | state.emitToActors('archivist:touch') |
| del | ✔ | state.emitToActors('archivist:del') |
Required Topic API
| signature | required | default implementation |
|---|---|---|
createKey(topic, index) |
{ topic: topic, index: index } |
|
serialize(value) |
{ mediaType: '', data: '', encoding: 'utf8/base64' } |
|
shard(value) |
✔ |
Service Discovery
The Service Discovery library is a library that allows you to announce and discover services on the local network, through different types of engines. It helps coordinating larges clusters of applications by providing a near zero configuration environment.
The service allows the user to provides metadata that is stored on the network and retrieved when discovering devices, code can then do filtering or retrieve basic configuration options from this metadata.
Warning! Metadata should never contain any sensitive data such as usernames, passwords, etc…! In the case of mDNS for example, data is broadcasted using UDP on the local network with no restriction, as such any device on the local network can access this metadata!
Configuration
To use service discovery, configuration is mandatory. Due to the nature of cloud based solutions such as Amazon EC2,
mDNS will fail to work correctly and you will need to setup an alternative. Possible alternatives are zookeeper
and
consul
.
The mDNS
configuration is as follows:
server:
serviceDiscovery:
engine: mdns
options:
description: "UniqueIdOnTheNetwork" # optional
The zookeeper
configuration is as follows:
server:
serviceDiscovery:
engine: zookeeper
options:
hosts: "<IP_ADDRESS>:2181,<IP_ADDRESS>:2181,etc..."
The consul
configuration is as follows:
server:
serviceDiscovery:
engine: consul
options:
interface: enp4s0
# optional
consul:
host: consul.service.dc.consul
The single
configuration is as follows:
server:
serviceDiscovery:
engine: single
API
Module
Methods
serviceDiscovery.createService(name, type)
, wherename
is the name of your service, andtype
is either tcp or udp. Returns aService
instance.
Service
Methods
service.announce(port, metadata, cb)
, whereport
is the port you want to announce,metadata
is a blob of data associated with the service andcb
is a callback called with a potential error object once the announcement is done.service.discover()
, starts the discovery process, firing events when nodes gets up or down on the network
Events
up
, fired when a node appear on the network (after an announcement), provides aServiceNode
instance.down
, fired when a node disappear from the network, also provides aServiceNode
instance.error
, fired when an error occur.
ServiceNode
Properties
host
, the hostname of the machine hosting the service.port
, the announced port.addresses
, a list of ips, either IPv4 or IPv6, see the methodgetIp
for a better way to access this list.data
, the metadata associated with the service.
Methods
getIp(version, network)
, allows you to retrieve an IP from the addresses list,version
is the IP version, either 4 or 6.network
is an array containing the network list where your service is. The CIDR notation is used to represent the networks.isLocal()
, whether the node is running on the current server or not.
Todo
- Add a method to iterate on ips by version
- Add a up/down event on the node itself?
Examples
Finding nodes on the network
// load the module
var serviceDiscovery = require('../serviceDiscovery'),
mage = require('../mage'),
logger = mage.core.logger.context('mysql'),
mysql = require('some-imaginary-mysql-client-module');
// create our service, let's say for mysql servers on a tcp socket
var service = serviceDiscovery.createService('mysql', 'tcp');
// when a mysql server appear on the network
service.on('up', function (service) {
// connect to the mysql service we just found
mysql.connect(service.getIp(4), service.port, function (error) {
if (error) {
// fatal
}
// do stuff
});
});
// or when one goes down
service.on('down', function (service) {
// mysql went down, start panicking!
});
// start discovering
service.discover();
Announcing your service to the world
// load the module
var serviceDiscovery = require('../serviceDiscovery'),
mage = require('../mage'),
logger = mage.core.logger.context('hello'),
hello = require('imaginary-hello-server');
// create our service, let's say this is a hello world HTTP server
var service = serviceDiscovery.createService('hello', 'tcp');
hello.listen(80, function (error) {
if (error) {
// error stuff
logger.emergency('Could not start hello server on port 80');
mage.fatalError();
return;
}
// announce our service to everyone, and also tell them that we support SPDY
service.announce(80, { spdySupport: true }, function (error) {
if (error) {
// error stuff
logger.emergency('No one can see us, announcing failed!');
return;
}
logger.debug('Server announced successfully!');
});
});
Single-server discovery engine
This engine allows you to have a fake service discovery engine, to have the service discovery library available when you are using only one server.
The services are stored directly in memory. In cluster mode, it uses the process messenger to exchange messages between master and workers.
Limitations
- It works only if you have no more than one server.
mDNS discovery engine
mDNS is a discovery protocol designed by Apple in 2002. It uses UDP multicasting to send on the network service announcements which can be used by other devices/services to connect or access servers without the need for any configuration, it was initially developed for printers and similar devices.
Limitations
Due to the nature of UDP broadcasts/multicasts, they can only be transfered on Level 2 LAN networks. That mean that any higher kind of network won’t be able to see/transmit those packets, and the discovery will fail. Amazon EC2 is an example of network that will not work with mDNS even when using Amazon VPC.
Configuration
In the server.serviceDiscovery.options
object the following options exists:
- description: A string that allows you to override the service name generated by the engine. It needs to be
unique on the network!
Debugging
When announcing with mDNS, there are several tools that allows seeing the network topography in great details, metadata included:
- Linux:
avahi-discover
will discover the whole network, clicking on services will display the metadata at the bottom of the screen. - macOS: BonjourBrowser will do that for you.
- Windows:
Bonjour Browser for Windows
should do that for you.
ZooKeeper discovery engine
Apache ZooKeeper is a project from the Apache Software Foundation that provides distributed datastores for large distributed systems. It is part of the Hadoop eco-system. It uses a hierarchical tree for storing data similar to a file-system.
Configuration
In the server.serviceDiscovery.options
object the following options exists:
- hosts: A string that contains a list of host:port separated with commas.
- options: The
options
object accepted by createClient.
The node-zookeeper-client
library applies 0 retries as a default, which we override with 3
.
You can tweak these settings further by configuring the options
above.
Limitations
ZooKeeper is written in Java so is very heavy, moreover watchers and other niceties are far too lightweight and causes a lot more work than should be necessary when events happen (like having to manually build the difference when nodes change).
Debugging
When debugging, it is possible to retrieve the zookeeper install file and connects using the
bin/zkCli.sh
file provided in the SDK using the following command:
bin/zkCli.sh -server <serverip>:<port>
The CLI will then connect and display a list of commands available, the important ones are:
ls <path>
, show the children list for that path.get <path>
, retrieve the metadata and details for a particular node.
HTTP Server
Registering routes
httpServer.addRoute(pathMatch, handlerFunction, type)
This registers a route (a string or a regular expression) on the HTTP server. Incoming requests will be expected to be handled by the handler function you pass. Based on the type you specify, your handler function will receive different arguments.
“simple”: handler(req, res, path, query, urlInfo)
- req: the IncomingMessage object.
- res: the ServerResponse object.
- path: the path part of the URL.
- query: the parsed query string.
- urlInfo: all information that came out of
url.parse()
.
“callback”: handler(req, path, query, callback)
- req: the IncomingMessage object.
- path: the path part of the URL.
- query: the parsed query string.
- callback: call this when you’ve constructed your HTTP response.
The callback accepts the following arguments in order:
- httpCode: the HTTP status code you want to return.
- out: a string or buffer that you want to send to the client.
- headers: an object with headers.
“websocket”: handler(client, urlInfo)
- client: a WebSocket client connection object. See the
ws
documentation. - urlInfo: all information that came out of
url.parse()
.
“proxy”: endpoint handler(req, urlInfo)
- req: the IncomingMessage object.
- urlInfo: all information that came out of
url.parse()
.
Your handler function must return an endpoint object to connect to. For syntax, please read the
net.connect()
documentation.
httpServer.delRoute(pathMatch)
Removes the handler function registered on the given route.
Simple file serving
httpServer.serveFolder(route, folderPath, [defaultFile], [onFinish])
Registers a route to lead directly to a folder on disk. If you want to be notified when a request finishes, you may pass
an onFinish
function. It may receive an error as its first argument. If you decide to pass this function, logging the
error will be your responsibility.
Example:
mage.core.httpServer.serveFolder('/source', './lib');
If you provide a defaultFile
file name argument, serving up a folder by its name will serve up a default file if it
exists.
Example:
mage.core.httpServer.serveFolder('/source', './lib', 'index.html');
httpServer.serveFile(route, filePath, [onFinish])
Registers a route to lead directly to a file on disk. If you want to be notified when a request finishes, you may pass
an onFinish
function. It may receive an error as its first argument. If you decide to pass this function, logging the
error will be your responsibility.
check.txt
By default, the HTTP server will serve a file “check.txt” from the root folder of your project, if it exists. This can be useful for services such as load-balancers to poll for the availability of your application.
Custom Favicon
httpServer.setFavicon(buffer, [mimetype])
Registers a route “/favicon.ico” and serves the given buffer as content. The mime type defaults to
image/x-icon
and may be overridden.
Cross-Origin Resource Sharing (CORS)
If you want your application to span multiple domains, you need to enable CORS. For information on what CORS is, please consult the following websites.
Performance
Keep in mind that using CORS will often cause so-called “preflight” requests. A preflight request
is an HTTP OPTIONS
request to the server to confirm if a request is allowed to be made in a manner
that is considered safe for the user. Only after this confirmation will a real GET
or POST
request be made. All this happens invisible to the end user and developer, but there is a
performance penalty that you pay, caused by this extra round trip to the server.
Using authentication and CORS
If you use Basic
or any other HTTP authentication mechanism, you cannot configure CORS to allow
any origin using the *
wildcard symbol. In that case, you must specify exactly which origin is
allowed to access your server.
Configuration
In your configuration, you can enable CORS like this:
server:
clientHost:
cors:
methods: "GET, POST, OPTIONS"
origin: "http://mage-app.wizcorp.jp"
credentials: true
methods
(optional) lists which HTTP request methods are acceptable.origin
(optional) sets the required origin, may be (and defaults to)"*"
credentials
(optional) must be set totrue
if you want cookies and HTTP authentication to be usable at all. You can then no longer use the wildcard origin.
CORS in IE9
The XMLHttpRequest object exist on IE8+ but sadly CORS is not supported on IE8 and IE9. So the idea is to use the XDomainRequest object for those old version of IE but here is the catch: XDomainRequest doesn’t hold a status code.
How to deal with IE9 without status code then?
You may already have it in place in your app. If not, here what you could do.
When communicating with the server, the request will time out and generate a ‘network’ error.
You can listen to it from the msgServer msgServer.on('io.error.network', doSomething);
.
The suggestion here is to retry (msgServer.resend();
) on a network error and to reload the app
after n retries. If the app is in maintenance it would probably show the maintenance page.
What does it change for other browsers?
Absolutely nothing. And you can still apply the retry logic on network errors, it’s not a bad idea.
Log Configuration
server:
quietRoutes: # Filter out debug and verbose logs for URLs matching these regex
- ^\/check\.txt
- ^\/favicon\.ico
longRoutes: # Filter out long warnings for URLs matching these regex
- ^\/msgstream
longThreshold: 500 # The number of milliseconds before a request is considered to be taking too long
The following logs will be filtered out when the url for the request matches any regex in the
quietRoutes
:
m-28019 - 19:58:44.830 <debug> [MAGE http] Received HTTP GET request: /check.txt
m-28019 - 19:58:44.830 <verbose> [MAGE http] Following HTTP route /check.txt
The following log will be shown for any http request that takes longer than the configured
longThreshold
and can be filtered out when the url for the request matches any reges in the
longRoutes
:
m-876 - 20:08:58.193 <warning> [MAGE http] /app/pc/landing completed in 1181 msec
Message Server
The message server is in charge of message propagation through the network and between the clients and the backend.
Cluster Identification
Currently the message server system will identify itself as being a part of a cluster by using the application root package name and version. However in environment where multiple instances of the same application and version are run, there will be conflicts with the messaging system. (e.g. inside single box which houses multiple test environments).
To prevent pollution and contamination of messages, the server.serviceName
configuration entry needs to be set, to
give each environment a unique identifier.
Example:
server:
serviceName: applicationName-environmentID
Subsystems
The systems that make the message server and their configuration are described below.
MMRP
The message server uses the MMRP library to ensure communication between the different MAGE instances on the network. In order to find these instances, you need to have service discovery set up.
To use MMRP, please configure it to bind to a particular port and pick a network on which it is expected to communicate.
Example:
server:
mmrp:
bind:
host: "*" # asterisk for <IP_ADDRESS>, or a valid IP address
port: "*" # asterisk for any port, or a valid integer
network:
- "192.168.2" # formatted according to https://www.npmjs.com/package/netmask
For more information, please read the MMRP documentation.
Message stream
The messages sent by the server inevitably make their way to a client through a message stream.
For more information, please read the Message stream documentation.
Sending messages
It is recommended to use the State class to send and broadcast messages to users, but if you want more control over or a better understanding about how messages are sent, please keep on reading.
Keep in mind that the system makes no guarantees that the message will actually be delivered, although as long as the address and clusterId you pass are valid, and that user remains logged in, the message should be received.
Sending a single message
You can send a message directly to a user using the following call.
mage.core.msgServer.send(address, clusterId, message);
Where:
address
is the recipient’s session key.clusterId
refers to the physical location in the network where that session’s messages are being stored for delivery.message
is a string or Buffer containing valid JSON, or an array of strings/Buffers if you want to send a batch.
Broadcasting a message to all logged-in users
You can send a message to all logged-in users, by using the following call.
mage.core.msgServer.broadcast(message);
Where:
message
is a string or Buffer, or when you want to send multiple messages, an array of strings/Buffers.
Receiving messages on a web client
The MAGE client will receive messages and emit them as events. You can listen for them through mage.eventManager
.
mage.eventManager.on('event.name', function (message) {
/* use the message */
});
For now, the only way to receive these messages is to make sure messages are sent or broadcast with the following format:
[
["event.name", {"key":5,"another":"value"}],
["another.event", {"etc":false}],
]
Testing your application with AerisCloud and Marathon
When using AerisCloud to test your application across a cluster of servers using the Marathon service, please make sure
to include the following configuration in config/marathon.yaml
. Over time, the addresses below may change, so always
stay in touch with your friendly local system administrator to make sure these make sense.
server:
clientHost:
expose: null
bind:
port: PORT0
serviceDiscovery:
engine: "zookeeper"
options:
hosts: "<IP_ADDRESS>:2181"
mmrp:
bind:
port: PORT1
network:
- "192.168.2"
Your Makefile needs to replace the magical constants PORT0 and PORT1 in this file when running. Here’s an example of the marathon Make target:
.PHONY: marathon
marathon:
source /opt/nvm/nvm.sh && \
sed -i "s/PORT0/${PORT0}/" config/marathon.yaml && \
sed -i "s/PORT1/${PORT1}/" config/marathon.yaml && \
sed -i "s/PORT/${PORT}/" config/marathon.yaml && \
nvm use 0.10.33 && \
NODE_ENV=production,marathon HOME=${MESOS_DIRECTORY} make deps && \
NODE_ENV=production,marathon node .
When pushing the application to Marathon, you need to select ports for PORT0 and PORT1. An example:
aeriscloud marathon office/jp push -p 80 16001
For more information about using Marathon, please read the documentation
MMRP
General concept
MMRP (MAGE Message Relay Protocol) is the messaging layer between node instances. It is used to enable communication between multiple MAGE instances and between the different node processes run by MAGE. In the end, it allows messages to flow from one user to another.
Imagine a network set up like the following, where 3 servers each host a process-cluster.
 If MAGE is running in single-node mode (that is, the cluster consists of only a single process), it will instantiate MMRP as both a relay and as a client. Through this topology, messages can be sent between all processes (or nodes) in the MAGE network.
MMRP depends on service discovery to announce relays on the network to each process. The protocol and library used to communicate between processes is ZeroMQ.
MMRP Nodes
Node API
Construction
To get started, you must have access to an MMRP node object, or create one like this:
var MmrpNode = mage.core.msgServer.mmrp.MmrpNode;
var role = 'relay'; // or 'client', or 'both'
var cfg = { host: '*', port: '*' };
var clusterId = require('os').hostname();
var node = new MmrpNode(role, cfg, clusterId);
node.relayUp(uri, clusterId)
Notify the node that another relay has become available at a particular URI with a particular clusterId. If the node is a relay, it should connect to it. If the node is a client, it should connect to it only if it’s the relay of the same clusterId as the client.
node.relayDown(uri)
Notify the node that another relay has become unavailable. If we are connected to it, we should use this as an advice to disconnect.
node.send(envelope, attempts)
Sends an envelope across the network according to the route inside it. If attempts is passed, the node will try to resend the envelope if routing failed. Retrying should normally not be needed, as ZeroMQ manages connections and buffers for you.
node.broadcast(envelope)
Will broadcast the envelope to every node on the network.
Node events
delivery
When an envelope is received, a delivery is always fired, even if the envelope is not intended for this cluster or even this node in the cluster! It is up to the receiver to decide to act on it or not. You can listen for the event based on the type of the envelope that is being delivered.
For example, if the type of the envelope is "my.special.namespace"
, a total of four events will
be emitted with the envelope as the sole argument, in the following order:
- delivery.my.special.namespace
- delivery.my.special
- delivery.my
- delivery
Envelopes
MMRP communicates by sending envelopes from node to node. An envelope is a type of object that is instantiated by MMRP when it receives one, and you must instantiate yourself when you want to send one. The public API for that is described below.
Envelope API
Construction
You can get a hold of the Envelope class through MMRP, and instantiate it as follows.
var Envelope = mage.core.msgServer.mmrp.Envelope;
var type = 'something.i.made.up';
var message = ['hello', 'world'];
var route = ['someClusterId', 'someAddressOnThatCluster'];
var returnRoute = [];
var flags = ['TRACK_ROUTE'];
var envelope = new Envelope(type, message, route, returnRoute, flags);
To and from ZeroMQ
To create an envelope from an array received through ZeroMQ, call:
var envelope = Envelope.fromArgs(myArgs);
To turn an envelope into an array that can be sent through ZeroMQ, call:
var args = envelope.toArgs();
The message
envelope.setMessage(type, message)
Resets the envelope’s type and message, where message is either a string, buffer or an array of strings and buffers.
envelope.addMessage(message)
Adds the message to the list of messages currently wrapped inside the envelope. The message must be a buffer or a string.
The route
envelope.setRoute(route)
Resets the route of the envelope. The route must be an array, or a string (that becomes a 1-element array) or falsy (to empty the route).
envelope.consumeRoute(identity)
Removes the identity from the start of the route, if it’s there. If no identity is given, the entire route is emptied.
envelope.routeRemains()
Returns true if there is a route left, false otherwise.
envelope.getFinalDestination()
Returns the final address in the route.
The return route
envelope.setReturnRoute(route)
Like the setRoute(route)
function, and applying the same rules to its only argument, this sets
the return route of the message. In other words, if we want to send something back to the sender of
this envelope, we should use this returnRoute.
envelope.injectSender(identity)
Prepends an identity to the returnRoute.
envelope.getInitialSource()
Returns the envelope’s original sender identity.
Flags
There is currently only 1 flag, namely ‘TRACK_ROUTE’. When this flag is active, the returnRoute must be kept around in the envelope as it travels across the network.
envelope.isFlagged(flag)
Returns true if a flag is turned on for this envelope, false otherwise. The flag may be a string, or its integer representation.
envelope.getFlags()
Returns an array with string representations of all flags that are turned on.
envelope.setFlag(flag)
Turns a flag on.
Message Stream
The protocol
The serialization protocol used by the message stream is the same regardless of the transport type used. Packages of messages are formatted in JSON as follows:
{
"1": [
["some.event.name"],
["another.event", { "some": "data" }]
],
"2": [
["eventname", { "any": "data" }]
]
}
The key represents the message pack ID, which increments by 1 for each batch of messages emitted. Messages must be processed in order of message pack ID and the array order in which they occur. The content of the message is an array of 1 or 2 elements.
The first element is a string that represents the type of the message. This may be used as the event name in a client-side event emitter. The optional second element is the real payload of the message.
Message types may be dot-separated. A client may choose to implement delivery of messages through
the emission of an event in chunks from most-relevant to least-relevant. For example, the message of
type shop.purchase
with data { "item": "boots" }
may be emitted twice:
shop.purchase
with data{ "item": "boots" }
shop
with data{ "item": "boots" }
Because networks (especially mobile) are not always reliable, MAGE will keep all messages on the server until the client actively reports successful delivery. This is done per message pack ID. Successful reception of these messages must be confirmed back to the server. Each transport implements this mechanism in its own way.
Transport types
HTTP short-polling
With short-polling, the endpoint for message retrieval becomes:
GET http://domain/msgstream?sessionKey=abc&transport=shortpolling&confirmIds=1,2,3
Always be sure to pass the active session key for this user. Without it, no messages can be
delivered. If messages are available, they will be returned in the format described in
“The protocol”, with content-type application/json
and HTTP status code 200
. If no messages are
available, HTTP status code 204
will be returned without a response body.
Message pack IDs you want to confirm are to be sent back in the URL’s query string named
confirmIds
and must be comma separated.
HTTP long-polling
With long-polling, the endpoint for message retrieval becomes:
GET http://domain/msgstream?sessionKey=abc&transport=longpolling&confirmIds=1,2,3
Always be sure to pass the active session key for this user. Without it, no messages can be
delivered. If messages are available, they will be returned in the format described in
“The protocol”, with content-type application/json
and HTTP status code 200
. This is no
different from short-polling. If there are no messages available however, the connection will hang
and wait for messages to arrive. The moment messages are ready for delivery, the HTTP request will
return with HTTP status code 200
and content-type application/json
.
In order to reliably detect clients’ unavailability, the server will periodically close the
connection to the client. It is up to the client to detect this, and reconnect with the server to
make its presence known. The disconnect will happen with HTTP status code 204
and does not carry a
response body.
Message pack IDs you want to confirm are to be sent back in the URL’s query string named
confirmIds
and must be comma separated.
WebSocket
The WebSocket endpoint is:
ws://domain/msgstream?sessionKey=abc
Always be sure to pass the active session key for this user. Without it, no messages can be delivered. If messages are available, they will be returned in the format described in “The protocol”.
Message pack IDs you want to confirm are to be sent back in single strings that are simply all the
IDs comma separated. For example: 1,2,3
.
Command Center
The Command Center handles the exposure and processing of MAGE module user commands (ie: RPC).
MAGE currently only supports the MAGE user command custom protocol, through httpBatchHandler
It uses the following endpoint: /<appname>/
.
Access from the client
MAGE automatically adds functions to your modules to have easy access to your user commands. If you have a gift
user
command in a gifting
module, you can use mage.gifting.gift()
.
If you want to use the command directly, you have to use the msgServer
module.
var mage = require('mage');
mage.msgServer.sendCommand('gifting.gift', parameters, callback);
User commands
A complete example is available in the walkthrough: Writing modules.
Configuration
Configuration is hierarchical, and based upon the NODE_ENV
environment variable. Configuration
can be found in a number of places, and may be either JSON or YAML. In ascending importance:
- Module
config
files - Game
default
file - Personal configuration file
- Optional
environment
file
For example, if a field exists in the game default configuration, and also in a module default configuration, then the game configuration wins.
Module default files
A typical module is a folder that may look like:
missions
usercommands/
index.js
client.js
component.json
config.yaml
When you tell Mage to use a module, it will automatically know that it should look for and load the
default configuration file. It may be a YAML or a JSON file. Your choice. If no file is found, then
Mage doesn’t mind and assumes that no configuration was necessary. A config.yaml
file that
looks like:
hello: world
will be loaded into the Mage configuration object under module.missions
. The module
field is
fixed, and the missions
field is taken from the name of the module in the file system. The mage
configuration object will look like (considering only the added data):
{
"module": {
"missions": {
"hello": "world"
}
}
}
Remember, this is the lowest importance of configuration. You can choose to override it with the game default configuration file, or a personal configuration file.
Game default file
The game default file should contain fields common to, or sensible as defaults in, all
environments. Remember, the production and personal configuration files override the content of
this file, so having things in here does not limit you. This file lives in the top level of your
game project, in the config/
folder.
Personal configuration file
Finally, the environment variable NODE_ENV
is read to determine which file to use as a personal
configuration. These files live in the config/
directory along with the default
configuration
file. The value of NODE_ENV
should be equal to the name of the configuration file you want to
use, sans extension. i.e. If your NODE_ENV
resolves to me
, then the personal configuration file
to be used should be called me.json
or me.yaml
(don’t have both in the directory, just don’t).
Make sure you set this environment variable in your .bash_profile
or .profile
so you don’t have
to keep typing it.
Environment file
The environment file should be called environment.json
or environment.yaml
and allows the user
to define variables in the configuration that might be overridden by environment variables, it takes
the same format as the original configuration files but instead of a value the name of the environment
variable should be used.
For example if one wants database.mysql.url
to be overridden by an optional DB_URL
variable:
database:
mysql:
url: DB_URL
Environment variables also support casting to int
, float
or bool
by adding a colon and the
type right after it inside your environment
file:
foo:
variable_int: VAR_INT:int # will cast VAR_INT to int
variable_float: VAR_FLOAT:float
bar:
variable_bool: VAR_BOOL:bool
Methods
The methods of the configuration object should be avoided most of the time and are mainly for mage internal use. However, there may be cases for which configuration is automatically generated etc. , when these methods may be useful.
mage.core.config
exposes the following methods:
get(path, [alt]);
getSource(path)
getMatryoshka()
loadModuleConfig(moduleName, absolutePathToConfigFile);
setDefaults(absolutePathToConfigFile);
initialize(mageLogger)
get
A safe getter function. The path
should be an array of keys of increasing depth, but can
optionally be a dot delimited string. The optional alt
parameter is a value to use if there is
no value found at this path. By default undefined
will be returned if the object does not have
this path. Unlike earlier versions of configuration, get
is necessary to extract configuration
data.
getSource
This uses a path array (or string) just like get
, but returns information about the source of the
configuration.
getMatryoshka
This returns a copy of the underlying data structure that contains your aggregated configuration. This may be useful for tools or for debugging.
loadModuleConfig
This is used by Mage only. This function is used when Mage is setting up a module, and is
responsible for loading the default configuration file, if one exists. Just place a file called
config.<json, yaml, js>
and Mage will pick it up.
setDefaults
Merges an object into an existing configuration, without overriding existing content. This is intended for Mage internal use and should not be used in games.
initialize
This is used by Mage to trigger a load of configuration files. It may only be called once.
Logging
Terminology
Channel
Loggers have multiple channels (described at the bottom of this file) that define the verbosity or severity of a log entry.
Message
The message is the primary line of information you are logging. In every case, whenever you pass a message into a logger, you are encouraged to send multiple arguments and not do any serialization (JSON or other) yourself. The reason for this is that some messages may be surpressed by the configuration, based on verbosity, and needless serialization would needlessly hurt performance.
Loggers can serialize any data type, including Error instances (which will yield a stack trace).
Contexts
A logger can be given one or more contexts in order to make the source of a log-call a bit clearer.
Details
Additional details can be passed into a logger to clarify a situation with more human readable output where needed. This is especially helpful when observing log output in a terminal.
Data
Loggers may be passed additional key/value data. This helps certain logging services (like Graylog2) which allow users to query for data.
API
The logger
module is a logger instance that exposes multiple methods. Loggers expose several
methods.
logger.context(context1, .., contextN)
Returns a new logger object that has the given contexts on top of the ones that logger had. The typical use case for this is that your module (eg: ‘shop’) has the following code at the top:
var logger = mage.logger.context('shop');
// all logging should now be done on logger
logger.<channel>(arg1, .., argN)
Each channel name is a method on a logger. This allows you to write:
mage.logger.emergency('Crucial file missing');
mage.logger.warning('Stamina too low to quest:', stamina, 'quest:', questId);
mage.logger.verbose('Reading from file:', filePath);
At the same time, the channel is not just a function, but an object that exposes a few more
methods. All those methods can be chained. Because you are no longer using the channel name as a
function, you have to call the log
method at the end of the chain to provide the log message (see
examples below).
logger.<channel>.details(arg1, .., argN)
To provide additional human readable details. This function can be called multiple times to provide multiple lines of information.
logger.<channel>.context(context1, .., contextN)
This adds the contexts to this one log entry, on top of the ones that were already set before.
logger.<channel>.data(key, value)
Add a queryable key/value pair to the log entry.
logger.<channel>.data(valueMap)
Add many key/value pairs to the queryable data of this log entry.
logger.simulator(name)
This will return a shim for any of the supported 3rd party logger libraries. This can be useful when interfacing with 3rd party modules that have a strong dependency on these loggers.
Examples
try-catch block
try {
throw new Error('You are too grey for me');
} catch (error) {
mage.logger.error(error);
}
Passing rich data that certain logging services may be able to query
mage.logger.debug
.data({
playerId: 'abc',
questId: questId,
conditions: {
stamina: player.stamina,
xp: player.xp
}
})
.log('Trying to run quest', questId);
Passing details
mage.logger.debug
.details('Used Facebook mobile login')
.details('URL', facebookServiceUrl)
.log('User logged in', actorId);
HTML5 Client API
If configured, the client module will also expose all the channels, so you can log just like you do
on the server. The API is however limited to logging a message
. At this time, contexts, details
and key/value data are not supported. Stack traces of Error objects are automatically logged in the
data part however. The client can log to console, but also to the server.
Built-in Logging Channels
The built-in channels are designed to be granular and meaningful; this should help production operation by allowing you to throw alerts properly (only 3-4 emergencies or alerts a minute should alert operation, but it might take 100’s of user errors a minute to trigger the same alerting).
| Channel | Description |
|---|---|
| verbose | Low-level debug information (I/O details, etc). Reserved to MAGE internals. |
| debug | Game server debugging information. |
| info | User command request information |
| notice | Services state change notification (example: third-party services and databases) |
| warning | An unusual situation occured, which should required analysis. |
| error | A user request caused an error. The user should still be able to continue using the services. |
| critical | A user is now stuck in a broken state which the system cannot naturally repair. |
| alert | Internal service (datastore API calls failed, etc) or external service are failing. |
| emergency | The app cannot boot or stopped unexpectedly. |
Configuration
The logger allows for logging in different “writers”. At this time, the following are available on the server:
- terminal: log to the console
- file: write to log files on disk
- syslog: write to syslog through UDP
- graylog: www.graylog2.com
- websocket: streams the log on a Savvy websocket
And the following are available on the client:
- console: outputs to console.log/warn/error
- server: send log entries to the server to be reported in server-side writers
Configuration happens in your config file in:
logging:
server: {}
html5: {}
Any time you see a channel range configuration, it may be a string describing a range of channels,
or an array of multiple of these strings. A range may contain >
, >=
, <
and <=
comparison
operators. The special string all
represents all channels.
Server: Terminal
logging:
server:
terminal:
channels: [">=info"]
config:
jsonIndent: 2
jsonOutput: false # defaults to false, useful when piping output to another app
theme: default
Available themes: default
, dark
, light
.
Server: File
logging:
server:
file:
channels: [">=debug"]
config:
path: "./logs"
# optional:
jsonIndent: 2
mode: "0600" # make sure this is a string!
fileNames:
"app.log": "all" # this is configured by default and you may override it
"access.log": "info"
"error.log": ">=warning"
Server: Syslog
logging:
server:
syslog:
channels: [">=debug"]
config:
host: localhost # host to connect to (IP or hostname)
port: 514 # UDP port to connect to
appName: myGame
facility: 1 # see syslog documentation
format:
multiLine: true # allow newline characters
indent: 2 # indentation when serializing data in multiLine mode
Server: Graylog
logging:
server:
graylog:
channels: [">=info"]
config:
servers:
- { host: "<IP_ADDRESS>", port: 12201 }
- { host: "<IP_ADDRESS>", port: 12201 }
facility: Application identifier
format:
multiLine: true # allow newline characters
embedDetails: false # embed log details into the message
embedData: false # embed data into the message
Server: WebSocket
logging:
server:
websocket: {}
Please note that channels are not supported in the websocket configuration. The channel description array is to be passed as JSON into the connection once it has been established.
HTML5: All writers
MAGE will override the client’s console object with it’s own version that supports logging to the server and hiding output on the client. If you do not want this behavior you can set disableOverride to true, but MAGE will no longer be able to collect logs from the client.
HTML5: Console
Logs to the browser’s console.
logging:
html5:
disableOverride: false
console:
channels: [">=verbose"]
server:
channels: [">=error"]
Sampler
The sampler library is an interface for Panopticon. It uses configuration to handle the setup of panoptica, and exposes the methods of these panoptica as a group. Sampler handles the sending of data to all of the panoptica, so you only need to worry about the API.
API
sampler.set(path, id, n)
, wheren
, a finite number, may replace a previousn
for thisid
.sampler.inc(path, id, n)
, wheren
is added to the previous value ifn
is a finite number. Ifn
is not a finite number, then it defaults to1
.sampler.sample(path, id, n)
, which keeps track of the max, min, average and standard deviation ofn
over an interval.sampler.timedSample(path, id, dt)
, which is like sample, but takes the output of a high resolution timerdt
(or rather the difference between two timers).
These methods take the same arguments as a panopticon, so please see the Panopticon documentation for more details.
Configuration
Sampler is configured by giving it one ore more (named) intervals. For each such interval, data will be aggregated and served on the HTTP and WebSocket Savvy routes.
Other configuration that may optionally be supplied are:
sampleMage (boolean) Indicates whether MAGE itself should automatically collect its internal metrics. This provides information on data store operations, user commands, and more.
bufferLength (integer) Indicates how many aggregated results should be kept around in memory. More data means that services querying for it can look further back into the past.
Example:
sampler:
sampleMage: true
bufferLength: 100
intervals:
realtimeish: 1000
bytheminute: 60000
Access
The data can be accessed through the Savvy route: /savvy/sampler
, for example by doing an
HTTP GET
request to http://localhost/savvy/sampler
. The same URL (with the ws:
protocol) can
be used to create a WebSocket data stream. The “ws” module comes with a connector utility, so this
should work from your project (please do mind the hostname and port):
./node_modules/mage/node_modules/.bin/wscat --connect "ws://localhost/savvy/sampler/"
You can dig into the dataset by augmenting the path you query. For example, if all you care about is
the PID of the master process, you may query http://localhost/savvy/sampler/realtimeish/data/pid/values/master/val
.
While connected through a WebSocket, you may retrieve the buffered samples by sending a message
containing the string sendBufferedData
. This will send all deliveries, filtered through the path
you have used in the URL while connecting.
Savvy
API
Savvy provides an HTTP server for various management interfaces. When running your project without
a master node, it will use the normal MAGE HTTP server. If there is a master process, Savvy will sit
in this master process, catching all HTTP requests to /savvy/...
. This makes it ideal for serving
data that the master process may be aggregating from workers. Two examples of this are the logger,
and the sampler subsystems.
getBaseUrl()
addRoute(pathMatch, handler, type)
The base URL is configuration driven, and can be resolved with the getBaseUrl
function.
addRoute
is identical to the addRoute
method of the MAGE HTTP server, except that it will make
sure you are not registering a route outside of the /savvy/
URL space. Also, when calling this on
a worker node, it will be a no-operation. Savvy only registers your routes on the master node.
Process Manager
A MAGE game is really a collection of processes; one master process which spawns workers which are in turn responsible for executing game logic.
Mage CLI & Tab Completion
CLI
This module uses commander to manage the command-line interface of MAGE.
Those are the commands you can pass to Mage:
To run those, cd
to the root folder of your game,
and use node . [command]
where [command]
is one of the commands listed below.
show-config [options] [trail]
output the full configuration, or the sub-config at the given trail in JSONarchivist-create [vaults]
create database environments for all configured vaultsarchivist-drop [vaults]
destroy database environments for all configured vaults (use with caution!)archivist-migrate [version]
migrates all vaults to the current version, or to the version requestedstart
start the application daemonizedstop
stop the daemonized applicationrestart
restart the daemonized applicationreload
recycle all workers with zero-downtime (not to be used on version changes)status
output the status of the daemonized application
Tab Completion
How to install
Recommended way:
make dev
Which will setup the githooks and the tab completion.
You can also:
make dev-autocomplete
Which will only setup the tab completion.
Implementation Details
The current implementation of tab completion uses Tabalot.
Tabalot pretty much works this way:
node . completion -- word
Provides tabcompletion for any command starting withword
Because of a few tweaks we made to simplify the installation process, you can use:
node . completion --filename test_file --bash_completion bash_completion_file --save
Which will:- read information from your package.json to get the necessary information to create a tab completion script
- save this tab completion script in
test_file
- add a line to
bash_completion_file
(usually~/.bash_completion
), sourcingtest_file
Typing make dev-autocomplete at the root folder of your Mage project will basically do this for you.
You will then have to source
your bash_completion
(usually ~/.bash_completion
) file
or restart your shell to benefit from the tab completion.
Every command added to the program object in lib/cli/index.js
will benefit from tab completion.
API
cli.run()
Parse the process arguments obtained via process.argv
.
cli.program
Commander object which allows you to extend the provided CLI.
Example:
var cli = require('mage').cli;
cli.program.option('--clown', 'Enables clown mode');
cli.run();
With the previous code, you should obtain the following: “` $ ./game –verbose –help
Usage: game [options] [command] … Options: … –clown Enables clown mode ”`