Getting Started¶
Installation¶
Monstache is just a single binary without dependencies on runtimes like Ruby, Python or PHP.
If you simply want to run Monstache you just need to download the latest version.
Which version should I use?¶
| Monstache version | Git branch (used to build plugin) | Docker tag | Description | Elasticsearch | MongoDB | Status |
|---|---|---|---|---|---|---|
| 6 | rel6 | rel6, latest | MongoDB, Inc. go driver | Version 7+ | Version 2.6+ | Supported |
| 5 | rel5 | rel5 | MongoDB, Inc. go driver | Version 6 | Version 2.6+ | Supported |
| 4 | master | rel4 | mgo community go driver | Version 6 | Version 3 | Deprecated |
| 3 | rel3 | rel3 | mgo community go driver | Versions 2 and 5 | Version 3 | Deprecated |
Unzip the download and adjust your PATH variable to include the path to the folder for your platform.
Let's make sure Monstache is set up as expected. You should see a similar version number in your terminal:
monstache -v
# 6.7.4
You can also build monstache from source. Monstache uses vgo.
cd ~/build # somewhere outside your $GOPATH
git clone https://github.com/rwynn/monstache.git
cd monstache
git checkout <branch-or-tag-to-build>
go install
# monstache binary should now be in $GOPATH/bin
Usage¶
Monstache uses the MongoDB oplog as an event source. You will need to ensure that MongoDB is configured to produce an oplog by deploying a replica set. If you haven't already done so, follow the 5 step procedure to initiate and validate your replica set. For local testing your replica set may contain a single member.
Note
If you have enabled security in MongoDB you will need to give the user in your connection string certain privileges:
For MongoDB versions prior to 3.6 the user in the connection string will need to be able read the local
database (to read from the oplog) and any user databases that you wish to synch data from.
When using the resume or clustering features the user will need to be able to write to and
create indexes for the monstache
database, or more generally, whatever you configure the
option config-database-name
to be.
When using change streams you will need to implement the changes in the documentation for access control. Monstache defaults to opening the change stream against the entire deployment.
Without any explicit configuration monstache will connect to Elasticsearch and MongoDB on localhost on the default ports and begin tailing the MongoDB oplog. Any changes to MongoDB while Monstache is running will be reflected in Elasticsearch.
To see the indexes created by Monstache you may want to issue the following command which will show the indices in Elasticsearch. By default, the index names will match the db.collection name in MongoDB.
curl localhost:9200/_cat/indices?v
Monstache uses the TOML format for its configuration. You can run monstache with an explicit configuration by passing the -f flag.
monstache -f /path/to/config.toml
The following shows how to specify options in a TOML config file.
Note
It is highly recommended that you start with only your MongoDB and Elasticsearch connection settings and only specify additional options as needed.
# connection settings
# connect to MongoDB using the following URL
mongo-url = "mongodb://someuser:password@localhost:40001"
# connect to the Elasticsearch REST API at the following node URLs
elasticsearch-urls = ["https://es1:9200", "https://es2:9200"]
# frequently required settings
# if you need to seed an index from a collection and not just listen and sync changes events
# you can copy entire collections or views from MongoDB to Elasticsearch
direct-read-namespaces = ["mydb.mycollection", "db.collection", "test.test", "db2.myview"]
# if you want to use MongoDB change streams instead of legacy oplog tailing use change-stream-namespaces
# change streams require at least MongoDB API 3.6+
# if you have MongoDB 4+ you can listen for changes to an entire database or entire deployment
# in this case you usually don't need regexes in your config to filter collections unless you target the deployment.
# to listen to an entire db use only the database name. For a deployment use an empty string.
change-stream-namespaces = ["mydb.mycollection", "db.collection", "test.test"]
# additional settings
# if you don't want to listen for changes to all collections in MongoDB but only a few
# e.g. only listen for inserts, updates, deletes, and drops from mydb.mycollection
# this setting does not initiate a copy, it is only a filter on the change event listener
namespace-regex = '^mydb\.mycollection$'
# compress requests to Elasticsearch
gzip = true
# generate indexing statistics
stats = true
# index statistics into Elasticsearch
index-stats = true
# use the following user name for Elasticsearch basic auth
elasticsearch-user = "someuser"
# use the following password for Elasticsearch basic auth
elasticsearch-password = "somepassword"
# use 4 go routines concurrently pushing documents to Elasticsearch
elasticsearch-max-conns = 4
# use the following PEM file to connections to Elasticsearch
elasticsearch-pem-file = "/path/to/elasticCert.pem"
# validate connections to Elasticsearch
elastic-validate-pem-file = true
# propogate dropped collections in MongoDB as index deletes in Elasticsearch
dropped-collections = true
# propogate dropped databases in MongoDB as index deletes in Elasticsearch
dropped-databases = true
# do not start processing at the beginning of the MongoDB oplog
# if you set the replay to true you may see version conflict messages
# in the log if you had synced previously. This just means that you are replaying old docs which are already
# in Elasticsearch with a newer version. Elasticsearch is preventing the old docs from overwriting new ones.
replay = false
# resume processing from a timestamp saved in a previous run
resume = true
# do not validate that progress timestamps have been saved
resume-write-unsafe = false
# override the name under which resume state is saved
resume-name = "default"
# use a custom resume strategy (tokens) instead of the default strategy (timestamps)
# tokens work with MongoDB API 3.6+ while timestamps work only with MongoDB API 4.0+
resume-strategy = 1
# exclude documents whose namespace matches the following pattern
namespace-exclude-regex = '^mydb\.ignorecollection$'
# turn on indexing of GridFS file content
index-files = true
# turn on search result highlighting of GridFS content
file-highlighting = true
# index GridFS files inserted into the following collections
file-namespaces = ["users.fs.files"]
# print detailed information including request traces
verbose = true
# enable clustering mode
cluster-name = 'apollo'
# do not exit after full-sync, rather continue tailing the oplog
exit-after-direct-reads = false
See Configuration for details about each configuration option.


{% extends "base.html" %} {% load staticfiles %} {% block title %}Tutorial{% endblock title %} {% block content %}
This tutorial will walk you through the following steps
A user account is required to save projects to the server and post comments. An account is not required to access the model or view other user's saved projects.
Account registration is free and takes less than 5 minutes. Simply visit the registration page, and fill out the form with your name, email and password. Your password will be encrypted and your email will not be vieweble by any other users other than the website administration.
After submitting the form, you will soon receive an activation email in your inbox containing a link to activate your account. Click the link and you will see a confirmation that your account has been activated.
Once you have activated your account, you can log in with your chosen username and password at the login page. If you have forgotten your password, click the link to reset it.
After logging in, you will be redirected to the WIRM homepage and you should see your log-in status and link to log out in the upper right corner.
To log out, simple click the logout button in the upper right corner of the navigation bar.
To open the model application, either
Click Model
> New Project
on the navigation bar
Important: if you plan to save a project to the server you will need to login first. If you login after setting up a new project, you will lose any changes to the parameter values.
Once the application is loaded, the user is presented with the New Project interface. A Project is an application of the model for a specific river and under specific conditions. Each New Project is based on the default parameter values. The user needs to adjust each of these parameters based on the characteristics of the river of interest.
The model parameters are grouped into two tabs:
Changes to the parameters values can be undone by clicking the Reset
button, which resets the parameters to the default values.
Once the parameter values have been adjusted to match a specific river, the user can save the project to the server by clicking the Save
button.
The user must then fill out the new project form by setting the Title
, Location
and Description
of the project, and clicking the Save
button.
After the project is saved to the server, the browser will automatically be redirected to the URL assigned to the new project.
A saved proejct can be loaded by either:
Click Model
> Saved Projects
on the navigation bar
The Saved Projects page will list all existing projects that the user has saved to the server. A project can be loaded by clicking its title, or the Load
button. Projects can also be deleted by clicking the Delete
button. Note that this list only shows the projects created by the current user, and does not include projects created by other users.
Existing project can only be modified by the user who created them (i.e. the "owner"). The owner can edit the project information by clicking the Edit Project Info
button, delete the project by clicking the Delete Project
button, or save changes to the parameter values by clicking the Save
button. Other users may change the parameter values to investigate the parameter sensitivity or effect of changing initial discharge concentration, but they cannot save changes to the server.
Changes to the parameters values can be undone by clicking the Reset
button, which resets the parameter values to the values that were last saved to the server.
Comments can be added to any saved project by clicking on the Comments
tab. Any registered user can leave comments on any project.
Once a project has been saved to the server, it can be easily shared with other users by clicking the Share
button.
The Share Project window provides a unique URL to the current project, which can be copied to the clipboard and pasted into an email or document. Other users can simple copy and paste this URL into their browser and immediately load the saved project. Note that all projects are publically accessible and thus other users do not need to register or log in to WIRM in order to view exisitng projects.