A Basic introduction to Mongo DB
Mongo DB has rapidly grown to become a popular database for web applications and is a perfect fit for Node.JS applications, letting you write Javascript for the client, backend and database layer. Its schemaless nature is a better match to our constantly evolving data structures in web applications, and the integrated support for location queries is a bonus that’s hard to ignore. Throw in Replica Sets for scaling, and we’re looking at really nice platform to grow your storage needs now and in the future.
Now to shamelessly plug my driver. It can be downloaded via npm, or fetched from the github repository. To install via npm, do the following:
npm install mongodb
or go fetch it from github at https://github.com/christkv/node-mongodb-native
Once this business is taken care of, let’s move through the types available for the driver and then how to connect to your Mongo DB instance before facing the usage of some CRUD operations.
Mongo DB data types
So there is an important thing to keep in mind when working with Mongo DB, and that is the slight mapping difference between types Mongo DB supports and native Javascript data types. Let’s have a look at the types supported out of the box and then how types are promoted by the driver to fit as close to native Javascript types as possible.
- Float is a 8 byte and is directly convertible to the Javascript type Number
- Double class a special class representing a float value, this is especially useful when using capped collections where you need to ensure your values are always floats.
- Integers is a bit trickier due to the fact that Javascript represents all Numbers as 64 bit floats meaning that the maximum integer value is at a 53 bit. Mongo has two types for integers, a 32 bit and a 64 bit. The driver will try to fit the value into 32 bits if it can and promote it to 64 bits if it has to. Similarly it will deserialize attempting to fit it into 53 bits if it can. If it cannot it will return an instance of Long to avoid loosing precession.
- Long class a special class that let’s you store 64 bit integers and also let’s you operate on the 64 bits integers.
- Date maps directly to a Javascript Date
- RegExp maps directly to a Javascript RegExp
- String maps directly to a Javascript String (encoded in utf8)
- Binary class a special class that let’s you store data in Mongo DB
- Code class a special class that let’s you store javascript functions in Mongo DB, can also provide a scope to run the method in
- ObjectID class a special class that holds a MongoDB document identifier (the equivalent to a Primary key)
- DbRef class a special class that let’s you include a reference in a document pointing to another object
- Symbol class a special class that let’s you specify a symbol, not really relevant for javascript but for languages that supports the concept of symbols.
As we see the number type can be a little tricky due to the way integers are implemented in Javascript. The latest driver will do correct conversion up to 53 bit’s of complexity. If you need to handle big integers the recommendation is to use the Long class to operate on the numbers.
Getting that connection to the database
Let’s get around to setting up a connection with the Mongo DB database. Jumping straight into the code let’s do direct connection and then look at the code.
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(!err) {
console.log("We are connected");
}
});
Let’s have a quick look at how the connection code works. The Db.connect method let’s use use a uri to connect to the Mongo database, where localhost:27017 is the server host and port and exampleDb the db we wish to connect to. After the url notice the hash containing the auto_reconnect key. Auto reconnect tells the driver to retry sending a command to the server if there is a failure during it’s execution.
Another useful option you can pass in is
poolSize, this allows you to control how many tcp connections are opened in parallel. The default value for this is 5 but you can set it as high as you want. The driver will use a round-robin strategy to dispatch and read from the tcp connection.
We are up and running with a connection to the database. Let’s move on and look at what collections are and how they work.
Mongo DB and Collections
Collections are the equivalent of tables in traditional databases and contain all your documents. A database can have many collections. So how do we go about defining and using collections. Well there are a couple of methods that we can use. Let’s jump straight into code and then look at the code.
the requires and and other initializing stuff omitted for brevity
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
db.collection('test', function(err, collection) {});
db.collection('test', {w:1}, function(err, collection) {});
db.createCollection('test', function(err, collection) {});
db.createCollection('test', {w:1}, function(err, collection) {});
});
Three different ways of creating a collection object but slightly different in behavior. Let’s go through them and see what they do
db.collection('test', function(err, collection) {});
This function will not actually create a collection on the database until you actually insert the first document.
db.collection('test', {strict:true}, function(err, collection) {});
Notice the {strict:true} option. This option will make the driver check if the collection exists and issue an error if it does not.
db.createCollection('test', function(err, collection) {});
This command will create the collection on the Mongo DB database before returning the collection object. If the collection already exists it will ignore the creation of the collection.
db.createCollection('test', {strict:true}, function(err, collection) {});
The {strict:true} option will make the method return an error if the collection already exists.
With an open db connection and a collection defined we are ready to do some CRUD operation on the data.
 The Mongo DB wire protocol is built around 4 main operations insert/update/remove/query. Most operations on the database are actually queries with special json objects defining the operation on the database. But I’m getting ahead of myself. Let’s go back and look at insert first and do it with some code.
the requires and and other initializing stuff omitted for brevity
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
var collection = db.collection('test');
var doc1 = {'hello':'doc1'};
var doc2 = {'hello':'doc2'};
var lotsOfDocs = [{'hello':'doc3'}, {'hello':'doc4'}];
collection.insert(doc1);
collection.insert(doc2, {w:1}, function(err, result) {});
collection.insert(lotsOfDocs, {w:1}, function(err, result) {});
});
A couple of variations on the theme of inserting a document as we can see. To understand why it’s important to understand how Mongo DB works during inserts of documents.
Mongo DB has asynchronous insert/update/remove operations. This means that when you issue an insert operation its a fire and forget operation where the database does not reply with the status of the insert operation. To retrieve the status of the operation you have to issue a query to retrieve the last error status of the connection. To make it simpler to the developer the driver implements the {w:1} options so that this is done automatically when inserting the document. {w:1} becomes especially important when you do update or remove as otherwise it’s not possible to determine the amount of documents modified or removed.
Now let’s go through the different types of inserts shown in the code above.
collection.insert(doc1);
Taking advantage of the async behavior and not needing confirmation about the persisting of the data to Mongo DB we just fire off the insert (we are doing live analytics, loosing a couple of records does not matter).
collection.insert(doc2, {w:1}, function(err, result) {});
That document needs to stick. Using the {w:1} option ensure you get the error back if the document fails to insert correctly.
collection.insert(lotsOfDocs, {w:1}, function(err, result) {});
A batch insert of document with any errors being reported. This is much more efficient if you need to insert large batches of documents as you incur a lot less overhead.
Right that’s the basics of insert’s ironed out. We got some documents in there but want to update them as we need to change the content of a field. Let’s have a look at a simple example and then we will dive into how Mongo DB updates work and how to do them efficiently.
the requires and and other initializing stuff omitted for brevity
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
var collection = db.collection('test');
var doc = {mykey:1, fieldtoupdate:1};
collection.insert(doc, {w:1}, function(err, result) {
collection.update({mykey:1}, {$set:{fieldtoupdate:2}}, {w:1}, function(err, result) {});
});
var doc2 = {mykey:2, docs:[{doc1:1}]};
collection.insert(doc2, {w:1}, function(err, result) {
collection.update({mykey:2}, {$push:{docs:{doc2:1}}}, {w:1}, function(err, result) {});
});
});
Alright before we look at the code we want to understand how document updates work and how to do the efficiently. The most basic and less efficient way is to replace the whole document, this is not really the way to go if you want to change just a field in your document. Luckily Mongo DB provides a whole set of operations that let you modify just pieces of the document Atomic operations documentation. Basically outlined below.
- $inc - increment a particular value by a certain amount
- $set - set a particular value
- $unset - delete a particular field (v1.3+)
- $push - append a value to an array
- $pushAll - append several values to an array
- $addToSet - adds value to the array only if its not in the array already
- $pop - removes the last element in an array
- $pull - remove a value(s) from an existing array
- $pullAll - remove several value(s) from an existing array
- $rename - renames the field
- $bit - bitwise operations
Now that the operations are outline let’s dig into the specific cases show in the code example.
collection.update({mykey:1}, {$set:{fieldtoupdate:2}}, {w:1}, function(err, result) {});
Right so this update will look for the document that has a field mykey equal to 1 and apply an update to the field fieldtoupdate setting the value to 2. Since we are using the {w:1} option the result parameter in the callback will return the value 1 indicating that 1 document was modified by the update statement.
collection.update({mykey:2}, {$push:{docs:{doc2:1}}}, {w:1}, function(err, result) {});
This updates adds another document to the field docs in the document identified by {mykey:2} using the atomic operation $push. This allows you to modify keep such structures as queues in Mongo DB.
Let’s have a look at the remove operation for the driver. As before let’s start with a piece of code.
the requires and and other initializing stuff omitted for brevity
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
var collection = db.collection('test');
var docs = [{mykey:1}, {mykey:2}, {mykey:3}];
collection.insert(docs, {w:1}, function(err, result) {
collection.remove({mykey:1});
collection.remove({mykey:2}, {w:1}, function(err, result) {});
collection.remove();
});
});
Let’s examine the 3 remove variants and what they do.
collection.remove({mykey:1});
This leverages the fact that Mongo DB is asynchronous and that it does not return a result for insert/update/remove to allow for synchronous style execution. This particular remove query will remove the document where mykey equals 1.
collection.remove({mykey:2}, {w:1}, function(err, result) {});
This remove statement removes the document where mykey equals 2 but since we are using {w:1} it will back to Mongo DB to get the status of the remove operation and return the number of documents removed in the result variable.
collection.remove();
This last one will remove all documents in the collection.
Time to Query
Queries is of course a fundamental part of interacting with a database and Mongo DB is no exception. Fortunately for us it has a rich query interface with cursors and close to SQL concepts for slicing and dicing your datasets. To build queries we have lots of operators to choose from Mongo DB advanced queries. There are literarily tons of ways to search and ways to limit the query. Let’s look at some simple code for dealing with queries in different ways.
the requires and and other initializing stuff omitted for brevity
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
var collection = db.collection('test');
var docs = [{mykey:1}, {mykey:2}, {mykey:3}];
collection.insert(docs, {w:1}, function(err, result) {
collection.find().toArray(function(err, items) {});
var stream = collection.find({mykey:{$ne:2}}).stream();
stream.on("data", function(item) {});
stream.on("end", function() {});
collection.findOne({mykey:1}, function(err, item) {});
});
});
Before we start picking apart the code there is one thing that needs to be understood, the find method does not execute the actual query. It builds an instance of Cursor that you then use to retrieve the data. This lets you manage how you retrieve the data from Mongo DB and keeps state about your current Cursor state on Mongo DB. Now let’s pick apart the queries we have here and look at what they do.
collection.find().toArray(function(err, items) {});
This query will fetch all the document in the collection and return them as an array of items. Be careful with the function toArray as it might cause a lot of memory usage as it will instantiate all the document into memory before returning the final array of items. If you have a big resultset you could run into memory issues.
var stream = collection.find({mykey:{$ne:2}}).stream();
stream.on("data", function(item) {});
stream.on("end", function() {});
This is the preferred way if you have to retrieve a lot of data for streaming, as data is deserialized a data event is emitted. This keeps the resident memory usage low as the documents are streamed to you. Very useful if you are pushing documents out via websockets or some other streaming socket protocol. Once there is no more document the driver will emit the end event to notify the application that it’s done.
collection.findOne({mykey:1}, function(err, item) {});
This is special supported function to retrieve just one specific document bypassing the need for a cursor object.
That’s pretty much it for the quick intro on how to use the database. I have also included a list of links to where to go to find more information and also a sample crude location application I wrote using express JS and mongo DB.


 Charges and interest rates apply.
Min-max deposit: 10-10,000 (in USD, GBP, or EUR) Processing time: Instant
|
Fund now
|
|
|
|
|
Paysafecard
A convenient payment service that doesn’t need a bank account or credit card. Accepted by thousands of merchants worldwide.
Min-max deposit: 5-1,000 (in USD, GBP, or EUR) Processing time: Instant
|
Fund now
|
|
|
|
|
View all payment options
|
Fund your account in 3 easy steps
|
|
|
|
Once your deposit has been processed, the amount will appear in your [account no.] account balance and you can begin trading.
|
|
Make a deposit
|


Week 2 Wireframe Blog
Since week 1's blog template was done in a hurry, much of the planned structure was simplified and I will be reimplementing the full structure this week as well as adding responsive design.
The side navigation bar on the left with links to previous posts was coded to have fixed size last week and now it will be fluid size with fixed position. Same with the sidebar on the right with external links. Each set of links will also now be coded as unorder lists to clean up the visual formatting for clarity.
The footer that was omitted from last week's design will be added back in the same style as Facebook where it is the last block of the right sidebar.
The header didn't have the text aligned to center last week, so it will be centered now. The text will also be a link back to the index of my site.
Titles were also conspicuously absent from last week's blog template. Other minor changes were made.
There isn't a wireframe graphic for this, but in order for this to be more accessible to smartphones and tablets, the nav sidebar will be eliminated for both tablet and smartphone resolutions and the external links and footer will be moved to below the content area in smartphone view


Home
About
Movies
This Page is just a simulation to a real website where the tool will be used on!
This Page is just a simulation to a real website where the tool will be used on!


Try out our Premium Member services: Virtual Legal Assistant, Query Alert Service and an ad-free experience. Free for one month and pay only if you like it.
IN THE HIGH COURT OF JUDICATURE FOR RAJASTHAN AT JAIPUR BENCH JAIPUR ORDER S.B.CIVIL WRIT PETITION NO.9685/2009 M/s.Rao Jogendra Grit Udhyog Versus Raj. State Pollution Control Board DATE OF ORDER --- August 12,2009 PRESENT HONBLE MR.JUSTICE PREM SHANKER ASOPA Mr.S.K.Shukla,for the petitioner Counsel inter alia submits that the Board alone is empowered u/s 31-A of Air (Prevention and Control of Pollution) Act, 1981, while order impugned dated 21.7.2009 has been passed by the Member Secretary, to whom no power has been delegated by the authority competent under the Act, 1981. Issue notice along with a copy of this order to respondents returnable within four weeks. Notices be given Dasti.
In the meanwhile, operation of the impugned order dated 21.7.2009 shall remain stayed till further orders.
(Prem Shanker Asopa)J.


--- layout: post title: With @jonathannelson 40k feet in the air date: 2010-01-16 13:50:23.000000000 -08:00 type: post published: true status: publish categories: [] tags: [] meta: {} author: ---


home
contact
about
This project took me about 14 hours (8 straight from 8pm to 4am).
Hope you like it and thank you


Name and Surname:
*REQUIRED*.
Events Room:
Room1
Room2
Room3
Room4
Coffe Break:
Cofferom1
Cofferom2
Search
Name:
Room:
Coffe Break:
Actions:
Name:
Room:
Coffe:


Text with different SearchTypes examples
Cancel
Cancel
Cancel
Cancel
-------------
more enter inside div
more enter inside div
more enter inside div
more enter inside div
more enter inside div


Earthquakes —
{{ date ? ( date | date: 'MMM d, yyyy' ) : 'Last 30 days' }}
Earthquake on {{quake.properties.time | date: 'MMM d, h:mm a'}}
Distance from you:
{{nearestDistance | number: 2 }}
miles


Enactus Unical is just one enactus team amongst the various enactus teams in:
1,600 Colleges & Universities in 115 Countries worldwide
We are in the business of Empowering our community
In Enactus Unical, our team consists of A Faculty Officer, Our Alumni, Our Business Advisory board members, and the Students who dedicate their time to empowering the society, while indirectly empowering themselves through the skills and experience they acquire in the process.


| Time |
Services |
Location |
Customer |
| 9:30AM |
Grand - 7 to 9 Baskets Travel: Leave at 8:45AM |
Cyn (Austin, Lakeway, The Hills,
West Lake Hills)
|
Nicole Zuber |
| 9:30AM |
Grand - 7 to 9 Baskets Travel: Leave at 8:45AM |
Cyn (Austin, Lakeway, The Hills,
West Lake Hills)
|
Nicole Zuber |
| 9:30AM |
Grand - 7 to 9 Baskets Travel: Leave at 8:45AM |
Cyn (Austin, Lakeway, The Hills,
West Lake Hills)
|
Nicole Zuber |
| 9:30AM |
Grand - 7 to 9 Baskets Travel: Leave at 8:45AM |
Cyn (Austin, Lakeway, The Hills,
West Lake Hills)
|
Nicole Zuber |
| 9:30AM |
Grand - 7 to 9 Baskets Travel: Leave at 8:45AM |
Cyn (Austin, Lakeway, The Hills,
West Lake Hills)
|
Nicole Zuber |


Gowing up my parents used to own a hobby store.
That being said I always grew up around electronics, soldering, and tinkering.
As weird as it is to say, this rotation really sent me back to my childhood.
Soldering came back to me relatively easily and was just a matter of plugging and chugging throughout the assignments.
The first project we were to make was a strobe light.
This proved to be a easy project and was completed relatively quickly.
The second assignment was to make a heart.
The hearts color was determined by the color led's that came with the kit.
Another easy project, this went relatively well but was just time consuming


Call Us
(+9)<PHONE_NUMBER>
Email Us
<EMAIL_ADDRESS>Global Certificate
ISO 9001:2017
This is a template that is great for small businesses. It doesn't have too much fancy flare to it, but it makes a great use of the standard Bootstrap core components. Feel free to use this template for any project you want!
Call to Action!


Advanced¶
Versions¶
| Monstache version | Git branch (used to build plugin) | Docker tag | Description | Elasticsearch | MongoDB | Status |
|---|---|---|---|---|---|---|
| 6 | rel6 | rel6, latest | MongoDB, Inc. go driver | Version 7+ | Version 2.6+ | Supported |
| 5 | rel5 | rel5 | MongoDB, Inc. go driver | Version 6 | Version 2.6+ | Supported |
| 4 | master | rel4 | mgo community go driver | Version 6 | Version 3 | Deprecated |
| 3 | rel3 | rel3 | mgo community go driver | Versions 2 and 5 | Version 3 | Deprecated |
Note
You can use monstache rel5
and rel6
with MongoDB versions back to 2.6 with the following caveats.
If you have MongoDB 3.6 then you must explicitly enumerate collections in your change-stream-namespaces
setting because change streams against databases and entire deployments was not introduced until MongoDB version
4.0. Alternatively, you can disable change events entirely with disable-change-events
. You must also set
resume-strategy
to 1
to use a token-based resume strategy compatibile with MongoDB API 3.6.
If you have MongoDB 2.6 - 3.5 then you must omit any mention of change-stream-namespaces
in your config file
because change streams were first introduced in 3.6. To emulate change events you must turn on the option
enable-oplog
. Alternatively, you can disable change events entirely with disable-change-events
.
Warning
Your MongoDB binary version does not always mean that the feature compatibility is at that same level. Check your feature compatibility version from the MongoDB console to ensure that MongoDB is not operating in a lesser capability mode. This sometimes happens when MongoDB is upgraded in place or MongoDB is started with a data directory of a previous installation. Sometimes there are reasons to stay at a lower feature compatibility so check before you upgrade it.
GridFS Support¶
Monstache supports indexing the raw content of files stored in GridFS into Elasticsearch for full
text search. This feature requires that you install an Elasticsearch plugin which enables the field type attachment
.
For versions of Elasticsearch prior to version 5 you should install the
mapper-attachments plugin.
For version 5 or later of Elasticsearch you should instead install the
ingest-attachment plugin.
Once you have installed the appropriate plugin for Elasticsearch, getting file content from GridFS into Elasticsearch is as simple as configuring monstache. You will want to enable the index-files option and also tell monstache the namespace of all collections which will hold GridFS files. For example in your TOML config file,
index-files = true
direct-read-namespaces = ["users.fs.files", "posts.fs.files"]
file-namespaces = ["users.fs.files", "posts.fs.files"]
file-highlighting = true
The above configuration tells monstache that you wish to index the raw content of GridFS files in the users
and posts
MongoDB databases. By default, MongoDB uses a bucket named fs
, so if you just use the defaults your collection name will
be fs.files
. However, if you have customized the bucket name, then your file collection would be something like mybucket.files
and the entire namespace would be users.mybucket.files
.
When you configure monstache this way it will perform an additional operation at startup to ensure the destination indexes in
Elasticsearch have a field named file
with a type mapping of attachment
.
For the example TOML configuration above, monstache would initialize 2 indices in preparation for indexing into Elasticsearch by issuing the following REST commands:
For Elasticsearch versions prior to version 5...
POST /users.fs.files
{
"mappings": {
"fs.files": {
"properties": {
"file": { "type": "attachment" }
}}}}
POST /posts.fs.files
{
"mappings": {
"fs.files": {
"properties": {
"file": { "type": "attachment" }
}}}}
For Elasticsearch version 5 and above...
PUT /_ingest/pipeline/attachment
{
"description" : "Extract file information",
"processors" : [
{
"attachment" : {
"field" : "file"
}
}
]
}
When a file is inserted into MongoDB via GridFS, monstache will detect the new file, use the MongoDB api to retrieve the raw
content, and index a document into Elasticsearch with the raw content stored in a file
field as a base64
encoded string. The Elasticsearch plugin will then extract text content from the raw content using
Apache Tika, tokenize the text content, and allow you to query on the content of the file.
To test this feature of monstache you can simply use the mongofiles
command to quickly add a file to MongoDB via GridFS. Continuing the example above one could issue the following command to put a
file named resume.docx
into GridFS and after a short time this file should be searchable in Elasticsearch in the index users.fs.files
.
mongofiles -d users put resume.docx
After a short time you should be able to query the contents of resume.docx in the users index in Elasticsearch
curl -XGET "http://localhost:9200/users.fs.files/_search?q=golang"
If you would like to see the text extracted by Apache Tika you can project the appropriate sub-field
For Elasticsearch versions prior to version 5...
curl -H "Content-Type:application/json" localhost:9200/users.fs.files/_search?pretty -d '{
"fields": [ "file.content" ],
"query": {
"match": {
"file.content": "golang"
}
}
}'
For Elasticsearch version 5 and above...
curl -H "Content-Type:application/json" localhost:9200/users.fs.files/_search?pretty -d '{
"_source": [ "attachment.content" ],
"query": {
"match": {
"attachment.content": "golang"
}
}
}'
When file-highlighting is enabled you can add a highlight clause to your query
For Elasticsearch versions prior to version 5...
curl -H "Content-Type:application/json" localhost:9200/users.fs.files/_search?pretty -d '{
"fields": ["file.content"],
"query": {
"match": {
"file.content": "golang"
}
},
"highlight": {
"fields": {
"file.content": {
}
}
}
}'
For Elasticsearch version 5 and above...
curl -H "Content-Type:application/json" localhost:9200/users.fs.files/_search?pretty -d '{
"_source": ["attachment.content"],
"query": {
"match": {
"attachment.content": "golang"
}
},
"highlight": {
"fields": {
"attachment.content": {
}
}
}
}'
The highlight response will contain emphasis on the matching terms
For Elasticsearch versions prior to version 5...
"hits" : [ {
"highlight" : {
"file.content" : [ "I like to program in golang.\n\n" ]
}
} ]
For Elasticsearch version 5 and above...
"hits" : [{
"highlight" : {
"attachment.content" : [ "I like to program in golang." ]
}
}]
Workers¶
You can run multiple monstache processes and distribute the work between them. First configure the names of all the workers in a shared config.toml file.
workers = ["Tom", "Dick", "Harry"]
In this case we have 3 workers. Now we can start 3 monstache processes and give each one of the worker names.
monstache -f config.toml -worker Tom
monstache -f config.toml -worker Dick
monstache -f config.toml -worker Harry
monstache will hash the id of each document using consistent hashing so that each id is handled by only one of the available workers.
High Availability¶
You can run monstache in high availability mode by starting multiple processes with the same value for cluster-name. Each process will join a cluster which works together to ensure that a monstache process is always syncing to Elasticsearch.
High availability works by ensuring one active process in the monstache.cluster
collection in MongoDB at any given time. Only the process in
this collection will be syncing for the cluster. Processes not present in this collection will be paused. Documents in the
monstache.cluster
collection have a TTL assigned to them. When a document in this collection times out it will be removed from
the collection by MongoDB and another process in the monstache cluster will have a chance to write to the collection and become the
new active process.
When cluster-name is supplied the resume feature is automatically turned on and the resume-name becomes the name of the cluster. This is to ensure that each of the processes is able to pick up syncing where the last one left off.
You can combine the HA feature with the workers feature.
Index Mapping¶
When indexing documents from MongoDB into Elasticsearch the default mapping is as follows:
For Elasticsearch prior to 6.2
Elasticsearch index name <= MongoDB database name . MongoDB collection name
Elasticsearch type <= MongoDB collection name
Elasticsearch document _id <= MongoDB document _id
For Elasticsearch 6.2+
Elasticsearch index name <= MongoDB database name . MongoDB collection name
Elasticsearch type <= _doc
Elasticsearch document _id <= MongoDB document _id
If these default won't work for some reason you can override the index and type mapping on a per collection basis by adding the following to your TOML config file:
[[mapping]]
namespace = "test.test"
index = "index1"
type = "type1"
[[mapping]]
namespace = "test.test2"
index = "index2"
type = "type2"
With the configuration above documents in the test.test
namespace in MongoDB are indexed into the index1
index in Elasticsearch with the type1
type.
If you need your index and type mapping to be more dynamic, such as based on values inside the MongoDB document, then see the sections Middleware and Routing.
Warning
It is not recommended to override the default type of _doc
if using Elasticsearch 6.2+ since this will be the supported path going forward.
Also, using _doc
as the type will not work with Elasticsearch prior to 6.2.
Make sure that automatic index creation is not disabled in elasticsearch.yml or create your target indexes before using Monstache.
If automatic index creation must be controlled, whitelist any indexes in elasticsearch.yml that monstache will create.
Namespaces¶
When a document is inserted, updated, or deleted in MongoDB a document is appended to the oplog representing the event. This document has a field ns
which is the namespace. For inserts, updates, and deletes the namespace is the database name and collection name of the document changed joined by a dot. E.g. for use test; db.foo.insert({hello: "world"});
the namespace for the event in the oplog would be test.foo
.
In addition to inserts, updates, and deletes monstache also supports database and collection drops. When a database or collection is dropped in MongoDB an event is appended to the oplog. Like the other types of changes this event has a field ns
representing the namespace. However, for drops the namespace is the database name and the string $cmd
joined by a dot. E.g. for use test; db.foo.drop()
the namespace for the event in the oplog would be test.$cmd
.
Middleware¶
monstache supports embedding user defined middleware between MongoDB and Elasticsearch. Middleware is able to transform documents, drop documents, or define indexing metadata. Middleware may be written in either Javascript or in Golang as a plugin.
Warning
It is HIGHLY recommended to use a golang plugin in production over a javascript plugin due to performance differences. Currently, golang plugins are orders of magnitude faster than javascript plugins. This is due to concurrency and the need to perform locking on the javascript environment. Javascript plugins are very useful for quickly prototyping a solution, however at some point it is recommended to convert them to golang plugins.
If you enable a Golang plugin then monstache will ignore an javascript middleware in your configuration. This may change in the future but for now the choice of middleware language is mutually exclusive.
Golang¶
monstache supports golang plugins. You should have golang version 1.11 or greater installed and will need to perform the build on the Linux or OSX platform. Golang plugins are not currently supported on the Windows platform due to limits in golang.
To implement a plugin for monstache you need to implement specific function signatures, use the go command to build a .so file for your plugin, and finally pass the path to your plugin .so file when running monstache.
See this wiki page for an example using Docker.
Warning
Golang plugins must be built with the exact same source code (including dependencies) of the loading program.
If you don't build your plugin this way then monstache may fail to load it at runtime due to source code mismatches.
To create a golang plugin for monstache
- git clone
monstache
somewhere outside your $GOPATH - git checkout a specific monstache version tag (e.g.
v6.7.4
). See Versions above. - in the monstache root directory run
go install
to build themonstache
binary. It should now be in $GOPATH/bin - create a .go source file for your plugin in the monstache root directory with the package name
main
- implement one or more of the following functions:
Map
,Filter
,Pipeline
,Process
func Map(input *monstachemap.MapperPluginInput) (output *monstachemap.MapperPluginOutput, err error)
func Filter(input *monstachemap.MapperPluginInput) (keep bool, err error)
func Pipeline(ns string, changeStream bool) (stages []interface, err error)
func Process(input*monstachemap.ProcessPluginInput) error
Compile your plugin to a .so with
go build -buildmode=plugin -o myplugin.so myplugin.go
Run the binary, the one you built above with go install
(not a release binary), with the following arguments
$GOPATH/bin/monstache -mapper-plugin-path /path/to/myplugin.so
The following example plugin simply converts top-level string values to uppercase
package main
import (
"github.com/rwynn/monstache/monstachemap"
"strings"
)
// a plugin to convert document values to uppercase
func Map(input *monstachemap.MapperPluginInput) (output *monstachemap.MapperPluginOutput, err error) {
doc := input.Document
for k, v := range doc {
switch v.(type) {
case string:
doc[k] = strings.ToUpper(v.(string))
}
}
output = &monstachemap.MapperPluginOutput{Document: doc}
return
}
The input parameter will contain information about the document's origin database and collection:
| field | meaning |
|---|---|
Document |
MongoDB document updated or inserted |
UpdateDescription |
If available, the update description |
Namespace |
Operation namespace as described above |
Database |
MongoDB database from where the event came |
Collection |
MongoDB collection where the document was inserted, deleted or updated |
Operation |
Which kind of operation triggered this event, see gtm.mapOperation(). "i" for insert, "u" for update, "d" for delete and "c" for invalidate. The Map function will only receive inserts and updates. To handle deletes or invalidates implement the Process function described below. |
Session |
*mgo.Session. You need not Close the session as monstache will do this automatically when the function exits |
The output parameter will contain information about how the document should be treated by monstache:
| field | meaning |
|---|---|
Document |
an updated document to index into Elasticsearch |
Index |
the name of the index to use |
Type |
the document type |
ID |
override the document ID |
Routing |
the routing value to use |
Drop |
set to true to indicate that the document should not be indexed but removed |
Passthrough |
set to true to indicate the original document should be indexed unchanged |
Parent |
the parent id to use |
Version |
the version of the document |
VersionType |
the version type of the document (internal, external, external_gte) |
Pipeline |
the pipeline to index with |
RetryOnConflict |
how many times to retry updates before failing |
Skip |
set to true to indicate the the document should be ignored |
For detailed information see monstachemap/plugin.go
Few examples are:
To skip the document (direct monstache to ignore it) set output.Skip = true
.
To drop the document (direct monstache not to index it but remove it) set output.Drop = true
.
To simply pass the original document through to Elasticsearch, set output.Passthrough = true
To set custom indexing metadata on the document use output.Index
, output.Type
, output.Parent
and output.Routing
.
Note
If you override output.Index
, output.Type
, output.Parent
or output.Routing
for any MongoDB namespaces in a
golang plugin you should also add those namespaces to the routing-namespaces
array in your config file.
This instructs Monstache to query the document metadata so that deletes of the document work correctly.
If would like to embed other MongoDB documents (possibly from a different collection) within the current document
before indexing, you can access the *mgo.Session
pointer as input.Session
. With the mgo session you can use the mgo API to find documents in MongoDB and embed them in the Document set on output.
When you implement a Filter
function the function is called immediately after reading inserts and updates from the oplog. You can return false from this function to completely ignore a document. This is different than setting output.Drop
from the mapping function because when you set output.Drop
to true, a delete request is issued to Elasticsearch in case the document had previously been indexed. By contrast, returning false from the Filter
function causes the operation to be completely ignored and there is no corresponding delete request issued to Elasticsearch.
When you implement a Pipeline
function the function will be called to setup an aggregation pipeline for both direct reads and any change streams that you have configured. The aggregation pipeline stages that you return may be different depending if applied to a direct read or to a change stream. For direct reads the root document will be the document in the collection. For change streams the root document will be a change event with a fullDocument
field inside it. Use the boolean parameter changeStream
to alter the stages that you return from this function accordingly.
When you implement a Process
function the function will be called after monstache processes each event. This function has full access to the MongoDB and Elasticsearch clients (including the Elasticsearch bulk processor) in the input and allows you to handle complex event processing scenarios. The input parameter for the Process
function will have all the same fields as the input to a Map
function described above plus the following:
| field | meaning |
|---|---|
Document |
MongoDB document updated, inserted, or deleted |
ElasticClient |
A full featured Elasticsearch client |
ElasticBulkProcessor |
The same bulk processor monstache uses to index documents. You need only Add requests to the processor and they will be flushed in bulk automatically. Note you must delete the _id field from any argument to the bulk processor Add function |
Timestamp |
The MongoDB timestamp of the change event from the oplog. In the case of direct reads the timestamp is the time at which the document was read from MongoDB. |
Note
Under the docker/plugin
folder there is a build.sh
script to help you build a plugin. There is a README file in that directory with instructions.
Javascript¶
Monstache supports plugins written in Javascript. You may find that Javascript plugins give you much less performance than golang plugins. You also may reach some limits of what can be done in the Javascript. This is due to the implementation of the Javascript environment and the locking required under high load. Javascript plugins are still very useful for quick prototypes and small data sets.
Transformation¶
Monstache uses the amazing otto library to provide transformation at the document field level in Javascript. You can associate one javascript mapping function per MongoDB collection. You can also associate a function at the global level by not specifying a namespace. These javascript functions are added to your TOML config file, for example:
[[script]]
namespace = "mydb.mycollection"
script = """
var counter = 1;
module.exports = function(doc) {
doc.foo += "test" + counter;
counter++;
return doc;
}
"""
[[script]]
namespace = "anotherdb.anothercollection"
path = "path/to/transform.js"
routing = true
[[script]]
# this script does not declare a namespace
# it is global to all collections
script = """
module.exports = function(doc, ns, updateDesc) {
// the doc namespace e.g. test.test is passed as the 2nd arg
// if available, an object containing the update description is passed as the 3rd arg
return _.omit(doc, "password", "secret");
}
"""
The example TOML above configures 3 scripts. The first is applied to mycollection
in mydb
while the second is applied
to anothercollection
in anotherdb
. The first script is inlined while the second is loaded from a file path. The path can be absolute or relative to the directory monstache is executed from.
The last script does not specify a namespace, so documents from all collections pass through it. Global scripts are run before scripts which are
linked to a specific namespace.
You will notice that the multi-line string feature of TOML is used to assign a javascript snippet to the variable named
script
. The javascript assigned to script must assign a function to the exports property of the module
object. This
function will be passed the document from MongoDB just before it is indexed in Elasticsearch. Inside the function you can
manipulate the document to drop fields, add fields, or augment the existing fields.
The this
reference in the mapping function is assigned to the document from MongoDB.
When the return value from the mapping function is an object
then that mapped object is what actually gets indexed in Elasticsearch.
For these purposes an object is a javascript non-primitive, excluding Function
, Array
, String
, Number
, Boolean
, Date
, Error
and RegExp
.
Filtering¶
You can completely ignore documents by adding filter configurations to your TOML config file. The filter functions are executing immediately after inserts or updates are read from the oplog. The correspding document is passed into the function and you can return true or false to include or ignore the document.
[[filter]]
namespace = "db.collection"
script = """
module.exports = function(doc, ns, updateDesc) {
return !!doc.interesting;
}
"""
[[filter]]
namespace = "db2.collection2"
path = "path/to/script.js"
Aggregation Pipelines¶
You can alter or filter direct reads and change streams by using a pipeline definition. Note, when building a pipeline for a change stream the
root of the document will be the change event and the associated document will be under a field named fullDocument
.
For more information on the properties of the root document for change streams see Change Events.
You can scope a pipeline to a particular namespace using the namespace
attribute or leave it off to have the pipeline applied to all namespaces.
[[pipeline]]
script = """
module.exports = function(ns, changeStream) {
if (changeStream) {
return [
{ $match: {"fullDocument.foo": 1} }
];
} else {
return [
{ $match: {"foo": 1} }
];
}
}
"""
Warning
You should not replace the root using $replaceRoot
for a change stream since monstache needs this information. You should only make
modifications to the fullDocument
field in a pipeline.
Dropping¶
If the return value from the mapping function is not an object
per the definition above then the result is converted into a boolean
and if the boolean value is false
then that indicates to monstache that you would not like to index the document. If the boolean value is true
then
the original document from MongoDB gets indexed in Elasticsearch.
This allows you to return false or null if you have implemented soft deletes in MongoDB.
[[script]]
namespace = "db.collection"
script = """
module.exports = function(doc) {
if (!!doc.deletedAt) {
return false;
}
return true;
}
"""
In the above example monstache will index any document except the ones with a deletedAt
property. If the document is first
inserted without a deletedAt
property, but later updated to include the deletedAt
property then monstache will remove, or drop, the
previously indexed document from the Elasticsearch index.
Note
Dropping a document is different that filtering a document. A filtered document is completely ignored. A dropped document results in a delete request being issued to Elasticsearch in case the document had previously been indexed.
Scripting Features¶
You may have noticed that in the first example above the exported mapping function closes over a var named counter
. You can
use closures to maintain state between invocations of your mapping function.
Finally, since Otto makes it so easy, the venerable Underscore library is included for you at
no extra charge. Feel free to abuse the power of the _
.
Embedding Documents¶
In your javascript function you have access to the following global functions to retreive documents from MongoDB for embedding in the current document before indexing. Using this approach you can pull in related data.
function findId(documentId, [options]) {
// convenience method for findOne({_id: documentId})
// returns 1 document or null
}
function findOne(query, [options]) {
// returns 1 document or null
}
function find(query, [options]) {
// returns an array of documents or null
}
function pipe(stages, [options]) {
// returns an array of documents or null
}
Each function takes a query
type object parameter and an optional options
object parameter.
The options object takes the following keys and values:
var options = {
database: "test",
collection: "test",
// to omit _id set the _id key to 0 in select
select: {
age: 1
},
// only applicable to find...
sort: ["name"],
limit: 2
}
If the database or collection keys are omitted from the options object, the values for database and/or collection are set to the database and collection of the document being processed.
Here are some examples:
This example sorts the documents in the same collection as the document being processed by name and returns the first 2 documents projecting only the age field. The result is set on the current document before being indexed.
[[script]]
namespace = "test.test"
script = """
module.exports = function(doc) {
doc.twoAgesSortedByName = find({}, {
sort: ["name"],
limit: 2,
select: {
age: 1
}
});
return doc;
}
"""
This example grabs a reference id from a document and replaces it with the corresponding document with that id.
[[script]]
namespace = "test.posts"
script = """
module.exports = function(post) {
if (post.author) { // author is a an object id reference
post.author = findId(post.author, {
database: "test",
collection: "users"
});
}
return post;
}
"""
This example runs an aggregation pipeline and stores the results in an extra field in the document
[[script]]
namespace = "test.test"
script = """
module.exports = function(doc, ns) {
doc.extra = pipe([
{ $match: {foo: 1} },
{ $limit: 1 },
{ $project: { _id: 0, foo: 1}}
]
// optional , { database: "foo", collection: "bar"} // defaults to same namespace
);
return doc;
}
"""
Indexing Metadata¶
You can override the indexing metadata for an individual document by setting a special field named
_meta_monstache
on the document you return from your Javascript function.
The _meta_monstache
object supports the following properties.
| prop | meaning |
|---|---|
routing |
the routing value |
index |
the name of the index to use |
type |
the document type |
parent |
the document parent |
version |
the document version |
versionType |
the document version type |
pipeline |
the name of a pipeline to apply to the document |
retryOnConflict |
control how retry works on conflicts |
skip |
set this boolean to true to skip indexing |
id |
override the ID used to index the document |
Assume there is a collection in MongoDB named company
in the test
database.
The documents in this collection look like either
{ "_id": "london", "type": "branch", "name": "London Westminster", "city": "London", "country": "UK" }
or
{ "_id": "alice", "type": "employee", "name": "Alice Smith", "branch": "london" }
Given the above the following snippet sets up a parent-child relationship in Elasticsearch based on the incoming documents from MongoDB and updates the ns (namespace) from test.company to company in Elasticsearch
[[script]]
namespace = "test.company"
routing = true
script = """
module.exports = function(doc, ns) {
// var meta = { type: doc.type, index: 'company' };
var meta = { type: doc.type, index: ns.split(".")[1] };
if (doc.type === "employee") {
meta.parent = doc.branch;
}
doc._meta_monstache = meta;
return _.omit(doc, "branch", "type");
}
"""
The snippet above will route these documents to the company
index in Elasticsearch instead of the
default of test.company
, if you didn't specify a namespace, it'll route all documents to indexes named as the collection only without the database db.collection (MongoDB) => collection (Elasticsearch). Also, instead of using company
as the Elasticsearch type, the type
attribute from the document will be used as the Elasticsearch type. Finally, if the type is
employee then the document will be indexed as a child of the branch the person belongs to.
We can throw away the type and branch information by deleting it from the document before returning
since the type information will be stored in Elasticsearch under _type
and the branch information
will be stored under _parent
.
The example is based on the Elasticsearch docs for parent-child
For more on updating the namespace name, check the Delete Strategy
Routing¶
Routing is the process by which Elasticsearch determines which shard a document will reside in. Monstache supports user defined, or custom, routing of your MongoDB documents into Elasticsearch.
Consider an example where you have a comments
collection in MongoDB which stores a comment and
its associated post identifier.
use blog;
db.comments.insert({title: "Did you read this?", post_id: "123"});
db.comments.insert({title: "Yeah, it's good", post_id: "123"});
In this case monstache will index those 2 documents in an index named blog.comments
under the id
created by MongoDB. When Elasticsearch routes a document to a shard, by default, it does so by hashing
the id of the document. This means that as the number of comments on post 123
grows, each of the comments
will be distributed somewhat evenly between the available shards in the cluster.
Thus, when a query is performed searching among the comments for post 123
Elasticsearch will need to query
all of those shards just in case a comment happened to have been routed there.
We can take advantage of the support in Elasticsearch and in monstache to do some intelligent
routing such that all comments for post 123
reside in the same shard.
First we need to tell monstache that we would like to do custom routing for this collection by setting routing
equal to true on a custom script for the namespace. Then we need to add some metadata to the document telling
monstache how to route the document when indexing. In this case we want to route by the post_id
field.
[[script]]
namespace = "blog.comments"
routing = true
script = """
module.exports = function(doc) {
doc._meta_monstache = { routing: doc.post_id };
return doc;
}
"""
Now when monstache indexes document for the collection blog.comments
it will set the special _routing
attribute
for the document on the index request such that Elasticsearch routes comments based on their corresponding post.
The _meta_monstache
field is used only to inform monstache about routing and is not included in the source
document when indexing to Elasticsearch.
Now when we are searching for comments and we know the post id that the comment belongs to we can include that post id in the request and make a search that normally queries all shards query only 1 shard.
$ curl -H "Content-Type:application/json" -XGET 'http://localhost:9200/blog.comments/_search?routing=123' -d '
{
"query":{
"match_all":{}
}
}'
You will notice in the response that only 1 shard was queried instead of all your shards. Custom routing is great way to reduce broadcast searches and thus get better performance.
The catch with custom routing is that you need to include the routing parameter on all insert, update, and delete operations. Insert and update is not a problem for monstache because the routing information will come from your MongoDB document. Deletes, however, pose a problem for monstache because when a delete occurs in MongoDB the information in the oplog is limited to the id of the document that was deleted. But monstache needs to know where the document was originally routed in order to tell Elasticsearch where to look for it.
Monstache has 3 available strategies for handling deletes in this situation. The default strategy is stateless and uses
a term query into Elasticsearch based on the ID of the document deleted in MongoDB. If the search into Elasticsearch returns
exactly 1 document then monstache will schedule that document for deletion. The 2nd stategy monstache uses is stateful and requires
giving monstache the ability to write to the collection monstache.meta
. In this collection monstache stores information about
documents that were given custom indexing metadata. This stategy slows down indexing and takes up space in MongoDB.
However, it is precise because it records exactly how each document was indexed. The final stategy simply punts on deletes and
leaves document deletion to the user. If you don't generally delete documents in MongoDB or don't care if Elasticsearch contains
documents which have been deleted in MongoDB, this option is available. See Delete Strategy for more information.
For more information see Customizing Document Routing
In addition to letting your customize the shard routing for a specific document, you can also customize the Elasticsearch
index
and type
using a script by putting the custom information in the meta attribute.
[[script]]
namespace = "blog.comments"
routing = true
script = """
module.exports = function(doc) {
if (doc.score >= 100) {
// NOTE: prefix dynamic index with namespace for proper cleanup on drops
doc._meta_monstache = { index: "blog.comments.highscore", type: "highScoreComment", routing: doc.post_id };
} else {
doc._meta_monstache = { routing: doc.post_id };
}
return doc;
}
"""
Joins¶
Elasticsearch 6 introduces an updated approach to parent-child called joins. The following example shows how you can accomplish joins with Monstache. The example is based on the Elasticsearch documentation.
This example assumes Monstache is syncing the test.test
collection in MongoDB with the test.test
index in Elasticsearch.
First we will want to setup an index mapping in Elasticsearch describing the join field.
curl -XPUT 'localhost:9200/test.test?pretty' -H 'Content-Type: application/json' -d'
{
"mappings": {
"_doc": {
"properties": {
"my_join_field": {
"type": "join",
"relations": {
"question": "answer"
}
}
}
}
}
}
'
Warning
The above mapping uses _doc as the Elasticsearch type. _doc is the recommended type for new versions Elasticsearch but it only works with Elasticsearch versions 6.2 and greater. Monstache defaults to using _doc as the type when it detects Elasticsearch version 6.2 or greater. If you are using a previous version of Elasticsearch monstache defaults to using the MongoDB collection name as the Elasticsearch type. The type Monstache uses can be overriden but it is not recommended from Elasticsearch 6.2 on.
Next will will configure Monstache with custom Javascript middleware that does transformation and routing. In a file called CONFIG.toml.
[[script]]
namespace = "test.test"
routing = true
script = """
module.exports = function(doc) {
var routing;
if (doc.type === "question") {
routing = doc._id;
doc.my_join_field = {
name: "question"
}
} else if (doc.type === "answer") {
routing = doc.question;
doc.my_join_field = {
name: "answer",
parent: routing
};
}
if (routing) {
doc._meta_monstache = { routing: routing };
}
return doc;
}
"""
The mapping function adds a my_join_field
field to each document. The contents of the field are based on the type
attribute in the MongoDB
document. Also, the function ensures that the routing is always based on the _id of the question document.
Now with this config in place we can start Monstache. We will use verbose to see the requests.
monstache -verbose -f CONFIG.toml
With Monstache running we are now ready to insert into MongoDB
rs:PRIMARY> use test;
switched to db test
rs:PRIMARY> db.test.insert({type: "question", text: "This is a question"});
rs:PRIMARY> db.test.find()
{ "_id" : ObjectId("5a84a8b826993bde57c12893"), "type" : "question", "text" : "This is a question" }
rs:PRIMARY> db.test.insert({type: "answer", text: "This is an answer", question: ObjectId("5a84a8b826993bde57c12893") });
When we insert these documents we should see Monstache generate the following requests to Elasticsearch
{"index":{"_id":"5a84a8b826993bde57c12893","_index":"test.test","_type":"_doc","routing":"5a84a8b826993bde57c12893","version":6522523668566769665,"version_type":"external"}}
{"my_join_field":{"name":"question"},"text":"This is a question","type":"question"}
{"index":{"_id":"5a84a92b26993bde57c12894","_index":"test.test","_type":"_doc","routing":"5a84a8b826993bde57c12893","version":6522524162488008705,"version_type":"external"}}
{"my_join_field":{"name":"answer","parent":"5a84a8b826993bde57c12893"},"question":"5a84a8b826993bde57c12893","text":"This is an answer","type":"answer"}
This looks good. We should now have a parent/child relationship between these documents in Elasticsearch.
If we do a search on the test.test
index we see the following results:
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [
{
"_index" : "test.test",
"_type" : "_doc",
"_id" : "5a84a8b826993bde57c12893",
"_score" : 1.0,
"_routing" : "5a84a8b826993bde57c12893",
"_source" : {
"my_join_field" : {
"name" : "question"
},
"text" : "This is a question",
"type" : "question"
}
},
{
"_index" : "test.test",
"_type" : "_doc",
"_id" : "5a84a92b26993bde57c12894",
"_score" : 1.0,
"_routing" : "5a84a8b826993bde57c12893",
"_source" : {
"my_join_field" : {
"name" : "answer",
"parent" : "5a84a8b826993bde57c12893"
},
"question" : "5a84a8b826993bde57c12893",
"text" : "This is an answer",
"type" : "answer"
}
}
]
}
To clean up our documents in Elasticsearch a bit we can omit the information that we don't really need in the source docs by
updating our mapping function. This information needs not be at the top-level since it is duplicated in my_join_field
.
return _.omit(doc, "type", "question");
If your parent and child documents are in separate MongoDB collections then you would set up a script for each collection. You can tell if the doc is a parent or child by the collection it comes from. The only other difference would be that you would need to override the index dynamically in addition to the routing such that documents from both MongoDB collections target the same index.
doc._meta_monstache = { routing: routing, index: "parentsAndChildren" };
Warning
You must be careful when you route 2 or more MongoDB collections to the same Elasticsearch index that the document _ids across the MongoDB collections do not collide for any 2 docs because they will be used as the _id in the target index.
Time Machines¶
If you are not just interested in what the current value of a document in MongoDB is, but also would like to see how it has changed over time use time machine namespaces. For example, you've inserted and later updated a document with id 123 in the test.test
collection in MongoDB. If test.test
is a time machine namespace you will have 2 documents representing those changes in the log.test.test.2018-02-20
index (timestamp will change) in Elasticsearch. If you later want all the changes made to that document in MongoDB you can issue a query like this:
$ curl -XGET 'http://localhost:9200/log.test.test.*/_search?routing=123' -d '
{
"query":{
"sort" : [
{ "_oplog_ts" : {"order" : "desc"}}
],
"filtered":{
"query":{
"match_all":{}
},
"filter":{
"term":{
"_source_id":"123"
}
}
}
}
}'
That query will be very efficient because it only queries the shard that all the change docs went to for MongoDB document id 123. It filters the documents on that shard by _source_id
, or id from MongoDB, to only give us the changes to that document. Finally, it sorts by the _oplog_ts
which gives us the most recent change docs first.
The index pattern in the query is a wildcard to pick up all the timestamped indexes that we've acculated for the test.test
namespace.
Merge Patches¶
A unique feature of monstache is support for JSON Merge Patches rfc-7396.
If merge patches are enabled monstache will add an additional field to documents indexed into Elasticsearch. The
name of this field is configurable but it defaults to json-merge-patches
.
Consider the following example with merge patches enabled...
db.test.insert({name: "Joe", age: 16, friends: [1, 2, 3]})
At this point you would have the following document source in Elasticsearch.
"_source" : {
"age" : 16,
"friends" : [
1,
2,
3
],
"json-merge-patches" : [
{
"p" : "{\"age\":16,\"friends\":[1,2,3],\"name\":\"Joe\"}",
"ts" :<PHONE_NUMBER>,
"v" : 1
}
],
"name" : "Joe"
}
As you can see we have a single timestamped merge patch in the json-merge-patches array.
Now let's update the document to remove a friend and update the age.
db.test.update({name: "Joe"}, {$set: {age: 21, friends: [1, 3]}})
If we now look at the document in Elasticsearch we see the following:
"_source" : {
"age" : 21,
"friends" : [
1,
3
],
"json-merge-patches" : [
{
"p" : "{\"age\":16,\"friends\":[1,2,3],\"name\":\"Joe\"}",
"ts" :<PHONE_NUMBER>,
"v" : 1
},
{
"p" : "{\"age\":21,\"friends\":[1,3]}",
"ts" :<PHONE_NUMBER>,
"v" : 2
}
],
"name" : "Joe"
}
You can see that the document was updated as expected and an additional merge patch was added.
Each time the document is updated in MongoDB the corresponding document in Elasticsearch gains a timestamped merge patch. Using this information we can time travel is the document's history.
There is a merge patch for each version of the document. To recreate a specific version we simply need to apply the merge patches in order up to the version that we want.
To get version 1 of the document above we start with {} and apply the 1st merge patch.
To get version 2 of the document above we start with {}
- apply the 1st merge patch to get v1
- apply the 2nd merge patch to v1 to get v2
The timestamps associated with these merge patches are in seconds since the epoch, taken from the timestamp recorded in the oplog when the insert or update occured.
To enable the merge patches feature in monstache you need to add the following to you TOML config:
enable-patches = true
patch-namespaces = ["test.test"]
You need you add each namespace that you would like to see patches for in the patch-namespaces array.
Optionally, you can change the key under which the patches are stored in the source document as follows:
merge-patch-attribute = "custom-merge-attr"
Merge patches will only be recorded for data read from the MongoDB oplog. Data read using the direct read feature will not be enhanced with merge patches.
Most likely, you will want to turn off indexing for the merge patch attribute. You can do this by creating an index template for each patch namespace before running monstache...
PUT /_template/test.test
{
"template" : "test.test",
"mappings" : {
"test" : {
"json-merge-patches" : { "index" : false }
}
}
}
Systemd¶
Monstache has support built in for integrating with systemd. The following monstache.service
is an example
systemd configuration.
[Unit]
Description=monstache sync service
[Service]
Type=notify
ExecStart=/usr/local/bin/monstache -f /etc/monstache/config.toml
WatchdogSec=30s
Restart=always
[Install]
WantedBy=multi-user.target
Systemd unit files are normally saved to /lib/systemd/system
. Verify same with your OS documentation.
After saving the monstache.service file you can run systemctl daemon-reload
to tell systemd to reload
all unit files.
You can enable the service to start on boot with systemctl enable monstache.service
and start the service with systemctl start monstache.service
.
With the configuration above monstache will notify systemd when it has started successfully and then notify systemd repeatedly at half the WatchDog interval to signal liveness. The configuration above causes systemd to restart monstache if it does not start or respond within the WatchdDog interval.
Docker¶
There are Docker images available for Monstache on Docker Hub
You can pull and run the latest images with
docker run rwynn/monstache:rel6 -v
docker run rwynn/monstache:rel5 -v
You can pull and run release images with
docker run rwynn/monstache:6.7.4 -v
docker run rwynn/monstache:5.7.4 -v
For example, to run monstache via Docker with a golang plugin that resides at ~/plugin/plugin.so
on the host you can use a bind mount
docker run --rm --net=host -v ~/plugin:/tmp/plugin rwynn/monstache:6.7.4 -mapper-plugin-path /tmp/plugin/plugin.so
HTTP Server¶
Monstache has a built in HTTP server that you can enable with --enable-http-server. It listens on :8080 by default but you can change this with --http-server-addr.
When using monstache with kubernetes this server can be used to detect liveness and act accordingly
The following GET endpoints are available
/started¶
Returns the uptime of the server
/healthz¶
Returns at 200 status code with the text "ok" when monstache is running
/stats¶
Returns the current indexing statistics in JSON format. Only available if stats are enabled
/instance¶
Returns information about the running monstache process including whether or not it is currently enabled (a cluster will have one enabled process) and the most recent change event timestamp read from MongoDB.
/debug (if pprof is enabled)¶
If the pprof setting is enabled the following endpoints are also made available:
- /debug/pprof/
- /debug/pprof/cmdline
- /debug/pprof/profile
- /debug/pprof/symbol
- /debug/pprof/trace
MongoDB Authentication¶
Check the following link for all available options that you can specify in the MongoDB connection string related to authentication.
For more information on how the MongoDB driver processes authentication configuration see the driver docs.
AWS Signature Version 4¶
Monstache has included AWS Signature Version 4 request signing. To enable the AWS Signature Version 4 support add the following to your config file:
elasticsearch-urls = ["https://<endpoint_from_aws_overview_screen>:443"]
[aws-connect]
access-key = "XXX"
secret-key = "YYY"
region = "ZZZ"
See the docs for aws-connect for the different stategies available for configuring a credential provider.
Notice how the elasticsearch-url
references the port number 443
in the connection string. This is because AWS makes
your cluster available on the standard https port and not the default Elasticsearch port of 9200
. If you have connection
problems make sure you are using the correct port. You cannot omit the port because the driver will default to 9200
if a
port is not specified.
You can read more about Signature Version 4 and Amazon Elasticsearch Service.
For information on how to obtain the access-key
and secret-key
required to connect you can read this
blog post.
In short, you will need to create or use an existing IAM user in your AWS account. You will then need to give this user
access to your Elasticsearch domain. The access-key
and secret-key
you put in your configuration file are those associated
with the IAM user.
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account-id>:user/<iam-user-name>"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:<account-id>:domain/<elasticsearch-domain-name>/*"
}
Watching changes on specific fields only¶
If you are using MongoDB 3.6+ you can use a change stream pipeline to only listen for change events on specific fields.
For example, if you wanted to listen for create
, delete
, and update
events on the namespace test.test
, but you only
wanted to sync changes when the foo
or bar
field changed on the doc, you could use the following configuration.
If, for example, a field named count
changed on the document, then this change would be ignored by monstache.
change-stream-namespaces = ["test.test"]
[[pipeline]]
namespace = "test.test"
script = """
module.exports = function(ns, changeStream) {
if (changeStream) {
return [
{
$match: {
$or: [
{ "updateDescription": {$exists: false} },
{ "updateDescription.updatedFields.foo": {$exists: true}},
{ "updateDescription.updatedFields.bar": {$exists: true}}
]
}
}
];
} else {
return [];
}
}
"""
To build complicated change stream pipelines see Change Events for information on the structure of change events. This information will shape your pipeline.
MongoDB view replication¶
You may have a situation where you want to replicate a MongoDB view in Elasticsearch.
Or you have a collection that should trigger sync of another collection. You can use the relate
config to do this.
Consider you have a collections thing
and state
. A thing has an associated state and a thing is linked to a state via
a field s
which points to the _id
of the associated state in the state
collection.
You can create a view in MongoDB that uses a $lookup
to pull the state information in and present a view of things with the state
information included.
use thingdb;
db.createView("thingview", "thing", [ {$lookup: {from: "state", localField: "s", foreignField: "_id", as: "s"}}])
Given this view you can use the following config to keep things up to date in a things
index in Elasticsearch.
direct-read-namespaces = ["thingdb.thingview"] # read direct from the view of the collection to seed index
change-stream-namespaces = ["thingdb.thing", "thingdb.state"] # change events happen on the underlying collections not views
[[mapping]]
namespace = "thingdb.thing" # map change events on the thing collection to the things index
index = "things"
[[mapping]]
namespace = "thingdb.thingview" # map direct reads of the thingview to the same things index
index = "things"
[[relate]]
namespace = "thingdb.thing" # when a thing changes look it up in the assoicated view by _id and index that
with-namespace = "thingdb.thingview"
keep-src = false # ignore the original thing that changed and instead just use the lookup of that thing via the view
[[relate]]
namespace = "thingdb.state" # when a state changes trigger a thing change event since thing is associated to a state
with-namespace = "thingdb.thing"
src-field = "_id" # use the _id field of the state that changed to lookup associated things
match-field = "s" # only trigger change events for the things where thing.s (match-field) = state._id (src-field).
keep-src = false
Warning
Be careful of the expense of using relate
with a view. In the example above, if there were many things
associated to a single state then a change to that state would trigger n+1 queries to MongoDB when n is the
number of things related to the state. 1 query would be used to find all associated things and n queries would be
used to lookup each thing in the view.
Amazon DocumentDB (with MongoDB compatibility)¶
Monstache support for Amazon DocumentDB is currently experimental. Support for the change streams API in MongoDB was recently added to Amazon DocumentDB. Consult the DocumentDB documentation for instructions on enabling change streams for your collections.
Since Amazon DocumentDB only supports compatibility with MongoDB API 3.6 you will want to ensure that your change stream configuration targets collections and that your resume strategy is set to use tokens and not the default of timestamps.
Ensure that your MongoDB connection URI is set with a primary read preference: e.g. ?readPreference=primary
.
# ensure you target collections in your change stream namespaces
change-stream-namespaces = ["db1.col1", "db2.col2"]
# ensure that resuming, if enabled, is done based on tokens and not timestamps
resume = true
resume-strategy = 1


Average Home Price and Count of Home Sales in the Bay Area (2015 - 2020)
Bar - Average Price of Homes (2015 - 2020)
Bar - Count of Home Sales (2015 - 2020)
Line - Average Price Trend (2015 - 2020)
Line - Count of Home Sales Trend (2015 - 2020)
Close


Key Summary:
6+ years Professional Experience
Mobile & Web full-stack developer
Interests SW Dev & Big Data
Fluent in Agile (Scrum)
Professional Experiences
- Mar 2015 - Present
-
Details
Finnish Geospatial Research Institute
Research Scientist - Feb 2015 - Feb 2015
- Feb 2014 - Jul 2014
-
Details
University of Helsinki Masters Thesis
Research Assistance - May 2013 - Aug 2013
-
Details
Nokia Solutions and Networks
Seasonal Trainee - Nov 2009 - Aug 2012
-
Details
IMS Health Ltd.
Software Developer
Technical Skills
Progamming Language
Android
8.5 / 10
Java
8 / 10
HTML / HTML5
60%
JavaScript / jQuery
60%
Query Languages
MySQL
80%
CSS / CSS3
80%
ASP.Net
40%
NodeJs
60%
C++
80%
Tools & Frameworks
Academic Background
- Aug 2012 - Apr 2016
-
Details
University of Helsinki M.Sc. Degree
Computer Science (Networking and Services) - Dec 2004 - Oct 2009
-
Details
Bangladesh University of Engineering & Technology B.Sc. Degree
Computer Science & Engineering
Connect with Me


PortableDevSync
new
remove
download
revisions
update software
sync now
Adding a new monitor
Name
?
Folder to synchronize
?
Time between syncs
?
Name
Last updated
Files synced
GRIDNET
PortableDevSync
This is a folder
This is another folder
previous page
1
2
3
next page


Welcome, fellow traveller!
This is your homepage
You are logged in as user "{{account.firstName}} {{account.lastName}}".
If you want to
sign in, you can try the default accounts:
- Administrator (login="admin" and password="admin")
- User (login="user" and password="user").
What about
If you like JHipster, don't forget to give us a star on GitHub!


source code on
Github
Coins:
Workers:
Get a coin
Get a worker (
coins)
There is no more content in the game
Reset game


Mister Mime
Mister Mime named Mimi is Ash and Delia Ketchum's loyal Pokémon, the family's helper. Now he is in Professor Seris's laboratory, looking after Ash.


Source code
The City of Philadelphia is hosting the content below that was removed from the EPA website.


about Pommes
Pommes R.
Yo noche comiendo menudo sudoroso.
Websites
- alexh.name - this page
- 0101010.one - tech blog
formerly
FacebookAlex H
Links
Pommes
I'm a female student in her 14ths.
I'm studying Frittieröll and cakes.
Cosmetic stuff is all self-taught.
My main interest in not getting eat.
Selected Writings
- Hosting a (decently) secure XMPP / Jabber server
- Hosting a solid Mail Server (netqmail + dovecot + roundcube)
Projects
Contact
Email: pomm. es at de


It is straight-forward to design and auto-generate
a fully functional application system, and be up
and running within hours (or even minutes).
We emphasize a practical and thorough
implementation based on standards and best-practices.
Generated systems consist of the integration of numerous
technologies working together on one platform. Each area
is carefully and thoroughly implemented.
Configuration files, management components, tests
and testing infrastructure are created along with
generated implementation.
The savings in cost is immeasurable. The quality and
thoroughness of the resulting generated application
system, management components, user-interface(s), and
tests far outweighs any efforts that could be done by hand.
By default, auto-generation creates multiple levels of tests
for each primary component of the target system.
Unit-tests are created to use
.
Multiple levels of integration tests and system tests
are created to use JUnit together with the valuable
and
.
A comprehensive and structured approach for testing
is needed for the purpose of being thorough and effective
in detecting and reducing potential problems.

Tests are structured to run in multiple test contexts.
This is important since certain conditions and certain
errors may only likely occur in one test environment,
while not in others - such problems can otherwise
be very hard to detect and fix.
Although hand-coding is not needed,
it is prefereble at times and is
completely easy to integrate into the auto-generated system.
It is straight-forward to integrate with external systems
such as a gateways, incoming message streams, data-sources or
external processing or payment systems.
Sample target systems of varying types can be designed,
generated, built, tested and experimented with, and compared
against each other, in nearly the quickest possible turnaround times.
This allows an organization to try-out various potential
solutions in different environments and in different contexts
- mixing and matching different approaches, constructs, services and
communication methods.


I'm intrested in social media ads and how to increase business using digital marketing,
and learning how to build web site using programming languages.
I have experience in sport field i'm working in Jordan Football Association(JFA). For 3 years because this i love sport and ask you questions about it and what it mean to me.
My goal is to build my own website or application and go viral.


PROJECT MANAGEMENT OFFICE
HOME
TEAM
GALLERY
FILE REPOSITORY
CONTACT
GALLERY
OUR GALLERY
PROJECT MANAGEMENT OFFICE
Home
Gallery
Album 1
Click Me
Album 2
Click Me
Album 3
Click Me
Album 4
Click Me
Album 5
Click Me
Album 6
Click Me


#include <UBitArray.h>
Public Member Functions | |
| UBitArray () | |
| class UBitArray | |
| UBitArray (unsigned char *data, unsigned int bitCount) | |
| ~UBitArray () | |
| void | Resize (unsigned int newSize) |
| resizes the bit array to the size specified. | |
| void | PushBit (unsigned int value) |
| push/pop/peek bit functionality. | |
| void | PushBits (unsigned int value, unsigned int count) |
| void | PushBits (int value, unsigned int count) |
| unsigned int | PopBit () |
| unsigned int | PopBits (unsigned int count) |
| int | PopBitsSignExtend (unsigned int count) |
| unsigned int | PeekTopBit () |
| void | SetBitValue (unsigned int idx, unsigned int value) |
| get/set bit functionality. | |
| unsigned int | GetBitValue (unsigned int idx) const |
| const unsigned char * | GetBits () const |
| returns a pointer to bit data. | |
| unsigned int | GetBitCount () const |
File: UBitArray.h Author: Kevin Bray Created: 09-22-06 Copyright © 2004 Bootstrap Studios. All rights reserved. std c++ includes. class UBitArray
Definition at line 18 of file UBitArray.h.
| UBitArray::UBitArray | ( | ) |
class UBitArray
ctor & dtor
always start with room for 32 bits.
Definition at line 24 of file UBitArray.cpp.
| UBitArray::UBitArray | ( | unsigned char * | data, | |
| unsigned int | bitCount | |||
| ) |
Definition at line 32 of file UBitArray.cpp.
| UBitArray::~UBitArray | ( | ) |
Definition at line 40 of file UBitArray.cpp.
| unsigned int UBitArray::GetBitCount | ( | ) | const [inline] |
Definition at line 43 of file UBitArray.h.
| const unsigned char* UBitArray::GetBits | ( | ) | const [inline] |
| unsigned int UBitArray::GetBitValue | ( | unsigned int | idx | ) | const |
| unsigned int UBitArray::PeekTopBit | ( | ) |
| unsigned int UBitArray::PopBit | ( | ) |
get the top value and decrement the bit count.
return the value.
Definition at line 88 of file UBitArray.cpp.
| unsigned int UBitArray::PopBits | ( | unsigned int | count | ) |
Definition at line 104 of file UBitArray.cpp.
| int UBitArray::PopBitsSignExtend | ( | unsigned int | count | ) |
get the return value.
extend the sign bit.
return the value.
Definition at line 114 of file UBitArray.cpp.
| void UBitArray::PushBit | ( | unsigned int | value | ) |
push/pop/peek bit functionality.
check to see if we need to add another dword.
now set the bit.
Definition at line 60 of file UBitArray.cpp.
| void UBitArray::PushBits | ( | int | value, | |
| unsigned int | count | |||
| ) |
Definition at line 81 of file UBitArray.cpp.
| void UBitArray::PushBits | ( | unsigned int | value, | |
| unsigned int | count | |||
| ) |
Definition at line 73 of file UBitArray.cpp.
| void UBitArray::Resize | ( | unsigned int | newSize | ) |
resizes the bit array to the size specified.
public methods
make sure we have enough DWORDS to fit the number of bits specified.
Definition at line 52 of file UBitArray.cpp.
| void UBitArray::SetBitValue | ( | unsigned int | idx, | |
| unsigned int | value | |||
| ) |
get/set bit functionality.
make sure the incoming parameters are valid.
get the index into the array of DWORDs.
set the bit's value.
Definition at line 140 of file UBitArray.cpp.