Installing and configuring the MerMEId editor
Digital Infrastruktur og Service,
The Royal Library,
Copenhagen
<EMAIL_ADDRESS>|
|
|
|
Dansk Center for Musikudgivelse
The Royal Library,
Copenhagen
<EMAIL_ADDRESS>0. Table of Contents
For information on how to use the editor, please refer to the manual
Migration from MEI 2010 to 2012
If you have an installed base of mei documents you need to migrate them in order to use the new software. Please refer to HOW TO migrate from MEI 2010 to 2012. Please note that this will only migrate the meiHead
Copyright 2013 Danish Center for Music Publication
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
1. Overview
2. System requirements
This system is lightweight. It is only tested under Apache2 HTTPD with Tomcat6 servlet container under Red Hat and SUSE Linux. We have not been able to run it on less then 3GB, and recommend 4GB internal memory. You should allocate at least 2GB to the java instance that runs the various components.
It requires the following readily available software on the server:
- Apache HTTPD
- Apache Tomcat 6 running
- xerces2-j (i.e., the java version)
- xalan-j (xslt processor, java version)
Software can run on a modest system. We have it on a virtual server with 4GB of internal memory. 80GB of hard-disk would be a lot.
3. Installation
Unpack and install the MerMEId: We suggest that you create a directory called /home/xml-store and unpack it there. There should then appear the following:
| Directory | Content |
|---|---|
| documents | empty, more or less |
| filter |
a number of JAVA servlets, builds a war file using maven. You'll need to edit the java property file filter/src/main/resources/http_filter.xml (see Appendix A. When done you must recompile the package using maven: cd <path to source>/filter/ mvn install cp target/filter*.war /<PATH TO YOUR>/tomcat6/webapps/filter.war If you don't have maven installed, you can always unzip the filter.war coming with the distribution, edit http_filter.xml and then repack it as a war file. For documentation of the filter software itself, see its API javadoc. |
| mermeid | the forms. The configuration here lives in forms/mermeid_configuration.xml |
| orbeon | A JSP script which should be installed as <your tomcat>/webapps/orbeon/xforms-jsp/mei-form/index.jsp -- this is basically an xinclude processor |
| transforms | some transforms used for presentation, filtering and also for migration of mei2010 to 2012 just copy them on top of the old ones |
| xqueries | used for search and navigation |
In addition, there are a number of files. filter.war, editor.war INSTALL.html (this file).
You have a choice. You can deliver the form through Apache HTTPD. That is an advantage if you are developer and need to customize the code code. But if you like to keep it as it is, then it might be easier to deploye it in your tomcat.
cp editor.war /<PATH TO YOUR>/tomcat6/webapps/editor.war
See below how apache is configured in the two cases.
-
Install Apache-Tomcat: We have recently run it on versions 6.0.20 to 6.0.36 of Apache-Tomcat. If the one provided in your Linux distribution is 6.0.20 or better I suggest that you use that. The advantage is that you get all the java needed without further ado.
In the following we assume that tomcat lives in
/<PATH TO>/tomcat6
-
Install Orbeon: We are using version 3.9.0.
cp orbeon.war /<PATH TO>/tomcat6/webapps/
We are using some extra files, among those is a jsp script for connecting our form to orbeon. They need to be installed inside the orboen webappcd /home/xml-store/editor mkdir /<PATH TO>/tomcat6/webapps/orbeon/xforms-jsp/mei-form cp xquery/mei_form.jsp /<PATH TO>/tomcat6/webapps/orbeon/xforms-jsp/mei-form/index.jsp
The jsp is dependent on a package called commons-httpclient. We've added it in the distribution. -
Install eXist: We're using version 1.4.1 Prepare
the code and build the application by first running the installer
java -jar eXist-<version number>.jar
This will unpack the distribution into a directory of your choice. Let's call it eXist. In that directory you can repack it as a *.war file which can be installed in you tomcat6.cd eXist ./build.sh dist-war
That will create a file called something like dist/exist-<version number>.war. This file should be installed as exist.war in you tomcat6.cp dist/exist-*.war /<PATH TO YOUR>/tomcat6/webapps/exist.war
-
Install Orbeon: We are using version 3.9.0.
- Install apache httpd: If it isn't installed already. We're using the one that came with the distribution Apache/2.2.3 Ensure that you have mod_proxy and mod_proxy_ajp installed.
4. Configuration
Ensure that Apache HTTPD and Apache Tomcat6 both starts with the operating system. A requisite for the following is that they are healthy, up and running. We assume they that they are running on port numbers 80 and 8080 respectively on a server called mei.example.org.
-
Figure 1. eXist interface. The Admin functions are down to the left.
Figure 2. eXist admin client. What you see is the situation after eXist is installed and configured. You see the dcm collection owned by the dcm user, and also the XQuery list_files.xq which is used for many tasks. See main text.Configure eXist: The database engine is entirely web based, and all work of it can be done via an ordinary browser or a web service client bundled with the eXist. Please not that the Royal Library cannot not provide support for this or indeed any other software we haven't authored in house.
Direct your browser to http://mei.example.org:8080/exist/ (i.e., to where you installed exist). You will get a page as in Figure 1.
- Set a password for the database admin. This is done under Admin < User management.
- Create the dcm collection and the user dcm. We do all the adminstrative tasks using the JNLP based client IcedTea. It should work on all platforms. Click on "Web Start Client" and log on as admin with the user credentials you just created. The admin's credentials should be put into the configuration files Figure 2 shows the client after installation and configuration.
- Create a collection called library. Used to keep a standard set of bibliographic references that are often used.
- Load some content:
After configuring eXist, it is just to populate the database with some content. The easiest way to do that is to use the admin client (see above).
-
Load the query/queries.
While still in xquery, you can do this from the command line.
# This is how to configure stuff in Apache: Alias "/editor" "/home/xml-store/MerMEId/mermeid <Location "/editor"> Options None Order allow,deny Allow from all </Location> ProxyPass /storage "ajp://mei.example.org:8009/exist/rest/db" ProxyPass /orbeon "ajp://mei.example.org:8009/orbeon" ProxyPass /filter "ajp://mei.example.org:8009/filter"
On the other hand, if you put them in the application server through deploying the editor.war. Enter the following in your apache configuration.
ProxyPass /editor "ajp://mei.example.org:8009/editor ProxyPass /storage "ajp://mei.example.org:8009/exist/rest/db" ProxyPass /orbeon "ajp://mei.example.org:8009/orbeon" ProxyPass /filter "ajp://mei.example.org:8009/filter"
5. Testing and Getting Started
The main component in the user interface is the list_files.xq which directs the editorial staff to all the other components. That script should have the URI:
http://mei.example.org/storage/list_files.xq
If everything is up and running you should be able to get an HTML version, a form and a raw XML version for each object in the xml_store.
6. Authentication & Security
Figure 3. Setting the permissions in eXist DB.
The seasoned sysadmin should already have noticed that there is only one component which is protected by some Authentication and Authorization Infrastructure (AAI), and that is the eXist database engine. The sysadmin should also have noticed that the web application filter actually performs these operations and that it has not been mentioned that it is protected.
It is not protected. If you run this application on the open Internet you need to do the following.
- Place the MerMEId server behind a firewall that closes tomcat's port 8080 for external users
-
Protect both the vulnerable components /orbeon and
/filter. Currently we do this using basic
authentication. We put the following in our apache configuration:
<LocationMatch ^/+(orbeon|filter)/+.*> <Limit POST PUT DELETE> AuthType Basic AuthName "MerMEId users & developers." AuthUserFile /home/xml-store/passwordfile Require valid-user </Limit> </LocationMatch>
Note that you need to set
tomcatAuthentication="false"
On the appropriate entry in <your tomcat>/conf/server.xml, i.e. change from
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
to
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" tomcatAuthentication="false" />
This will protect the database from the vulnerabilities built into the forms and the filter. You can create the passwordfile like this:
htpasswd -bc passwordfile trusted_user magic_word
This will create a new passwordfile for your trusted_user containing its very secret magic_word. The htpasswd utility should come with the Apache distribution. You could obviously use any AAI scheme supported by the web server.
For the system to be secure, you need to set the update and write permissions in the eXist database. See Figure 3.
7. Backup & restore
Figure 4. Backup (top) and restore(bottom).
At DCM we have eXist DB and its application server on a Linux machine that are backed-up regularly. In addition we take a backup of the XML-files using the admin client. See Figure 4. It is also possible to add the content of an entire directory to the database through the client.
We have some other simple tools:
- The load_exist.pl (see above) can be used to retrieve files as well as storing them
-
transforms/mei/utils/backup_exist.xsl
can be used for making backup of a large number of files in a
directory on eXist. It transforms an eXist directory listing into a
shell script that uses lwp-mirror to mirror individual files. First
generate the script
xsltproc --stringparam base_uri \ http://mei.example.org/storage/dcm/ \ transforms/mei/utils/backup_exist.xsl \ http://distest.kb.dk/storage/dcm/ > backup.sh
Then make it executable and run itchmod a+x backup.sh ./backup.sh
The script contains code as below.#!/bin/sh if ! [ -d ./db/dcm ]; then mkdir -p ./db/dcm fi lwp-mirror http://mei.example.org/storage/dcm/db/dcm/1285256669.xml \ ./db/dcm/1285256669.xml lwp-mirror http://mei.example.org/storage/dcm/db/dcm/1285323711.xml \ ./db/dcm/1285323711.xml lwp-mirror http://mei.example.org/storage/dcm/db/dcm/1285750781.xml \ ./db/dcm/1285750781.xml lwp-mirror http://mei.example.org/storage/dcm/db/dcm/1289382773.xml \ ./db/dcm/1289382773.xml ...
The lines have been folded. - eXist DB comes with a backup.sh script
Appendix A. Configuration files
Filter configuration
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"> <properties> <comment>HTTP Filter parameters</comment> <!-- The filter is meant to operate on top of an eXist XML database --> <entry key="exist.host">mei.example.org</entry> <entry key="exist.port">8080</entry> <entry key="exist.realm">exist</entry> <entry key="exist.context">/exist/rest/db/dcm</entry> <entry key="exist.user">admin</entry> <entry key="exist.password">your very secret word</entry> <!-- get is the retrieval filter --> <entry key="get">http://mei.example.org/editor/transforms/mei/filter_get.xsl</entry> <entry key="get.mime">application/xml</entry> <entry key="get.charset">UTF-8</entry> <!-- put is for updating --> <entry key="put">http://mei.example.org/editor/transforms/mei/filter_put.xsl</entry> <entry key="put.mime">application/xml</entry> <entry key="put.charset">UTF-8</entry> <!-- del is for removing stuff --> <entry key="del.redirect">http://mei.example.org/storage/list_files.xq</entry> <!-- Create is for what it seems to be. A URI that is good to use for creating new documents. We're doing that using uri templates (RFC6570). The URI should look like http://mei.example.org/orbeon/xforms-jsp/mei-form/?uri=http://mei.example.org/form/dcm/&doc=cnw0361.xml when ready --> <entry key="create">http://mei.example.org/orbeon/xforms-jsp/mei-form/{?uri,dir,doc}</entry> <entry key="create.uri">http://mei.example.org/editor/forms/mei/edit-work-case.xml</entry> <entry key="create.dir">http://mei.example.org/filter/</entry> </properties>
Form configuration
<parameters xmlns="http://www.kb.dk/dcm"> <document_root>storage/dcm/</document_root> <exist_dir>/storage/</exist_dir> <orbeon_dir>http://mei.example.org/orbeon/xforms-jsp/mei-form/</orbeon_dir> <form_home>http://mei.example.org/editor/forms/mei/</form_home> <crud_home>http://mei.example.org/filter/</crud_home> <server_name>http://mei.example.org/</server_name> <xml_file/> <!-- default editor settings --> <!-- Set boolean values to 'true' or nothing --> <!-- Enable attribute editor? --> <attr_editor/> <!-- Enable xml:id display component? --> <show_id/> <!-- Enable xml:id display component? --> <code_inspector/> </parameters>


--- layout: post title: "Collatz Sea Fan" date: 2017-04-30 19:26:00 categories: experiments custom_js: - p5/p5.min.js - p5/addons/p5.dom.js - local/fs.js ---


Id = 14
Nombre Capitulo = Giant's Town
Tipo = Fantasy town
Dimension = Fantasy Dimension
Personaje = ['https://rickandmortyapi.com/api/character/89', 'https://rickandmortyapi.com/api/character/399', 'https://rickandmortyapi.com/api/character/400']
Numero Personaje = ['89', '399', '400']
URL Locacion = https://rickandmortyapi.com/api/location/14


Design your own game!
Defining
Learners
Who are your learners? How many? How are their language skills? What’s the target age/grade?
Defining
Learning Goals
What do you want players to learn? Vocabulary/Sentence Structure/Speak Freely/Grammar?
Defining
Game Dynamics
What are core interactions of the game? What are players' main action/behavior in the game?
Defining
Game Mechanics
What to use to add random factors in the game? What are the scoring/incentives that players have?
Defining
Game Aesthetics
How is the game "fun"? What kind of emotion is the game eliciting that makes the game fun?
Cueing is by indirectly giving another player cues of certain words/objects/person. Cues can be given via verbal information, acting, drawing, iconic representation, etc.


Component Library
The component library for creating portfolio website.
@typography
The biggest heading is H1
The second largest heading is H2
This is a paragraph which is like the most common way of displaying the
text on the website
This is comparatively smaller text
@container
fluid container
This is a fluid container and it will go till the end of the page
center container
This is a center container and it will go till the specified width
@links
Primary Link
Secondary Link
@lists
unordered list
- Item one
- Item two
- Item three
unordered inline list
- Inline Item one
- Inline Item two
- Inline Item three
reverse ordered list
- milk
- eggs
- toast
@nav
@header
Software Engineer at Arcesium
@section
white section
This will have white background
off white section
This will have off white background
@footer


| git init |
The first step is to initialize a new Git repo locally in your project root. You can do so with the command below |
| git add . |
To add all files in your project to the staging area |
| git status |
Shows the status of the current repository including staged, unstaged, and untracked files |
| git commit -m "your commit message here" |
A commit message is made up of a short summary of changes. This command is used to commit your changes |
| git commit -a -m"your commit message here" |
You can add and commit tracked files with a single command by using the -a and -m options |
| git log |
This command shows the commit history for the current repository |
| git branch |
You can view all created branches using the git branch command. Highlights current branch with an asterisk |
| git checkout -b branch_name |
In a single command, you can create and switch to a new branch right away |
| git branch -d branch_name |
When you are done working with a branch and have merged it, you can delete it using this command |
| git merge branch_name |
To merge the history of the branch you are currently in with the branch_name |
| git push |
When all your work is ready to be saved on a remote repository, you can push all changes using the command below |
| git pull |
If other team members are working on your repository, you can retrieve the latest changes made to the remote repository with the command below |
| git push -u origin branch_name |
If you want to push a branch to a remote repository you can use the command below. Just remember to add -u to create the branch upstream |


Basic Usage
Dependency
Plugin Files To Include
Initialization Code for AnyPicker (with Default settings)
Initialization Code for AnyPicker DateTime Mode
Initialization Code for AnyPicker Select Mode
This is a quick quide which will help you to start using AnyPicker.
- jQuery
Javascript
- anypicker.min.js (required)
CSS
- anypicker-all.min.css (required, combines all files listed below)
OR
- anypicker.min.css (required)
- anypicker-ios.css (optional, for iOS
theme
) - anypicker-android.css (optional, for Android
theme
) - anypicker-windows.css (optional, for Windows
theme
) - anypicker-font.css (optional, Windows
theme
and ButtonrowsNavigation
icon font)
Font
You have to include fonts folder in distribution file. Fonts folder contains -
- anypicker-font (Font Icon - Up & Down Buttons and Set, Cancel, Now & Clear Buttons in Windows theme)
HTML
<input type="text" id="input-1" readonly>
Javascript
$("#input-1").AnyPicker({});
HTML
<input type="text" id="input-1" readonly>
Javascript
$("#input-1").AnyPicker( { mode: "datetime", dateTimeFormat: "MMM dd, yyyy" });
HTML
<input type="text" id="input-1" readonly>
Javascript
$("#input-1").AnyPicker( { mode: "select", components: [ { component: 0, name: "currency", label: "Currency" }], dataSource: [ { component: 0, data: [ { val: "USD", label: "$ - USD" }, { val: "EUR", label: "€ - EUR" }, { val: "INR", label: "₹ - INR" }, { val: "ILS", label: "₪ - ILS" }, { val: "JPY", label: "￥ - JPY" }] }] });
Parameter Options List
This is a list of the plugin settings parameters. This list includes parameters, their default values, all possible values that you can set and describes how to use parameters.
Callback Functions List
This is a list of callback functions that are triggered on reaching the specified condition.
Build with Grunt
QuickStart
Preconfigured Tasks
- Install node from http://nodejs.org.
- Run
npm install
in the AnyPicker's root directory to install grunt's dependencies. - Run
grunt
, which will run the default task, which runsconcat
,copy
,uglify
andcssmin
.
-
grunt
- This is a Default Task.It is configured as an array of tasks.
-
concat
- Default concat task, which concatenates all .js files to create anypicker.js -
copy
- copies all .js and .css files from /src folder to /dist folder. -
uglify
- minifies javascript code -
cssmin
- concatenates /src/anypicker.css to /dist/CalenStyle.min.css and minifies css.
-
-
grunt AnyPickerJSConcat
- This command helps to build a /src/anypicker-custom.js by combining .js files for specified views. After concatenation,copy
,uglify
andcssmin
tasks will be performed.For example -This command will build a custom script by combining code for Select and DateTime.grunt AnyPickerJSConcat:Select:DateTime
A list of valid arguments for this commandSelect
DateTime


On-Demand Environments with Docker and AWS ECS
In this tutorial, we’ll look at how to spin up reproducible (and easily-destructible), on-demand test environments with Docker, Amazon EC2 Container Service (ECS), and Circle CI (for continuous integration and delivery).
We’ll be using the following tools…
| Tool | Use Cases | Version |
|---|---|---|
| Docker | Containerization and distribution | 17.03.2-ce |
| AWS ECS | Container orchestration and management | 1.14.4 (service agent) |
| Circle CI | Continuous integration | 2.0 |
| AWS JavaScript SDK | Interacting with AWS | 2.114.0 |
For a demo, check out the following video.
Contents
- Prerequisites
- Objectives
- Introduction
- Development Workflow
- Project Setup
- Manual AWS Setup
- Circle CI Setup
- AWS SDK Setup
- Deployment Script
- Testing
- Teardown
- Conclusion and Next Steps
Prerequisites
This post assumes prior knowledge of Docker and Docker Compose along with microservice architecture in general.
If you haven’t already, review the Developing Microservices - Node, React, and Docker blog post. This tutorial utilizes a slightly modified version of the finished project from that post, so be sure to review the code from the v3 tag of the microservice-movies repository, taking note of each service:
| Name | Service | Container | Tech |
|---|---|---|---|
| Web | Web | web | React, React-Router |
| Movies API | Movies | movies | Node, Express |
| Movies DB | Movies | movies-db | Postgres |
| Swagger | Movies | swagger | Swagger UI |
| Users API | Users | users | Node, Express |
| Users DB | Users | users-db | Postgres |
| Functional Tests | Test | n/a | TestCafe |
You should also be familiar with AWS in general along with the following AWS services - VPC, ELB, EC2, and IAM.
Objectives
By the end of this tutorial, you will be able to…
- Explain what on-demand environments are, why you would want to use them, and the overall development workflow
- Discuss the benefits of using on-demand environments
- Automate the building, configuring, deploying, and maintaining of on-demand environments on AWS
- Configure an Application Load Balancer and ECS to run a set of microservices
- Set up continuous integration and deployment to ECS via Circle CI
- Integrate Amazon EC2 Container Registry, an image registry, into the continuous integration process
- Create a teardown script to remove the environment once testing is complete
Introduction
The end goal of on-demand test environments is to allow developers to quickly and easily spin up multiple, independent testing environments.
In other words, testing environments are spun up as needed during the development process - when code is checked-in, for example - and, then, when no longer needed, spun down just as quickly to free up resources and keep costs down.
With microservices, testing a single service can be difficult, time consuming, and expensive since you often have to spin up a plethora of services, especially if you have resource and/or config-heavy services. On-demand environments simplify this, making testing and trial easy, decreasing the feedback cycle between review and development.
Such environments can used for running integration and end-to-end tests, QA, UAT, troubleshooting, and basic trial.
NOTE: For more on the benefits of using reproducible, on-demand test environments, review the The Business Case for On-Demand Test Services whitepaper.
Development Workflow
With on-demand environments, the development workflow looks like…
(1) Local development
- Create a new feature branch from the master branch
- Make code changes
- Commit and push code to GitHub
(2) Continuous integration
- Open a new PR against the development branch
- A new build is then triggered on Circle CI
- If the build passes, manually merge the PR
- A new build is triggered again on Circle CI
- If the build passes, deployment occurs…
(3) Deployment on AWS via deployment scripts
- Images are created, tagged, and pushed to ECR
- Task Definitions are registered on ECS
- Target Groups are created
- A Listener is added to the load balancer and Rules are added
- ECS Services are created
(4) Testing
- End-to-end tests
- Acceptance tests
- UAT
(5) Teardown
- All AWS Resources are torn down
Project Setup
Fork the microservice-movies repo, clone it down, and then check out the v3 tag to a new branch called aws-docker-on-demand
:
$ git clone https://github.com/YOUR_GITHUB_NAME/microservice-movies
$ cd microservice-movies
$ git checkout tags/v3 -b aws-docker-on-demand
Set the environment variables:
$ export NODE_ENV=test
$ export REACT_APP_USERS_SERVICE_URL=http://localhost:3000
$ export REACT_APP_MOVIES_SERVICE_URL=http://localhost:3001
Build the images and fire up the containers:
$ docker-compose -f docker-compose-review.yml build
$ docker-compose -f docker-compose-review.yml up -d
Run the tests:
$ docker-compose -f docker-compose-review.yml \
run users-service-review npm test \ &&
docker-compose -f docker-compose-review.yml \
run movies-service-review npm test \ &&
testcafe firefox tests/**/*.js
Finally, ensure you can view the app in your browser at http://localhost:3007, and then try logging in with username michael
and password herman
.
Manual AWS Setup
Before deploying, we need to configure the the following AWS resources:
- EC2 Key Pair
- ECS Cluster
- EC2 Container Registry (ECR)
- Application Load Balancer (ALB)
- Security Group
Keep in mind that you could create each of these resources dynamically as well. Clusters can take a bit of time to fire up since EC2 instances need to be spun up though. By setting up the Cluster beforehand, EC2 instances are up and running, ready for Tasks to be added.
EC2 Key Pair
Within the EC2 Dashboard, click “Key Pairs” on the navigation pane, and then click the “Create Key Pair” button. Name the key microservicemovies-review
. Save the file in a safe place - i.e., “~/.ssh”.
ECS Cluster
An ECS Cluster is just a group of EC2 container instances managed by ECS. To set up, navigate to the ECS Console, and then select the region for the Cluster on the right-side of the nav bar.
NOTE: This tutorial uses the
US West (Oregon)
/us-west-2
region. Feel free to use the region of your choice. For more info, review the Regions and Availability Zones guide.
Next, click the “Create Cluster” button:
- “Cluster name”:
microservicemovies-review
- “EC2 instance type”:
t2.medium
- “Number of instances”:
3
- “Key pair”:
microservicemovies-review
- Create a new VPC and Security Group
Create the Cluster.
Navigate to the Cluster once it’s created, and then click the “ECS Instances” tab. From there, click the “Actions” dropdown and select “View Cluster Resources”. Take note of the VPC and Security Group:
ECR
Within the ECS Console, click “Repositories” on the navigation pane, and then click the “Create repository” button. Add the following repositories:
microservicemovies/users-db-review
microservicemovies/movies-db-review
microservicemovies/users-service-review
microservicemovies/movies-service-review
microservicemovies/web-service-review
microservicemovies/swagger-review
Application Load Balancer
Elastic Load Balancing, as of writing, supports two types of load balancers - Classic and Application. Either will work with ECS, but the Application Load Balancer (ALB) is the better of the two since it:
- Dynamically maps container services to ports
- Distributes traffic evenly across the entire ECS Service
- Runs status health checks against each service
- Allows for zero-downtime deploys
To set up, navigate to the EC2 Dashboard, update the region (if necessary), and then click “Load Balancers” in the navigation pane. Click the “Create Load Balancer” button. Select “Application Load Balancer”, and then go through each of the steps to configure the load balancer…
- Configure Load Balancer:
- “Name”:
microservicemovies-review
- “VPC”: Select the VPC that was just created
- “Availability Zones”: Select at least two available subnets
- “Name”:
- Configure Security Settings: Skip this for now
- Configure Security Groups: Select the Security Group that was just created
- Configure Routing:
- “Name”:
review-default
- “Port”:
80
- “Path”:
/
- “Name”:
- Register Targets: Do not assign any instances manually since this will be managed by ECS
Along with the Application Load Balancer, this will also create a default Target Group called review-default
:
Security Group
Finally, let’s add some ports to work with to the Security Group. Within the EC2 Dashboard, click “Security Groups” in the navigation pane, and then select the Security Group that was just created. On the “Inbound Rules” pane, click the “Edit” button and the “Add another rule button”:
- “Type”: “Custom TCP Rule”
- “Protocol”: “TCP (6)”
- “Port Range”:
30000-50000
- “Source”:
<IP_ADDRESS>/0
Click the “Save” button.
That’s it for the basic AWS resources.
Circle CI Setup
Next, sign up for Circle CI (if necessary) and follow the basic steps to configure Circle 2.0, making sure to enable access to your GitHub repos. Then, create a new folder called “.circleci” in the project root of “microservice-movies” and add a config.yml file to that directory:
version: 2
jobs:
build:
docker:
- image: docker:17.03.2-ce-git
working_directory: ~/microservice-movies
environment:
- NODE_ENV: test
- REACT_APP_USERS_SERVICE_URL: http://localhost:3000
- REACT_APP_MOVIES_SERVICE_URL: http://localhost:3001
parallelism: 1
steps:
- checkout
- setup_remote_docker:
reusable: true
exclusive: false
- run:
name: Install dependencies
command: |
apk add --no-cache \
py-pip=9.0.0-r1 \
bash \
jq \
curl \
nodejs
pip install \
docker-compose==1.12.0 \
awscli==1.11.76
- run:
name: Build Docker images
command: docker-compose -f docker-compose-review.yml build
- run:
name: Spin up Docker containers
command: docker-compose -f docker-compose-review.yml up -d
- run:
name: Test the user service
command: docker-compose -f docker-compose-review.yml run users-service-review npm test
- run:
name: Test the movies service
command: docker-compose -f docker-compose-review.yml run movies-service-review npm test
Review this file. Refer to the Circle documentation as needed. Commit and push your code to GitHub once done. On Circle, navigate to the “Add Projects” page and click the “Build Project” button next to your project. This will trigger a new build, which should pass.
AWS SDK Setup
Since the microservice stack is built with Node, we’ll develop the deployment script in JavaScript with the AWS JavaScript SDK.
Add a package.json to the project root:
{
"name": "microservice-movies",
"dependencies": {
"aws-sdk": "2.114.0"
}
}
Install the dependency:
$ npm install
Add a new folder to the project root called “ecs”, and then create a new folder called “scripts” within that folder. Finally, add a new file called setup.js to “scripts”:
const AWS = require('aws-sdk');
// globals
const AWS_ACCOUNT_ID = process.env.AWS_ACCOUNT_ID;
const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID;
const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY;
const AWS_USERNAME = process.env.AWS_USERNAME;
const AWS_CONFIG_REGION = 'us-west-2';
const clusterName = 'microservicemovies-review';
// config
AWS.config = new AWS.Config();
AWS.config.accessKeyId = AWS_ACCESS_KEY_ID;
AWS.config.secretAccessKey = AWS_SECRET_ACCESS_KEY;
AWS.config.region = AWS_CONFIG_REGION;
// init aws services
const ecs = new AWS.ECS();
const iam = new AWS.IAM();
// methods
function ensureAuthenticated() {
return new Promise((resolve, reject) => {
const params = { UserName: AWS_USERNAME };
iam.getUser(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function confirmRegion() {
return new Promise((resolve, reject) => {
if (AWS.config.region !== AWS_CONFIG_REGION) {
reject('Something went wrong!');
}
resolve(AWS_CONFIG_REGION);
});
}
function getCluster() {
return new Promise((resolve, reject) => {
const params = { clusters: [ clusterName ] };
ecs.describeClusters(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
return confirmRegion();
})
.then((region) => {
console.log(`AWS Region -> ${region}`);
return getCluster();
})
.then((cluster) => {
if (!cluster.clusters.length) {
console.log('Cluster does not exist!');
return;
}
console.log(`ECS Cluster -> ${cluster.clusters[0].clusterName}`);
})
.catch((err) => { console.log(err); });
In essence, this confirms that the AWS user credentials are correct and that the AWS Region and ECS Cluster are configured properly.
Set the following environment variables locally:
AWS_ACCOUNT_ID
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_USERNAME
Run the script:
$ node ecs/scripts/setup.js
You should see:
Welcome AWS_USERNAME!
AWS Region -> us-west-2
ECS Cluster -> microservicemovies-review
Add another step to the bottom of .circleci/config.yml:
- run:
name: Deploy
command: |
npm install
node ecs/scripts/setup.js
Add the following environment variables to Circle so that we can properly configure the AWS SDK:
AWS_ACCOUNT_ID
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_USERNAME
Commit your changes, and then push to GitHub to trigger a new build. Make sure it passes before moving on.
Deployment Script
Moving right along, let’s start building the deployment scripts to set up the AWS resources and deploy the app.
Main Steps:
- Tag and push images to ECR
- Get open port for the Listener
- Register Task Definitions
- Create Target Groups
- Add the Listener and Rules
- Create new Services
(1) Tag and push images to ECR
Create a new file in “ecs/scripts” called ecr.sh:
#!/usr/bin/env bash
# config
set -e
ECS_REGION="us-west-2"
NAMESPACE="microservicemovies"
IMAGE_BASE="microservicemovies"
ECR_URI="${AWS_ACCOUNT_ID}.dkr.ecr.${ECS_REGION}.amazonaws.com"
SHORT_GIT_HASH=$(echo $CIRCLE_SHA1 | cut -c -7)
TAG=$SHORT_GIT_HASH
# helpers
configure_aws_cli() {
echo "Configuring AWS..."
aws --version
aws configure set default.region $ECS_REGION
aws configure set default.output json
echo "AWS configured!"
}
tag_and_push_images() {
echo "Tagging and pushing images..."
$(aws ecr get-login --region "${ECS_REGION}")
# tag
docker tag ${IMAGE_BASE}_users-db-review ${ECR_URI}/${NAMESPACE}/users-db-review:${TAG}
docker tag ${IMAGE_BASE}_movies-db-review ${ECR_URI}/${NAMESPACE}/movies-db-review:${TAG}
docker tag ${IMAGE_BASE}_users-service-review ${ECR_URI}/${NAMESPACE}/users-service-review:${TAG}
docker tag ${IMAGE_BASE}_movies-service-review ${ECR_URI}/${NAMESPACE}/movies-service-review:${TAG}
docker tag ${IMAGE_BASE}_web-service-review ${ECR_URI}/${NAMESPACE}/web-service-review:${TAG}
docker tag ${IMAGE_BASE}_swagger-review ${ECR_URI}/${NAMESPACE}/swagger-review:${TAG}
# push
docker push ${ECR_URI}/${NAMESPACE}/users-db-review:${TAG}
docker push ${ECR_URI}/${NAMESPACE}/movies-db-review:${TAG}
docker push ${ECR_URI}/${NAMESPACE}/users-service-review:${TAG}
docker push ${ECR_URI}/${NAMESPACE}/movies-service-review:${TAG}
docker push ${ECR_URI}/${NAMESPACE}/web-service-review:${TAG}
docker push ${ECR_URI}/${NAMESPACE}/swagger-review:${TAG}
echo "Images tagged and pushed!"
}
# main
configure_aws_cli
tag_and_push_images
Each deploy is associated with a different commit and, thus, a different version of the code. To link the commit back to a specific image, we just used part of the Git commit SHA as the image tag:
SHORT_GIT_HASH=$(echo $CIRCLE_SHA1 | cut -c -7)
TAG=$SHORT_GIT_HASH
NOTE: Instead of using the associated Git commit SHA, you could get a bit creative and build a Heroku-like random name generator.
Update the Deploy
command in .circle/config.yml:
- run:
name: Deploy
command: |
npm install
node ecs/scripts/setup.js
sh ecs/scripts/ecr.sh
Commit and push your code to GitHub to trigger a new Circle build. Once done, ensure the images are up on ECS with the appropriate tag:
(2) Get open port for the Listener
Remember how we set a range of open ports, 30000
- 50000
, when we set up the Security Group? Well, each new environment will be assigned to a port within this range.
Add a new file called listener.js to “ecs/scripts”:
const AWS = require('aws-sdk');
// globals
const AWS_ACCOUNT_ID = process.env.AWS_ACCOUNT_ID;
const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID;
const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY;
const AWS_USERNAME = process.env.AWS_USERNAME;
const AWS_CONFIG_REGION = 'us-west-2';
const LOAD_BALANCER_ARN = 'UPDATE_ME';
// config
AWS.config = new AWS.Config();
AWS.config.accessKeyId = AWS_ACCESS_KEY_ID;
AWS.config.secretAccessKey = AWS_SECRET_ACCESS_KEY;
AWS.config.region = AWS_CONFIG_REGION;
// init aws services
const elbv2 = new AWS.ELBv2();
// methods
function getPort() {
return new Promise((resolve, reject) => {
var params = {
LoadBalancerArn: LOAD_BALANCER_ARN
};
elbv2.describeListeners(params, (err, data) => {
if (err) { reject(err); }
const max = data.Listeners.reduce((prev, current) => {
return (prev.Port > current.Port) ? prev : current;
});
if (parseInt(max.Port) === 80) {
resolve(30000);
} else {
const port = parseInt(max.Port) + 1;
resolve(port);
}
});
});
}
module.exports = {
getPort
};
Add then update the following variable:
const LOAD_BALANCER_ARN = 'UPDATE_ME';
Then, take note of the following if
statement:
if (parseInt(max.Port) === 80) {
resolve(30000);
} else {
const port = parseInt(max.Port) + 1;
resolve(port);
}
Why do we need this?
Since the default Target Group, review-default
, was configured to listen on port 80, we need to account for it if it’s the only Listener set up since we’re bumping the port by 1 each time.
We’ll test this out in the next section…
(3) Register Task Definitions
A Task Definition is used to create an application from one or more containers. It’s similar to a Docker Compose file.
Steps:
- Create Task Definition files
- Create script to register Task Definitions
Create Task Definition files
Create a new directory called “tasks” within “ecs”, and then add the following files:
- users-review_task.js
- movies-review_task.js
- web-review_task.js
Users - users-review_task.js
function createUsersTaskDefinition(accountID, region, tag, family, revision) {
const taskDefinition = {
containerDefinitions: [
{
name: 'users-service-review',
image: `${accountID}.dkr.ecr.${region}.amazonaws.com\/microservicemovies\/users-service-review:${tag}`,
essential: true,
memoryReservation: 300,
cpu: 300,
portMappings: [
{
containerPort: 3000,
hostPort: 0,
protocol: 'tcp'
}
],
environment: [
{
name: 'DATABASE_URL',
value: 'postgres://postgres:postgres@users-db-review:5432/users_dev'
},
{
name: 'DATABASE_TEST_URL',
value: 'postgres://postgres:postgres@users-db-review:5432/users_test'
},
{
name: 'NODE_ENV',
value: 'test'
},
{
name: 'TOKEN_SECRET',
value: 'changeme'
}
],
links: [
'users-db-review'
],
logConfiguration: {
logDriver: 'awslogs',
options: {
'awslogs-group': 'microservicemovies',
'awslogs-region': region
}
}
},
{
name: 'users-db-review',
image: `${accountID}.dkr.ecr.${region}.amazonaws.com\/microservicemovies\/users-db-review:${tag}`,
essential: true,
memoryReservation: 300,
cpu: 300,
portMappings: [
{
containerPort: 5432
}
],
environment: [
{
name: 'POSTGRES_USER',
value: 'postgres'
},
{
name: 'POSTGRES_PASSWORD',
value: 'postgres'
}
],
logConfiguration: {
logDriver: 'awslogs',
options: {
'awslogs-group': 'microservicemovies',
'awslogs-region': region
}
}
}
],
family: 'microservicemovies-review-users-td'
};
return taskDefinition;
}
module.exports = {
createUsersTaskDefinition
};
Most of this should be fairly straightforward since the container definition relates back to the Docker Compose file. Review the Task Definition Parameters guide for more info.
Two things to note:
- We set the host port for the users service to
0
so that a port is dynamically assigned when the Task is fired up. - We also configured logs, via LogConfiguration, to pipe logs to CloudWatch. To set up, we need to create a new Log Group. Simply navigate to CloudWatch, click “Logs” on the navigation pane, click the “Actions” drop-down button, and then select “Create log group”. Name the group
microservicemovies-review
.
Movies - movies-review_task.js
function createMoviesTaskDefinition(accountID, region, tag, family, revision) {
const taskDefinition = {
containerDefinitions: [
{
name: 'movies-service-review',
image: `${accountID}.dkr.ecr.${region}.amazonaws.com\/microservicemovies\/movies-service-review:${tag}`,
essential: true,
memoryReservation: 300,
cpu: 300,
portMappings: [
{
containerPort: 3000,
hostPort: 0,
protocol: 'tcp'
}
],
environment: [
{
name: 'DATABASE_URL',
value: 'postgres://postgres:postgres@movies-db-review:5432/movies_dev'
},
{
name: 'DATABASE_TEST_URL',
value: 'postgres://postgres:postgres@movies-db-review:5432/movies_test'
},
{
name: 'NODE_ENV',
value: 'test'
},
{
name: 'TOKEN_SECRET',
value: 'changeme'
}
],
links: [
'movies-db-review'
],
logConfiguration: {
logDriver: 'awslogs',
options: {
'awslogs-group': 'microservicemovies',
'awslogs-region': region
}
}
},
{
name: 'movies-db-review',
image: `${accountID}.dkr.ecr.${region}.amazonaws.com\/microservicemovies\/movies-db-review:${tag}`,
essential: true,
memoryReservation: 300,
cpu: 300,
portMappings: [
{
containerPort: 5432
}
],
environment: [
{
name: 'POSTGRES_USER',
value: 'postgres'
},
{
name: 'POSTGRES_PASSWORD',
value: 'postgres'
}
],
logConfiguration: {
logDriver: 'awslogs',
options: {
'awslogs-group': 'microservicemovies',
'awslogs-region': region
}
}
},
{
name: 'swagger-review',
image: `${accountID}.dkr.ecr.${region}.amazonaws.com\/microservicemovies\/swagger-review:${tag}`,
essential: true,
memoryReservation: 300,
cpu: 300,
portMappings: [
{
containerPort: 3001,
hostPort: 0,
protocol: 'tcp'
}
],
environment: [
{
name: 'NODE_ENV',
value: 'test'
}
],
logConfiguration: {
logDriver: 'awslogs',
options: {
'awslogs-group': 'microservicemovies',
'awslogs-region': region
}
}
}
],
family: 'microservicemovies-review-movies-td'
};
return taskDefinition;
}
module.exports = {
createMoviesTaskDefinition
};
The first two containers should be almost identical to the containers in the previous Task Definition. Review the Swagger container definition.
Web - web-review_task.js
function createWebTaskDefinition(accountID, region, tag, usersURL, moviesURL) {
const taskDefinition = {
containerDefinitions: [
{
name: 'web-service-review',
image: `${accountID}.dkr.ecr.${region}.amazonaws.com\/microservicemovies\/web-service-review:${tag}`,
essential: true,
memoryReservation: 300,
cpu: 300,
portMappings: [
{
containerPort: 9000,
hostPort: 0,
protocol: 'tcp'
}
],
environment: [
{
name: 'NODE_ENV',
value: 'test'
},
{
name: 'REACT_APP_USERS_SERVICE_URL',
value: usersURL
},
{
name: 'REACT_APP_MOVIES_SERVICE_URL',
value: moviesURL
}
],
logConfiguration: {
logDriver: 'awslogs',
options: {
'awslogs-group': 'microservicemovies',
'awslogs-region': region
}
}
},
],
family: 'microservicemovies-review-web-td'
};
return taskDefinition;
}
module.exports = {
createWebTaskDefinition
};
Notice how we’re passing in values for the environment variables. You could also use HashiCorp’s Vault project to better manage secrets and variables.
Create script to register Task Definitions
Next, add a new file called tasks.js to “ecs/scripts”:
const AWS = require('aws-sdk');
const createUsersTaskDefinition = require('../tasks/users-review_task').createUsersTaskDefinition;
const createMoviesTaskDefinition = require('../tasks/movies-review_task').createMoviesTaskDefinition;
const createWebTaskDefinition = require('../tasks/web-review_task').createWebTaskDefinition;
const port = require('./listener').getPort;
// globals
const AWS_ACCOUNT_ID = process.env.AWS_ACCOUNT_ID;
const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID;
const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY;
const AWS_USERNAME = process.env.AWS_USERNAME;
const AWS_CONFIG_REGION = 'us-west-2';
const SHORT_GIT_HASH = process.env.CIRCLE_SHA1.substring(0, 7);
const LOAD_BALANCER_DNS = 'http://microservicemovies-review-476947634.us-west-2.elb.amazonaws.com';
// config
AWS.config = new AWS.Config();
AWS.config.accessKeyId = AWS_ACCESS_KEY_ID;
AWS.config.secretAccessKey = AWS_SECRET_ACCESS_KEY;
AWS.config.region = AWS_CONFIG_REGION;
// init aws services
const ecs = new AWS.ECS();
const iam = new AWS.IAM();
// methods
function ensureAuthenticated() {
return new Promise((resolve, reject) => {
const params = { UserName: AWS_USERNAME };
iam.getUser(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function registerTaskDef(task) {
return new Promise((resolve, reject) => {
const params = task;
ecs.registerTaskDefinition(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function registerUsersTD() {
const task = createUsersTaskDefinition(AWS_ACCOUNT_ID, AWS_CONFIG_REGION, SHORT_GIT_HASH);
return registerTaskDef(task)
.then((res) => {
console.log('Task Registered!');
console.log(res.taskDefinition.taskDefinitionArn);
})
.catch((err) => {
console.log(err);
});
}
function registerMoviesTD() {
const task = createMoviesTaskDefinition(AWS_ACCOUNT_ID, AWS_CONFIG_REGION, SHORT_GIT_HASH);
return registerTaskDef(task)
.then((res) => {
console.log('Task Registered!');
console.log(res.taskDefinition.taskDefinitionArn);
})
.catch((err) => {
console.log(err);
});
}
function registerWebTD(usersURL, moviesURL) {
const task = createWebTaskDefinition(AWS_ACCOUNT_ID, AWS_CONFIG_REGION, SHORT_GIT_HASH, usersURL, moviesURL);
return registerTaskDef(task)
.then((res) => {
console.log('Task Registered!');
console.log(res.taskDefinition.taskDefinitionArn);
})
.catch((err) => {
console.log(err);
});
}
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
return port();
})
.then((port) => {
const usersURL = `${LOAD_BALANCER_DNS}:${port}/users`;
const moviesURL = `${LOAD_BALANCER_DNS}:${port}/movies`;
registerUsersTD();
registerMoviesTD();
registerWebTD(usersURL, moviesURL);
})
.catch((err) => { console.log(err); });
Here, we pulled in the getPort
function from the listener.js file, created the individual Task Definitions, and then passed the definitions to the registerTaskDefinition
function to register them on AWS. Review the official AWS SDK documentation for more info.
Update the Deploy
command in .circle/config.yml:
- run:
name: Deploy
command: |
npm install
node ecs/scripts/setup.js
sh ecs/scripts/ecr.sh
node ecs/scripts/tasks.js
Commit and push your code to GitHub. Once the Circle build passes, make sure the Task Definitions were created:
Also, make sure the REACT_APP_USERS_SERVICE_URL
and REACT_APP_MOVIES_SERVICE_URL
environment variables were added correctly to the microservicemovies-review-web-td
Task Definition by clicking the Task Definition name, selecting the latest revision, and then expanding the web-service-review
container:
(4) Create Target Groups
Target Groups are used to link the Application Load Balancer (ALB) to the container instances, which we just defined.
Create a new script in “scripts” called alb.js:
const AWS = require('aws-sdk');
// globals
const AWS_ACCOUNT_ID = process.env.AWS_ACCOUNT_ID;
const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID;
const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY;
const AWS_USERNAME = process.env.AWS_USERNAME;
const AWS_CONFIG_REGION = 'us-west-2';
const SHORT_GIT_HASH = process.env.CIRCLE_SHA1.substring(0, 7);
const VPC_ID='UPDATE_ME';
let USERS_TARGET_GROUP_ARN;
let MOVIES_TARGET_GROUP_ARN;
let WEB_TARGET_GROUP_ARN;
// config
AWS.config = new AWS.Config();
AWS.config.accessKeyId = AWS_ACCESS_KEY_ID;
AWS.config.secretAccessKey = AWS_SECRET_ACCESS_KEY;
AWS.config.region = AWS_CONFIG_REGION;
// init aws services
const elbv2 = new AWS.ELBv2();
const iam = new AWS.IAM();
// methods
function ensureAuthenticated() {
return new Promise((resolve, reject) => {
const params = { UserName: AWS_USERNAME };
iam.getUser(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function addTargetGroup(service, port, path) {
return new Promise((resolve, reject) => {
var params = {
Name: `${SHORT_GIT_HASH}-${service}`,
Port: port,
Protocol: 'HTTP',
VpcId: VPC_ID,
HealthCheckPath: path
};
elbv2.createTargetGroup(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
return addTargetGroup('users', '3000', '/users/ping');
})
.then((res) => {
USERS_TARGET_GROUP_ARN = res.TargetGroups[0].TargetGroupArn;
console.log('Target Group Added!');
return addTargetGroup('movies', '3000', '/movies/ping');
})
.then((res) => {
MOVIES_TARGET_GROUP_ARN = res.TargetGroups[0].TargetGroupArn;
console.log('Target Group Added!');
return addTargetGroup('web', '9000', '/');
})
.then((res) => {
WEB_TARGET_GROUP_ARN = res.TargetGroups[0].TargetGroupArn;
console.log('Target Group Added!');
})
.catch((err) => { console.log(err); });
Make sure to add the correct VPC ID in the script, and then take note of the addTargetGroup
function:
function addTargetGroup(service, port, path) {
return new Promise((resolve, reject) => {
var params = {
Name: `${SHORT_GIT_HASH}-${service}`,
Port: port,
Protocol: 'HTTP',
VpcId: VPC,
HealthCheckPath: path
};
elbv2.createTargetGroup(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
Again, the name
should be unique, so we used a portion of the Git commit SHA in it. The port
refers back to the container port, and the HealthCheckPath
is used by the load balancer to ensure the container is running.
| Task Name | Port | Path |
|---|---|---|
| users-service-review | 3000 | /users/ping |
| movies-service-review | 3000 | /movies/ping |
| web-review | 9000 | / |
After the Target Groups are created, we grabbed the returned Amazon Resource Name (ARN) and assigned it to a variable, which we’ll end up using shortly when we set up the Listeners. For more on registering Target Groups, review the official documentation.
Update the Deploy
command in .circle/config.yml:
- run:
name: Deploy
command: |
npm install
node ecs/scripts/setup.js
sh ecs/scripts/ecr.sh
node ecs/scripts/tasks.js
node ecs/scripts/alb.js
Commit and push your changes to GitHub. Make sure the Circle build passes. Within the EC2 Dashboard, click “Target Groups” on the navigation pane, and you should see three new Target Groups:
(5) Add the Listener and Rules
A Listener forwards traffic from the load balancer to a specific Target Group.
Add Listener
Add the following function to alb.js:
function addListener(port) {
return new Promise((resolve, reject) => {
var params = {
DefaultActions: [
{
TargetGroupArn: DEFAULT_TARGET_GROUP_ARN,
Type: 'forward'
}
],
LoadBalancerArn: LOAD_BALANCER_ARN,
Port: port,
Protocol: 'HTTP'
};
elbv2.createListener(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
And add then update the following variable:
const DEFAULT_TARGET_GROUP_ARN = 'UPDATE_ME';
const LOAD_BALANCER_ARN = 'UPDATE_ME';
Get open port
Before we can create a new Listener, we need to find an open port on the Application Load Balancer to add it to. So, import the getPort
function that we created before:
const port = require('./listener').getPort;
Update the promise chain:
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
return addTargetGroup('users', '3000', '/users/ping');
})
.then((res) => {
USERS_TARGET_GROUP_ARN = res.TargetGroups[0].TargetGroupArn;
console.log('Target Group Added!');
return addTargetGroup('movies', '3000', '/movies/ping');
})
.then((res) => {
MOVIES_TARGET_GROUP_ARN = res.TargetGroups[0].TargetGroupArn;
console.log('Target Group Added!');
return addTargetGroup('web', '9000', '/');
})
.then((res) => {
WEB_TARGET_GROUP_ARN = res.TargetGroups[0].TargetGroupArn;
console.log('Target Group Added!');
return addListener(MAX_PORT);
})
.then((res) => {
LISTENER_ARN = res.Listeners[0].ListenerArn;
console.log(`Listener added on port ${res.Listeners[0].Port}!`);
})
.catch((err) => { console.log(err); });
Be sure to add let LISTENER_ARN;
to the top as well.
Add rules
Finally, add some rules to the Listener:
function addRule(targetGroup, pattern, listener, priority) {
return new Promise((resolve, reject) => {
var params = {
Actions: [
{
TargetGroupArn: targetGroup,
Type: 'forward'
}
],
Conditions: [
{
Field: 'path-pattern',
Values: [pattern]
}
],
ListenerArn: listener,
Priority: priority
};
elbv2.createRule(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
Update the promise chain again:
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
return addTargetGroup('users', '3000', '/users/ping');
})
.then((res) => {
USERS_TARGET_GROUP_ARN = res.TargetGroups[0].TargetGroupArn;
console.log('Target Group Added!');
return addTargetGroup('movies', '3000', '/movies/ping');
})
.then((res) => {
MOVIES_TARGET_GROUP_ARN = res.TargetGroups[0].TargetGroupArn;
console.log('Target Group Added!');
return addTargetGroup('web', '9000', '/');
})
.then((res) => {
WEB_TARGET_GROUP_ARN = res.TargetGroups[0].TargetGroupArn;
console.log('Target Group Added!');
return port();
})
.then((port) => {
return addListener(port);
})
.then((res) => {
LISTENER_ARN = res.Listeners[0].ListenerArn;
console.log(`Listener added on port ${res.Listeners[0].Port}!`);
return addRule(USERS_TARGET_GROUP_ARN, '/users*', LISTENER_ARN, 1);
})
.then((res) => {
console.log('Rule Added!');
return addRule(MOVIES_TARGET_GROUP_ARN, '/movies*', LISTENER_ARN, 2);
})
.then((res) => {
console.log('Rule Added!');
return addRule(MOVIES_TARGET_GROUP_ARN, '/docs*', LISTENER_ARN, 3);
})
.then((res) => {
console.log('Rule Added!');
return addRule(WEB_TARGET_GROUP_ARN, '/*', LISTENER_ARN, 4);
})
.then((res) => {
console.log('Rule Added!');
})
.catch((err) => { console.log(err); });
Commit and push your changes. After the build passes, make sure the Listener was added to the Application Load Balancer:
Also, make sure the Listener has four Rules:
(6) Create a new Service
A Service is used to spin up and run a number of Tasks within the ECS Cluster. Since we’ve already pre-configured the Cluster and assigned EC2 instances to it, we can simply create a new Service that uses the Cluster resources.
Start by adding a new file called services.js to “ecs/scripts”:
const AWS = require('aws-sdk');
// globals
const AWS_ACCOUNT_ID = process.env.AWS_ACCOUNT_ID;
const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID;
const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY;
const AWS_USERNAME = process.env.AWS_USERNAME;
const AWS_CONFIG_REGION = 'us-west-2';
const SHORT_GIT_HASH = process.env.CIRCLE_SHA1.substring(0, 7);
// config
AWS.config = new AWS.Config();
AWS.config.accessKeyId = AWS_ACCESS_KEY_ID;
AWS.config.secretAccessKey = AWS_SECRET_ACCESS_KEY;
AWS.config.region = AWS_CONFIG_REGION;
// init aws services
const ecs = new AWS.ECS();
const iam = new AWS.IAM();
// methods
function ensureAuthenticated() {
return new Promise((resolve, reject) => {
const params = { UserName: AWS_USERNAME };
iam.getUser(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function addService(service) {
return new Promise((resolve, reject) => {
const params = service;
ecs.createService(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
})
.catch((err) => { console.log(err); });
Next, add a new directory to “ecs” called “services”, and then add the following files:
- users-review_service.js
- movies-review_service.js
- web-review_service.js
Users - users-review_service.js
function createUsersService(cluster, name, targetGroup) {
const params = {
cluster: cluster,
serviceName: name,
taskDefinition: 'microservicemovies-review-users-td',
loadBalancers: [
{
targetGroupArn: targetGroup,
containerName: "users-service-review",
containerPort: 3000
}
],
desiredCount: 1,
role: "ecsServiceRole"
};
return params;
}
module.exports = {
createUsersService
};
Movies - movies-review_service.js
function createMoviesService(cluster, name, targetGroup) {
const params = {
cluster: cluster,
serviceName: name,
taskDefinition: 'microservicemovies-review-movies-td',
loadBalancers: [
{
targetGroupArn: targetGroup,
containerName: "movies-service-review",
containerPort: 3000
}
],
desiredCount: 1,
role: "ecsServiceRole"
};
return params;
}
module.exports = {
createMoviesService
};
Web - web-review_service.js
function createWebService(cluster, name, targetGroup) {
const params = {
cluster: cluster,
serviceName: name,
taskDefinition: 'microservicemovies-review-web-td',
loadBalancers: [
{
targetGroupArn: targetGroup,
containerName: "web-service-review",
containerPort: 9000
}
],
desiredCount: 1,
role: "ecsServiceRole"
};
return params;
}
module.exports = {
createWebService
};
Update services.js:
const AWS = require('aws-sdk');
const createUsersService = require('../services/users-review_service').createUsersService;
const createMoviesService = require('../services/movies-review_service').createMoviesService;
const createWebService = require('../services/web-review_service').createWebService;
// globals
const AWS_ACCOUNT_ID = process.env.AWS_ACCOUNT_ID;
const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID;
const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY;
const AWS_USERNAME = process.env.AWS_USERNAME;
const AWS_CONFIG_REGION = 'us-west-2';
const SHORT_GIT_HASH = process.env.CIRCLE_SHA1.substring(0, 7);
const LOAD_BALANCER_ARN = 'arn:aws:elasticloadbalancing:us-west-2:046505967931:loadbalancer/app/microservicemovies-review/493be740ee6aea54';
const CLUSTER_NAME = 'microservicemovies-review';
let USERS_TARGET_GROUP_ARN;
let MOVIES_TARGET_GROUP_ARN;
let WEB_TARGET_GROUP_ARN;
// config
AWS.config = new AWS.Config();
AWS.config.accessKeyId = AWS_ACCESS_KEY_ID;
AWS.config.secretAccessKey = AWS_SECRET_ACCESS_KEY;
AWS.config.region = AWS_CONFIG_REGION;
// init aws services
const ecs = new AWS.ECS();
const iam = new AWS.IAM();
const elbv2 = new AWS.ELBv2();
// methods
function ensureAuthenticated() {
return new Promise((resolve, reject) => {
const params = { UserName: AWS_USERNAME };
iam.getUser(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function addService(service) {
return new Promise((resolve, reject) => {
const params = service;
ecs.createService(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function getTargetGroups() {
return new Promise((resolve, reject) => {
var params = {
LoadBalancerArn: LOAD_BALANCER_ARN
};
elbv2.describeTargetGroups(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
return getTargetGroups();
})
.then((res) => {
const groups = res.TargetGroups.filter((group) => {
return group.TargetGroupName.includes(SHORT_GIT_HASH);
});
for (const group of groups) {
if (group.TargetGroupName.includes('users')) {
USERS_TARGET_GROUP_ARN = group.TargetGroupArn;
}
if (group.TargetGroupName.includes('movies')) {
MOVIES_TARGET_GROUP_ARN = group.TargetGroupArn;
}
if (group.TargetGroupName.includes('web')) {
WEB_TARGET_GROUP_ARN = group.TargetGroupArn;
}
}
const userServiceParams = createUsersService(
CLUSTER_NAME, `${SHORT_GIT_HASH}-users`, USERS_TARGET_GROUP_ARN);
return addService(userServiceParams);
})
.then((res) => {
console.log('Service Added!');
const moviesServiceParams = createMoviesService(
CLUSTER_NAME, `${SHORT_GIT_HASH}-movies`, MOVIES_TARGET_GROUP_ARN);
return addService(moviesServiceParams);
})
.then((res) => {
console.log('Service Added!');
const webServiceParams = createWebService(
CLUSTER_NAME, `${SHORT_GIT_HASH}-web`, WEB_TARGET_GROUP_ARN);
return addService(webServiceParams);
})
.then((res) => {
console.log('Service Added!');
})
.catch((err) => { console.log(err); });
Update the Deploy
command in .circle/config.yml:
- run:
name: Deploy
command: |
npm install
node ecs/scripts/setup.js
sh ecs/scripts/ecr.sh
node ecs/scripts/tasks.js
node ecs/scripts/alb.js
node ecs/scripts/services.js
Again, commit and push to GitHub, and then ensure:
- The Circle build passes
- The Services were created on the
microservicemovies-review
Cluster - Each Service has a running Task associated with it
- Three Target Groups were created, each with health targets
- A Listener was added to the Application Load Balancer with four Rules
- Logs were added to CloudWatch
- The following endpoints work-
- http://LOAD_BALANCER_DNS:LISTENER_PORT
- http://LOAD_BALANCER_DNS:LISTENER_PORT/users/ping
- http://LOAD_BALANCER_DNS:LISTENER_PORT/movies/ping
Testing
To run the end-to-end tests, first change each of the URLs in tests/sample.test.js from http://localhost:3007
to http://LOAD_BALANCER_DNS:LISTENER_PORT
, and then run the tests locally:
$ testcafe firefox tests/**/*.js
They should pass.
Teardown
Next, let’s add a script to handle the tearing down of the AWS resources after testing is complete.
Main Steps:
- Remove Listener
- Remove Target Groups
- Remove Services
Start by adding a new script called teardown.js to “ecs/scripts”.
(1) Remove Listener
const AWS = require('aws-sdk');
// globals
const AWS_ACCOUNT_ID = process.env.AWS_ACCOUNT_ID;
const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID;
const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY;
const AWS_USERNAME = process.env.AWS_USERNAME;
const AWS_CONFIG_REGION = 'us-west-2';
const LOAD_BALANCER_ARN = 'UPDATE_ME';
const ARGS = process.argv.slice(2);
const USAGE_MESSAGE = '\nusage:\n teardown.js LISTENER_PORT COMMIT_SHA\n';
if (!ARGS[0] || !ARGS[1]) {
console.log(USAGE_MESSAGE);
return;
}
const SHORT_GIT_HASH = ARGS[1].substring(0, 7);
// config
AWS.config = new AWS.Config();
AWS.config.accessKeyId = AWS_ACCESS_KEY_ID;
AWS.config.secretAccessKey = AWS_SECRET_ACCESS_KEY;
AWS.config.region = AWS_CONFIG_REGION;
// init aws services
const iam = new AWS.IAM();
const elbv2 = new AWS.ELBv2();
// methods
function ensureAuthenticated() {
return new Promise((resolve, reject) => {
const params = { UserName: AWS_USERNAME };
iam.getUser(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function getListeners() {
return new Promise((resolve, reject) => {
var params = {
LoadBalancerArn: LOAD_BALANCER_ARN
};
elbv2.describeListeners(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function removeListener(listener) {
return new Promise((resolve, reject) => {
var params = {
ListenerArn: listener
};
elbv2.deleteListener(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
return getListeners();
})
.then((res) => {
const listener = res.Listeners.filter((listener) => {
return parseInt(listener.Port) === parseInt(ARGS[0]);
})[0];
if (!listener) {
throw new Error('Listener does not exist.');
}
return removeListener(listener.ListenerArn);
})
.then((res) => {
console.log('Listener Removed!');
})
.catch((err) => { console.log(err); });
Here, we grabbed all Listeners in getListeners
and then filtered them by the provided listener port. From there, we deleted the listener. Be sure to update LOAD_BALANCER_ARN
before continuing.
Test this out:
$ node ecs/scripts/teardown.js LISTENER_PORT COMMIT_SHA
(2) Remove Target Groups
Add the following two functions to teardown.js:
function getTargetGroups() {
return new Promise((resolve, reject) => {
var params = {};
elbv2.describeTargetGroups(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function removeTargetGroup(targetgroup) {
return new Promise((resolve, reject) => {
var params = { TargetGroupArn: targetgroup };
elbv2.deleteTargetGroup(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
Then update the promise chain:
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
return getListeners();
})
.then((res) => {
const listener = res.Listeners.filter((listener) => {
return parseInt(listener.Port) === parseInt(ARGS[0]);
})[0];
if (!listener) {
throw new Error('Listener does not exist.');
}
return removeListener(listener.ListenerArn);
})
.then((res) => {
console.log('Listener Removed!');
return getTargetGroups();
})
.then((res) => {
const targets = res.TargetGroups.filter((group) => {
return group.TargetGroupName.includes(SHORT_GIT_HASH);
});
if (!targets.length) {
throw new Error('Targets do not exist.');
}
const promises = targets.map((target) => {
return removeTargetGroup(target.TargetGroupArn);
});
return Promise.all(promises);
})
.then(() => {
console.log('Target Groups Removed!');
})
.catch((err) => { console.log(err); });
(3) Remove Services
Finally, add the following functions to remove the Services:
function updateServiceCount(serviceName) {
return new Promise((resolve, reject) => {
var params = {
service: serviceName,
desiredCount: 0,
cluster: CLUSTER_NAME
};
ecs.updateService(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
function removeService(serviceName) {
return new Promise((resolve, reject) => {
var params = {
service: serviceName,
cluster: CLUSTER_NAME
};
ecs.deleteService(params, (err, data) => {
if (err) { reject(err); }
resolve(data);
});
});
}
Add the variables to the top:
const CLUSTER_NAME = 'microservicemovies-review';
const ecs = new AWS.ECS();
Then, update the promise chain:
// main
return ensureAuthenticated()
.then((data) => {
console.log(`Welcome ${data.User.UserName}!`);
return getListeners();
})
.then((res) => {
const listener = res.Listeners.filter((listener) => {
return parseInt(listener.Port) === parseInt(ARGS[0]);
})[0];
if (!listener) {
throw new Error('Listener does not exist.');
}
return removeListener(listener.ListenerArn);
})
.then((res) => {
console.log('Listener Removed!');
return getTargetGroups();
})
.then((res) => {
const targets = res.TargetGroups.filter((group) => {
return group.TargetGroupName.includes(SHORT_GIT_HASH);
});
if (!targets.length) {
throw new Error('Targets do not exist.');
}
const promises = targets.map((target) => {
return removeTargetGroup(target.TargetGroupArn);
});
return Promise.all(promises);
})
.then(() => {
console.log('Target Groups Removed!');
return updateServiceCount(`${SHORT_GIT_HASH}-users`);
})
.then(() => {
console.log('Service Updated!');
return removeService(`${SHORT_GIT_HASH}-users`);
})
.then(() => {
console.log('Service Removed!');
return updateServiceCount(`${SHORT_GIT_HASH}-movies`);
})
.then(() => {
console.log('Service Updated!');
return removeService(`${SHORT_GIT_HASH}-movies`);
})
.then((res) => {
console.log('Service Removed!');
return updateServiceCount(`${SHORT_GIT_HASH}-web`);
})
.then((res) => {
console.log('Service Updated!');
return removeService(`${SHORT_GIT_HASH}-web`);
})
.then((res) => {
console.log('Service removed!');
})
.catch((err) => { console.log(err); });
Test it out again:
$ node ecs/scripts/teardown.js LISTENER_PORT COMMIT_SHA
You should see:
Welcome AWS_USERNAME!
Listener Removed!
Target Groups Removed!
Service Updated!
Service Removed!
Service Updated!
Service Removed!
Service Updated!
Service removed!
Make sure all associated AWS Resources were removed as well.
Conclusion and Next Steps
That’s it!
Development Workflow
Let’s quickly review the development workflow…
- Local development
- Create a new feature branch from the master branch
- Make code changes
- Commit and push code to GitHub
- Continuous integration
- Open a new PR against the development branch
- A new build is then triggered on Circle CI
- If the build passes, manually merge the PR
- A new build is triggered again on Circle CI
- If the build passes, deployment occurs…
- Deployment on AWS via deployment scripts
- Testing
- Update then run the end-to-end tests
- Teardown
- Run the teardown script
What’s next?
Did you test out the Swagger docs? They don’t work. What’s happening? Fix this on your own.
Developers will inevitably forget to run the teardown script. Configure an AWS Lambda function to run nightly to tear down all AWS resources associated with the on-demand test environments.
Add Vault and Consul into the mix to handle secrets and environment variables.
Grab the final code from the v4 tag of the microservice-movies repo. Please add questions and/or comments below. Cheers!


Sinn Féin gets further £0.5m from benefactor who 'never mentioned Irish politics'
THE man responsible for giving Sinn Féin what is thought to be Northern Ireland's largest ever political donation "never even mentioned Irish politics", according to a friend.
Dave Morton said the late Billy Hampton, a former market trader, left his fortune to Sinn Féin "out of spite" and "to say 'up you' to the British establishment".
The remarks came as it emerged that Sinn Féin has received a further tranche of £500,000 from Mr Hampton, on top of the £1.5 million already pledged posthumously to the party.
The 82-year-old died in January last year in Pembrokeshire in Wales, where he was living in a nursing home. His ashes were later buried in west Belfast in a "solemn ceremony" attended by Gerry Adams.
Mr Hampton's gravestone in Hannahstown Cemetery reads: "True friend of Ireland. Remembered by his friends and comrades in Sinn Féin."
According to the BBC, additional money has been paid to the party and is expected to be confirmed when the Electoral Commission publishes updated information on political donations at the end of the month.
Mr Morton told the broadcaster that Mr Hampton was unhappy during his life because he could not access all the inheritance he felt he was entitled in a single lump sum.
He had inherited money from his father, Ted, who had extensive business interests in Fenstanton, Cambridgeshire.
Mr Morton said the deceased man left his his own fortune to Sinn Féin "out of spite" and "to say 'up you' to the British establishment."
"He had a really wicked sense of humour and he would have thought that was funny," Mr Morton said.
"Basically no one helped him, and he got it into his head that the establishment was out to get him. He felt round every corner someone was watching him which they weren't."
According to Mr Morton, the Sinn Féin benefactor "never even mentioned Irish politics".
"All he said was when he was over in Ireland he generally had a good time," he said.
Subscribe to our most read headlines in the evening newsletter


channel.986b6b __
channel.fbfbfb __
channel.242f25 __
channel.59eb94 __
channel.fb5e5e __
octree __
channel.fd0000 __
binary __
channel.01fa64 __
channel.a00000 __
channel.867d83 __


Indian Legal Dataset Analysis
IIT Delhi
National
KL Divergence
Mean Disposal Time
Std Dev Disposal Time
Zero Day Disposal Percentage
Tail Bounds - 3 months
Tail Bounds - 6 months
Tail Bounds - 1 year
Tail Bounds - 3 years
States
Andhra Pradesh
Assam
Bihar
Chattisgarh
Delhi
Goa
Gujarat
Harayana
Himachal Pradesh
Jammu and Kashmir
Jharkhand
Karnataka
Kerala
Lakshwadeep
Madhya Pradesh
Maharashtra
Odisha
Punjab
Rajasthan
Sikkim
Tamil Nadu
Telangana
Tripura
Uttarakhand
Uttar Pradesh
West Bengal
Compare
Rankings
by Zero Day Disposals
by KL Divergence
Ranking 3
FAQ
About Us
Nandurbar
Dhule
Jalgaon
Chittoor
East Godavari
Belgaum
Bagalkot
Bengaluru rural
Gulbarga
Nandurbar
Dhule
Jalgaon
Chittoor
East Godavari
Belgaum
Bagalkot
Bengaluru rural
Gulbarga


Choose the bid type of your spot requests.
Persistent: Ensure that requests are resubmitted after its instance is terminated by you or by Amazon EC2, until you cancel the bid request. This enables you to automate launching Spot instances any time the Spot price is below your maximum price.
Volatile: Requests are not resubmitted after its instance is terminated.


Azure Kinect with Open3D¶
Azure Kinect is only officially supported on Windows and Ubuntu 18.04.
Installation¶
Install the Azure Kinect SDK¶
Follow the guide to install the Azure Kinect SDK (K4A).
On Ubuntu, you’ll need to set up a udev rule to use the Kinect camera without
sudo
, follow
this instruction.
After installation, you may run k4aviewer
from the Linux terminal or
k4aviewer.exe
on Windows to make sure that the device is working.
Currently, Open3D supports the Azure Kinect SDK version v1.2.0
, though future
versions might also be compatible.
If you’re using Ubuntu 16.04, the Azure Kinect SDK is not officially supported. We have an unofficial workaround, see Unofficial Ubuntu 16.04 workaround for details.
Using Open3D from Pip or Conda¶
If you’re using Open3D installed via Pip or Conda, Open3D Azure Kinect features shall work out-of-the box if K4A is installed in the system in the recommended way. Open3D will try to load the K4A dynamic library automatically at runtime, when a K4A related feature within Open3D is used.
On Ubuntu, the default search path follows the Linux convention.
On Windows, Open3D will try to load the shared library from the default
installation path. For example, for K4A v1.2.0
, the default path is
C:\Program Files\Azure Kinect SDK v1.2.0
. If this doesn’t work, copy
depthengine_x_x.dll
, k4a.dll
and k4arecord.dll
to where Open3D
Python module is installed if you’re using Open3D with Python, or to the same
directory as your C++ executable.
You can get Open3D’s Python module path by:
python -c "import open3d as o3d; import os; print(os.path.dirname(o3d.__file__))"
Compile from Source¶
To build Open3D from source with K4A support, set BUILD_AZURE_KINECT=ON
at
CMake config step. That is
cmake -DBUILD_AZURE_KINECT=ON -DOTHER_FLAGS ..
Open3D Azure Kinect Viewer¶
Open3D Azure Kinect Viewer is used for previewing RGB and depth image stream captured by the Azure Kinect sensor.
Open3D provides Python and C++ example code of Azure Kinect viewer. Please
see examples/Cpp/AzureKinectViewer.cpp
and
examples/Python/ReconstructionSystem/sensors/azure_kinect_viewer.py
for details.
We’ll use the Python version as an example.
python examples/Python/ReconstructionSystem/sensors/azure_kinect_viewer.py --align_depth_to_color
When recording at a higher resolution at a high framerate, sometimes it is helpful to use the raw depth image without transformation to reduce computation.
python examples/Python/ReconstructionSystem/sensors/azure_kinect_viewer.py
When the visualizer window is active, press ESC
to quit the viewer.
You may also specify the sensor config with a json
file.
python examples/Python/ReconstructionSystem/sensors/azure_kinect_viewer.py --config config.json
An sensor config will look like the following. For the full list of available configs, refer to here.
{
"camera_fps" : "K4A_FRAMES_PER_SECOND_30",
"color_format" : "K4A_IMAGE_FORMAT_COLOR_MJPG",
"color_resolution" : "K4A_COLOR_RESOLUTION_720P",
"depth_delay_off_color_usec" : "0",
"depth_mode" : "K4A_DEPTH_MODE_WFOV_2X2BINNED",
"disable_streaming_indicator" : "false",
"subordinate_delay_off_master_usec" : "0",
"synchronized_images_only" : "false",
"wired_sync_mode" : "K4A_WIRED_SYNC_MODE_STANDALONE"
}
Open3D Azure Kinect Recorder¶
Open3D Azure Kinect Viewer is used for recording RGB and depth image stream to a MKV file.
Open3D provides Python and C++ example code of Azure Kinect viewer. Please
see examples/Cpp/AzureKinectRecord.cpp
and
examples/Python/ReconstructionSystem/sensors/azure_kinect_recorder.py
for details.
We’ll use the Python version as an example.
python examples/Python/ReconstructionSystem/sensors/azure_kinect_recorder.py --output record.mkv
You may optionally specify the camera config when running the recorder script.
When the visualizer window is active, press SPACE
start or pause the
recording, press ESC
to quit the recorder.
Open3D Azure Kinect MKV Reader¶
The recorded MKV file uses K4A’s custom format which contains both RGB and depth information. Regular video player may only support playing back the color channel or not supporting the format at all. To view the customized MKV file, use the Open3D Azure Kinect MKV Reader.
Open3D provides Python and C++ example code of Open3D Azure Kinect MKV Reader.
Please see examples/Cpp/AzureKinectMKVReader.cpp
and
examples/Python/ReconstructionSystem/sensors/azure_kinect_mkv_reader.py
for details.
python examples/Python/ReconstructionSystem/sensors/azure_kinect_mkv_reader.py --input record.mkv
Note that even though the recorder records the unaligned raw depth image, the reader can correctly wrap the depth image to align with the color image.
To convert the MKV video to color and depth image frames, specify the --output
flag.
python examples/Python/ReconstructionSystem/sensors/azure_kinect_mkv_reader.py --input record.mkv --output frames
Unofficial Ubuntu 16.04 workaround¶
The Azure Kinect SDK is not officially supported on Ubuntu 16.04. We provide unofficial support for experimental purposes.
Using Open3D Python packages¶
For Python Open3D, run
pip install open3d_azure_kinect_ubuntu1604_fix
The open3d_azure_kinect_ubuntu1604_fix
package contains 4 shared libs:
libstdc++.so
: copied from ubuntu 18.04, needed bylibdepthengine.so
libdepthengine.so
: copied from K4A installation on ubuntu 18.04libk4a.so
: build from source on ubuntu 16.04libk4arecord.so
: build from source on ubuntu 16.04
The open3d_azure_kinect_ubuntu1604_fix
will preload the shared libs and set
LD_LIBRARY_PATH
which are then used by dlopen
when the Kinect library
is loaded from the compiled module.
After installing open3d_azure_kinect_ubuntu1604_fix
, import Open3D as usual
with import open3d
, Open3D will try to load the shared libraries at
initialization time in __init__.py
.
Compiling Open3D from source¶
First, install open3d_azure_kinect_ubuntu1604_fix
package in your Python
environment. This package contains the headers that are required for compiling
Open3D with Azure Kinect support.
When building Open3D from source, set the flag -DBUILD_AZURE_KINECT=ON
at CMake configure time. CMake will then try to detect the location of the
open3d_azure_kinect_ubuntu1604_fix
package using the Python
executable
available from the current PATH
. Therefore, when running CMake, make sure
that the same Python environment where open3d_azure_kinect_ubuntu1604_fix
was installed is activated.
If you build a C++ binary, you’ll still need to ensure that
LD_LIBRARY_PATH
contains the directory which contains libstdc++.so
and
libdepthengine.so
at runtime. If you build the compiled Open3D Python
module, Open3D’s __init__.py
will try to import
open3d_azure_kinect_ubuntu1604_fix
to append the LD_LIBRARY_PATH
automatically.


{# Copyright (c) 2015 The Pycroft Authors. See the AUTHORS file. This file is part of the Pycroft project and licensed under the terms of the Apache License, Version 2.0. See the LICENSE file for details. #} {% macro topnav_link(blueprint_name, endpoint_name, text) -%}


Installation¶
It is recommended that GOSINT be installed on a GNU/Linux system with the latest version of the Go language available.
The document was prepared specifically for Ubuntu Server 16.04.2 LTS.
Warnings¶
Package managers may not provide up to date versions of the software and should be tested to ensure compatibility.
It is strongly recommended that Go be installed with the latest version from https://golang.org/dl/
Package managers may name packages differently depending on the specific package manager or OS release repository.
For example, php-fpm may not exist; php7.0-fpm may be the correct name of the package
Prerequisites¶
GOSINT requires
- A working and up to date Go environment
- Mongo DB (Community Edition is ok)
- A reverse proxy/web server (NGINX preferred)
- PHP
You can use your preferred package manager to install most of these environments and applications. For aptitude:
sudo apt-get install mongodb php-fpm nginx git
- Install MongoDB and ensure it is ONLY listening on your local loopback interface (<IP_ADDRESS>/localhost) if you are running it on the same host as GOSINT.
- Allowing your database to listen on any externally facing ports is a security risk, and should not be done without proper precautions taken to prevent unauthorized access.
- You can use aptitude to install an older version with the command
sudo apt-get install mongodb
, or you can follow the instructions at https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/ to install a more up to date version from the MongoDB repositories.
- Install PHP (v5 or higher) and verify the installation was successful.
- Install NGINX (or your preferred web server).
- You will need to configure NGINX to listen on a public interface at a port you specify.
- It is recommended that you install a valid certificate for HTTPS and enable some form of authorization (local auth or LDAP) to prevent unauthorized access to GOSINT.
- Please find the base nginx configuration file at NGINX Configuration
Step by Step¶
- Create a user for GOSINT to run on with minimal privileges.
This user will run the backend binary which is responsible for pulling indicators and exposing an API for the frontend to use:
sudo useradd -m gosint sudo su gosint
- Install and test the Go environment.
- Download the GNU/Linux Go 1.8 package.
- 64 Bit:
cd ~ && wget https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz
- 32 Bit:
cd ~ && wget https://storage.googleapis.com/golang/go1.8.linux-386.tar.gz
- Decompress archive.
- 64 Bit:
tar zxvf go1.8.linux-amd64.tar.gz
- 32 Bit:
tar zxvf go1.8.linux-386.tar.gz
Create project workspace and setup the environment:
mkdir ~/projects export GOROOT=$HOME/go export PATH=$PATH:$GOROOT/bin export GOPATH=$HOME/projects export GOBIN=$GOPATH/bin export PATH=$GOPATH:$GOBIN:$PATH
Test Go environment using the instructions at https://golang.org/doc//install/source#testing
Install godep vendor management:
go get github.com/tools/godep go install github.com/tools/godep
Clone GOSINT repository into your
src
directory in your go environment and build it:cd ~/projects/src git clone https://github.com/ciscocsirt/GOSINT cd GOSINT godep go build -o gosint chmod +x gosint
Test GOSINT build:
./gosint
- GOSINT will start and then error out trying to connect to the database if MongoDB has not yet been installed.
- For ease of use, it is recommended you use a terminal multiplexer such as GNU screen to keep the terminal open that GOSINT is running in: screen -dm ./gosint
- If an alternate IP is needed to be specified for the Mongo DB server, you can use the flag -mongo to change it from the default <IP_ADDRESS>.
- Type ./gosint -h for a list of available flags.
- If GOSINT starts up without any errors, and you have NGINX setup properly, you should now be able to navigate to the address and port specified in your webserver configuration and access the GOSINT web interface.
NGINX Configuration¶
server {
listen 80;
root /home/gosint/projects/src/gosint/website;
index index.php index.html index.htm;
try_files $uri $uri/ @apachesite;
server_name someserver.yourcompany.com;
gzip on;
gzip_proxied any;
gzip_types
text/css
text/javascript
text/xml
text/plain
application/javascript
application/x-javascript
application/json;
#location / {
# try_files $uri $uri/ =404;
#}
error_page 404 /404.html;
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
location @apachesite {
proxy_pass http://localhost:8000;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
Updates¶
Updating is simple and encouraged as bugs are reported and fixed or new features are added. To update your instance of GOSINT, pull the latest version of GOSINT from the repository and re-run the build command to compile the updated binary:
godep go build -o gosint


A module for managing parser functions. Each specified function is added to a central module that then gets included into the Scope class.
@api public
Return the number of arguments a function expects.
@param [Symbol] name the function @return [Integer] The arity of the function. See {newfunction} for
the meaning of negative values.
@api public
# File lib/puppet/parser/functions.rb, line 221 def self.arity(name) func = get_function(name) func ? func[:arity] : -1 end
Accessor for singleton autoloader
@api private
# File lib/puppet/parser/functions.rb, line 35 def self.autoloader @autoloader ||= Puppet::Util::Autoload.new( self, "puppet/parser/functions", :wrap => false ) end
Get the module that functions are mixed into corresponding to an environment
@api private
# File lib/puppet/parser/functions.rb, line 45 def self.environment_module(env = nil) if env and ! env.is_a?(Puppet::Node::Environment) env = Puppet::Node::Environment.new(env) end @modules.synchronize { @modules[ (env || Environment.current || Environment.root).name ] ||= Module.new } end
Determine if a function is defined
@param [Symbol] name the function
@return [Symbol, false] The name of the function if it’s defined,
otherwise false.
@api public
# File lib/puppet/parser/functions.rb, line 167 def self.function(name) name = name.intern func = nil @functions.synchronize do unless func = get_function(name) autoloader.load(name, Environment.current) func = get_function(name) end end if func func[:name] else false end end
# File lib/puppet/parser/functions.rb, line 185 def self.functiondocs autoloader.loadall ret = "" merged_functions.sort { |a,b| a[0].to_s <=> b[0].to_s }.each do |name, hash| ret += "#{name}\n#{"-" * name.to_s.length}\n" if hash[:doc] ret += Puppet::Util::Docs.scrub(hash[:doc]) else ret += "Undocumented.\n" end ret += "\n\n- *Type*: #{hash[:type]}\n\n" end ret end
Create a new Puppet DSL function.
**The {newfunction} method provides a public API.**
This method is used both internally inside of Puppet to define parser functions. For example, template() is defined in {file:lib/puppet/parser/functions/template.rb template.rb} using the {newfunction} method. Third party Puppet modules such as [stdlib](forge.puppetlabs.com/puppetlabs/stdlib) use this method to extend the behavior and functionality of Puppet.
See also [Docs: Custom Functions](docs.puppetlabs.com/guides/custom_functions.html)
@example Define a new Puppet DSL Function
>> Puppet::Parser::Functions.newfunction(:double, :arity => 1, :doc => "Doubles an object, typically a number or string.", :type => :rvalue) {|i| i[0]*2 } => {:arity=>1, :type=>:rvalue, :name=>"function_double", :doc=>"Doubles an object, typically a number or string."}
@example Invoke the double function from irb as is done in RSpec examples:
>> scope = Puppet::Parser::Scope.new_for_test_harness('example') => Scope() >> scope.function_double([2]) => 4 >> scope.function_double([4]) => 8 >> scope.function_double([]) ArgumentError: double(): Wrong number of arguments given (0 for 1) >> scope.function_double([4,8]) ArgumentError: double(): Wrong number of arguments given (2 for 1) >> scope.function_double(["hello"]) => "hellohello"
@param [Symbol] name the name of the function represented as a ruby Symbol.
The {newfunction} method will define a Ruby method based on this name on the parser scope instance.
@param [Proc] block the block provided to the {newfunction} method will be
executed when the Puppet DSL function is evaluated during catalog compilation. The arguments to the function will be passed as an array to the first argument of the block. The return value of the block will be the return value of the Puppet DSL function for `:rvalue` functions.
@option options [:rvalue, :statement] :type (:statement) the type of function.
Either `:rvalue` for functions that return a value, or `:statement` for functions that do not return a value.
@option options [String] :doc (”) the documentation for the function.
This string will be extracted by documentation generation tools.
@option options [Integer] :arity (-1) the
[arity](http://en.wikipedia.org/wiki/Arity) of the function. When specified as a positive integer the function is expected to receive _exactly_ the specified number of arguments. When specified as a negative number, the function is expected to receive _at least_ the absolute value of the specified number of arguments incremented by one. For example, a function with an arity of `-4` is expected to receive at minimum 3 arguments. A function with the default arity of `-1` accepts zero or more arguments. A function with an arity of 2 must be provided with exactly two arguments, no more and no less. Added in Puppet 3.1.0.
@return [Hash] describing the function.
@api public
# File lib/puppet/parser/functions.rb, line 121 def self.newfunction(name, options = {}, &block) name = name.intern Puppet.warning "Overwriting previous definition for function #{name}" if get_function(name) arity = options[:arity] || -1 ftype = options[:type] || :statement unless ftype == :statement or ftype == :rvalue raise Puppet::DevError, "Invalid statement type #{ftype.inspect}" end # the block must be installed as a method because it may use "return", # which is not allowed from procs. real_fname = "real_function_#{name}" environment_module.send(:define_method, real_fname, &block) fname = "function_#{name}" environment_module.send(:define_method, fname) do |*args| if args[0].is_a? Array if arity >= 0 and args[0].size != arity raise ArgumentError, "#{name}(): Wrong number of arguments given (#{args[0].size} for #{arity})" elsif arity < 0 and args[0].size < (arity+1).abs raise ArgumentError, "#{name}(): Wrong number of arguments given (#{args[0].size} for minimum #{(arity+1).abs})" end self.send(real_fname, args[0]) else raise ArgumentError, "custom functions must be called with a single array that contains the arguments. For example, function_example([1]) instead of function_example(1)" end end func = {:arity => arity, :type => ftype, :name => fname} func[:doc] = options[:doc] if options[:doc] add_function(name, func) func end
Reset the list of loaded functions.
@api private
# File lib/puppet/parser/functions.rb, line 20 def self.reset @functions = Hash.new { |h,k| h[k] = {} }.extend(MonitorMixin) @modules = Hash.new.extend(MonitorMixin) # Runs a newfunction to create a function for each of the log levels Puppet::Util::Log.levels.each do |level| newfunction(level, :doc => "Log a message on the server at level #{level.to_s}.") do |vals| send(level, vals.join(" ")) end end end
Generated with the Darkfish Rdoc Generator 2.


Team Member
Name:
Intelligence:
1
2
3
4
5
6
7
8
9
Strength:
1
2
3
4
5
6
7
8
9
Endurance:
1
2
3
4
5
6
7
8
9
Spicy Food Tolerance:
1
2
3
4
5
6
7
8
9
Candidate Application
Name:
Intelligence:
1
2
3
4
5
6
7
8
9
Strength:
1
2
3
4
5
6
7
8
9
Endurance:
1
2
3
4
5
6
7
8
9
Spicy Food Tolerance:
1
2
3
4
5
6
7
8
9
Calculate
Score's List


Access
Java Programming Resources for CSC 120 via this link.
CSC 120 Lab 7
In this lab we will practice with for
loops and drawing repetitive patterns with loops
- Create a new NetBeans project called Lab7a, and copy source
files to it.
- File | New Project, etc. Make sure the Project Location is in
your CSC120\Java folder, and the Project Name is Lab7a.
- Right-click on the following link and save this .zip file in your CSC120 folder:
Lab7aStarterFiles.zip
- Use Windows Explorer to Extract All from the .zip file you saved in the previous
step, and then copy BOTH files that were extracted,
and paste them in the src folder of your project.
- Run the program.
- Run the program as it is given. Five lines are displayed in the Text Area
on screen. Then examine the program in "Source" view. Look at the
source code of the program -- there is only one
outputArea.append
statement inside the
for
loop
(it is spread across several lines), which
is executed multiple times because of the loop in the program.
- Make changes to the given program.
- Make the value of the variable
down
take on the values 7, 14, 21, 28, 35 as the
program is executed. Use the same type of code to do this as is used in the program to
cause over
take on a series of values. If you do this properly, the values
of down
shown above will be printed in the Text Area on screen.
Run your program and
see if you can obtain correct output.
- Make the program print 20 lines of output in the Text Area. Run the program and count to make
sure that 20 lines of output are printed.
- Add code to change other variables.
- Inside the body of the constructor method for the
MUFrame
class,
declare two more Integer
variables called width
and height
.
- Make these variables take
on the following values:
width
should start with 700, and repeatedly decrease
by 10 each time through the loop; height
should start with 1, and double itself
each time through the loop.
- Add code to the
outputArea.append
statement inside the loop so that all four
variable values
are printed in the Text Area on one line, with appropriate labels preceding each numeric
value.
- Run the program, test, and fix as needed.
- Create a new NetBeans project called Lab7b, and copy source
files to it.
- Close the current Lab7a project in NetBeans.
- Choose "File | New Project, etc." from the NetBeans top-level menu bar. Make sure
the Project Location is in your CSC120\Java folder, and the Project Name is Lab7b.
- Right-click on the following link and save this .zip file in your CSC120 folder:
Lab7bStarterFiles.zip
- Use Windows Explorer to Extract All from the .zip file you saved in the previous
step, and then copy ALL THREE files that were extracted,
and paste them in the src folder of your project.
- In NetBeans, open the file MUFrame.java, and in "Design" view, change the title of the frame
so that it displays your name instead of "Samantha Student".
- Run the program as it is given. Notice that 7 blue bars are displayed in the output window.
Examine the MUPanel.java file and make sure you understand why the pattern that is drawn shows up as
it does.
- Draw patterns using loops.
- Change the code in the given
MUPanel
so that the patterns that may be viewed
by clicking THIS LINK
are drawn in the output window.
- Keep changing the program from pattern to pattern as you go along -- you don't need to keep
each version of the program in a separate NetBeans project (unless you want to do that).
- Hint: each trip through the loop, after all the drawing is done, changing the values
of
over
, down
, width
and/or
height
will cause the next shape to be drawn differently the
next time through the loop. What is changing in the pattern from
shape to shape? Those variables are what need to be changed inside
the loop body.
- Hint: to make every other color (or every third color, etc.) change to a
different color, you'll need to use the
% operation to calculate the remainder of a division problem.
- Ask your instructor
for help with any pattern you have trouble generating with a loop.
- Submit your Lab7b project for grading.
Turn in ONLY the final version of the Lab7b project into
the
D2L dropbox for this assignment. Be sure to submit a
screen capture of your final pattern along with the .zip file
of the entire NetBeans project for this part of the lab.
Grading Rubric:
Lab07 Grading Rubric


The purpose of web scraping is to extract online data in a large scale by automation. Nevertheless, some of the website content may be under copyright protection called Digital Millennium Copyright Act (even pornography is protected by DMCA). This adds a little complication to our projects. Even open source data is subjected to fair use, yet the interpretation of fair use is always trivial. There have been many lawsuits around web scraping. Sometimes the scrapers lose sometimes the scrapers win. The actual verdict varies case by case. Law is equal to anyone. Except those with deep pockets are more equal than the others. Nobody really wants to be that sucker who pays the fine. I am not licensed to offer any practical legal counselling but I do have a couple of useful hints for you.
In most cases, you can find terms of web scraping in the declaration of the content rights or other similar legal sections. These terms are long, sophisticated and incomprehensible (insert the joke here, can you speak English 😁 ). The most straightforward way is to add /robots.txt behind any website domain. You will find out the policy set by the website administrator. Usually it tells you what the restrictions are.
For instance
https://www.lemonde.fr/robots.txt
In English, all user agents (* refers to all) are allowed for the following 2 sub directories.
/ws/1/live/*
/ws/1/related_content/*
But forbidden for the following directories
/ajah/
/api/
/beta
/cgi-bin/ACHATS/
/element/commun/afficher/
/petites-annonces/
/qui-sommes-nous/
/txt/
/verification/source/*
/noscript/
/ws/*
/recherche/*
An ethical way to do web scraping is to pause your request from time to time. Using distributed web scrapers, the algorithms send requests much faster than clicking website in a web browser. Some of the small websites could encounter server overload and service shutdown. If you are being too aggressive, you might trigger an anti-abuse system. Usually it results in IP address blocking, perhaps temporary for a few hours or days, unlikely to be permanent. In worst cases, your scraping could be identified as DDoS attack. State sponsored hacking groups normally launch these sorts of attacks. These attacks are serious offense under Computer Fraud and Abuse Act. You could be convicted in criminal cases and end up in jail. Hence, there is no harm in waiting for a few extra seconds to disguise as a human viewer.
In Python, we just do
import time
import random as rd
time.sleep(rd.randint(1,5))
Thanks to European Union, now we have General Data Protection Regulation. Apart from GDPR, California is about to roll out California Consumer Privacy Act. This brings challenges to the thriving business of data broker. Ideally most of us build web scrapers to scrape open data source. If you are scraping résumé from LinkedIn or stalking someone on social media, and the person is fallen under the jurisdiction of European Court of Justice, then it will be tricky. Is it considered obtaining and processing personal data without someone's consent? I can't tell. As a consumer, I fully support the regulation safeguarding my personal data. As a coder, I would've recused myself from building a web scrapper with potential violations.
Last but not least, if there is API which implies it is legitimate, always use API instead of building your own toolbox. Why reinvent the wheels and bear the risks?
Click the icon below to be redirected to GitHub Repository


--- layout: post title: Look ma, I’m on a podcast! date: '2015-10-25T16:58:00.002-04:00' author: Allison McMillan tags: - remote - ruby - podcast modified_time: '2015-10-25T16:58:41.771-04:00' blogger_id: tag:blogger.com,1999:blog-6054545261088815023.post-7187012264098239734 blogger_orig_url: http://allisonsherenmcmillan.blogspot.com/2015/10/look-ma-im-on-podcast.html ---


Creates an autoscaler that automatically chooses and sets the number of pods that run in a kubernetes cluster.
Looks up a Deployment, ReplicaSet, StatefulSet, or ReplicationController by name and creates an autoscaler that uses the given resource as a reference. An autoscaler can automatically increase or decrease number of pods deployed within the system as needed.
Examples:
# Auto scale a deployment "foo", with the number of pods between 2 and 10, no target CPU utilization specified so a default autoscaling policy will be used:
kubectl autoscale deployment foo --min=2 --max=10
# Auto scale a replication controller "foo", with the number of pods between 1 and 5, target CPU utilization at 80%:
kubectl autoscale rc foo --max=5 --cpu-percent=80
kubectl autoscale (-f FILENAME | TYPE NAME | TYPE/NAME) [--min=MINPODS] --max=MAXPODS [--cpu-percent=CPU] [options]
Use "kubectl options" for a list of global command-line options (applies to all commands).