--- title: "Write your own Gameday scraper with XML2R and pitchRx" date: "2014-09-24" categories: [Baseball, R] ---
MLBAM’s website hosts a wealth of data that informs the MLB Gameday application. Most notably, this website hosts pitch-by-pitch (ie, PITCHf/x) and play-by-play data. pitchRx provides a simple interface to acquire a good portion of Gameday data (see Table 1 of my paper), but there is a significant amount that isn’t directly accessible using pitchRx::scrape
. This post will show you some general techniques to acquire and manipulate data from this website.
When scraping data off the web, typically the first task is to construct urls that point to the information of interest. Sometimes this is difficult, but thankfully, the Gameday site structure urls in a sensible fashion. In fact, if we are searching for game specific data, the full address can be constructed from the Gameday identifier. To demonstrate, let’s grab Gameday identifiers for MLB and Triple-A games played in August of 2014 (this post has more on searching Gameday identifiers with regular expressions).
library(pitchRx)
# Load gameday identifiers
data(gids, package = "pitchRx")
data(nonMLBgids, package = "pitchRx")
# Subset MLB identifiers to August, 2014
MLBids <- gids[grepl("2014_08_[0-9]{2}", gids)]
# Subset to Triple-A games in August, 2014
AAAids <- nonMLBgids[grepl("2014_08_[0-9]{2}_[a-z]{3}aaa_[a-z]{3}aaa", nonMLBgids)]
ids <- c(MLBids, AAAids)
ids[1]
## [1] "gid_2014_08_01_phimlb_wasmlb_1"
The function pitchRx::makeUrls
can then be used to pre-append the path required to reach ‘game directories’.
paths <- makeUrls(gids = ids)
paths[1]
## [1] "http://gd2.mlb.com/components/game/mlb/year_2014/month_08/day_01/gid_2014_08_01_phimlb_wasmlb_1"
Within each game directory, there are a bunch of files (to see them, click on the link above). Note that pitchRx::scrape
has support for acquiring: miniscoreboard.xml, players.xml, inning/inning_all.xml, and inning/inning_hit.xml files; but any of these XML files can be harvested using XML2R.
While developing pitchRx, I found myself writing a lot of redundant R code that repeatedly transformed similarly structured XML documents into a list of data frames. Through this discovery, I found a reasonable way to abstract the common tasks into a small number of functions/conventions which are bundled into a separate R package named XML2R. In fact, I have a vignette which explains how pitchRx uses XML2R to acquire PITCHf/x data. I intend to write more sports data scrapers on top of XML2R (e.g., bbscrapeR) and possibly combine them into a central R package.
The fundamental idea behind XML2R is that any XML structure can be coerced into a list of observations. Here I will transform a bunch of rawboxscore.xml files into a list of observations using XML2Obs
:
library(XML2R)
urls <- paste0(paths, "/rawboxscore.xml")
obs <- XML2Obs(urls) # this will take a minute
The names
of this list keeps track of the XML hierarchy. In some sense, this hierarchy can be thought as defining an observational unit. XML2R has functionality for combining and creating links between observations, but we won’t cover that here. Instead, let’s take a peek at the top 5 observational units in terms of the count of observations within each level.
nms <- names(obs)
top <- sort(table(nms), decreasing = TRUE)
head(top)
## nms
## boxscore//team//batting//batter
## 26205
## boxscore//team//pitching//pitcher//strikeout
## 14047
## boxscore//linescore//inning_line_score
## 8652
## boxscore//team//pitching//pitcher
## 7153
## boxscore//team//pitching//pitcher//walk
## 5799
## boxscore//umpires//umpire
## 3349
Just because, here are two observations from the most popular observational unit:
batter <- obs[grepl("^boxscore//team//batting//batter$", nms)][1:2]
batter
## $`boxscore//team//batting//batter`
## hr sac name_display_first_last pos rbi id lob bis_avg
## [1,] "0" "0" "Ryan Howard" "1B" "0" "429667" "0" ".220"
## name bb bis_s_hr bam_s_so fldg bam_s_bb hbp d e so a
## [1,] "Howard" "0" "16" "124" "1.000" "45" "0" "0" "0" "0" "1"
## bis_s_t bam_s_t sf bam_s_r bam_s_hr h bat_order bis_s_r bis_s_rbi
## [1,] "1" "1" "0" "48" "16" "0" "400" "48" "63"
## t bis_s_so bis_s_h bam_s_h ao bam_s_rbi r bam_s_e sb bis_s_d
## [1,] "0" "124" "87" "87" "1" "63" "0" "7" "0" "11"
## bam_s_d bis_s_e po ab bis_s_bb bam_avg go
## [1,] "11" "7" "12" "4" "45" ".220" "3"
## url
## [1,] "http://gd2.mlb.com/components/game/mlb/year_2014/month_08/day_01/gid_2014_08_01_phimlb_wasmlb_1/rawboxscore.xml"
##
## $`boxscore//team//batting//batter`
## hr sac name_display_first_last first_sac pos id rbi lob
## [1,] "0" "1" "Roberto Hernandez" "262" "P" "433584" "0" "2"
## bis_avg name bb bis_s_hr bam_s_so fldg
## [1,] ".056" "Hernandez, R" "0" "0" "17" "1.000"
## first_two_out_risp_lob bam_s_bb hbp d e so a sf bam_s_r
## [1,] "95" "0" "0" "0" "0" "0" "0" "0" "0"
## bam_s_hr two_out_risp_lob h bat_order bis_s_r bis_s_rbi t
## [1,] "0" "1" "0" "900" "0" "1" "0"
## bis_s_so bis_s_h bam_s_h ao bam_s_rbi r sb po ab bis_s_bb go
## [1,] "17" "2" "2" "0" "1" "0" "0" "2" "2" "0" "3"
## bam_avg
## [1,] ".056"
## url
## [1,] "http://gd2.mlb.com/components/game/mlb/year_2014/month_08/day_01/gid_2014_08_01_phimlb_wasmlb_1/rawboxscore.xml"
Note that each observation (or each list element) is a character matrix with one row. This makes it easy to row bind observations at a later point.
str(batter)
## List of 2
## $ boxscore//team//batting//batter: chr [1, 1:46] "0" "0" "Ryan Howard" "1B" ...
## ..- attr(*, "dimnames")=List of 2
## .. ..$ : NULL
## .. ..$ : chr [1:46] "hr" "sac" "name_display_first_last" "pos" ...
## $ boxscore//team//batting//batter: chr [1, 1:43] "0" "1" "Roberto Hernandez" "262" ...
## ..- attr(*, "dimnames")=List of 2
## .. ..$ : NULL
## .. ..$ : chr [1:43] "hr" "sac" "name_display_first_last" "first_sac" ...
For our purposes, I just want to see how audience attendance differs across ballparks in both the MLB and Triple-A. To investigate, I’ll just grab the “boxscore” observational unit:
bs <- obs[grepl("^boxscore$", nms)]
bs[1]
## $boxscore
## wind game_type venue_name home_sport_code
## [1,] "15 mph, L to R" "R" "Nationals Park" "mlb"
## attendance official_scorer game_pk date status_ind
## [1,] "28,410" "Dave Matheson" "382175" "August 1, 2014" "F"
## home_league_id elapsed_time game_id venue_id
## [1,] "104" "2:28" "2014/08/01/phimlb-wasmlb-1" "3309"
## weather gameday_sw
## [1,] "79 degrees, cloudy" "P"
## url
## [1,] "http://gd2.mlb.com/components/game/mlb/year_2014/month_08/day_01/gid_2014_08_01_phimlb_wasmlb_1/rawboxscore.xml"
Now, since scores
is a list of observations all with the same observational unit, we can collapse the observations into a matrix.
unique(names(bs))
bs.mat <- collapse_obs(bs)
bs.mat[1:5, !grepl("url", colnames(bs.mat))] #first 5 records without "url" column
## wind game_type venue_name
## [1,] "15 mph, L to R" "R" "Nationals Park"
## [2,] "3 mph, L to R" "R" "Oriole Park at Camden Yards"
## [3,] "7 mph, In from CF" "R" "Progressive Field"
## [4,] "8 mph, In from CF" "R" "Comerica Park"
## [5,] "0 mph, None" "R" "Marlins Park"
## home_sport_code attendance official_scorer game_pk date
## [1,] "mlb" "28,410" "Dave Matheson" "382175" "August 1, 2014"
## [2,] "mlb" "39,487" "Jim Henneman" "382177" "August 1, 2014"
## [3,] "mlb" "27,009" "Bob Maver" "382179" "August 1, 2014"
## [4,] "mlb" "39,052" "Dan Marowski" "382170" "August 1, 2014"
## [5,] "mlb" "20,410" "Ronald Jernick" "382169" "August 1, 2014"
## status_ind home_league_id elapsed_time game_id
## [1,] "F" "104" "2:28" "2014/08/01/phimlb-wasmlb-1"
## [2,] "F" "103" "2:29" "2014/08/01/seamlb-balmlb-1"
## [3,] "F" "103" "3:23" "2014/08/01/texmlb-clemlb-1"
## [4,] "F" "103" "2:56" "2014/08/01/colmlb-detmlb-1"
## [5,] "F" "104" "2:58" "2014/08/01/cinmlb-miamlb-1"
## venue_id weather gameday_sw
## [1,] "3309" "79 degrees, cloudy" "P"
## [2,] "2" "78 degrees, cloudy" "P"
## [3,] "5" "76 degrees, clear" "P"
## [4,] "2394" "81 degrees, clear" "P"
## [5,] "4169" "75 degrees, roof closed" "P"
In R, matrices have homogeneous column types and XML2R will always return a character matrix. For this reason, some data cleaning is usually required. In this case, we coerce attendance to an integer type and collect a subset of the columns into a data frame (which can handle heterogeneous column types).
attendance <- as.integer(gsub("\\,", "", boxscore[,"attendance"]))
venue <- boxscore[,"venue_name"]
# this is here to tell ggplot to sort venues by their median attendance
parks <- names(sort(tapply(attendance, INDEX = venue, median)))
df <- data.frame(attendance, venue = factor(venue, levels = parks),
league = boxscore[,"home_sport_code"])
Lastly, let’s visualize the distribution of single game attendance across ballparks. Not a whole lot of surprise here, but I think it is interesting that the variance is MUCH higher in the major leagues (except for the few teams with a very loyal following – I’m looking at you Red Sox & Giants fans).
library(ggplot2)
qplot(data = df, x = venue, y = attendance, geom = "boxplot",
fill = league) + coord_flip() +
theme(legend.position = c(0.8,0.2))


|
Panda3D
|
This defines all of the TextProperties structures that might be referenced by name from an embedded text string. More...
Public Member Functions | |
| clearGraphic (str name) | |
| Removes the named TextGraphic structure from the manager. More... | |
| clearProperties (str name) | |
| Removes the named TextProperties structure from the manager. More... | |
| TextGraphic | getGraphic (str name) |
| Returns the TextGraphic associated with the indicated name. More... | |
| TextProperties | getProperties (str name) |
| Returns the TextProperties associated with the indicated name. More... | |
| bool | hasGraphic (str name) |
| Returns true if a TextGraphic structure has been associated with the indicated name, false otherwise. More... | |
| bool | hasProperties (str name) |
| Returns true if a TextProperties structure has been associated with the indicated name, false otherwise. More... | |
| setGraphic (str name, const NodePath model) | |
| This flavor of set_graphic implicitly creates a frame for the model using the model's actual computed bounding volume, as derived from NodePath::calc_tight_bounds(). More... | |
| setGraphic (str name, const TextGraphic graphic) | |
| Defines the TextGraphic associated with the indicated name. More... | |
| setProperties (str name, const TextProperties properties) | |
| Defines the TextProperties associated with the indicated name. More... | |
| write (Ostream out, int indent_level) | |
Static Public Member Functions | |
| static TextPropertiesManager | getGlobalPtr () |
| Returns the pointer to the global TextPropertiesManager object. More... | |
This defines all of the TextProperties structures that might be referenced by name from an embedded text string.
A text string, as rendered by a TextNode, can contain embedded references to one of the TextProperties defined here, by enclosing the name between \1 (ASCII 0x01) characters; this causes a "push" to the named state. All text following the closing \1 character will then be rendered in the new state. The next \2 (ASCII 0x02) character will then restore the previous state for subsequent text.
For instance, "x\1up\1n\2 + y" indicates that the character "x" will be rendered in the normal state, the character "n" will be rendered in the "up" state, and then " + y" will be rendered in the normal state again.
This can also be used to define arbitrary models that can serve as embedded graphic images in a text paragraph. This works similarly; the convention is to create a TextGraphic that describes the graphic image, and then associate it here via the set_graphic() call. Then "\5name\5" will embed the named graphic.
| clearGraphic | ( | str | name | ) |
Removes the named TextGraphic structure from the manager.
| clearProperties | ( | str | name | ) |
Removes the named TextProperties structure from the manager.
|
static |
Returns the pointer to the global TextPropertiesManager object.
| TextGraphic getGraphic | ( | str | name | ) |
Returns the TextGraphic associated with the indicated name.
If there was not previously a TextGraphic associated with this name, a warning is printed and then a default TextGraphic structure is associated with the name, and returned.
Call has_graphic() instead to check whether a particular name has been defined.
| TextProperties getProperties | ( | str | name | ) |
Returns the TextProperties associated with the indicated name.
If there was not previously a TextProperties associated with this name, a warning is printed and then a default TextProperties structure is associated with the name, and returned.
Call has_properties() instead to check whether a particular name has been defined.
| bool hasGraphic | ( | str | name | ) |
Returns true if a TextGraphic structure has been associated with the indicated name, false otherwise.
Normally this means set_graphic() has been called with this name, but because get_graphic() will implicitly create a default TextGraphic structure, it may also mean simply that get_graphic() has been called with the indicated name.
| bool hasProperties | ( | str | name | ) |
Returns true if a TextProperties structure has been associated with the indicated name, false otherwise.
Normally this means set_properties() has been called with this name, but because get_properties() will implicitly create a default TextProperties structure, it may also mean simply that get_properties() has been called with the indicated name.
| setGraphic | ( | str | name, |
| const NodePath | model | ||
| ) |
This flavor of set_graphic implicitly creates a frame for the model using the model's actual computed bounding volume, as derived from NodePath::calc_tight_bounds().
Create a TextGraphic object first if you want to have explicit control of the frame.
| setGraphic | ( | str | name, |
| const TextGraphic | graphic | ||
| ) |
Defines the TextGraphic associated with the indicated name.
When the name is subsequently encountered in text embedded between \5 characters in a TextNode string, the specified graphic will be embedded in the text at that point.
If there was already a TextGraphic structure associated with this name, it is quietly replaced with the new definition.
| setProperties | ( | str | name, |
| const TextProperties | properties | ||
| ) |
Defines the TextProperties associated with the indicated name.
When the name is subsequently encountered in text embedded between \1 characters in a TextNode string, the following text will be rendered with these properties.
If there was already a TextProperties structure associated with this name, it is quietly replaced with the new definition.
| write | ( | Ostream | out, |
| int | indent_level | ||
| ) |


Family Privacy Protection Act Policy
(Enacted Pursuant to S.C. Code Section 30-2-10 et. seq.)*
The University is a complex agency which has a variety of legitimate public purposes encompassing teaching, research and extension. It is in the University's best interest to promote and support affiliated foundations and entities. In some cases the University’s interests can be served by providing goods and services to its students, parents, alumni and employees. The University provides much of this support and many services internally; however there are a number of supportive functions which for various reasons are performed by affiliated entities or private contractors.
To this end, the university will implement the Family Privacy Protection Act (FPPA) as follows:
- Responsibility. It shall be the responsibility of the Office of Institutional Research to implement compliance with the FPPA, and specifically Section 30-2-40 A.
- Evaluation. The university will engage in an ongoing evaluation to ensure that the personal information it gathers in its various programs is necessary for the performance of the university’s legitimate public purposes.
- Personal Rights. The university will continue to inform all persons from whom the university collects personal information (hereinafter, ‘affected persons’) of their rights under applicable laws, including the South Carolina Personal Privacy Protection Act (SC Code Section 30-2-10 et. seq.) and the Family Educational and Right to Privacy Act (20 USC 1232g), as appropriate.
- Campus Directory Information. The university has determined that publication of its annual campus telephone directory is a business necessity. The university will release this directory to third parties only after notifying the requestor that the information therein may not be used for commercial solicitation and receiving written acknowledgement that the material will not be used for commercial solicitation. The telephone directory and any Web directories containing personal information will contain the warnings referred to in Section 30-2-50 of the FPPA.
- Foundations and Affiliated Entities. The university will release necessary personal information concerning affected persons to an affiliated entity, that is, an institute, association, foundation, etc, in direct support of the university, provided:
- The use for which the material is requested is for a legitimate public purpose in support of the University’s public mission;
- The entity acknowledges that the material is to be used only for the purpose or purposes specified;
- The entity acknowledges that the material will not be released to a third party without the university’s written consent;
- Any contractors of the entity must acknowledge in writing that the material is to be used only for the contracted purpose, and will not be released to third parties without written permission from the University;
- Only so much information is released as is necessary to accomplish the stated purpose for the request.
- Contractors. The University will release necessary personal information of affected persons to a contractor performing tasks which are essential to the fulfillment of the university’s mission, provided:
- The contract was reached after compliance with applicable procurement requirements;
- The contract is in support of an essential element of one of the University’s legitimate public purposes;
- The contractor acknowledges that the material will only be used to fulfill the contract;
- The contractor acknowledges that the material will not be disclosed to third parties, without the university’s written consent;
- Only so much information is released as is necessary to accomplish the stated purpose of the contract.
- Third Parties. Subject to the provisions of the S.C. Freedom of Information Act or the Family Educational Right to Privacy Act, personal information which identifies an individual may be released to third parties, provided:
- The university first notifies the requestor that: “The information that you have requested is personal information under South Carolina law. It is a violation of South Carolina law to use this personal information for commercial solicitation”;
- The requestor provides written acknowledgment that: “I understand that it is a violation of South Carolina criminal law to use the personal information that I have requested for purposes of commercial solicitation”;
- Only so much personal information is released as is necessary to accomplish the stated purpose of the request;
- Exemptions are contained in Section 30-2-30 (3). A party claiming to be exempt from the application of FPPA will cite the source of the claimed exemption. The requesting party will be advised that the personal information is provided solely for its use under the stated exemption and that any use for commercial solicitation by any other party is illegal under S.C. law.
- University Use. The university may use any of the personal information it collects from affected persons as necessary to accomplish, or in support of, any of its legitimate public purposes. The university may provide mailing services to or for third parties, provided that the purpose of such services is not commercial solicitation.
Policy last reviewed October 20 2010
*Family Privacy Protection Act of 2002
Section 30-2-10. Short Title.
This chapter shall be designated as the "Family Privacy Protection Act of 2002."
HISTORY: 2002 Act No.225, Section 1.
Section 30-2-20.
All state agencies, boards, commissions, institutions, departments, and other state entities, by whatever name known, must develop privacy policies and procedures to ensure that the collection of personal information pertaining to citizens of the State is limited to such personal information required by any such agency, board, commission, institution, department or other state entity and necessary to fulfill a legitimate public purpose.
HISTORY: 2002 Act No.225, Section 1.
Section 30-2-30.
For purposes of this act, the following terms have the following meanings:
- "Personal information" means information that identifies or describes an individual including, but not limited to, an individual's photograph or digitized image, Social Security number, date of birth, driver's identification number, name, home address, home telephone number, medical or disability information, education level, financial status, bank account numbers, account or identification number issued, used, or both by any federal or state governmental agency or private financial institution, employment history, height, weight, race, other physical details, signature, biometric identifiers, and any credit records or reports. "Personal information" does not mean information about boating accidents, vehicular accidents, driving violations, boating violations or driver status, or names and addresses from any registration documents filed with the Department of Revenue as a business address which also may be a personal address.
- "Legitimate public purpose" means a purpose or use which falls clearly within the statutory charge or mandates of an agency, board, commission, institution, department, or other state entity.
- "Commercial solicitation" means contact by telephone, mail, or electronic mail for the purpose of selling or marketing a consumer product or service. "Commercial solicitation" does not include contact by whatever means for the purpose of:
- offering membership in a credit union;
- notification of continuing education opportunities;
- selling or marketing banking, insurance, securities, or commodities services provided by an institution or entity defined in or required to comply with the Federal Gramm-Leach-Bliley Financial Modernization Act, 113 Stat. 1338;
- contacting persons for political purposes using information on file with state or local voter registration offices.
- "Medical information" includes, but is not limited to, blood samples and test results obtained and kept by the Department of Health and Environmental Control pursuant to Section 44-37-30.
HISTORY: 2002 Act No.225, Section 1; 2003 Act No. 20, Section 1; 2003 Act No. 69, Section 3.11.
Section 30-2-40.
- Any state agency, board, commission, institution, department, or other state entity which hosts, supports, or provides a link to page or site accessible through the World Wide Web must clearly display its privacy policy and the name and telephone number of the agency, board, commission, institution, department, or other state entity person responsible for administration of the policy.
- Where personal information is authorized to be collected by an entity covered by this section, the entity must at the time of collection advise the citizen to whom the information pertains that the information is subject to public scrutiny or release.
- Subsection (B) does not apply to criminal justice or judicial agencies, or both.
HISTORY: 2002 Act No.225, Section 1.
Section 30-2-50.
- A person or private entity shall not knowingly obtain or use any personal information obtained from a state agency for commercial solicitation directed to any person in this State.
- Each state agency shall provide a notice to all requestors of records pursuant to this chapter and to all persons who obtain records pursuant to this chapter that obtaining or using public records for commercial solicitation directed to any person in this State is prohibited.
- All state agencies shall take reasonable measures to ensure that no person or private entity obtains or distributes personal information obtained from a public record for commercial solicitation.
- A person knowingly violating the provisions of subsection (A) is guilty of a misdemeanor and, upon conviction, must be fined an amount not to exceed five hundred dollars or imprisoned for a term not to exceed one year, or both.
- This chapter does not apply to a local governmental entity of a subdivision of this state or local government.
HISTORY: 2002 Act No.225, Section 1; 2003 Act No. 20, Section 2.


I am
Kushagra
. Hover me to see the tooltip!
Kushagra Agarwal
HTML/CSS expert
Frontend Designer
View more mork
×


Using Docker
Abstract
Docker images are created on development/build machines and published to a remote registry. On the deployment side, docker images are pulled from the remote registry, stored in the local registry and then instantiated at runtime as Docker containers.
Prerequisites
- A private registry host, such as Artifactory or GitLab must be setup as place to store the docker images
- Docker and all dependencies have to be installed on both build and deployment environments
Build Setup
In the root directory of the project, create a Dockerfile similar to this:
FROM node:8
# Create app directory
WORKDIR /usr/src/app
# Install app dependencies
COPY package.json .
COPY package.json package-lock.json ./
ENV mso_portal_url="http://localhost:3010"
RUN npm install
# Bundle app source
COPY . .
EXPOSE 3000
CMD [ "npm", "start" ]
Build the Docker image. It will be stored in your local Docker registry.
docker build -t community.cablelabs.com:4567/micronets-docker/micronets-reg-server .
Publish the Docker image to your remote registry.
docker login community.cablelabs.com:4567; docker push community.cablelabs.com:4567/micronets-docker/micronets-reg-server
Deployment
Docker containers can be named at runtime. Doing so will ensure that the container created with docker run
can be stopped and restarted using the name given to the container. It also allows the container to be started and stopped using the standard unix systemd
facility.
Docker image installation
To download a Docker image from a remote repository, use the docker pull
command:
# Log in with your remote registry credentials
docker login
# Retrieve the image
docker pull community.cablelabs.com:4567/micronets-docker/micronets-reg-server:latest`
# Remove the container first, if it exists (cannot create a named container with the same name)
docker rm -f micronets-reg-server
# Run the image (create a container) and give it a name.
docker run -d --name=micronets-reg-server -e someenv -p 3000:3000 community.cablelabs.com:4567/micronets-docker/micronets-reg-server:latest
Create a systemd configuration file
This will ensure that your Docker container will start every time the system reboots.
Add a configuration file: /etc/systemd/system/micronets-reg-server.service
[Unit]
Description=Micronets Registration Server Container
Requires=docker.service
After=docker.service
[Service]
Restart=always
RestartSec=5
ExecStart=/usr/bin/docker start -a micronets-reg-server
ExecStop=/usr/bin/docker stop -t 2 micronets-reg-server
ExecRestart=/usr/bin/docker restart -t 2 micronets-reg-server
[Install]
WantedBy=default.target
Enable the service
sudo systemctl enable micronets-reg-server
Starting and Stopping your container
# systemd
sudo systemctl start micronets-reg-server
sudo systemctl stop micronets-reg-server
sudo systemctl restart micronets-reg-server
# docker
docker start -a micronets-reg-server
docker stop -t 2 micronets-reg-server
docker restart -t 2 micronets-reg-server


Angular wouldn't be what it is today without its communities, from all over the world. Check out this other meetups & conferences!


 I'm going to do my best to fill your mind full of all kinds of technobabble.
First off, let's start with version control. What is it? Well it's this nifty little thing that lets you save your work at certain points. I'm really quite fond on the analogy that it's like a save point in a video game. It is a point that you can go back to when something goes wrong instead of going all the way back to the beginning.
Git helps you keep track of changes through version control. When you use git you create commits that are like the "save points" that I had mentioned before. They create the points that you can rollback to is something in your code goes wrong.
The nice thing about doing this on GitHub is that it's an online repository for the work you do. Now you might wonder why you need it to be online since you are doing this on your own. That is exactly why you do it online, to do it with other people. It allows you to do your coding with other people. GitHub allows for others to see what changes you make in your code and offer suggestions and further commits to what you have already done. It makes it a great way to colaberate with other minds out there in cyber space!


If you want your site's content to be known by search engines like Google, then it's a good idea to provide the search engine (I'll be referring to Google from now on) with a sitemap.xml
file.
Google has a set of Webmaster Tools (GWT) that can help your site have better results on Google's search results.
Yellowblue auto genarates the sitemap.xml
file for you, which you can submit to GWT. Here is an artricle from Google, that explains what a sitemap is and why you may or may not need one.
If your site's base URL is www.myblog.com, your sitemap's url will be www.myblog.com/sitemap.xml
. In other words, to get your sitemap's URL, just append sitemap.xml
to your sit's base URL.
Before deploying your site, do not forget to enter your site's base URL in the url
variable in the _config.yml
file!
If you want to have a look at the code that generates the sitemap, just open the sitemap.xml
file.
If you want to take control of the sitemap's variables, you can define them in your posts' or pages' Front Matter section, like so:
sitemap:
lastmod: 2015-09-18
priority: 0.7
changefreq: 'monthly'
exclude: 'yes'
The exclude
variable is for pages, whereas the published
variable is for posts. So if you don't want a page or a post to be included in the sitemap, just set the appropriate variable to no
.
If you omit the above section, the default values will be used, which are just fine.
Full credit for the sitemap code goes to David Ensinger and this wonderful post of his. Thank you!


public class Loader<D>
extends java.lang.Object
android.content.Loader
.
Used to write apps that run on platforms prior to Android 3.0. When running
on Android 3.0 or above, this implementation is still used; it does not try
to switch to the framework's implementation. See the framework SDK
documentation for a class overview.| Modifier and Type | Class and Description |
|---|---|
class |
Loader.ForceLoadContentObserver
An implementation of a ContentObserver that takes care of connecting
it to the Loader to have the loader re-load its data when the observer
is told it has changed.
|
static interface |
Loader.OnLoadCompleteListener<D>
Interface that is implemented to discover when a Loader has finished
loading its data.
|
| Constructor and Description |
|---|
Loader(Context context)
Stores away the application context associated with context.
|
| Modifier and Type | Method and Description |
|---|---|
void |
abandon()
Tell the Loader that it is being abandoned.
|
void |
commitContentChanged()
Commit that you have actually fully processed a content change that
was returned by
takeContentChanged() . |
java.lang.String |
dataToString(D data)
For debugging, converts an instance of the Loader's data class to
a string that can be printed.
|
void |
deliverResult(D data)
Sends the result of the load to the registered listener.
|
void |
dump(java.lang.String prefix,
java.io.FileDescriptor fd,
java.io.PrintWriter writer,
java.lang.String[] args)
Print the Loader's state into the given stream.
|
void |
forceLoad()
Force an asynchronous load.
|
Context |
getContext() |
int |
getId() |
boolean |
isAbandoned()
Return whether this loader has been abandoned.
|
boolean |
isReset()
Return whether this load has been reset.
|
boolean |
isStarted()
Return whether this load has been started.
|
protected void |
onAbandon()
Subclasses implement this to take care of being abandoned.
|
void |
onContentChanged()
Called when
Loader.ForceLoadContentObserver detects a change. |
protected void |
onForceLoad()
Subclasses must implement this to take care of requests to
forceLoad() . |
protected void |
onReset()
Subclasses must implement this to take care of resetting their loader,
as per
reset() . |
protected void |
onStartLoading()
Subclasses must implement this to take care of loading their data,
as per
startLoading() . |
protected void |
onStopLoading()
Subclasses must implement this to take care of stopping their loader,
as per
stopLoading() . |
void |
registerListener(int id,
Loader.OnLoadCompleteListener<D> listener)
Registers a class that will receive callbacks when a load is complete.
|
void |
reset()
Resets the state of the Loader.
|
void |
rollbackContentChanged()
Report that you have abandoned the processing of a content change that
was returned by
takeContentChanged() and would like to rollback
to the state where there is again a pending content change. |
void |
startLoading()
Starts an asynchronous load of the Loader's data.
|
void |
stopLoading()
Stops delivery of updates until the next time
startLoading() is called. |
boolean |
takeContentChanged()
Take the current flag indicating whether the loader's content had
changed while it was stopped.
|
java.lang.String |
toString() |
void |
unregisterListener(Loader.OnLoadCompleteListener<D> listener)
Remove a listener that was previously added with
registerListener(int, android.support.v4.content.Loader.OnLoadCompleteListener<D>) . |
public Loader(Context context)
context
- used to retrieve the application context.public void deliverResult(D data)
data
- the result of the loadpublic Context getContext()
public int getId()
public void registerListener(int id, Loader.OnLoadCompleteListener<D> listener)
Must be called from the process's main thread.
public void unregisterListener(Loader.OnLoadCompleteListener<D> listener)
registerListener(int, android.support.v4.content.Loader.OnLoadCompleteListener<D>)
.
Must be called from the process's main thread.public boolean isStarted()
startLoading()
has been called and no calls to stopLoading()
or
reset()
have yet been made.public boolean isAbandoned()
public boolean isReset()
reset()
has been called.public final void startLoading()
stopLoading()
will stop the delivery of callbacks.
This updates the Loader's internal state so that
isStarted()
and isReset()
will return the correct
values, and then calls the implementation's onStartLoading()
.
Must be called from the process's main thread.
protected void onStartLoading()
startLoading()
. This is not called by clients directly,
but as a result of a call to startLoading()
.public void forceLoad()
startLoading()
this will ignore a previously
loaded data set and load a new one. This simply calls through to the
implementation's onForceLoad()
. You generally should only call this
when the loader is started -- that is, isStarted()
returns true.
Must be called from the process's main thread.
protected void onForceLoad()
forceLoad()
.
This will always be called from the process's main thread.public void stopLoading()
startLoading()
is called.
Implementations should not invalidate their data at this point --
clients are still free to use the last data the loader reported. They will,
however, typically stop reporting new data if the data changes; they can
still monitor for changes, but must not report them to the client until and
if startLoading()
is later called.
This updates the Loader's internal state so that
isStarted()
will return the correct
value, and then calls the implementation's onStopLoading()
.
Must be called from the process's main thread.
protected void onStopLoading()
stopLoading()
. This is not called by clients directly,
but as a result of a call to stopLoading()
.
This will always be called from the process's main thread.public void abandon()
reset()
to have it retain its current data but not report
any new data.protected void onAbandon()
onReset()
-- it means that
the client is no longer interested in any new data from the loader,
so the loader must not report any further updates. However, the
loader must keep its last reported data valid until the final
onReset()
happens. You can retrieve the current abandoned
state with isAbandoned()
.public void reset()
startLoading()
may later be called at which point it must be
able to start running again.
This updates the Loader's internal state so that
isStarted()
and isReset()
will return the correct
values, and then calls the implementation's onReset()
.
Must be called from the process's main thread.
protected void onReset()
public boolean takeContentChanged()
public void commitContentChanged()
takeContentChanged()
. This is for use with
rollbackContentChanged()
to handle situations where a load
is cancelled. Call this when you have completely processed a load
without it being cancelled.public void rollbackContentChanged()
takeContentChanged()
and would like to rollback
to the state where there is again a pending content change. This is
to handle the case where a data load due to a content change has been
canceled before its data was delivered back to the loader.public void onContentChanged()
Loader.ForceLoadContentObserver
detects a change. The
default implementation checks to see if the loader is currently started;
if so, it simply calls forceLoad()
; otherwise, it sets a flag
so that takeContentChanged()
returns true.
Must be called from the process's main thread.
public java.lang.String dataToString(D data)
public java.lang.String toString()
toString
in class java.lang.Object
public void dump(java.lang.String prefix, java.io.FileDescriptor fd, java.io.PrintWriter writer, java.lang.String[] args)
prefix
- Text to print at the front of each line.fd
- The raw file descriptor that the dump is being sent to.writer
- A PrintWriter to which the dump is to be set.args
- Additional arguments to the dump request.


Chaitanya
I have a background in Aerospace,Event management, Sales,
e-commerce,Casino management.
I am looking to apply My new development skills to use an innovative,
user-centric approach to solve problems for others.
I believes that problem solving should be organic, data-driven, and iterative.


rasterizer¶
-
class
RasterGlyph
(sourceGlyph)[source]¶ An object to scan glyphs and rasterize them into element components.
-
lib_key_coordenates
= 'coordenates.rasterizer.com.hipertipo'¶
-
lib_key_margins
= 'margins.rasterizer.com.hipertipo'¶
-
-
get_esize
(font, element_src='_element')[source]¶ Get a font’s element size from its bounding box.
Parameters: - font (RFont) – The font to which the element glyph belongs.
- element_src (str) – The name of the element glyph.
-
randomize_elements
(glyph, esize, rand_size)[source]¶ Randomize the size of element shapes in the current glyph.
Parameters: - glyph (RGlyph) – The glyph in which the element shapes will be transformed.
- esize (int) – The current base size of the element shape.
- rand_size (tuple) – The scale factors for minimum and maximum random element sizes.
-
set_element
(font, size, type='rect', magic=None, element_src='_element')[source]¶ Set the shape of the element glyph in the font.
Parameters: - font (RFont) – A font in which to create the element glyph.
- size (int) – The size of the element shape.
- type (str) – The type of the element shape:
rect
,oval
orelement
. - magic (float) – A number indicating the roundness of shapes of type
element
. - element_src (str) – The glyph in which the element shape will be drawn.


Audi Gets One Over On BMW And Mercedes
BMW doesn't seem happy about it.
by Jared RosenholtzA few months ago, Audi was in search of a new CEO as its former CEO was still being charged with fraud, indirect discrimination, and false advertising during the Dieselgate scandal. Audi picked the man for the job - Markus Duesmann, former BMW engine development and purchasing expert (and Mercedes engineer) - but BMW raised objections and would not release Duesmann from his non-compete clause.
Duesmann's contract with BMW has now ended and Audi has announced he will take over as CEO starting on April 1, 2020. The 50-year-old mechanical engineer takes over for interim CEO Bram Schot, who has held the position since June 2018.
"As an excellent engineer, Markus Duesmann will do everything in his power to leverage the great potential of the Audi brand and will once again demonstrate the promise of Vorsprung durch Technik," said Dr. Herbert Diess, Chairman of the Supervisory Board of Audi AG.
Audi is about to reveal a slew of new models at the 2019 Los Angeles Auto Show, ushering in a new era of performance for the brand. We've already seen some of these models like the RS6 Avant and RS7, which will be making their North American debuts in LA, but there will also be some global reveals like the e-tron Sportback and RS Q8. Having Duesmann as the new CEO should aid Audi as it continues to improve its engine technology and move towards electrification.
You Might Also Like
"We expect Markus Duesmann and his management team to ensure stable capacity utilization at the plants and to promote more courage to take the lead through technology. We look forward to positive cooperation in the interests of the employees and the company," Chairman of the General Works Council of Audi AG and Deputy Chairman of the Supervisory Board, Peter Mosch commented.
Schot was "the right man at Audi at the right time. He started a cultural transformation towards fewer hierarchies, a clear value system, and more openness. In mid-2018, Schot assumed responsibility for the company in a very difficult situation," he added.


{%- include head.html -%}
{%- include header.html -%}
{%- include social.html -%}
{{ content }} {%- include footer.html -%} {% include analytics.html %}


The SIGNAL procedure signals the occurrence of the named alert.
SIGNAL(name VARCHAR2, message VARCHAR2)
Parameters
name
Name of the alert.
message
Information to pass with this alert.
Examples
The following anonymous block signals an alert for alert_test.
DECLARE v_name VARCHAR2(30) := 'alert_test';BEGIN DBMS_ALERT.SIGNAL(v_name,'This is the message from ' || v_name); DBMS_OUTPUT.PUT_LINE('Issued alert for ' || v_name);END;Issued alert for alert_test


Download summary files from Census Bureau <https://www2.census.gov/> and extract data, in particular high resolution data at block, block group, and tract level, from decennial census and American Community Survey 1-year and 5-year estimates.
| Version: | 0.5.1 |
| Depends: | R (≥ 3.2.1) |
| Imports: | stringr (≥ 1.2.0), data.table (≥ 1.10.1), magrittr (≥ 1.5), purrr (≥ 0.2.4), utils (≥ 3.3.2) |
| Suggests: | knitr, rmarkdown, ggmap, ggplot2 |
| Published: | 2018-12-08 |
| Author: | Guanglai Li |
| Maintainer: | Guanglai Li <liguanglai at gmail.com> |
| BugReports: | https://github.com/GL-Li/totalcensus/issues |
| License: | MIT + file LICENSE |
| URL: | https://github.com/GL-Li/totalcensus |
| NeedsCompilation: | no |
| CRAN checks: | totalcensus results |
| Reference manual: | totalcensus.pdf |
| Package source: | totalcensus_0.5.1.tar.gz |
| Windows binaries: | r-devel: totalcensus_0.5.1.zip, r-release: totalcensus_0.5.1.zip, r-oldrel: totalcensus_0.5.1.zip |
| OS X binaries: | r-release: totalcensus_0.5.1.tgz, r-oldrel: totalcensus_0.5.1.tgz |
| Old sources: | totalcensus archive |
Please use the canonical form https://CRAN.R-project.org/package=totalcensus to link to this page.


Udacity Blog Part Two: Design Concepts
What is grid based design?
One of the main uses for a grid based design is to basically arranging your elements in order and algined keeping your page concept neat and managable as grids encourages form. By utlizing a grid system, you're creating a set concept and way of organising all your elements promoting seamless layout.
Whats all the hype about responsive webpages?
Responsive website design ensures users have a good viewing experience no matter what type of device they use. Today Google prioritizes mobile-friendly sites in its search results algorithm. Using HTML and CSS to resize, hide, shrink, enlarge, or move the content to make it look good on on a variety of devices and window or screen sizes. Recent work also considers the viewer proximity as part of the viewing context as an extension for RWD.
What is semantic structure?
Semantic structure is a fancy term for an organization that represents meaning. For example, an English sentence is a semantic structure. Consider the following sentence structure: subject - verb - object.


--- layout: page ---
These four positions are the required by the UQ Union and the society’s constitution. They are responsible for many of the administration and coordination tasks, including interaction with the UQ Union Clubs & Societies office and relevant UQ schools, such as the School of Information Technology and Electrical Engineering (ITEE).
|
President |
The President is responsible for all the club’s activities and administration. The role includes chairing meetings, ensuring that all communication with club members is carried out in a friendly and civilised manner, maintaining a relationship with the UQ Union Clubs and Societies office, and fulfilling the society’s obligations to the Union. |
|
Vice President |
The Vice-President is responsible for assisting the President in their duties. The role includes acting as chair of a meeting if the President is absent, and ensuring the responsibilities of the President (above) are carried out. |
|
Secretary |
The Secretary is responsible for many of the administration duties of the society. The role includes ensuring correspondence between the executive committee and the society members, compiling agendas and minutes of meetings, organising venues for events, and responding to communication on behalf of the members. |
|
Treasurer |
The Treasurer is responsible for keeping and maintaining the society’s bank account and financial records. The role includes keeping cheque and deposit books, receipts, and petty cash; preparing budgets; and ensuring that all authorised payments are carried out. |
|
System Administrator |
The system administrator is responsible for maintaining the society’s website and technical services. The role includes becoming familiar with the existing setup of the club’s website and informing the executive on how to use it; performing any fixes if it stops working; suggesting changes or additions, and acting on any of management’s decisions to change the technical services. |
|
General Committee Member |
The general committee members are responsible for giving an extra opinion and vote at committee meetings. The role involves attending management meetings and contributing to discussion and decisions that positively affect the society. |


|
Ocular Engine
|
#include <Exposable.hpp>
Public Member Functions | |
| void | getAllExposedNames (std::vector< std::string > &names) const |
| bool | getVariable (std::string const &name, ExposedVariable &var) |
| template<typename T > | |
| bool | getVariableValue (std::string const &name, T &var) |
| template<typename T > | |
| bool | setVariableValue (std::string const &name, T const &value) |
| virtual void | onVariableModified (std::string const &varName) |
Protected Member Functions | |
| void | exposeVariable (std::string const &name, std::string const &type, bool isPointer, bool isExposed, void *data) |
Protected Attributes | |
| std::unordered_map< std::string, ExposedVariable > | m_ExposedVariables |
Friends | |
| class | Buildable |
When inheriting from Exposable, an object is allowing for selected variables and pointers to be inspected, and potentially modified, by any outside source.
While external behaviour can not be guaranteed, the Ocular Engine will only inspect or modify variables when generic access is required. This typically falls under the following scenarios:
- Loading object from file - Saving object to file - Ocular Editor inspection
Many fundamental classes inherit from Exposable (including SceneObjects, Renderables, Routines, various Resources, etc.) in order to allow for the above scenarios to take place.
A class that inherits from Exposable may also see special automatic behaviour when also inheriting from the Buildable class. This typically includes automatic building from, and saving to, BuilderNode chains.
|
protected |
Exposes the specified member variable so that it's value can be retrieved or modified by anyone. This includes protected and private variables.
Internally (ie by the Ocular engine itself) this is typically only used for saving/loading purposes (or by the editor application). But it's use is in no way limited to just the engine.
| [in] | name | Full unique case-sensitive name of the member variable |
| [in] | type | String representation of the member variable type (see OCULAR_TYPE macro) |
| [in] | isPointer | Set to TRUE if the variable is a pointer |
| [in] | isExposed | Set to TRUE if the variable inherits from Exposable |
| [in] | ptr | Pointer to the member variable to expose |
| void Ocular::Core::Exposable::getAllExposedNames | ( | std::vector< std::string > & | names | ) | const |
Fills a vector with the names of all exposed variables. These names can then be used with the various get/set variable methods.
| [out] | names |
| bool Ocular::Core::Exposable::getVariable | ( | std::string const & | name, |
| ExposedVariable & | var | ||
| ) |
Attempts to retrieve the specified exposed variable.
| [in] | name | |
| [out] | var |
|
inline |
Attempts to retrieve the value of the specified exposed variable.
| [in] | name | |
| [out] | var |
|
virtual |
Optional method that should be called after modifying an exposed variable. This allows the implementation a chance to react to the modification if necessary.
Reimplemented in Ocular::Math::Transform, Ocular::Core::SceneObject, and Ocular::Core::PointLight.
|
inline |
Attempts to set the value of the specified exposed variable.
| [in] | name | |
| [in] | value |


A website where users may enter a GitHub username into a form, submit it, and see names and descriptions of that person's public repositories. A person can also look for repositories
Users Picture
{{search.bio}}
Twitter_Handle:{{search.twitter_username}}
Where Located:{{search.location}}
Status:{{search.hireable}}
Account Created on :{{search.created_at|date|uppercase}}
Updated at:{{search.updated_at |date|uppercase}}
{{search.followers}} followers
{{search.following}} following
{{search.public_gist}}
{{search.public_repo}}


--- layout: page title: "Contact" description: "Have questions? I have answers (maybe)." header-img: "img/hero-punch.jpg" ---
Want to get in touch with me? Shoot me a text or an email with your information and I'll try to respond as quickly as possible!
There are links to my GitHub and Twitter accounts below.
Email<EMAIL_ADDRESS>Phone:<PHONE_NUMBER>


SBI Life Insurance, ICICI Prudential likely to be included in MSCI index
ICICI Direct believes these stocks will be chosen on the higher free float market cap criteria.
SBI Life Insurance Company and ICICI Prudential Life Insurance are likely to be included in the MSCI index from December onwards, ICICI Direct has said.
These stocks are likely to be included in the upcoming MSCI reviews, it added.
The half-yearly review for MSCI India will be announced on November 10, 2019, which will be applicable from December 1, 2019.
The brokerage believes these stocks will be chosen on the higher free float market cap criteria.
Insurance stocks have been the biggest gainers in the last one year with ICICI Prudential rising 45 percent and SBI Life up 68 percent. Most experts believe these stocks are expected to be in demand due to low penetration of life insurance sector in India compared to developed countries.
On the other side, ICICI Direct believes due to the lower free float market cap, Indiabulls Housing Finance, and Glenmark Pharma may be excluded from the MSCI India index.Special Thursday Expiry on 10th 7th Nov
Early Closure to The Great Diwali Discount. Last Chance to get Moneycontrol PRO for a year @ Rs. 289/- only
Coupon code: DIWALI


ANTLR Development Tools
There are plug-ins for Intellij, NetBeans, Eclipse, Visual Studio Code, Visual Studio IDE, and jEdit.Intellij Plugin for ANTLR 4
We have a ANTLR v4 plugin for Intellij. Features:
- Syntax highlighting and syntax error checking
- Navigation window
- Code completion for tokens, rule names
- Live grammar interpreter for grammar preview
- Config per grammar, autobuild on save
- Use the meta-key while moving the mouse and it will show you token information; click on the token that's underlined and it will show the location in the ANTLR grammar editor window of the grammar element that matched that token
- Meta-j pops up a list of live templates, just like it does for Java programming
- Ctrl-return, or whatever you have configured for the generate pop-up, will bring up a list of things you can generate
- Alt-key while moving the mouse shows the region of input associated with parser rule that matched it
- Clicking at the same time jumps the grammar to the associated rule. A pop up shows the parser call stack leading to that parse region.
- Sophisticated profiler tab that shows ambiguities, lookahead depth, etc...
Sample usage showing Java grammar to the right.
ANTLRWorks2 and NetBeans Plugin for ANTLR 3 and 4
ANTLRWorks 2. This IDE is a sophisticated editor for ANTLR v3/v4 grammars as well as StringTemplate templates. It can run the ANTLR tool to generate recognizers and can run the TestRig (grun on command line) to test grammars. To integrate ANTLR-generated recognizers into your application, you will still need at least the runtime library.
ANTLR4 IDE Eclipse Plugin for ANTLR 4
Edgar Espina has created an Eclipse plugin for ANTLR v4. Features:
- Advanced Syntax Highlighting
- Automatic Code Generation (on save)
- Manual Code Generation (through External Tools menu)
- Code Formatter (Ctrl+Shift+F)
- Syntax Diagrams
- Advanced Rule Navigation between files (F3 or Ctrl+Click over a rule)
- Quick fixes
AntlrDT Tools Suite for Eclipse
1. AntlrDT — an ANTLR v4 grammar editor and builder
- full syntax-directed editor with outline view
- hyperlinked navigation between rules and rule subterms
- automatic builder with real-time problem feedback markers
- ANTLR grammar and Java native code formatter
- Parse Tree view – graphical visualization of generated parse trees
- builds parse trees using full (non-interpreted) grammar builds and parsing
- displays lexed tokens list and full parser/lexer errors list
- supports use of custom token emitters, token factories, and error strategies
- Paths view – graphical visualization of rule connectivity paths
- shows all possible parse paths from the start rule to any other selected rule
- particularly useful in the design of XVisitor grammars and ANTLR XPaths
2. XVisitorDT — an XVisitor grammar editor and builder
- full syntax-directed editor with outline view
- hyperlinked navigation between rules
- automatic builder with real-time problem feedback markers
- builder performs automatic visitor code generation
- utilizes fully symbolic XPath-style rule path references
- ANTLR grammar and Java native code formatter
3. StringTemplateDT — a StringTemplate v4 template editor
- full syntax-directed editor with outline view
- for template ('.st') and template group ('.stg') files
Update site: http://www.certiv.net/updates/
Visual Studio Code plugin for ANTLR 4
Mike Lischke created an ANTLR 4 plug-in for Visual Studio Code. Features:
- Syntax coloring for ANTLR grammars (.g and .g4 files).
- An own color theme, which not only includes all the recommended groups, but also some special rules for grammar elements that you don't find in other themes.
- Parser generation on save.
- Syntax and some semantic error checking (symbol matching)
- Quick navigation via ctrl/cmd+click.
- The symbol list for quick navigation (via shift+ctrl/cmd+O).
- Hovers (tooltips) with symbol information.
- Rule reference counts via code lens.
- Railroad diagrams for all types of rules (parser, lexer, fragment lexer).
- ATN graphs for all rule types. This is a visualization of the internal ATN that drives lexers + parsers. It uses D3.js for layout and interaction. Nodes can be repositioned with the mouse and you can drag and zoom the image. The transformation and position state is restored when reopening a graph.
Visual Studio IDE extension for ANTLR 4
Ken Domino created an ANTLR 4 extention for Visual Studio 2019. Features:
- Syntax highlighting for ANTLR grammars (.g and .g4 files).
- Quick navigation for next and previous rules in grammar.
- Intellisense command completion and tooltips.
- Go to def/refs.
- Go to C# visitor and listener methods, with generation if missing.
- Reformat using Codebuff.
- Support for integrated C# parser generation via Antlr Java tool, compile, and debug.
- Templates for NET Code and Framework Antlr examples.
Visual Studio IDE extension for ANTLR 4
Zev Spitz has created a Visual Studio debugging visualizer that supports Antlr.Runtime.Standard.DLL, as well as the older Antlr.Runtime.DLL. Features:
- Multiple views: token list, parse tree nodes, source
- Selection sync across views
- Targets
System.String
,BufferedTokenStream
,RuleContext
, and subclasses - Filter by text, whitespace, and/or errors; and by specific token types or rule contexts
- UI controls can be embedded in third-party applications
jEdit plugin for ANTLR 4
The jEdit editor has a plugin with:
- Syntax highlighting
- Code navigation
- Code generation
- Error checking


name: Private-Terraform-Enterprise-Lab class: center,middle,title-slide count: false ![:scale 40%](images/Terraform_VerticalLogo_FullColor.png) Private Terraform Enterprise Lab ========================= Installing and Troubleshooting PTFE In Restricted Corporate Networks ??? Welcome to the Private Terraform Enterprise lab! I'm your instructor, YOURNAME, and today I'll help you tame the beast that is PTFE. During the training you will install Private Terraform inside of a simulated corporate network with a proxy and firewall rules. Along the way we'll point out common mistakes and pitfalls to avoid. Let's get started! --- name: Introductions class: center,middle,title-slide count: false Introductions ========================= .contents[ * Your Name * Job Title * Experience with Terraform * Fun fact about you ] ??? Let's start with some brief introductions. Please share your name, your job title, any experience you've had with Terraform and perhaps an interesting fact about yourself. --- name: Your-Instructor class: center,middle,title-slide count: false Your Instructor ========================= .center[ ![:scale 80%](./images/c64.jpg) ] Sean Carolan at the Seattle Computer Museum ??? Introduce yourself to the class, share a bit about your background. You can also add an image or text to this slide if you like. --- name: Its-Dangerous-to-Go-Alone class: center, middle, title-slide count: false ![:scale 100%](./images/its-dangerous-to-go-alone.png) ??? Be prepared! This workshop is designed to give you enough practice to install and troubleshoot PTFE on your own. --- name: Follow-Along class: center, middle, title-slide count: false Link to Slide Deck ========================= https://scarolan.github.io/ptfe-lab ------------------------- Login with password: .darkgreen[hashicorptraining] ??? This entire slide presentation was created in Markdown. Because Powerpoint is for the Muggles. If you want to skip ahead, that's fine but we won't be answering questions about slides we haven't covered yet. You can use the slide deck to get caught up or review in case you fall behind. I encourage you to work together and collaborate on the exercises. You can also use this slide deck to give the training yourself. All of the content is stored on github, so if you want to contribute it's as easy as editing a text file and submitting a pull request. --- name: Table-of-Contents class: center,middle Table of Contents ========================= .contents[ 1. Intro to Private Terraform Enterprise 2. Lab Setup 3. Install Bitbucket Server 4. Install Private Terraform Enterprise 5. Connect PTFE to Bitbucket 6. Provision an App Server ] ??? Here are the different chapters of this slide deck and training lab. We'll start with a brief introduction to Private Terraform Enterprise, and then we'll provision a brand new lab environment using Terraform. Once we have access to our lab environments, each of you will install a Bitbucket Server. Next we'll install PTFE, and finally we'll connect PTFE to Bitbucket and provision an application server using some sample Terraform code. After we have everything configured correctly, we will snapshot your environment so you can continue to use it later for demos, practice, and troubleshooting. --- name: Chapter-1 class: center,middle .section[ Chapter 1 Intro to Private Terraform Enterprise (PTFE) ] --- name: What-is-PTFE What is Private Terraform Enterprise? ------------------------- * Software provided to customers * Runs in a private, isolated environment * Customer is responsible for installation, maintenance and backups * Containerized application that runs on Docker * License management and GUI provided by Replicated * Can run on bare metal, VMs or cloud ??? Terraform Enterprise was originally offered as a SaaS (Software as a Service) application but we quickly realized that many customers wish to run it behind their firewalls. PTFE was created to fill this need. Originally it was based on an AMI, which worked ok but was restricted to AWS only. During the first major rewrite, legacy components of the application were removed, and the installer was updated to use the Replicated platform. --- name: PTFE-Architecture PTFE Architecture ------------------------- ![:scale 100%](./images/tfe-architecture.mermaid.png) ??? TODO: Some notes about PTFE architecture --- name: Replicated Replicated ------------------------- .content[.center[![:scale 50%](./images/replicated_logo.png) http://www.replicated.com] Replicated provides SaaS vendors a container-based platform for easily deploying their cloud-native applications inside customers' environments. It provides license management, online and airgap installation, and a control panel for your docker-based application.] ??? Replicated definintely has some rough edges and bumps to be aware of. We'll cover some of those during the training today. --- name: Lab-Network-Architecture Lab Network Architecture ------------------------- ![:scale 100%](./images/network_diagram.png) ??? This is what your lab environment will look like from a network architecture point of view. Note that there are both private and public networks in this VPC. Machines deployed onto the private network have no internet access except via this proxy server. You can use this proxy to simulate the feeling of a restircted corporate network. --- name: Chapter-1-Review class: middle,center .review[ Chapter 1 Review Intro to PTFE ] .contents[ ☑ Terraform Enterprise ☑ Replicated ☑ AWS lab network architecture ☑ Prerequisites [Appendix A](#Appendix-A) ] ??? So far we've covered Terraform Enterprise, Replicated, and your AWS lab network architecture. If you haven't already please take a look at the prequisites listed in Appendix A. --- name: Chapter-2 class: center,middle .section[ Chapter 2 Lab Setup ] --- name: Your-Training-Lab Your Training Lab Environment ------------------------- TODO: Put an image on this slide. Maybe showing PTFE dashboard or something. This lab environment costs around $8.40 per day to run. ??? You can demo your own completed environment here if you wish and time permits. --- name: Clone-Training-Repo Clone the Training Lab Git Repo ------------------------- Command: ```bash git clone https://github.com/scarolan/ptfe-lab ``` Output: ```asciidoc Cloning into 'ptfe-lab'... remote: Counting objects: 92, done. remote: Compressing objects: 100% (62/62), done. remote: Total 92 (delta 52), reused 66 (delta 30), pack-reused 0 Unpacking objects: 100% (92/92), done. Checking connectivity... done. ``` ??? This git repository contains Terraform code for building the lab. The accompanying slide deck is an HTML markdown slideshow stored in the docs directory. You can either open the file locally in a web browser, or access it here: https://scarolan.github.io/ptfe-lab TODO: Make this repository public, move it into HashiCorp organization. --- name: Configure-Your-Variables Configure Your Variables ------------------------- Commands: ```bash cd ptfe-lab cp terraform.tfvars.example terraform.tfvars ``` Edit the terraform.tfvars file you just created and update your variables. --- name: Configure-Your-Variables-2 Configure Your Variables ------------------------- File Contents: ```bash # Customize these variables with your own settings *name = "YOURNAME" workstationpw = "Thisisaverylongpassword1234" tag_owner =<EMAIL_ADDRESS>tag_ttl = "72" rhel_version = "7.5" bbs_http_port = "7990" bbs_https_port = "8443" domain_name = "hashidemos.io" zone_id = "Z2VGUC188F45PC" ``` ??? NOTE: Partners or users who are deploying the lab into an account other than the HashiCorp SE AWS Demos account should update `domain_name` and `zone_id` to match the domain that is managed in route 53 in their own account. Everyone in the training should update the name variable. --- name: Run-Terraform-Init Run Terraform Init ------------------------- Command: ```bash terraform init ``` Output: ```asciidoc ... Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ``` ??? Terraform init is used to fetch all the modules and providers you'll need to build the lab. --- name: Run-Terraform-Apply Run Terraform Apply ------------------------- Command: ```bash terraform apply ``` Output: ```asciidoc ... Workstation: ec2-18-236-70-20.us-west-2.compute.amazonaws.com Username: Administrator Password: Thisisaverylongpassword1234 Once you have logged onto the workstation, you may SSH to the other instances using the short hostname. For example: ssh ptfe ssh bitbucket ``` Pay close attention to the Terraform output. It contains instructions that you will need in Chapter 3. You may wish to copy the output into a text file for easy reference. --- name: Log-onto-Workstation Log on to your Workstation ------------------------- .center[![:scale 50%](./images/rdp_screenshot.png)] ??? Mac OSX users should download the official Remote Desktop client from the app store. Windows users will have RDP built in already. Linux users may use rdesktop to connect. We suggest running the lab in full-screen mode, on its own desktop workspace.  The first page you see will contain instructions on how to make Chrome your default browser. .center[![:scale 50%](./images/chrome_default.png)] ??? Let's go ahead and move in, spruce the place up a bit and configure our desktop environment. We'll start with the default web browser.   Then click the green icon in the upper left corner of the terminal window. This will bring you to the settings menu. You may wish to change some of the default settings. Here are some options: * Select 'Main' to adjust your font size and font. * Select 'Startup' to change your default shell to Powershell. * Select 'Colors' under 'Features' and you can change the color scheme for your terminal. ??? From the developer: "Cmder is a software package created out of pure frustration over the absence of nice console emulators on Windows. It is based on amazing software, and spiced up with the Monokai color scheme and a custom prompt layout, looking sexy from the start." In case you're curious, there is no emoji support and no 256 bit color support on Windows terminals. You can kind of hack 256 color support into it but it's a pain in the arse and requires running a shim that intercepts character codes and renders them correctly. This is the part where all you Mac and Linux users get to act all smug. --- name: Set-up-cmder-2 Set up cmder ------------------------- .center[![:scale 80%](./images/inject_conemuhk.png)] Go to the 'Features' menu. Unselect the box for Inject ConemuHK. This will ensure that the terminal works correctly on your remote machines. Save all your changes. ??? This step is here because we ran into a weird issue with Cmder, SSH, and vim on the target system. vim was starting in 'Replace' mode due to some weird control character that was getting injected into the session. Turning this off seems to make vim and xterm happy. Why is this inject thing here in the first place? Because of Windows bugs. https://conemu.github.io/en/MicrosoftBugs.html --- name: Verify-SSH Verify SSH Connectivity ------------------------- Commands: ```bash ssh ptfe hostname ssh bitbucket hostname ssh proxy hostname ``` Output: ```asciidoc ssh ptfe hostname The authenticity of host 'ptfe.YOURNAME.hashidemos.io (<IP_ADDRESS>)' can't be established. ECDSA key fingerprint is SHA256:dEYEIngoY7n+yVQW5ZQ0B+jIyaO2UlZ4qk74A1oLaT0. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ptfe.YOURNAME.hashidemos.io,<IP_ADDRESS>' (ECDSA) to the list of known hosts. ptfe.YOURNAME.hashidemos.io ``` ??? There are three Linux hosts on the network that you can SSH onto. The PTFE server, the Bitbucket server, and the Squid proxy server. For convenience we created an ssh config file in the Administrator user's home directory (~/.ssh/config). This is what allows you to use short hostnames to SSH to the other hosts. When you SSH to each host for the first time, you'll be prompted to accept the key fingerprint before you continue. This lets SSH know that you trust the remote machine. You can have multiple tabs or windows open, with each connected to a different machine. --- name: Chapter-2-Review class: middle,center .review[ Chapter 2 Review Lab Setup ] .contents[ ☑ Cloned the training repo ☑ Configured variables ☑ Installed Terraform modules and providers ☑ Provisioned the lab environment ☑ Verified connectivity ] ??? Go over what you covered in chapter 2. --- name: Chapter-3 class: middle, center .section[ Chapter 3 Install Bitbucket Server ] --- name: What-is-Bitbucket What is Bitbucket? ------------------------- .center[![:scale 100%](./images/bitbucket_website.png)] ??? Bitbucket is a hosted or on-premise implementation of Git that is built for collaboration among teams. It provides similar functionality to GitHub and GitLab. Today we'll be using a trial version of the Bitbucket Server version. --- name: Connect-to-BBS Connect to the Bitbucket Server ------------------------- Command: ```bash ssh bitbucket ``` --- name: Get-Root Gain a root shell ------------------------- Command: ```bash sudo /bin/su - root ``` Note: This command is run on your Bitbucket server, not your workstation. Sometimes students will get confused if they are on the wrong machine. Gently point out the hostname in the terminal as a way to get your bearings. --- name: Run-BBS-Installer Run the Bitbucket Installer ------------------------- Command: ```bash ./atlassian-bitbucket-5.12.0-x64.bin ``` 1. Select option 1 to start a new installation. 2. Select option 1 to install a Server instance. 3. Hit [Enter] to install in the default home directory. 4. Hit [Enter] to install in the default data directory. 5. Hit [Enter] to use the default http port 7990. 6. Hit Y to install Bitbucket as a service. 7. Hit i and [Enter] to install Bitbucket. 8. Answer y to launch Bitbucket. In other words, just use all the default settings. --- name: Verify-BBS-Running Verify that Bitbucket Server is Running ------------------------- ```bash service atlbitbucket status ``` If for some reason Bitbucket is not running, you can start it like this: ```bash service atlbitbucket start ``` ??? Bitbucket should now be running on port 7990. Proceed to the next step. --- name: Open-the-web-UI Open the Bitbucket Web UI ------------------------- .center[![:scale 80%](./images/bitbucket_setup_1.png)] Open the web UI using the URL shown in your Terraform output. It should look something like this: .center[http://bitbucket.YOURNAME.hashidemos.io:7990] ??? If it wasn't obvious, you need to change YOURNAME to match your own environment. This is same as the `name` variable that we set in Chapter 2. --- name: Continue-with-UI-Installer Continue with the UI Installer ------------------------- .center[![:scale 80%](./images/bitbucket_setup_2.png)] If you already have an Atlassian account you can use it to sign in here. If not click the "create an account" link. The easiest way to log in is by using your Google credentials. ??? If you don't have an account yet the easiest way to get a new one is simply to log on using your personal or work Google account. Atlassian will automatically set up your account and provision your license onto the local instance. Slick. --- name: Get-Trial-License Get a Trial License ------------------------- .center[![:scale 80%](./images/bitbucket_setup_3.png)] Choose the Bitbucket Server option on the left. Leave everything else as is and click on 'Generate License' at the bottom of the page. Continue through the installer and set up your admin username and password. Verify that you can log onto the server with your new credentials. ??? Verify that all your participants are able to log in before proceeding. --- name: Configure-BBS-Proxy Configure Bitbucket Proxy Settings ------------------------- Choose Your Text Editor: ```bash vim /opt/atlassian/bitbucket/5.12.0/bin/_start-webapp.sh ``` ```bash nano /opt/atlassian/bitbucket/5.2.0/bin/_start-webapp.sh ``` File Contents: ```asciidoc # Occasionally Atlassian Support may recommend that you set some specific JVM arguments. You can use this # variable to do that. Simply uncomment the below line and add any required arguments. Note however, if this # environment variable has been set in the environment of the user running this script, uncommenting the below # will override that. # JVM_SUPPORT_RECOMMENDED_ARGS="-Dhttps.proxyHost=proxy.YOURNAME.hashidemos.io -Dhttps.proxyPort=3128 -Dhttp.nonProxyHosts=ptfe.YOURNAME.hashidemos.io" ``` Edit the _start-webapp.sh file and look for a line that contains `JVM_SUPPORT_RECOMMENDED_ARGS`. Replace it with the settings in your Terraform output. ??? This is a safe space - no editor shaming. Unless you're using emacs. Then we are totally going to shame you. Emacs is an awesome operating system. If only it came with a decent text editor. It's the only text editor that comes with 'therapist' mode. You can always tell an emacs user by their buff crossfit pinky fingers. GNU EMACS stands for Generally Not Used, Except by Middle-Aged Computer Scientists. I've been using vi for 20 years...mostly because I can't figure out how to exit it. Vi has two modes: Beep repeatedly, and "break everything". I have created a foolproof random number generator. Stick a windows user in front of vi and ask them to exit the program. This might be a little tricky for people who aren't used to the command line. Suggest using nano if your users aren't comfy with vim. They should be able to copy this configuration right out of the Terraform output from Chapter 2.  It's important to stop here because you have to create a DNS record before proceeding. --- name: Create-DNS-TXT-Record Create a DNS TXT Record ------------------------- .center[![:scale 50%](./images/bitbucket_setup_4.png)] For this step you'll need to log onto the AWS console. Once you're logged in navigate to the Route 53 section. Select the domain name you'll be using in your lab. Use the 'Create Record Set' to create a new TXT record to verify your cert with LetsEncrypt. ??? Remember to set the record type to TXT, enclose the data in quotes, and set your TTL to 30 seconds. After you create your record wait at least 60 seconds before going back to the certbot script and hitting Enter. If you hit Enter too early, or have made a mistake the verification process will fail and you have to start all over. Patience is a virtue. ??? The key here is to wait at least one minute after creating your DNS text record. This ensures that your new DNS TXT record is available to the LetsEncrypt ACME service which will verify your request. --- name: Verify-SSL-Cert Verify your SSL certificate ------------------------- ```asciidoc Waiting for verification... Resetting dropped connection: acme-v02.api.letsencrypt.org Starting new HTTPS connection (2): acme-v02.api.letsencrypt.org Cleaning up challenges IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: * /etc/letsencrypt/live/bitbucket.YOURNAME.hashidemos.io/fullchain.pem Your key file has been saved at: * /etc/letsencrypt/live/bitbucket.YOURNAME.hashidemos.io/privkey.pem Your cert will expire on 2018-11-02. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" ``` Now you can hit [Enter] to finish the verification process. If it is successful you'll see a message like the one above. Your cert and key are saved in the /etc/letsencrypt/live directory. --- name: Convert-to-p12 Convert the SSL Cert to p12 Format ------------------------- Command: ```bash openssl pkcs12 -export -name bitbucket.YOURNAME.hashidemos.io -in /etc/letsencrypt/live/bitbucket.YOURNAME.hashidemos.io/cert.pem -inkey /etc/letsencrypt/live/bitbucket.YOURNAME.hashidemos.io/privkey.pem -out /root/keystore.p12 ``` Here we convert our certificate into p12 format. This is an intermediate step. Copy the command from your Terraform output and paste it into your cmder terminal. Choose a password that you can remember. --- name: Convert-to-JKS Convert the SSL Cert to JKS Format ------------------------- ```bash /opt/atlassian/bitbucket/5.12.0/jre/bin/keytool -importkeystore -destkeystore /var/atlassian/application-data/bitbucket/shared/config/ssl-keystore.jks -srckeystore /root/keystore.p12 -srcstoretype pkcs12 -alias bitbucket.YOURNAME.hashidemos.io ``` This command will convert your cert into the Java Keystore format expected by Bitbucket. Copy and paste the command from your Terraform output and it will put the JKS cert into the correct location. --- name: Edit-BBS-Properties Edit the bitbucket.properties file ------------------------- Commands: ```bash vim /root/bitbucket.properties cp /root/bitbucket.properties /var/atlassian/application-data/bitbucket/shared/bitbucket.properties ``` File contents: ```java server.port=8443 server.ssl.enabled=true server.ssl.key-store=/var/atlassian/application-data/bitbucket/shared/config/ssl-keystore.jks server.ssl.key-store-password=CHANGEME server.ssl.key-password=CHANGEME server.ssl.key-alias=bitbucket.YOURNAME.hashidemos.io ``` Update the passwords and the server.ssl.key-alias to match your settings. --- name: Restart-Bitbucket-Server Restart Bitbucket Server ------------------------- Command: ```bash service atlbitbucket restart ``` If all went as planned, your Bitbucket instance will restart with SSL enabled on port 8443. --- Log onto the https web UI ------------------------- .center[![:scale 80%](./images/bitbucket_setup_5.png)] You should now see a green padlock in the browser URL address bar. --- Fix the "Base URL Mismatch" Error ------------------------- .center[![:scale 80%](./images/bitbucket_setup_6.png)] Click on the yellow warning icon in the upper right corner. Update your Base URL with the new, https-enabled setting. Congratulations! You now have an SSL-enabled Bitbucket server instance. --- name: Chapter-3-Review class: middle,center .review[ Chapter 3 Review Bitbucket Installation ] .contents[ ☑ Installed Bitbucket ☑ Set up the proxy ☑ Generated an SSL cert with LetsEncrypt ☑ Configured SSL cert and Base URL ☑ Restarted Bitbucket in SSL mode ] --- name: Chapter-4 class: middle, center .section[ Chapter 4 Install Private Terraform Enterprise ] --- name: Log-on-to-PTFE Log on to the PTFE server ------------------------- Commands: ```bash ssh ptfe ``` ```bash sudo /bin/su - root ``` ??? Got Root? --- name: Install-Docker Install Docker ------------------------- Command: ```bash yum -y install docker ``` Output: ```asciidoc Installed: docker.x86_64 2:1.13.1-68.gitdded712.el7 Dependency Installed: atomic-registries.x86_64 1:1.22.1-22.git5a342e3.el7 container-selinux.noarch 2:2.66-1.el7 container-storage-setup.noarch 0:0.10.0-1.gitdf0dcd5.el7 device-mapper-event.x86_64 7:1.02.146-4.el7 device-mapper-event-libs.x86_64 7:1.02.146-4.el7 device-mapper-persistent-data.x86_64 0:0.7.3-3.el7 docker-client.x86_64 2:1.13.1-68.gitdded712.el7 docker-common.x86_64 2:1.13.1-68.gitdded712.el7 docker-rhel-push-plugin.x86_64 2:1.13.1-68.gitdded712.el7 libaio.x86_64 0:0.3.109-13.el7 lvm2.x86_64 7:2.02.177-4.el7 lvm2-libs.x86_64 7:2.02.177-4.el7 oci-register-machine.x86_64 1:0-6.git2b44233.el7 oci-systemd-hook.x86_64 1:0.1.16-1.git05bd9a0.el7 oci-umount.x86_64 2:2.3.3-3.gite3c9055.el7 python2-pytoml.noarch 0:0.1.17-1.el7 skopeo-containers.x86_64 1:0.1.31-1.dev.gitae64ff7.el7 yajl.x86_64 0:2.0.4-4.el7 Complete! ``` ??? Here we are installing docker from the RHEL Extras repository, which we've already enabled for you in your config. According to our documentation docker 1.13.1 is supported, even though the installer will warn you not to use it. ¯\_(ツ)_/¯ https://www.terraform.io/docs/enterprise/private/rhel-install-guide.html --- name: Remove-RHEL-Auth-Plugin Remove the RHEL Auth Plugin ------------------------- Command: ```bash vim /usr/lib/systemd/system/docker.service ``` File Contents: ```asciidoc Environment=GOTRACEBACK=crash Environment=DOCKER_HTTP_HOST_COMPAT=1 Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin ExecStart=/usr/bin/dockerd-current \ --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \ --default-runtime=docker-runc \ * --authorization-plugin=rhel-push-plugin \ --exec-opt native.cgroupdriver=systemd \ --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \ --init-path=/usr/libexec/docker/docker-init-current ``` Delete the highlighted line and save the file. ??? This plugin causes memory issues and therefore we disable it. This step is mandatory. https://www.terraform.io/docs/enterprise/private/rhel-install-guide.html#mandatory-configuration --- name: Start-Docker Start Docker ------------------------- Command: ```bash systemctl start docker ``` --- name: Testing-with-InSpec Testing with InSpec ------------------------- .centered[ ![:scale 60%](./images/inspec.png) ### http://www.inspec.io ] InSpec is an open source testing tool that can be used to inspect server configurations. It can be run on the local machine itself or remotely over SSH. We're going to use InSpec to run some pre-flight checks to ensure that our server is ready for PTFE installation. ??? Why InSpec? Because it's awesome and really easy to work with. Just declare what state you expect the machine to be in, and InSpec does the rest for you. Nothing required to be installed on the target machine at all. This is just another tool you can have in your kit to make your PTFE installations easier. You can also have your customer do this before you come on site to help them. --- name: Download-the-ChefDK Download the ChefDK ------------------------- .centered[ ![:scale 100%](./images/chefdk_download.png) ### https://downloads.chef.io/chefdk ] ??? The Chef Development Kit includes InSpec. It's free and is the easiest way to get InSpec onto your Windows workstation. Download the x86_64 bit installer for windows and install it. --- name: Install-the-ChefDK Install the ChefDK ------------------------- .centered[ ![:scale 50%](./images/chefdk_installer.png) Use the default settings and click Install. ] ??? Nothing too crazy here. The ChefDK is 350MB in size because it includes its own Ruby and all the dependencies you need. Deal with it or install your own Ruby and dependencies. --- name: Clone-Preflight-Repo Clone the Preflight Check Repo ------------------------- Command: ```bash git clone https://github.com/scarolan/ptfe-preflight-check ``` Output: ```asciidoc Cloning into 'ptfe-preflight-check'... remote: Counting objects: 44, done. remote: Total 44 (delta 0), reused 0 (delta 0), pack-reused 44 Unpacking objects: 100% (44/44), done. ``` Note: Do this on your workstation, NOT the PTFE server. --- name: Edit-InSpec-Variables Edit the InSpec Test Variables ------------------------- Command: ```bash code .\ptfe-preflight-check\controls\network.rb ``` File Contents: ```ruby # Network tests for PTFE preflight check # Set these to match your environment #proxy_url = 'http://my.proxy.server' #vcs_url = 'https://my.bitbucket.server' vcs_url = 'https://bitbucket.YOURNAME.hashidemos.io:8443' proxy_url = 'http://proxy.YOURNAME.hashidemos.io:3128' ``` You can use the `code` shortcut anywhere in Windows to edit files or open directories using Visual Studio Code. Edit the network.rb file and set vcs_url and proxy_url to match your environment. Save and close the file. --- name: Run-Preflight-Checks Run Preflight Checks on PTFE Server ------------------------- Command: ```bash inspec exec ptfe-preflight-check -t<EMAIL_ADDRESS>-i ~/.ssh/id_rsa --sudo ``` Output: ```asciidoc ... [PASS] docker-config: File /usr/lib/systemd/system/docker.service [PASS] File /usr/lib/systemd/system/docker.service content should not include "--authorization-plugin=rhel-push-plugin" [PASS] Command docker info 2> /dev/null | grep Authorization exit_status should eq 1 [PASS] storage-config: Command uname -r [PASS] Command uname -r stdout should cmp > "3.10.0-693" [PASS] Mount / should be mounted [PASS] Mount / type should eq "xfs" [PASS] Command xfs_info / stdout should include "ftype=1" [PASS] Filesystem / size should be >= 41943040 Profile Summary: 6 successful controls, 0 control failures, 0 controls skipped *Test Summary: 14 successful, 0 failures, 0 skipped ``` ??? The important thing here is that *all* of the preflight checks pass before you proceed. If you followed the instructions carefully everything should be working. --- name: Generate-a-Trial-License Download Your Trial PTFE License ------------------------- .centered[ ![:scale 50%](./images/replicated_login.png) ] Ask your friendly neighborhood HashiCorp Solutions Engineer to generate a trial license for you. You should receive a download link along with a license file, and a password to access your installer bundle. Copy and paste the link into Chrome inside your workstation environment. Enter the password you received from your SE to download the bundle. Your download link should look something like this:
https://get.replicated.com/airgap/#/terraformenterprise/74fd29441337448b545fac76fca6c7e0
??? If you are a HashiCorp SE, you should have access to https://vendor.replicated.com/login to do this. Make sure when you generate the license that `Airgap Download` is enabled. --- name: Download-Airgap-Installer Download the Airgap Installer ------------------------- .centered[ ![:scale 50%](./images/replicated_clipboard.png) ] Click on the little clipboard icon to copy the real download URL into your clipboard. Go back to your PTFE server command line and enter the following commands. Make sure you are connected and [logged on as root](#Log-on-to-PTFE). You *must* wrap your URL in quotes for this to work. ```bash export https_proxy=http://proxy.YOURNAME.hashidemos.io:3128 wget "YOURDOWNLOADURL" -O /root/ptfe.airgap ``` This should download a file called `ptfe.airgap` into root's home directory. --- name: Download-Install-Script Download the install.sh Script ------------------------- Command: ```bash curl https://install.terraform.io/ptfe/stable > /root/install.sh chmod +x install.sh ``` ??? This is buried way down near the bottom of the PTFE installation docs page: https://www.terraform.io/docs/enterprise/private/install-installer.html#run-the-installer-online --- name: Run-Install-Script Run the install.sh script ------------------------- Command: ```bash ./install.sh ``` Output: ```asciidoc Determining local address The installer was unable to automatically detect the private IP address of this machine. Please choose one of the following network interfaces: [0] eth0 <IP_ADDRESS> [1] docker0 <IP_ADDRESS> *Enter desired number (0-1): 0 The installer will use network interface 'eth0' (with IP address '<IP_ADDRESS>'). Determining service address The installer will use the proxy at 'http://proxy.sean.hashidemos.io:3128' (imported from env var 'https_proxy') This installer will upgrade your current version of Docker (1.13.1) to the recommended version: 17.12.1 *Do you want to allow this? (Y/n) N The installed version of Docker (1.13.1) may not be compatible with this installer. The recommended version is *Do you want to proceed anyway? (y/N) Y It does not look like Docker is set up with http proxy enabled. This script will automatically configure it now. *Do you want to allow this? (Y/n) Y ``` ??? Match your answers to what is highlighted in yellow. Select 0, N, Y and Y. For some reason the installer complains even though we explicitly state that we support it in the documentation. --- name: Leave-Service-IP-Blank Leave Service IP Address Blank ------------------------- Leave the Service IP address setting blank and hit [Enter] to continue. ```asciidoc Determining service address The installer was unable to automatically detect the service IP address of this machine. Please enter the address or leave blank for unspecified. *Service IP address: ``` --- name: Install-Script-Complete Install Script Complete ------------------------- At this point you should see the following output: ```asciidoc Operator installation successful SELinux is enabled. Switching to enforcing mode and running docker with the "--selinux-enabled" flag may cause some features to become unavailable. To continue the installation, visit the following URL in your browser: https://
:8800 ``` You may ignore the warning about SELinux for now. We have set it to 'permissive' mode. When the machine is rebooted it will be completely disabled. ??? During the provisioning of the lab we set `SELINUX=disabled` in the /etc/selinux/config file, and set the machine into permissive mode so we can still use it without a reboot. --- name: Open-GUI-Installer Proceed to the GUI Installer ------------------------- .center[ ![:scale 70%](./images/ptfe_ssl_warning.png) ] Open the URL of your PTFE console in Chrome. It should look like this: https://ptfe.YOURNAME.hashidemos.io:8800 Click on the "Advanced" link on the lower left to proceed. --- name: Generate-PTFE-SSL-Cert Generate an SSL Cert ------------------------- We're going to generate another SSL cert [just like we did for Bitbucket](#Generate-SSL-Cert), only this time we don't have to convert it into JKS format.  This will allow you to transfer them back onto your workstation. Command: ```bash cp -rL /etc/letsencrypt/live/ptfe.YOURNAME.hashidemos.io /home/ec2-user/ ``` --- name: SCP-Certs-to-Workstation Copy SSL Certs to Your Workstation ------------------------- Open a new cmder Window on your workstation and enter the following command. You should see a new directory appear on your desktop. ```bash scp -r ptfe:~/ptfe.YOURNAME.hashidemos.io C:\Users\Administrator\Desktop ``` --- name: Upload-SSL-Cert-and-Key Upload Your SSL Certificate and Key ------------------------- .centered[ ![:scale 100%](./images/ptfe_installer_1.png) ] ??? Upload the privkey.pem and fullchain.pem files, and enter your DNS hostname into the Hostname field. --- name: Upload-License-to-Workstation Upload License File to Your Workstation ------------------------- Your HashiCorp Solutions Engineer should have provided you with a license file via email. You can't simply copy and paste it into your workstation because it is a binary file. Use one of the following options: 1. Enable a shared drive in your RDP settings and copy the license over. 2. Log onto your email inside of the workstation and download the file. 3. Stick it on Dropbox or another file sharing service that you can reach from your workstation browser. Once you've managed to get the *.rli license file onto your workstation you can upload it into the UI. ??? This bit feels a bit hacky but there's not much we can do about it. Customers go through this stuff every single day. --- name: Select-Airgapped-Install Select 'Airgapped' ------------------------- .centered[ ![:scale 60%](./images/ptfe_installer_2.png) ] Select the 'Airgapped' option and continue. --- name: Enter-Path-to-Installer Enter the Path to the Installer ------------------------- .centered[ ![:scale 60%](./images/path_to_airgap_package.png) ] This should be `/root/ptfe.airgap`. If you downloaded it to `/home/ec2-user/ptfe.airgap` enter that instead. --- name: Proceed-Anyway Select 'Proceed Anyway' ------------------------- Next you'll select a password for the setup console, and another pre-flight check will run. One of the checks will fail because it can't reach the Internet: ```bash Can access releases.hashicorp.com - ERROR: timeout Can connect to releases.hashicorp.com:443. ``` You may safely ignore this failure and click 'Proceed Anyway'. --- name: Select-Installation-Type Select 'Demo' ------------------------- .centered[ ![:scale 100%](./images/install_type.png) ] On the next page select the 'Demo' installation type. --- name: Bypass-Proxy-for-BBS Add Bitbucket Server to Proxy Bypass ------------------------- .centered[ ![:scale 100%](./images/ptfe_proxy_bypass.png) ] Enter the hostname of your bitbucket server in the Proxy Bypass field. --- name: Save-and-Restart Click 'Save' and Restart ------------------------- Load the URL in your browser: https://ptfe.YOURNAME.hashidemos.io:8800 ![:scale 100%](./images/ptfe_replicated_console.png) Click on the 'Open' link to launch Terraform Enteprise. --- name: Chapter-4-Review class: middle,center .review[ Chapter 4 Review PTFE Installation ] .contents[ ☑ Installed Docker 1.13.1 from RHEL extras ☑ Ran a pre-flight check with InSpec ☑ Installed Private Terraform Enterprise ☑ Generated an SSL certificate for PTFE ☑ Configured the PTFE console settings ] --- name: Chapter-5 class: middle, center .section[ Chapter 5 Connect PTFE to Bitbucket ] --- name: Follow-the-BBS-PTFE-Docs Follow the Documentation! ------------------------- Please follow the official documentation to integrate your PTFE and Bitbucket servers. Be sure to report any issues or mistakes you find in the docs. .centered[ https://www.terraform.io/docs/enterprise/vcs/bitbucket-server.html ] --- name: Chapter-5-Review class: middle,center .review[ Chapter 5 Review Connect PTFE to Bitbucket ] .contents[ ☑ Created a web hook ☑ Generated an SSH key for PTFE ☑ Connected Bitbucket to PTFE ] --- name: Chapter-6 class: middle, center .section[ Chapter 6 Provision an App Server ] --- name: Create-a-Repo Create a Git Repository ------------------------- .center[ ![:scale 100%](./images/create_repository.png) ] Create a project and a repository on your Bitbucket server.  git push origin master ``` --- name: Create-a-Workspace Create a New Workspace ------------------------- .center[ ![:scale 80%](./images/create_workspace.png) ] Create a new workspace in your PTFE organization and connect it to the git repo you created. --- name: Configure-Terraform-Variables Configure Your Terraform Variables ------------------------- .center[ ![:scale 90%](./images/terraform_variables.png) ] --- name: Configure-Environment-Variables Configure Your Environment Variables ------------------------- .center[ ![:scale 100%](./images/environment_variables.png) ] --- name: Run-Terraform-Apply Run Terraform ------------------------- Queue up a run, or make changes to the code and push them to your bitbucket repo. PTFE will automatically kick off a plan any time new code is committed, and if you have auto-apply enabled it will run automatically. --- name: Chapter-6-Review class: middle,center .review[ Chapter 6 Review Provision an App Server ] .contents[ ☑ Created a git repo ☑ Cloned the git repo to the workstation ☑ Pushed some changes to the repo ☑ Deployed our application ] --- Appendixes ------------------------- --- name: Appendix-A Appendix A: Training Lab Prerequisites ------------------------- * AWS account with API credentials to create VPCs, EC2 resources, and Route 53 DNS records * Terraform configured on your laptop * A domain name you can use for DNS records, with a zone file in Route 53 * RDP access to the Internet --- Appendix B: Note to the Instructor ------------------------- Lorem ipsum dolor amet brooklyn fingerstache crucifix, aesthetic normcore PBR&B sustainable. Vinyl organic gentrify shaman skateboard keffiyeh waistcoat ennui woke pinterest bicycle rights kogi jianbing. Vaporware kombucha blog readymade snackwave. Cloud bread flannel hammock, raw denim ramps austin humblebrag bushwick tilde occupy. Snackwave art party godard, leggings normcore sustainable celiac irony fashion axe tumeric gluten-free vice humblebrag vape hammock. Occupy polaroid kinfolk tote bag paleo. Godard selfies literally woke gochujang prism neutra. Freegan green juice blue bottle, williamsburg kitsch lyft tattooed pitchfork humblebrag venmo shabby chic stumptown raw denim.