Collision, fuel leak, Sabine Pass, Texas
by Erofey SchkvarkinOffshore supply ship CHERAMIE BOTRUC 22 and tug MARIYA MORAN collided near the entrance to Sabine Pass, Port Arthur, Texas, in the morning Nov 14, reported US Coast Guard. Fuel tank of one of the vessels was breached, some 10 tons of diesel fuel leaked, according to USCG press-release. The source of the release has been secured and the spill has been contained. Boom and sorbent material have been placed around the vessel while it remains anchored outside of the channel.
Damaged vessel which leaked fuel wasn’t identified, but according to data and tracks, it’s supply ship CHERAMIE BOTRUC 22. She moved to Sabine and was berthed, on Nov 15.
Offshore supply ship CHERAMIE BO TRUC 22, IMO 7621865, dwt 500, built 1975, flag USA, manager L&M BOTRUC RENTAL LLC.
Tug MARIYA MORAN, IMO 9764128, GT 708, built 2015, flag USA, manager MORAN TOWING CORP.


Follow these steps to start local debugging for C# in VS Code:
Main.cs
file in a new VS Code window. It is important that you open the project directory itself, not the directory you ran lean init
in.lean backtest
command with the --debug vsdbg
option and wait until the CLI tells you to attach to the debugger.
$ lean backtest "My Project" --debug vsdbg 20210423 13:50:54.634 TRACE:: DebuggerHelper.Initialize(): waiting for debugger to attach...
After finishing C# debugging with VS Code you will see a message saying "The pipe program 'docker' exited unexpectedly with code 137.". You can safely ignore this message.


FEAR AND LOATHING IN LAS VEGAS RATINGS: 7 / 10 --> Good movie
Review Date: May 23, 1998 Director: Terry Gilliam Writers: Tod Davies and Alex Cox Producer: Patrick Cassavetti, Laila Nabulsi and Stephen Nemeth Actors: Johnny Depp as Hunter S. Thompson (AKA Raoul Duke) Benicio Del Toro as Dr. Gonzo Genre: Dramedy Year of Release: 1998
Film adaptation of Hunter S. Thompson's infamous semi-autobiographical hallucinogen-fueled book of the same title. Director Terry Gilliam of TWELVE MONKEYS (8/10) and BRAZIL (7/10) fame, took over the helm of this project after fellow director Alex Cox (SID AND NANCY (7.5/10)) "apparently alienated everyone associated with the movie", according to Gilliam.
PLOT: Writer Thompson (Depp) heads down to Las Vegas with his attorney Dr. Gonzo (Del Toro) to cover a motorcycle race. During their trip, they systematically consume "two bags of grass, seventy-five pellets of mescaline, five sheets of high powered blotter acid, a salt shaker half-full of cocaine, a whole galaxy of multi-colored uppers, downers, screamers, laughers, a quart of tequila, a quart of rum, a case of beer, a pint of raw ether and two dozen amyls. The movie presents us with the results of that heavy drug use.
CRITIQUE: I have given this movie two separate ratings because I believe that the enjoyment of this psychedelic picture is highly correlated with the amount of drugs or alcohol that would be floating around in the viewer's own mind, whilst inhaling this cinematic vision of excess. If you are prepared to get high or intoxicated before watching this film, I would say that this is one picture that you will thoroughly enjoy on a multitude of colorful levels. If, on the other hand, you decide to stray from the addition of nefarious elements to your system, I could not imagine you truly appreciating much of this drug-induced picture's entire ride (5.5/10 for all those sober dogs). Note: I have not read Thompson's book.
Having said that, JoBlo did engage in an alcohol-based consumatory session before (and during) the viewing of this film, so his critique of the film should be appreciated on that level. This movie relies heavily on style and peculiar humour, rather than substance or plot. It moves admirably from one scene to the next, without much basis of their being, while presenting us with the two days in the life of writer Hunter S. Thompson during which he seemed to consume more drugs and alcohol than anyone could ever imagine. It was 1971, and the times were apparently "a' changing" in the States.
 During the filming, Depp apparently become fast friends with real-life writer Thompson, and was known to wander off the set from time to time, for the sake of checking out the newest barmaid at the local watering hole. I thought he did seem to exaggerate his walk a little bit too much, but then again, this movie is supposed to be a wild exaggeration of everything and anything, so who am I to talk.
The one thing that did blow my mind was the actual physical transformation endured by actor Benicio Del Toro for his role as Dr. Gonzo. I couldn't believe that this fat, Samoan lawyer, was the same guy who played the slick mumbling criminal in THE USUAL SUSPECTS (7.5/10). Word on the street is that gained over 40 pounds for this role, and I must say that his look was deliciously reprehensible. Plenty of cameos also pepper this kaleidoscopic moving picture in the form of Ellen Barkin, Christina Ricci, Tobey Maguire and Cameron Diaz, along with a bunch of others.
Other than that, the soundtrack was expectedly eclectic, the style was not as wild as I thought it would be, and the ending was certainly not much of a barn-burner, but then again, who really noticed. This movie is about visions of bats floating through your head, Johnny Depp looking goofy and being bald, and the cornucopia of drug-ravaged scenes filling your own intoxicated system with ideas of anarchy, rebellion and the lost American Dream. And for all those who plan on seeing this movie without the partnership of a mean drink or a mighty doobie, I suggest you move further down the aisle, buy yourself a ticket to GODZILLA (6/10), and enjoy the visual fabrications manufactured for the unstimulated mind.
Little Known Facts: Depp and Del Toro snorted plenty of powdered milk instead of cocaine. Bill Murray also portrayed a Thompson-based character in the film WHERE THE BUFFALO ROAM. Johnny Depp turned down roles in THE THREE MUSKETEERS, SPEED (7.5/10), and LEGENDS OF THE FALL (7.5/10), for smaller and quirkier roles in BENNY AND JOON (6.5/10) and WHAT'S EATING GILBERT GRAPE? (7/10). In 1988, Depp told Rolling Stone magazine that he'd tried every drug by the age of 14. Johnny hung out with some of the members of Oasis while filming the uncompleted DIVINE RAPTURE in Ireland, and later played some slide guitar on the 1997 album, Be Here Now. Johnny was born in Kentucky, is a high-school dropout, has nicknamed himself "Mr. Stench", has been engaged to four women until now (including actress Winona Ryder, whose "Winona Forever" tattoo had to be altered to "Wino forever" after their breakup), currently plays guitar in a band called "P", and owns The Viper Room nightclub in L.A.
(c) 1998 Berge Garabedian
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


DIANA Fellows
The DIANA Fellows program provides support for 4 graduate students each year to spend 3 months intensively developing software tools in conjunction with collaborating institutions. The graduate student should have a base stipend from their home institution. The DIANA Fellows program will provides travel and subsistence funds to support direct collaboration with others developing software tools for general use at a second institution. See the Fellow Projects page for example projects of interest and details on how to propose your own project. For more information, contact<EMAIL_ADDRESS>DIANA Undergraduate Fellow
A DIANA Undergraduate Fellow position is also available each year. The student will work 10-12 weeks during the summer, either developing or using data-intensive tools. The DIANA project will provide a stipend for the summer period and in some cases can provide additional travel/subsistence support if they work away from their home institution. For more information on the DIANA Undergraduate Fellow position contact<EMAIL_ADDRESS>Current and past DIANA fellows
Pratyush Das, Institute of Engineering and Management (Kolkata) [Undergrad]
- Topic: Add write functionality to uproot - proposal
- Mentor: Jim Pivarski, Princeton University
- Dates/Location: summer, 2018 (FNAL)
Himadri Pandey, University of Cincinnati [Undergrad]
- Topic: Development of Machine Learning Algorithms and Software Implementations for Reconstructing Straight Line Trajectories - proposal, final report
- Mentor: Mike Williams, MIT
- Dates/Location: Jan-Mar, 2018
Juan Baptista
- Topic: Numerical Integration Precision Studies for Maximum Likelihood Fits - proposal, final report
- Mentor: Henry Schreiner, University of Cincinnati
- Dates/Location:
Matthew Feickert, Southern Methodist University [Grad]
- Topic: Investigation of use of Tensorflow/Theano for realistic physics statistics models - proposal, final report
- Mentor: Gilles Louppe/Vincent Croft, NYU
- Dates/Location:
Meghan Frate, University of California - Irvine [Grad]
- Topic: Software to allow application of Gaussian Processes to HEP problems - proposal, final report
- Mentor: Kyle Cranmer, NYU
- Dates/Location: Mar-Apr, 2017 (NYU)
Andrew Carnes, University of Florida [Grad]
- Topic: Additional Functionality in TMVA (Software R&D) Regression and Boosted Decision Trees - proposal, final report
- Mentor: Sergei Gleyzer, University of Florida
- Dates/Location: Summer/Fall, 2016 (CERN)


#!/usr/bin/env bash
#!/usr/bin/env bash
A (magic) shell script to deploy Git repositories
Read the annotated source
Repository: AlphaHydrae/deploy
Shamelessly inspired by: visionmedia/deploy
VERSION=3.1.0
COLORS=
COLOR_BOLD=1
COLOR_RED=31
COLOR_GREEN=32
COLOR_YELLOW=33
COLOR_MAGENTA=35
COLOR_CYAN=36
DEPLOY_ARGS=("$@")
DEPLOY_COMMAND=
DEPLOY_ENV=
DEPLOY_OPTIONS_VALUE_MAP=''
test -z "$DEPLOY_COLOR" && DEPLOY_COLOR=auto
test -z "$DEPLOY_CONFIG" && DEPLOY_CONFIG=./deploy.conf
test -z "$DEPLOY_UPDATE_PREFIX" && DEPLOY_UPDATE_PREFIX=/usr/local
test -z "$DEPLOY_UPDATE_REPO" && DEPLOY_UPDATE_REPO=https://github.com/AlphaHydrae/deploy.git
test -z "$DEPLOY_UPDATE_REV" && DEPLOY_UPDATE_REV=master
deploy is a shell script to deploy your Git projects through SSH.
Add a deploy.conf
file in your project’s directory.
Here’s an example for a Node.js project:
# deploy.conf
[production]
repo https://github.com/me/my-app
host my.server.com
user deploy
path /var/www/app
# describe how to deploy your app
env NODE_ENV=production
deploy npm install --production
post-deploy npm start
deploy is language-agnostic. For a Rails project, you could replace the last 3 lines with:
env RAILS_ENV=production
deploy bundle install --without development test
deploy rake assets:precompile
post-deploy rails server
Now run deploy!
deploy production setup
deploy production rev master
It will:
my.server.com
as the deploy
user through SSHsetup
command will set up a deployment structure and clone your repositoryrev
command will checkout the latest version of your master
branch and run the deployment hooks you defined (deploy
and post-deploy
in the configuration file)Read on to learn what to write in the configuration file or how to use each sub-command.
usage() {
cat <<-EOF
Usage: deploy [options] [env] [command]
Default command:
<env> [options] [rev] deploy a release (commit, branch or tag)
(same as the "rev" command)
Deployment commands:
<env> setup run host setup commands
<env> rev [options] [rev] deploy a release (commit, branch or tag)
<env> console [path] open an ssh session to the host
<env> exec <cmd> execute the given command on the host
<env> list list deployed releases
<env> cleanup [options] clean up old releases
Local commands:
[env] config [key] output whole config file or the value of one key
<env> config-all <key> output all values of a config key in an environment
[env] config-section output a config section
update [options] [rev] update this script to the latest version
General options:
-C, --chdir <dir> change the working directory
--color <mode> set color mode to "always", "never" or "auto" (defaults to "auto")
-c, --config <file> set config file path (defaults to ./deploy.conf)
-y, --yes accept all confirmation prompts (use with caution)
-v, -V, --version output current version
-h, --help output usage information
Deployment options:
-P, --path <dir> set the remote path to deploy the project to on the host
-r, --repo <url> set the Git URL to fetch source code from when deploying
SSH options:
-A, --forward-agent enable SSH agent forwarding
-H, --host <address> set the host to connect to
-i, --identity <file> select a file from which the identity (private key)
for public key authentication is read
-p, --port <n> set the port to connect to
-t, --tty force pseudo-terminal allocation
-u, --user <name> set the host user to connect as
Self-update options:
--update-prefix <dir> set the base directory to update the deploy script
(full path will be <dir>/bin/deploy)
--update-path <file> set the full path to the deploy script
(overrides --update-prefix)
Examples:
deploy prod setup set up the 'prod' host
deploy prod master deploy the 'master' branch in the 'prod' env (shortcut)
deploy prod rev master (same)
deploy dev foo deploy the 'foo' branch in the 'dev' env
deploy prod exec pm2 status run 'pm2 status' in the 'prod' env
deploy dev config user get the value of the 'user' key in the 'dev' env
deploy update update the deploy script
EOF
}
With npm:
npm install -g bash-deploy
With curl:
PREFIX=/usr/local/bin \
FROM=https://raw.githubusercontent.com \
&& curl -sSLo $PREFIX/deploy \
$FROM/AlphaHydrae/deploy/master/bin/deploy \
&& chmod +x $PREFIX/deploy
With wget:
PREFIX=/usr/local/bin \
FROM=https://raw.githubusercontent.com \
&& wget -qO $PREFIX/deploy \
$FROM/AlphaHydrae/deploy/master/bin/deploy \
&& chmod +x $PREFIX/deploy
Or download it yourself.
deploy is a one-file bash script which requires the following commands: cat
, cut
, date
, git
, grep
, ls
, mkdir
, readlink
, sed
, ssh
, tail
, tar
and wc
.
Most bash shells have all of those out of the box except perhaps Git.
It also optionally requires the chmod
, cp
and mktemp
commands to update itself.
check_requirements() {
for com in cat cut date git grep ls mkdir sed ssh tail tar wc; do
require_command $com
done
}
require_command() {
command_exists "$1" || abort "$1 command not found"
}
command_exists() {
command -v "$1" >/dev/null 2>&1
}
deploy reads its main configuration from a deploy.conf
file in the current directory.
It can also be customized with environment variables and command line options.
config_file_exists() {
test -f "$DEPLOY_CONFIG" && test -r "$DEPLOY_CONFIG"
}
The configuration file is basically a series of sections containing key/value pairs:
# deploy.conf
[staging]
host staging.example.com
user test
# how to deploy
post-deploy ./run-test.sh
[production]
host <IP_ADDRESS>
user root
# how to deploy
env NODE_ENV=production
deploy npm install --production
deploy npm run build
post-deploy pm2 start pm2.json
Each named section, delimited by [name]
, represents an environment (i.e. a host machine) to deploy to,
in this example the staging and production environments.
Lines beginning with #
are comments and are ignored.
Other lines are key/value pairs. A key is a sequence of characters containing no whitespace, followed by at least one tab or space.
For example, in the line deploy npm run build
, the key is deploy
and its value is npm run build
.
config_section_exists() {
grep "^\[$1\]$" "$DEPLOY_CONFIG" &> /dev/null
}
get_config_section() {
local env="$1"
grep "^\[$env\]$" -A 999 "$DEPLOY_CONFIG" \
| tail -n +2 \
| grep -v "^ *#" \
| grep -v "^ *$" \
| sed "/^\[/q" \
| sed "/^\[/d"
}
get_config_key_values_in_env() {
local key="$1"
local env="$2"
get_config_section $env \
| grep "^$key[ \t][ \t]*[^ \t]" \
| cut -d ' ' -f 2-999 \
| sed "s/^ *//" \
| sed "s/ *$//"
}
Some keys like repo
, host
, port
or user
are simple configuration properties that have one value per environment.
If multiple values are found, the last one is used.
get_last_config_key_value() {
local key=$1
local env_var=$(get_config_key_env_var $key)
These keys can be overriden through an environment variable of the same name in uppercase and prefixed by DEPLOY_
.
For example, the environment variable to override the repo
key is DEPLOY_REPO
.
local env_var_value=${!env_var}
if test -n "$env_var_value"; then
echo "$env_var_value"
else
test -n "$key" \
&& get_all_config_key_values "$key" \
| tail -n 1
fi
}
get_config_key_env_var() {
local key=$1
echo "DEPLOY_$key" | tr '-' '_' | tr '[a-z]' '[A-Z]'
}
Other keys like setup
, deploy
or post-deploy
are multiple-value keys used for hooks.
They can be present multiple time in the same environment and all their values will be used in order.
get_all_config_key_values() {
local env=$DEPLOY_ENV
local key=$1
test -n "$env" \
&& test -n "$key" \
&& get_config_key_values_inherited $key $env \
| tail -n +2
}
When deploying your project to multiple environments, there will probably be some configuration properties that are identical for multiple environments. To avoid repetition, an environment can inherit from one or multiple other environments:
# deploy.conf
[common]
user dev
deploy do-stuff
deploy do-more-stuff
[secure]
user sekret
deploy do-sensitive-stuff
[production]
inherits common
inherits secure
user root
deploy just-do-it
In this example, the production
environment inherits from common
and secure
(in that order).
The value of a single-value key like user
will be the last value found in the environment inheritance tree.
In this case, it will be root
for the production
environment (the values inherited from the common
and secure
environments are overwritten).
A multiple-value key like deploy
will include all values found in the entire inheritance tree.
In this case, it will have 4 values for the production
environments, taken in order from the common
, secure
and production
sections:
do-stuff
do-more-stuff
do-sensitive-stuff
just-do-it
get_config_key_values_inherited() {
local key="$1"
local env="$2"
shift 2
local without=("$@")
test -n "$env" && without+=("[$env]")
local values=
if ! config_section_exists $env; then
abort "[$env] config section not defined"
fi
local new_line=$'\n'
local old_ifs=$IFS
IFS=$'\n'
local inherits="$(get_config_key_values_in_env inherits $env)"
for inherits_key in $inherits; do
IFS="$old_ifs"
if ! value_is_in "[$inherits_key]" "${without[@]}"; then
local inherited_result="$(get_config_key_values_inherited $key $inherits_key "${without[@]}")"
local inherited_without=("$(echo "$inherited_result"|head -n 1)")
local inherited_values="$(echo "$inherited_result"|tail -n +2)"
without=("${without[@]}" "${inherited_without[@]}")
if test -n "$inherited_values"; then
test -n "$values" && values="$values$new_line"
values="$values$inherited_values"
fi
fi
IFS=$'\n'
done
IFS="$old_ifs"
For convenience in simple use cases, you can also add a default environment by including a nameless section in your configuration file:
# deploy.conf
[]
user dev
deploy do-stuff
deploy do-more-stuff
[development]
deploy do-it-somehow
[production]
user root
deploy do-it-seriously
All environments that have no inherits
key automatically inherit from the default environment.
if test -z "$inherits" && ! value_is_in "[]" "${without[@]}"; then
without+=("[]")
local default_environment_values="$(get_config_key_values_in_env $key)"
if test -n "$default_environment_values"; then
[ -n "$values" ] && values="$new_line$values"
values="$default_environment_values$values"
fi
fi
local current_environment_values="$(get_config_key_values_in_env $key $env)"
if test -n "$current_environment_values"; then
[ -n "$values" ] && values="$values$new_line"
values="$values$current_environment_values"
fi
echo "${without[@]}$new_line$values"
}
Hooks are user-defined commands that can be run during a deployment phase.
There are currently two phases defined: setup and deploy.
The setup phase happens when you run the setup
command,
while the deploy phase happens when you run the rev
command.
There are various hooks for each phase: some that run before, some during and some after deploy does its thing. These are the currently available hooks:
pre-setup
post-setup
pre-deploy
deploy
post-deploy
Hooks are multiple-value keys that are optional and can be used as many times as you want. They will all be run in order.
run_hooks() {
local name=$1
test -n "$name" || abort hook name required
local dir=$2
test -n "$dir" || abort hook working directory required
local path=`get_last_config_key_value path`
local commands=`get_all_config_key_values $name`
log hook $name
if test -z "$commands"; then
echo " nothing to do"
return 0
fi
local old_ifs=$IFS
IFS=$'\n'
for cmd in $(echo "$commands"); do
IFS="$old_ifs"
Each hook is run in a specific working directory and has access to various environment variables.
$DEPLOY_PATH
is always exported and indicates the deployment directory
(which is not necessarily the same as the hook’s working directory).
Additional user-defined variables may also be made available (see environment variables).
Here’s an example of how you could use hooks to cache your node_modules
directory after every deployment
to shorten future installation times:
# restore cache (if present)
deploy tar -xzf $DEPLOY_PATH/cache.gz -C . || exit 0
# install new dependencies
deploy npm install --production
# update cache
post-deploy tar -czf $DEPLOY_PATH/cache.gz node_modules
run "cd $dir \
&& $(build_export_env_command) \
&& $cmd 2>&1" \
|| abort $name hook failed
IFS=$'\n'
done
IFS="$old_ifs"
}
repo <url>
(or the $DEPLOY_REPO
variable) defines the Git URL from which your repository will be cloned at setup time.path <dir>
(or the $DEPLOY_PATH
variable) defines the directory into which your project will be deployed on the host.
keep <n>
(or the $DEPLOY_KEEP
variable) defines how many releases to keep after deploying.
Older releases will be deleted. It must be either all
(the default), or an integer greater than zero.
If it’s a number, it indicates the number of releases that should be kept after successful deployment.
For example, with keep
set to 3, if there are 5 old releases deployed before running the script, 3 will be deleted
(so that 2 old releases and the new one being deployed, 3 in total, remain).
It’s also possible to manually trigger cleanup with the cleanup command.
Various SSH options can be specified through the configuration file or environment variables:
build_ssh_command() {
host <address>
(or the $DEPLOY_HOST
variable) is mandatory for all environments.
It indicates which host or IP address to connect to. local host=$(get_last_config_key_value host)
require_options host
local url=$host
user <name>
(or the $DEPLOY_USER
variable) specifies the user to connect as.
By default, you connect with the same name as your local user. local user=$(get_last_config_key_value user)
test -n "$user" && url="$user@$url"
identity <file>
(or the $DEPLOY_IDENTITY
variable) specifies a file from which the private key for public key authentication is read. local identity="`get_last_config_key_value identity`"
test -n "$identity" && identity="-i $identity"
forward-agent yes
(or the $DEPLOY_FORWARD_AGENT
variable) enables agent forwarding. local forward_agent="`get_last_config_key_value forward-agent`"
test -n "$forward_agent" && forward_agent="-A"
port <number>
(or the $DEPLOY_PORT
variable) specifies the host port to connect to. local port="`get_last_config_key_value port`"
test -n "$port" && port="-p $port"
tty yes
(or the $DEPLOY_TTY
variable) forces pseudo-terminal allocation. local tty="`get_last_config_key_value tty`"
test -n "$tty" && tty="-t"
echo "ssh $tty $forward_agent $port $identity $url"
}
Commands executed on the host through SSH will be logged to the console.
run() {
log_command "$@"
run_no_log "$@"
}
For some commands, the output may be retrieved and displayed in a more user-friendly manner.
run_no_log() {
local shell
shell="$(build_ssh_command 2>&1)"
test $? -eq 0 || abort "$shell"
$shell $@
}
You can define environment variables that will be exported on the host when executing hooks.
build_export_env_command() {
The $DEPLOY_PATH
variable is always exported and indicates the deployment directory
configured by the user with the path
config key or the local $DEPLOY_PATH
variable.
local path="$(get_last_config_key_value path)";
local env_command="export DEPLOY_PATH=$path"
env <NAME>=<VALUE>...
defines one or multiple environment variables to export on the host when running hooks.
# deploy.conf
env FOO=BAR
env BAZ=QUX CORGE=GRAULT
env_command="$env_command $(echo `get_all_config_key_values env`|sed 's/\n/ /g')"
forward-env <NAME>...
defines the name(s) of one or multiple local environment variables
to export on the host when running hooks.
# deploy.conf
forward-env FOO
forward-env BAR BAZ QUX
for forward_env in $(echo `get_all_config_key_values forward-env`|sed 's/\n/ /g'); do
env_command="$env_command $forward_env=$(echo "${!forward_env}"|sed 's/ /\\ /g')"
done
echo "$env_command"
}
If you have a .env
file in your local project directory, it will automatically be sourced.
This can be handy to create local variables that you can forward to the host.
# .env
export FOO=BAR
export YEAR=$(date "+%Y")
(Note that if you use the -C, --chdir
command line option or the $DEPLOY_CHDIR
variable,
the .env
file is read after the working directory is changed, so setting $DEPLOY_CHDIR
in the file has no effect.)
load_env_files() {
test -f .env && { test -r .env || abort "environment file .env is not readable"; }
test -f .env && { . .env || abort "an error occurred while sourcing .env"; }
If you have an environment file named after the environment, it will also be sourced.
For example, when deploying in the “production” environment, a .env.production
file
would be sourced if present in the local project directory.
If both .env
and .env.production
are present, they are sourced in that order.
if test -n "$DEPLOY_ENV"; then
local file=".env.$DEPLOY_ENV"
test -f "$file" && { test -r "$file" || abort "environment file $file is not readable"; }
test -f "$file" && { . "$file" || abort "an error occurred while sourcing $environment_specific_file"; }
fi
}
The properties in the configuration file can also be specified through command line options or environment variables.
For example, the repo
property in the configuration file can also be specified:
-r, --repo <url>
command line option.$DEPLOY_REPO
environment variable.Their precedence is as follows:
-r, --repo <url>
) always takes precedence if specified.$DEPLOY_REPO
) takes precedence over the configuration file.repo
) is used.
DEPLOY_OPTIONS_MAP=(
-C, --chdir <dir>
(or the $DEPLOY_CHDIR
variable) changes deploy‘s local working directory before loading the configuration file.C:chdir:dir
--color always|never|auto
(or the $DEPLOY_COLOR
variable) enables/disables colors in the output of the script.
This defaults to auto
, which only enables colors if the current terminal is interactive.
:color:mode
-c, --config <file>
(or the $DEPLOY_CONFIG
variable) allows you to set a custom path for the configuration file (defaults to ./deploy.conf
).c:config:path
--help
prints usage information and exits.h:help
-v, -V, --version
prints the current version and exits.V:version
v:version
-y, --yes
(or the $DEPLOY_YES
variable) will automatically accept all confirmation prompts
(only valid for the rev
, cleanup
and update
sub-commands).
Use with caution: old releases may be deleted with the rev
or cleanup
sub-commands if a keep
option is configured.
y:yes
-r, --repo <url>
(or the $DEPLOY_REPO
variable) sets the Git URL to fetch the project’s source code from when deploying.r:repo:url
-P, --path <dir>
sets the remote path to deploy the project to on the host
(this is the path to the directory managed by deploy, not to the current release).P:path:dir
-k, --keep <n>
(or the $DEPLOY_KEEP
variable) changes the number of old releases that are kept
after successful deployment (see the keep
option).k:keep:n
-f, --force
(or the $DEPLOY_FORCE
variable) forces deployment to proceed even when you have uncommitted changes.f:force
These options are valid for all commands which connect to the remote host through SSH
(setup
, rev
, console
, exec
, list
and cleanup
):
-A, --forward-agent
(or the $DEPLOY_FORWARD_AGENT
variable) enables forwarding of the authentication agent connection.
Use with caution. Users with the ability to bypass file permissions on the remote host (for the agent’s UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.
A:forward-agent
-H, --host <address>
(or the $DEPLOY_HOST
variable) sets the host to connect to.H:host:address
-i, --identity <file>
(or the $DEPLOY_IDENTITY
variable) selects a file from which the identity (private key)
for public key authentication is read.
The default is ~/.ssh/id_dsa
, ~/.ssh/id_ecdsa
, ~/.ssh/id_ed25519
and ~/.ssh/id_rsa
.
i:identity:file
-p, --port <n>
(or the $DEPLOY_PORT
variable) sets the port to connect to.p:port:n
-t, --tty
(or the $DEPLOY_TTY
variable) forces pseudo-terminal allocation.
This can be used to execute arbitrary screen-based programs on a remote machine, which can be very useful, e.g. when implementing menu services.
t:tty
-u, --user <name>
(or the $DEPLOY_USER
variable) sets the remote user to connect as.u:user:name
These options can be used by deploy to update itself with the update
sub-command:
--update-path <file>
(or the $DEPLOY_UPDATE_PATH
variable) sets the path to save the updated script when using the update command.
This overrides the --update-prefix
or $DEPLOY_UPDATE_PREFIX
options.
:update-path:file
--update-prefix <dir>
(or the $DEPLOY_UPDATE_PREFIX
variable) sets the base directory when using the update command.
The updated script will be saved in bin/deploy
relative to this path.
(Note that setting --update-path
or $DEPLOY_UPDATE_PATH
overrides this option.)
:update-prefix:dir
)
Note that all of deploy‘s command line options can be used anywhere in a command. The following 3 commands are equivalent:
deploy -u deploy production --keep 3 --yes master
deploy -u deploy --keep 3 --yes production master
deploy production master -u deploy --keep 3 --yes
parse_script_options() {
local i=0
while test $i -lt "$(script_arg_count)"; do
local option="$(script_arg $i)"
if [[ "$option" != -* ]]; then
((i++))
continue
fi
local option_name=
local option_value=
local option_args_count=1
Some options have a short form which may be specified with one hyphen,
e.g. -h
to display usage or -f
to force deployment with uncommitted local changes.
if [[ "$option" =~ ^-[a-zA-Z](=..*)?$ ]]; then
option_name="$(get_option_name "${option:0:2}")"
If a short option takes a value, it may be provided either as the next argument
(e.g. -k 5
) or with an equal sign (e.g. -k=5
).
if [[ "$option" =~ ^..= ]]; then
option_value="${option:3}"
fi
All options have a long form which may be specified with two hyphens,
e.g. --help
or --force
.
elif [[ "$option" =~ ^--[a-zA-Z][a-zA-Z]*(-[a-zA-Z][a-zA-Z]*)*(=..*)?$ ]]; then
local long_option="$(echo "$option"|sed 's/=.*$//')"
option_name="$(get_option_name "$long_option")"
If a long option takes a value, it may be provided either as the next argument
(e.g. --keep 5
) or with an equal sign (e.g. --keep=5
).
if [[ "$option" =~ ^[^=]*= ]]; then
option_value="$(echo "$option"|sed 's/^[^=]*=//')"
fi
else
The following errors will cause the script to be interrupted and print an error:
---foo
, -bar
, --baz
). abort "unsupported option format '$option'"
fi
local option_arity="$(get_option_arity "$option_name")"
if test $option_arity -lt 0 || test $option_arity -gt 1; then
abort "option arity $option_arity is not supported"
--help=3
). elif test -n "$option_value" && test $option_arity -eq 0; then
abort "option $(describe_option "$option_name") does not take a value"
elif test -z "$option_value" && test $option_arity -eq 1; then
option_value="$(script_arg $((i+1)))"
option_args_count=2
elif test -z "$option_value" && test $option_arity -eq 0; then
option_value=1
fi
test -z "$option_value" && test $option_arity -ge 1 \
&& abort "option $(describe_option "$option_name") requires a value"
if [[ "$option_name" == "help" ]]; then
usage; exit 0
elif [[ "$option_name" == "version" ]]; then
echo $VERSION; exit 0
fi
-k 5
and --keep 3
). script_has_option "$option_name" && abort "option $(describe_option "$option_name") can only be given once"
DEPLOY_OPTIONS_VALUE_MAP="$(printf "$DEPLOY_OPTIONS_VALUE_MAP\n$option_name $option_value")"
script_remove_args $i $option_args_count
done
}
apply_script_options() {
local options="$@"
test -z "$options" && options="$(echo "$DEPLOY_OPTIONS_VALUE_MAP"|cut -d ' ' -f 1|tr '\n' ' ')"
for option in $options; do
local variable_name="$(get_config_key_env_var "$option")"
local variable_value="$(echo "$DEPLOY_OPTIONS_VALUE_MAP"|grep "^${option} "|cut -d ' ' -f 2-999)"
export $variable_name="$variable_value"
done
}
main() {
parse_script_options
By default, deploy looks in the current directory to find its configuration file
and optional environment files. This can be customized by setting the -c, --chdir <dir>
command line option or the $DEPLOY_CHDIR
variable.
If this option is set, deploy will first move to that directory before sourcing the environment files and reading the configuration file.
apply_script_options chdir
validate_working_directory
test -n "$DEPLOY_CHDIR" && cd "$DEPLOY_CHDIR"
apply_script_options
local subcommand=
while script_has_args && test -z "$subcommand"; do
DEPLOY_COMMAND="$(script_arg)"
case "$DEPLOY_COMMAND" in
exec) subcommand=exec_remote_command ;;
cleanup) subcommand=clean_up_old_releases ;;
console) subcommand=open_remote_console ;;
setup) subcommand=set_up_host ;;
list) subcommand=list_releases ;;
rev) subcommand=deploy_rev ;;
update) subcommand=update_self ;;
config) subcommand=print_config_or_value ;;
config-all) subcommand=print_config_values ;;
config-section) subcommand=print_config_section ;;
*)
if test -z "$DEPLOY_ENV"; then
DEPLOY_ENV="$DEPLOY_COMMAND"
else
break
fi
;;
esac
script_shift
done
load_env_files
deploy is usually used this way:
deploy prod setup
deploy prod list
deploy prod rev master
However, since the most often used sub-command is rev
,
it is also the default command. The following 2 commands are equivalent:
deploy prod master
deploy prod rev master
if test -z "$subcommand"; then
subcommand=deploy_rev
DEPLOY_COMMAND=rev
fi
$subcommand
}
These are the commands you will use to set up and deploy your projects.
Note that most (but not all) of deploy‘s sub-commands require an environment to be specified before the actual command name, for example:
deploy setup # error! no environment
deploy production setup # all good
require_env() {
test -n "$DEPLOY_ENV" || abort "<env> required"
config_section_exists $DEPLOY_ENV || abort "[$DEPLOY_ENV] config section not defined"
}
require_config() {
validate_config_file
}
require_no_args() {
if script_has_args; then
local message="$(printf "command '$DEPLOY_COMMAND' takes no more arguments\n$(script_arg_count) extra argument")"
test $(script_arg_count) -ne 1 && message="${message}s"
abort "$message found: ${DEPLOY_ARGS[@]}"
fi
}
require_options() {
local option_names="$@"
for option_name in $option_names; do
local option_value=$(get_last_config_key_value "$option_name")
test -n "$option_value" && continue
log_error
log_error "The '$option_name' option is required by the '$DEPLOY_COMMAND' command."
log_error "Configure it one of the following ways:"
log_error
log_error "* Add the '$option_name' key to the [$DEPLOY_ENV] section of the $DEPLOY_CONFIG configuration file."
log_error "* Set the \$$(get_config_key_env_var "$option_name") environment variable."
log_error "* Add the $(describe_option "$option_name") command line option to the command."
abort "$DEPLOY_COMMAND interrupted"
done
}
<env> setup
Perform first-time setup tasks on the host before deployment. You should only have to run this once before deploying the first time.
deploy production setup
deploy will create the following structure for you in the deployment directory defined by the path
config key (or the $DEPLOY_PATH
variable):
$DEPLOY_PATH/releases
will contain each deployment’s files$DEPLOY_PATH/repo
will be a bare clone of your Git repository$DEPLOY_PATH/tmp
will be used to store temporary files during deploymentThis phase has 3 hooks that are all executed in the deployment directory. Note that this directory must already exist on the host and be writable by the user you connect as.
This is what deploy will do during setup:
set_up_host() {
require_env
require_no_args
validate_deploy_options repo
require_options path repo
local path="$(get_last_config_key_value path)"
local repo="$(get_last_config_key_value repo)"
pre-setup
hooks (if any). run_hooks pre-setup "$path"
releases
and a tmp
directories if they don’t exist already. log running setup
run "{ test -d $path || exit 1; } && mkdir -p $path/releases $path/tmp" \
|| abort "failed to set up directories; make sure $path exists and is writable"
repo
directory if it isn’t already there. log cloning repo
run "test -d $path/repo || { git clone --bare $repo $path/repo || exit 1; }" \
|| abort "failed to clone repo"
post-setup
hooks (if any). run_hooks post-setup "$path"
echo
log_success "setup complete"
echo
}
deploy_rev() {
require_env
You must provide a Git revision, i.e. a commit, branch or tag to deploy,
either as the [rev]
argument, with the rev
config key or the $DEPLOY_REV
variable.
If your Git repository is private, make sure that the deployment user has access to it.
local rev="$(get_last_config_key_value rev)"
if script_has_args; then
rev="$(script_arg)"
script_shift
fi
test -n "$rev" || abort "no commit, branch or tag specified"
require_no_args
validate_deploy_options force keep yes
require_options path
For each deployment, a new release directory will be created in releases
in the deployment directory.
The name of a release directory is the current date and time in the YYYY-MM-DD-HH-MM-SS
format.
(You can list deployed releases with the list command.)
local path=$(get_last_config_key_value path)
local release=$(date -u '+%Y-%m-%d-%H-%M-%S')
local current_dir="$path/current"
local release_dir="$path/releases/$release"
local tmp_dir="$path/tmp"
local keep=$(get_last_config_key_value keep)
Note that deploy will refuse to deploy unless all your local changes are committed and pushed.
You can override this behavior with the -f|--force
option (or the $DEPLOY_FORCE
variable).
require_no_local_changes
This is what deploy will do during deployment:
log deploying
echo " revision: $rev"
echo " release: $release"
echo " keep: ${keep:-all}"
log checking current release
local get_current_release_dir_command="readlink $path/current"
log_command "$get_current_release_dir_command"
local current_release_dir
current_release_dir="$(run_no_log "$get_current_release_dir_command" 2>&1)"
test $? -ne 0 && test -n "$current_release_dir" && abort "$current_release_dir"
if test -n "$current_release_dir"; then
log_color $COLOR_CYAN "$current_release_dir"
else
log_color $COLOR_CYAN "none (pre-deploy hooks will not be run)"
fi
keep
option is configured, all previously deployed releases are listed,
with those to be deleted in red, and those to be kept and the current release in green, and
confirmation is asked that they may be deleted if the deployment is successful. prepare_to_delete_old_releases deploy
if test -n "$current_release_dir"; then
run_hooks pre-deploy "$current_release_dir"
else
log hook pre-deploy
log_color $COLOR_CYAN "skipping because there is no current release"
fi
log fetching updates
run "cd $path/repo \
&& git fetch origin '+refs/heads/*:refs/heads/*' \
&& git rev-list -n 1 $rev" \
|| abort "could not fetch changes"
log "extracting source at $rev"
run "cd $path/repo \
&& mkdir -p $release_dir \
&& git archive --output $tmp_dir/release-$release.tar $rev \
&& tar -C $release_dir -x --file $tmp_dir/release-$release.tar \
&& rm -f $tmp_dir/release-*.tar" \
|| abort "could not extract source from repository"
run_hooks deploy $release_dir
current
in the deployment directory. log linking current
run "ln -fns $release_dir $current_dir" \
|| abort "could not create current symlink"
User-defined post-deploy hooks (if any) are run in the current release directory (which is now the same as the new release directory that was just created).
You should define post-deploy hooks to execute or start your application at this stage.
run_hooks post-deploy $current_dir
keep
option (if configured and any were found). test -n "$releases_to_delete" && delete_old_releases "$releases_to_delete"
echo
log_success successfully deployed
echo
}
require_no_local_changes() {
if test -z "$DEPLOY_FORCE"; then
git --no-pager diff --exit-code --quiet || abort "commit or stash your changes before deploying"
git --no-pager diff --exit-code --quiet --cached || abort "commit your staged changes before deploying"
[ -z "`git rev-list @{upstream}.. -n 1`" ] || abort "push your changes before deploying"
fi
}
print_config_or_value() {
If called with no environment and no argument, it prints the whole configuration file:
$> deploy config
[]
user dev
[production]
user root
port 222
if ! script_has_args; then
validate_default_options
test -z "$DEPLOY_ENV" || abort "this commands only supports both [env] and [key] arguments or neither"
require_config
cat "$DEPLOY_CONFIG"
If called with the environment and a key, it prints the last value of that key for that environment (only the last value is printed even for a multiple-value key):
$> deploy production config user
root
Exits with status 1 if no value is found for the key.
else
local key="$(script_arg)"
script_shift
require_env
require_no_args
validate_default_options
require_config
local value=$(get_last_config_key_value "$key")
test -n "$value" && echo "$value"
fi
}
<env> config-all <key>
Print all values of a config key in the current environment and its inherited environments (all values are printed even for a single-value key):
$> deploy production config-all user
dev
root
Exits with status 1 if no value is found for the key.
print_config_values() {
require_env
local key="$(script_arg)"
script_shift
require_no_args
validate_default_options
require_config
test -n "$key" || abort "<key> missing"
local values=$(get_all_config_key_values "$key")
test -n "$values" && echo "$values"
}
[env] config-section
Print all values of a config section “as is” (with no inheritance). If no environment is specified, the default config section is printed.
Exits with status 1 if the config section is not found (including the default one).
print_config_section() {
require_no_args
validate_default_options
require_config
config_section_exists "$DEPLOY_ENV" && get_config_section "$DEPLOY_ENV"
}
open_remote_console() {
require_env
If an absolute path is specified, the session starts there.
deploy production console /var/www
local path="$(script_arg)"
script_shift
require_no_args
validate_deploy_options
If no path is specified, the session starts in the deployment directory.
deploy production console
if test -z "$path"; then
require_options path
path="$(get_last_config_key_value path)"
If a relative path is specified, it starts there relative to the deployment directory.
deploy production console current
elif [[ ! "$path" =~ ^\/ ]]; then
require_options path
path="$(get_last_config_key_value path)/$path"
fi
test -n "$path" || abort "path is required"
local shell
shell="$(build_ssh_command 2>&1)"
test $? -eq 0 || abort "$shell"
log_command "$shell -t \"cd $path; \$SHELL -il\""
$shell -t "cd $path; \$SHELL -il"
}
<env> cleanup
Deletes old releases based on the keep
option.
The number of releases to keep is determined from the keep
configuration file option,
the $DEPLOY_KEEP
variable or the -k|--keep
command line option.
The releases to delete will be listed and confirmation will be asked before they are actually deleted.
clean_up_old_releases() {
require_env
require_no_args
validate_deploy_options keep yes
require_options path
local keep=$(get_last_config_key_value keep)
log cleaning up
echo " keep: ${keep:-all}"
prepare_to_delete_old_releases cleanup
if test -z "$releases_to_delete"; then
echo
log_success "nothing to clean up"
echo
else
delete_old_releases
echo
fi
}
exec_remote_command() {
require_env
validate_deploy_options
require_options path
local path="$(get_last_config_key_value path)"
run "cd $path && $(build_export_env_command) && ${DEPLOY_ARGS[@]}"
}
<env> list
List the releases that have been deployed on the host.
$> deploy production list
2016-12-24-17-45-23
2017-01-01-02-03-43
2017-04-01-00-00-00
list_releases() {
require_env
require_no_args
validate_deploy_options
require_options path
local path=`get_last_config_key_value path`
local releases_path="$path/releases"
log "listing releases"
local list_releases_command="ls -1 '$releases_path'"
log_command "$list_releases_command"
local releases_list
releases_list="$(run_no_log "$list_releases_command")"
test $? -eq 0 || abort "could not list releases directory $releases_path; make sure it exists and is readable"
if test -z "$releases_list"; then
echo
echo "No releases found"
echo
exit 0
fi
log "checking current"
local check_current_command="readlink $path/current"
log_command "$check_current_command"
local current_release_dir
current_release_dir="$(run_no_log "$check_current_command")"
echo
for release in $releases_list; do
if [[ "$releases_path/$release" == "$current_release_dir" ]]; then
log_color $COLOR_GREEN "$releases_path/$release (current)"
else
log_color $COLOR_CYAN "$releases_path/$release"
fi
done
echo
local number_of_releases=$(echo "$releases_list"|wc -l)
local message="$number_of_releases release"
test $number_of_releases -ne 1 && message="${message}s"
log_success "$message found"
echo
}
update [rev]
Updates deploy to the latest version by downloading it from the Git repository and installing it at /usr/local/bin/deploy
(by default).
update_self() {
if script_has_args; then
DEPLOY_UPDATE_REV="$(script_arg)"
script_shift
fi
test -z "$DEPLOY_ENV" || abort "environment not supported for this command"
require_no_args
validate_update_options
In addition to the basic requirements, this sub-command also requires chmod
, cp
and mktemp
to be available in the shell.
for com in chmod cp mktemp; do
require_command $com
done
local rev="$DEPLOY_UPDATE_REV"
local path="$DEPLOY_UPDATE_PATH"
local repo="$DEPLOY_UPDATE_REPO"
test -z "$path" && path="$DEPLOY_UPDATE_PREFIX/bin/deploy"
The installation path must be a writable file or not exist. If the installation path does not already exist, its parent must be a writable directory.
test -e "$path" && ! test -f "$path" && abort "$path already exists and is not a file"
test -f "$path" && ! test -w "$path" && abort "$path is not writable"
local dir="$(dirname "$path")"
test -f "$path" || test -e "$dir" || abort "$dir does not exist"
test -f "$path" || test -d "$dir" || abort "$dir is not a directory"
test -f "$path" || test -w "$dir" || abort "$dir is not writable"
log updating
test -z "$DEPLOY_UPDATE_PATH" && echo " prefix: $DEPLOY_UPDATE_PREFIX"
echo " path: $path"
echo " version: $rev"
echo " repository: $repo"
if test -z "$DEPLOY_YES"; then
echo
local update_confirmation=
while ! prompt_is_yes "$update_confirmation" && ! prompt_is_no "$update_confirmation"; do
log_prompt "will update $path; please confirm (yes/no) "
read update_confirmation
done
if prompt_is_no "$update_confirmation"; then
abort "update interrupted"
fi
fi
log "cloning deploy from $rev @ $repo"
To perform the update, deploy will download its Git repository into a temporary directory that will be cleaned up when the update is done (or fails).
local tmp_dir=`mktemp -d -t deploy.XXX`
trap "local_cleanup $tmp_dir" EXIT
The correct revision of the script is then copied to the installation path and made executable.
cd "$tmp_dir" \
&& git clone "$repo" "$tmp_dir" || abort could not clone "$repo" \
&& { \
git rev-parse --verify $rev &>/dev/null && git reset --hard $rev \
|| { git fetch origin $rev && git reset --hard origin/$rev; }; \
} || abort could not find $rev \
&& cp bin/deploy $path \
&& chmod +x $path \
|| abort could not update deploy
echo
log_success "updated $VERSION -> `./bin/deploy --version`"
echo
}
local_cleanup() {
local tmp_dir="$1"
test -n "$tmp_dir" && test -d "$tmp_dir" && rm -fr "$tmp_dir"
}
deploy creates a new release directory for each deployment. By default, it keeps these directories forever, leaving you responsible for cleaning them up.
If given a keep
option, it can do it for you automatically after each deployment.
prepare_to_delete_old_releases() {
local command="$1"
local keep="$(get_last_config_key_value keep)"
local path=$(get_last_config_key_value path)
local release=$(date -u '+%Y-%m-%d-%H-%M-%S')
releases_to_delete=
if test -z "$keep" || [[ "$keep" == "all" ]]; then
return 0
fi
deploy will list the contents of the releases
directory on the host.
log listing previous releases
local list_releases_command="ls -1 $path/releases"
log_command "$list_releases_command"
local releases_list
releases_list="$(run_no_log "$list_releases_command 2>&1")"
test $? -eq 0 || abort "$releases_list"
local number_of_releases="$(echo "$releases_list"|wc -l)"
If no old release is found, deployment will proceed normally.
if test -z "$releases_list"; then
log_color $COLOR_CYAN "no old releases to delete"
return 0
fi
If the number of old releases is equal to or greater than the keep
number,
deploy will print the list of releases that will be deleted in red.
local releases_to_keep=
if { [[ "$command" == "deploy" ]] && test "$number_of_releases" -lt "$keep"; } || { [[ "$command" != "deploy" ]] && test "$number_of_releases" -le "$keep"; }; then
releases_to_keep="$releases_list"
else
local to_keep=$keep
[[ "$command" == "deploy" ]] && to_keep=$((to_keep-1))
local to_delete=$keep
[[ "$command" != "deploy" ]] && to_delete=$((to_delete+1))
releases_to_keep="$(echo "$releases_list"|tail -n "$to_keep")"
releases_to_delete="$(echo "$releases_list"|reverse|tail -n "+$to_delete"|reverse)"
fi
echo
for old_release in $releases_to_delete; do
log_color $COLOR_RED "$path/releases/$old_release (will be deleted)"
done
Old releases that will be kept will be shown in green.
for old_release in $releases_to_keep; do
log_color $COLOR_GREEN "$path/releases/$old_release (will be kept)"
done
if [[ "$command" == "deploy" ]]; then
During deployment, the current release being deployed is also indicated in green
(as it counts towards the keep
number).
log_color $COLOR_GREEN "$path/releases/$release (current)"
fi
If there are any releases to delete, deploy will ask for confirmation
(unless the -y, --yes
command line option or the $DEPLOY_YES
variable is specified).
if test -n "$releases_to_delete" && test -z "$DEPLOY_YES"; then
echo
local number_of_releases_to_delete="$(echo "$releases_to_delete"|wc -l)"
local message="$number_of_releases_to_delete release"
test "$number_of_releases_to_delete" -ne 1 && message="${message}s"
while test -z "$delete_releases"; do
log_prompt "$message will be deleted; please confirm (yes/no) "
read delete_releases
Responding “no” interrupts the deployment and prints a help message explaining
how to configure the keep
number.
if prompt_is_no "$delete_releases"; then
echo
echo "Use the \"keep\" configuration file option, the -k|--keep command line option"
echo "or the \$DEPLOY_KEEP variable to change the number of old releases that are kept"
echo "(use the value \"all\" to keep all releases forever)."
abort "$command interrupted"
elif ! prompt_is_yes "$delete_releases"; then
delete_releases=
fi
done
fi
}
The actual cleanup of old releases is always performed after successful deployment.
delete_old_releases() {
if test -z "$releases_to_delete"; then
return 0
fi
log deleting old releases
local path=$(get_last_config_key_value path)
local old_ifs=$IFS
IFS=$'\n'
local directories=
for release in $releases_to_delete; do
IFS="$old_ifs"
directories="$directories \"$path/releases/$release\""
IFS=$'\n'
done
IFS="$old_ifs"
A single rm
command is executed on the host through SSH,
deleting all appropriate old release directories in one go.
run "rm -fr $directories"
local number_of_releases_to_delete="$(echo "$releases_to_delete"|wc -l)"
local message="$number_of_releases_to_delete old release"
test "$number_of_releases_to_delete" -ne 1 && message="${message}s"
echo
log_success "$message deleted"
}
Those messages are printed in colors by default on interactive terminals
(this can be disabled with the --color never
option or $DEPLOY_COLOR
variable).
echo_color() {
if test -z "$COLORS"; then
if [ "$DEPLOY_COLOR" == "always" ]; then
COLORS=yes
elif [ "$DEPLOY_COLOR" != "never" ] && is_interactive; then
COLORS=yes
fi
fi
local color="$1"
shift
local message="$@"
if [ "$COLORS" == "yes" ]; then
echo -e "\033[${color}m${message}\033[0m"
else
echo "$message"
fi
}
is_interactive() {
test "${-#*i}" != "$-" || test -t 0 || test -n "$PS1"
}
log() {
echo
echo_color $COLOR_BOLD " ○ $@"
}
log_command() {
echo_color $COLOR_YELLOW "$@" | tr -s ' ' | sed 's/^/ /'
}
log_prompt() {
echo -n "$(echo_color $COLOR_MAGENTA "$@")" | tr -s ' ' | sed 's/^/ /'
}
log_color() {
local color="$1"
shift
local message="$@"
echo_color "$color" "$message" | tr -s ' ' | sed 's/^/ /'
}
log_success() {
echo_color $COLOR_GREEN "$@" | tr -s ' ' | sed 's/^/ /'
}
log_error() {
echo_color $COLOR_RED "$@"|sed 's/^ *//'|sed 's/^/ /' 1>&2
}
If deploy encounters an unrecoverable error, it will log an error and exit with status 1.
abort() {
echo
log_error "$@"
echo
exit 1
}
prompt_is_yes() {
[[ "$1" =~ ^(1|y|yes|t|true)$ ]]
}
prompt_is_no() {
[[ "$1" =~ ^(0|n|no|f|false)$ ]]
}
reverse() {
tac 2> /dev/null || tail -r
}
script_arg() {
local i="$1"
test -z "$i" && i=0
echo "${DEPLOY_ARGS[$i]}"
}
script_arg_count() {
echo ${#DEPLOY_ARGS[@]}
}
script_has_args() {
test ${#DEPLOY_ARGS[@]} -gt 0
}
script_has_option() {
value_is_in "$1" "$(script_option_names)"
}
script_option_names() {
echo "$DEPLOY_OPTIONS_VALUE_MAP"|sed 's/ .*$//'|tr '\n' ' '
}
script_shift() {
local by=$1
test -z "$by" && by=1
DEPLOY_ARGS=("${DEPLOY_ARGS[@]:$by}")
}
script_remove_args() {
local slice_index="$1"
local slice_length="$2"
test -z "$slice_length" && slice_length=1
local arg_count="${#DEPLOY_ARGS[@]}"
local rest_index=$((slice_index+slice_length))
local rest_length=$((arg_count-rest_index))
DEPLOY_ARGS=("${DEPLOY_ARGS[@]:0:$slice_index}" "${DEPLOY_ARGS[@]:$rest_index:$rest_length}")
}
get_option_arity() {
local option_name="$1"
local arity="$(get_option_properties "$option_name"|cut -d ':' -f 3 -s)"
test -n "$arity" && echo 1 || echo 0
}
get_option_name() {
local short_or_long_option="$1"
if [[ "$short_or_long_option" =~ ^-?[A-Za-z]$ ]]; then
local letter="$(echo "$short_or_long_option"|sed 's/^-//')"
echo "${DEPLOY_OPTIONS_MAP[@]}"|tr ' ' '\n'|grep "^${letter}:"|sed 's/^[^:]*://'|sed 's/:.*$//'
else
echo "$short_or_long_option"|sed 's/^-*//'
fi
}
get_option_properties() {
local option_name="$1"
echo "${DEPLOY_OPTIONS_MAP[@]}"|tr ' ' '\n'|grep -E "^[a-zA-Z]?:${option_name}(:|$)"
}
describe_option() {
local option_name="$1"
local description="--$option_name"
local short_option_letter="$(get_option_properties "$option_name"|sed 's/:.*$//')"
test -n "$short_option_letter" && description="-$short_option_letter, $description"
local option_value_name="$(get_option_properties "$option_name"|cut -d ':' -f 3 -s)"
test -n "$option_value_name" && description="$description <$option_value_name>"
echo "$description"
}
value_is_in() {
local value="$1"
shift
local args="$@"
for arg in $args; do
if [[ "$arg" == "$value" ]]; then
return 0
fi
done
return 1
}
validate_color() {
local color="$DEPLOY_COLOR"
[[ "$color" =~ ^(always|never|auto)$ ]] \
|| abort "invalid color option '$color': allowed values are always, never or auto"
}
validate_config_file() {
local file="$DEPLOY_CONFIG"
test -e "$file" || abort "invalid config file: $file does not exist"
test -f "$file" || abort "invalid config file: $file is not a file"
test -r "$file" || abort "invalid config file: $file cannot be read"
}
validate_identity_file() {
local file="$(get_last_config_key_value identity)"
test -z "$file" && return
test -e "$file" || abort "invalid identity file: $file does not exist"
test -f "$file" || abort "invalid identity file: $file is not a file"
test -r "$file" || abort "invalid identity file: $file cannot be read"
}
validate_keep() {
local keep="$(get_last_config_key_value keep)"
test -z "$keep" \
|| [[ "$keep" == "all" ]] \
|| [[ "$keep" =~ ^[1-9][0-9]*$ ]] \
|| abort "keep must be \"all\" or a positive integer greater than zero"
}
validate_port_number() {
local port="$(get_last_config_key_value port)"
test -z "$port" \
|| { [[ "$port" =~ ^[1-9][0-9]*$ ]] && test $port -ge 0 && test $port -le 65535; } \
|| abort "port must be an integer between 0 and 65535"
}
validate_working_directory() {
local dir="$DEPLOY_CHDIR"
test -z "$dir" && return
test -e "$dir" || abort "invalid working directory: '$dir' does not exist"
test -d "$dir" || abort "invalid working directory: '$dir' is not a directory"
test -x "$dir" || abort "invalid working directory: '$dir' cannot be accessed"
}
validate_options() {
whitelist_options "$@"
value_is_in color "$@" && validate_color
value_is_in config "$@" && validate_config_file
value_is_in identity "$@" && validate_identity_file
value_is_in keep "$@" && validate_keep
value_is_in port "$@" && validate_port_number
}
validate_default_options() {
validate_options chdir color config "$@"
}
validate_deploy_options() {
validate_default_options forward-agent host identity path port tty user "$@"
}
validate_update_options() {
validate_options color update-path update-prefix yes
}
whitelist_options() {
local allowed_options="$@"
for option_name in $(script_option_names); do
value_is_in "$option_name" "$allowed_options" \
|| abort "option $(describe_option "$option_name") is not applicable to command '$DEPLOY_COMMAND'"
done
}
main $@


I am a CSIRO Early Research Career (CERC) postdoctoral research fellow at CSIRO Astronomy and Space Science. My work primarily involves developing and using Bayesian inference techniques to learn more about the properties of pulsars - rotating neutron stars that emit beams of electromagnetic radiation from their magnetic poles. I am also the priniciple investigator of the Parkes `P574' young pulsar timing project, which uses the CSIRO 64-m Parkes radio telescope (AKA 'Murriyang') to monitor 270 energectic pulsars.
Other interests of mine include follow-up parameter estimation of gravitational-wave events seen by Advanced LIGO and Virgo, and searches for the stochastic gravitational-wave background with pulsar timing arrays.
News
- Jul 3 2023: This week I am at Macquarie University in Sydney for this years' Astronomical Society of Australia annual meeting. I'll be presenting my work on asteroids around PSR J0738-4042.
- Jun 29 2023: Big news! Today the international pulsar timing community announced tentative evidence for a stochastic gravitational-wave background in data collected by five major experiments. Results from the Parkes Pulsar Timing Array search can be found here. And here's a great summary of the results by study leads Daniel Reardon and Andrew Zic.
- Jun 22 2023: Today I am visiting the Jodrell Bank Centre for Astrophysics at the University of Manchester. I'll be giving a seminar on our recent work to understand the peculiar polarisation properties of a radio-loud magnetar.
- Jun 20 2023: Dr Lucy Oswald and I have won a Royal Society International Exhange Grant to work together on understanding the polarisation properties of radio pulsars!
- Jun 18 2023: Off to visit collaborators at the University of Oxford for two weeks!
- Jun 12 2023: Next stop Greece! This week I'm attending the 'Timing and Imaging of compact sources with SKA pathfinders' conference in the ancient village of Kerastari. I will be giving an invited presentation on how young pulsars spin-down over time.
- May 28 2023: Starting my month of European travel with a two-week visit to the Max Planck Institute for Radioastronomie!
- May 9 2023: Rotational and radio emission properties of PSR J0738-4042 over half a century has been submitted to MNRAS.
- Mar 17 2023: The 2022 high-energy outburst and radio disappearing act of the magnetar 1E 1547.0-5408 has now been officially been published. Read the open access article here.
- Mar 16 2023: George Younes, Zaven Arzoumanian and I wrote a short 'NICER/ISS Science Nugget' outlining our recent paper on 1E 1547.0-5408.
- Feb 14 2023: The 2022 high-energy outburst and radio disappearing act of the magnetar 1E 1547.0-5408 has been accepted to ApJ! Read the pre-print here: arXiv:2302.07397.
- Dec 21 2022: The 2022 high-energy outburst and radio disappearing act of the magnetar 1E 1547.0-5408 submitted to ApJ.
- Dec 16 2022: I have officially graduated from my PhD at Swinburne University of Technology!
Dr. Marcus Lower pic.twitter.com/VIj8hj5ufr
— Marcus Lower (@astromelow) December 19, 2022 - Dec 2 2022: Today I am giving the MQAAAstro colloquium at Macquarie University where I am talking about The Parkes magnetar monitoring programme.
- Dec 1 2022: FRB20221128A found by UTMOST-NS! The first detection of a fast radio burst using the newly upgraded North-South arm of the Molonglo telescope. Huge congratualtions to the UTMOST team on this terrific miletone!!
- Nov 11 2022: A birefringent pair plasma within the near-field environment of a radio-loud magnetar submitted to Nature Astronomy.
- Oct 31 2022: This week I'm at the Parkes Obervatory for the 2022 Australiasian (AKA: Orange) Pulsar and FRB workshop. I will be talking about the magnetar monitoring programme on Parkes.
- Sep 28 2022: Pulsar polarization: A broad-band population view with the Parkes Ultra-Wideband receiver submitted to MNRAS.
- Aug 1 2022: I am in Busan, South Korea for the next two weeks for the 2022 IAU General Assembly! Along with attending various talks across a wide range of astronomy, I'll be giving a talk on Radio-loud magnetars as Galactic fast radio burst analogues at the IAU Symposium 369.
- Jul 18 2022: This week I am attending the 2022 ATNF Science Retreat, being held in the Barossa Valley!
- 27 July 2022: This week I am in Hobart, Tasmania for the 2022 ASA Meeting. I'll be giving a talk on Untangling the twisted polarised emission of a radio-loud magnetar.
- Apr 22 2022: Propagation of a fast radio burst through a birefringent relativistic plasma submitted to PRD. Read the preprint here - arXiv:2204.10816.
- Mar 8 2022: Circularly polarized radio emission from the repeating fast radio burst source FRB 20201124A accepted to MNRAS! Read the updated preprint here - arXiv:2109.11535.
- Jan 21 2022: We performed follow-up observations of the FRB-emitting magnetar SGR 1935+2154 with Parkes/Murriyang after two recent outbursts. Read about our non-detections here - ATel #15172.
- Jan 9 2022: I have moved to sunny Sydney to start my CSIRO postdoc in-person!
- Nov 19 2021: I gave a talk on my recent paper on glitchy pulsars to the Monash gravitaional-wave research group. I was formerly a member of this group during my undergrad and honours years at Monash!
- Nov 16 2021: This evening I presented my work on the apparent braking of glitchy pulsars to the Oxford Pulsar group!
- Nov 8 2021: A warm welcome to the CSIRO Space and Astronomy community via the ATNF Daily Astronomy Picture!
- Oct 18 2021: I have started my new postdoctoral research position at CSIRO Astronomy and Space Science! For the next 3 years I'll be working on studying young pulsars and magnetars observed primarily with the Parkes radio telescope.
- Oct 4 2021: After 3-and-a-bit years, 4 published papers and two-(ish) still in prep., I have submitted my PhD thesis! This couldn't have been done without the support of my PhD advisors, nor my dozens of friend and colleagues at Swinburne, CSIRO and MeerTime collaborators.
- Sep 23 2021: Circularly polarized radio emission from the repeating fast radio burst source FRB 20201124A submitted to MNRAS. Read the pre-print here - arXiv:2109.11535.
- Sep 15 2021: The fourth and final publication of my PhD, The impact of glitches on young pulsar rotational evolution was accepted to MNRAS! Read the accepted manuscript here - arXiv:2109.07612.
- Aug 21 2021: Technical document posted to arXiv on A phenomenological model for measuring generalised Faraday rotation. Check it out here!
- Aug 5 2021: A short paper by Johston and Lower, A supernova remnant association for the fast-moving pulsar PSR J0908-4913 was published in MNRAS Letters! Read the accepted manuscript here - arXiv:2108.09827.
- Jul 28 2021: This week I am attending the FRB 2021 conference.
- Jul 25 2021: The Parkes Pulsar Timing Array weighs in on the common signal detected among millisecond pulsars! A fantastic paper, On the Evidence for a Common-spectrum Process in the Search for the Nanohertz Gravitational-wave Background with the Parkes Pulsar Timing Array, led by Boris Goncharov was accepted to ApJ Letters. Read the accepted manuscript here - arXiv:2107.12112.
- Jul 24 2021: A new glitch in the Vela pulsar was detected by Liam Dunn with the Molonglo North-South system! Read about it here - ATel #14807.
- Jul 19 2021: This week I am attending the 14th Edoardo Amaldi Conference on Gravitational Waves, where I am again talking about the Double Pulsar eclipses..
- Jul 16 2021: My poster at the 2021 ASA meeting (How do young pulsars spin down?) won 3rd place in the student poster prize!
- Jul 12 2021: This week I am attending the 2021 annual general meeting of the Astronomical Society of Australia.
- Jun 21 2021: This week I am attending the 2021 IPTA Science Week, where I giving a talk about observations of the Double Pulsar eclipses with MeerKAT.
- Apr 26 2021: I'm up in Sydney this week as the active Duty Astronomer for the Australia Telescope Compact Array.
- Apr 14 2021: Effects of periodicity in observation scheduling on parameter estimation of pulsar glitches by Dunn, Lower and Melatos accepted to MNRAS. Read the accepted manuscript here - arXiv:2104.07363.
- Mar 30 2021: FRB 20210630A found by UTMOST! Read about it here - ATel #14745.
- Mar 6 2021: Multi-frequency observations of SGR J1935+2154 accepted to MNRAS.
- Mar 3 2021: FRB 20210303A found by UTMOST, Molonglo's 1st FRB of 2021! Read about it here - ATel #14434.
- Feb 9 2021: The Relativistic Binary Programme on MeerKAT: Science objectives and first results accepted to MNRAS. Read the accepted manuscript here - arXiv:2102.05160.
- Feb 2 2021: My recent work on the dynamic magnetopshere of Swift J1818.0−1607 was featured in The Age, Unlocking the secrets of one of the universe’s strongest magnets.
- Jan 26 2021: Non-detection of radio pulses from GRB 210119A/Swift J1851.2-6148 with Parkes, ATel #14347.
- Dec 4 2020: My 3rd PhD paper, The dynamic magnetosphere of Swift J1818.0−1607, has been accepted to MNRAS. You can read the accepted manuscript here: arXiv:2011.12463.
- Oct 30 2020: The dynamic magnetosphere of Swift J1818.0−1607 submitted to MNRAS.
- Sep 21 2020: Bayesian inference for compact binary coalescences with Bilby: Validation and application to the first LIGO–Virgo gravitational-wave transient catalogue accepted to MNRAS. Read the accepted manuscript here - arXiv:2006.00714.
- Sep 16 2020: A great press release covering the first magnetar parallax measurement made with the Very Long Baseline Array was posted on the NRAO website: "VLBA Makes First Direct Distance Measurement to Magnetar".
- Sep 11 2020: ATel #14005 - We detected the new magnetar Swift J1818.0−1607 with the VLBA!
- Aug 28 2020: The paper I co-authored with fellow Swinburne student Hao Ding on the parallax of XTE J1810-197 was featured on Space Australia: "Ancient Distance Measuring Technique Applied to Magnetar".
- Aug 14 2020: A magnetar parallax accepted for publication in MNRAS. Read the accepted manuscript here - arXiv:2008.06438.
- Jul 7 2020: Both the 2020 Fast Radio Burst and Astronomical Society of Australia annual meetings are being held virtually this week. I'm giving a presentation on the wideband properties of the magnetar XTE J1810-197.
- Jun 30 3030: ATel #13840 - The magnetar XTE J1810-197 has undergone a resurgence in its radio brightness.
- Jun 1 2020: Spectropolarimetric properties of Swift J1818.0−1607: a 1.4 s radio magnetar accepted for publication in ApJ Letters. Read the accepted manuscript here - arXiv:2004.11522.
- May 29 2020: The MeerKAT Telescope as a Pulsar Facility: System verification and early science results from MeerTime accepted to PASA. Read the manuscript here - arXiv:2005.14366.
- May 15 2020: ATel #13737 - We didn't see any radio emission during a chance observation of the X-ray pulsar binary IGR J18179-1621.
- May 8 2020: ATel #13715, FRB 200508 detected by UTMOST - the first FRB seen by UTMOST in the new decade!
- May 1 2020: Nice article by Michelle Starr from Science Alert dissecting some of my work on the new magnetar Swift J1818.0−1607: "Strangely Flaring Dead Star Could Be The 'Missing Link' Between Magnetars And Pulsars".
- Apr 23 2020: Spectropolarimetric properties of Swift J1818.0−1607: a 1.4 s radio magnetar submitted to ApJ Letters. Read the pre-print here - arXiv:2004.11522.
- Mar 25 2020: Multi-band follow up observations of the new magnetar with Parkes - ATel #13587.
- Mar 22 2020: The Parkes Pulsar Timing Array Project: Second data release accepted to PASA. Read the accepted manuscript here - arXiv:2003.09780.
- Mar 17 2020: The new magentar emits radio pulses! We followed up this latest discovery by observing it with the MeerKAT telesope - ATel #13562.
- Mar 13 2020: A new magnetar called Swift J1818.0−1607 has been found by the Swift satellite (GCN #27373) and confirmed by the NICER X-ray telesope - ATel #13551.
- Feb 28 2020: The UTMOST pulsar timing programme II: Timing noise across the pulsar population accepted to MNRAS. You can read the accepted manuscript here - arXiv:2002.12481.
- Feb 26 2020: Today I was invited to give the weekly ATNF colloquium, where I spoke about Massive scale pulsar timing with the Molonglo Observatory Synthesis Telescope.
- Feb 16 2020: I'm up in Sydney to visit the Australia Telescope National Facility in Marsfield for three weeks.
- Jan 15 2020: The UTMOST Survey for Magnetars, Intermittent pulsars, RRATs and FRBs I: System description and overview accepted to MNRAS.
- Dec 23 2019: ATel #13363 - A Christmas FRB found by UTMOST!
- Dec 18 2019: I found a glitch in the pulsar PSR J0908−4913 using UTMOST! This is a pulsar that has been observed for over 20 years, and had never been seen to glitch. You can read the research note here, or the post on the UTMOST website.
- Nov 18 2019: I am at the 3rd OzGrav annual retreat in Lorne, Australia this week.
- Nov 7 2019: UTMOST found another fast radio burst: FRB191107 - ATel #13282.
- Sep 19 2019: New paper submitted - Lower et al. The UTMOST pulsar timing programme II: Timing noise across the pulsar population. Submitted to MNRAS.
- Aug 6 2019: Another FRB at UTMOST - ATel #12995.
- Jun 27 2019: ASKAP has localised a lone FRB to its host galaxy! My supervisor Dr Ryan Shannon is one of the lead scientists of this project. You can read all about it in the ASKAP/CRAFT paper published in Science.
- Jun 9 2019: I am in Pune, India for the next two weeks for the 2019 International Pulsar Timing Array meeting.
- Apr 2 2019: "Wideband polarized radio emission from the newly revived radio magnetar XTE J1810−197" and "The 2018 X-ray and Radio Outburst of Magnetar XTE J1810−197" have both been accepted and published in ApJ Letters!
- Mar 22 2018: FRB190322 - the 10th FRB seen to date by UTMOST.
- Feb 22 2019: "The 2018 X-ray and Radio Outburst of Magnetar XTE J1810−197" submitted to ApJ (arXiv:1902.08358).
- Feb 14 2019: "Wideband polarized radio emission from the newly revived radio magnetar XTE J1810−197" submitted to ApJ Letters (arxiv:1902.04689).
- Dec 30 2018: FRB181228 - one for the holidays!
- Dec 13 2018: radio emission from the magnetar XTE J1810-197 reactivated. You can read about our observations at Molonglo here and here.
- Our paper describing the Bilby parameter estimation code has been submitted to ApJ Supplementary Series (arXiv:1811.02042).
- 31 Oct 2018: my 1st ever paper has been finally published in PRD! You can read it on the Physical review website, or the pre-print on arXiv.
- Oct 18 2018: in an amazing cosmic coincidence, we managed to see two FRBs at Molonglo in only two days (ATel #12124)! You can read more about them on the UTMOST website.
- Aug 7 2018: we saw a glitch in the pulsar PSR J1709−4429. Read the research note on the RNAAS website.
- Jul 31 2018: started PhD at Swinburne University of Technology. I'll be working on applying Bayesian inference to pulsar timing.

- Mar 26 2018: started internship at Swinburne University of Technology
- Dec 3 2017: finished my undergrad with 1st class honours!


Issue 30
-
The Haskell.org Website Working Group (HWWG)
The goal of the Haskell Website Working Group is to make sure the Haskell website caters to the needs of Haskell programmers, particularly beginners. In doing so we hope to either combine or differentiate haskell.org and haskell-lang.org, and give people clear recommendations of what “downloading Haskell” means.
-
What if we have a service that needs to support people and computers? REST allows us to do this. Can Haskell’s Message Pack library actually perform as well or better than the venerable Aeson?
-
Comparative concurrency with Haskell
We’ll start with approaches very similar to other languages like Go and Rust, and then dive into techniques like Software Transactional Memory which provide a much improved concurrency experience for more advanced workflows. Finally we’ll dive into the async library, which provides some very high-level functions for writing concurrent code in a robust manner.
-
Clean alternatives with MaybeT
The Monad instance for Maybe covers a common case: given some sequence of functions which may fail, we want to try them all and if any of them fail then we’ll short circuit it all. However, that’s not the only case. Very often, you’ll want to take the first thing that succeeds, rather than failing unless everything works.
-
“Hello, World” with PureScript C++
This is a short-as-possible demonstration of using PureScript’s C++11 backend (a.k.a. pure11) to build a native command-line application.


There are two types of categorical variables: a nominal categorical variable and an ordinal categorical variable.
A nominal variable is a categorical variable without an implied order. This means that it is impossible to say that 'one is worth more than the other'. For example, think of the categorical variable animals_vector
with the categories "Elephant"
, "Giraffe"
, "Donkey"
and "Horse"
. Here, it is impossible to say that one stands above or below the other. (Note that some of you might disagree ;-) ).
In contrast, ordinal variables do have a natural ordering. Consider for example the categorical variable temperature_vector
with the categories: "Low"
, "Medium"
and "High"
. Here it is obvious that "Medium"
stands above "Low"
, and "High"
stands above "Medium"
.
# Animals
animals_vector <- c("Elephant", "Giraffe", "Donkey", "Horse")
factor_animals_vector <- factor(animals_vector)
factor_animals_vector
# Temperature
temperature_vector <- c("High", "Low", "High","Low", "Medium")
factor_temperature_vector <- factor(temperature_vector, order = TRUE, levels = c("Low", "Medium", "High"))
factor_temperature_vector
# Animals
animals_vector <- c("Elephant", "Giraffe", "Donkey", "Horse")
factor_animals_vector <- factor(animals_vector)
factor_animals_vector
# Temperature
temperature_vector <- c("High", "Low", "High","Low", "Medium")
factor_temperature_vector <- factor(temperature_vector, order = TRUE, levels = c("Low", "Medium", "High"))
factor_temperature_vector
msg <- "Do not change anything about the sample code. Simply hit the Submit Answer button and inspect the solution!"
test_object("animals_vector", undefined_msg = msg, incorrect_msg = msg)
test_object("temperature_vector", undefined_msg = msg, incorrect_msg = msg)
test_object("factor_animals_vector", undefined_msg = msg, incorrect_msg = msg)
test_output_contains("factor_animals_vector", incorrect_msg = msg)
test_object("factor_temperature_vector", undefined_msg = msg, incorrect_msg = msg)
test_output_contains("factor_temperature_vector", incorrect_msg = msg)
success_msg("Can you already tell what's happening in this exercise? Awesome! Continue to the next exercise and get into the details of factor levels.")
Just click the 'Submit Answer' button and look at the console. Notice how R indicates the ordering of the factor levels for ordinal categorical variables.


{{#available}} {{/available}}
{{#available}}| {{name}} | {{#minGrade}}({{minGrade}}th grade and above) | {{/minGrade}} {{#minAge}}(Must be {{minAge}} years of age) | {{/minAge}}
The intensives sign-up form is not available at the moment. Please try again later.
{{/available}}


PURPOSE: This project began as an exploratory data analysis (EDA) to let the data 'speak to us.' Presented in this website are data analysis on the impact of the novel Coronavirus (COVID:19) on the U.S. National Park Visitations.
BACKGROUND: Data used in this analysis came from the U.S. National Park Service statistics page (NPS Stats), which houses a rich collection of data on visitors to the U.S. National Parks. The data are provided in pre-formatted reports dating back to 1904 and cut by a variety of park charateristics (e.g., park type, park region, state location, acitivities, etc.).
RESEARCH QUESTION: To what extent did COVID:19 affect the park visitation numbers at the regional level before and during COVID:19 lock-down measures and policies took place? For the U.S., many states issued a stay-at-home lock-down policy around March 2020, when COVID:19 began to make its presence on the U.S. population. I present my portion of the research project that answers this research question at a regional level, which examines the 2019 and 2020 data.
TAKEAWAY:


The goals / steps of this project are the following:
This is the fifth Udacity Self driving car nanodegree assignment. I developed a processing pipeline using computer vision to detect and track cars on video. Using a Support Vector Machine Classifier for predictions, creating a heat map, potential boxes and sliding windows to isolate the predicted cars positions and drawing a bounding box aroud it.
0 Intro -> Car images -> basic model
To be able to find a car in an image, it's neccesary to have some idea of what we're looking for, we need to have something that defines the concept of a "car", this is a "model" of a car to compare later.
We can extract information from an image because the pixels from it represent "features" of information, this information will be useful.
The simplest and useful, means a resizing of the image and flatteing with ravel(). Spatial features represents the values and ordering of the pixels.
All happens in the method called get_bin_spatial_features(), this method used an RGB image with a size 16x16
The color histogram features identify the amount of certain color values. Using other color spaces like HSV will make more easy detect cars in an image (RGB doesn't work so well), identify a car using HSV is much more useful and allow us to read a lot of color edges.
A method called get_color_hist_features() do this work, the parameters are the numbers of bins.
One of the most usefult features and the most expensive (computationally). HOG "histogram of oriented gradients" and basically be though of as colors are changing by 'p' amount in 'x' direction.HOG provides a generic shape information, the method called get_hog_features() do this work. Parameters to tune: pixels per cell, number of orientation directions, cells per block.
###########################################################################################
# Initial imports
###########################################################################################
import numpy as np
import cv2
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.preprocessing import StandardScaler
from skimage.feature import hog
from scipy.ndimage.measurements import label
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import glob
import time
import random
#import imageio
#imageio.plugins.ffmpeg.download()
from moviepy.editor import VideoFileClip
from IPython.display import HTML
%matplotlib inline
print('Import done!')
###########################################################################################
# Plot helpers
###########################################################################################
# draw one big image (could fail because an unknown bug)
def one_plot(img):
# Plots one big image
fig, ax = plt.subplots(figsize=(20, 10))
ax.imshow(img)
# draw two medium images
def two_plots(img1, img2, img1_title=None, img2_title=None):
# Plots two images
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))
f.tight_layout()
ax1.imshow(img1)
if img1_title: ax1.set_title(img1_title, fontsize=30)
ax2.imshow(img2)
if img2_title: ax2.set_title(img2_title, fontsize=30)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)
print('Plot helpers done!')
###########################################################################################
# 1 -> Features
###########################################################################################
# return binned color features for an image
def get_bin_spatial_features(img, size=(16, 16)):
# Resize the image, and then flatten it out into a feature vector
return cv2.resize(img, size).ravel()
# return color histogram feature
def get_color_hist_features(img, nbins=32, bins_range=(0, 256)):
# calculate the histogram of the color channels separately
ch1_hist = np.histogram(img[:,:,0], bins=nbins, range=bins_range)[0]
ch2_hist = np.histogram(img[:,:,1], bins=nbins, range=bins_range)[0]
ch3_hist = np.histogram(img[:,:,2], bins=nbins, range=bins_range)[0]
# concatenate the histograms into a single feature vector
return np.concatenate((ch1_hist, ch2_hist, ch3_hist))
# return HOG features
def get_hog_features(img, orient, pix_per_cell, cell_per_block, vis=False, feature_vec=True):
if vis == True:
features, hog_image = hog(img, orientations=orient,
pixels_per_cell=(pix_per_cell, pix_per_cell),
cells_per_block=(cell_per_block, cell_per_block),
transform_sqrt=True,
visualise=vis, feature_vector=feature_vec)
return features, hog_image
else:
features = hog(img, orientations=orient,
pixels_per_cell=(pix_per_cell, pix_per_cell),
cells_per_block=(cell_per_block, cell_per_block),
transform_sqrt=True,
visualise=vis, feature_vector=feature_vec)
return features
# take a list of images files names and put it into a list of image features, using later into the classifier
def extract_features(imgs, color_space='HSV', spatial_size=(16,16), hist_bins=32,
orient=8, pix_per_cell=8, cell_per_block=2, hog_channel='ALL', precomputed_hog=None,
use_spatial_features=True, use_color_hist_features=True, use_hog_features=True):
features = []
for img in imgs:
img_features = []
if type(img) is str:
image = mpimg.imread(img)
if img.endswith('.png'):
image = image*255
else:
image = img
img_size = (64, 64)
if image.shape[:2] != img_size:
image = cv2.resize(image, img_size)
if precomputed_hog is not None and precomputed_hog.shape[:2] != img_size:
precomputed_hog = cv2.resize(precomputed_hog, img_size)
# a basic color conversion
original_image = np.copy(image)
if color_space == 'RGB':
image = np.copy(image)
else:
color = eval('cv2.COLOR_RGB2{}'.format(color_space))
image = cv2.cvtColor(image, color)
# spatial features
if use_spatial_features == True:
spatial_features = get_bin_spatial_features(original_image, size=spatial_size)
img_features.append(spatial_features)
# color histogram Features
if use_color_hist_features == True:
hist_features = get_color_hist_features(image, nbins=hist_bins)/255
img_features.append(hist_features)
# HOG Features
if use_hog_features == True:
if precomputed_hog is not None:
img_features.append(precomputed_hog*255)
else:
if hog_channel == 'ALL':
hog_features = []
for channel in range(image.shape[2]):
hog_features.append(get_hog_features(
image[:,:,channel],
orient,
pix_per_cell,
cell_per_block,
vis=False,
feature_vec=True
))
hog_features = np.ravel(hog_features)
else:
hog_features = get_hog_features(
image[:,:,hog_channel],
orient,
pix_per_cell,
cell_per_block,
vis=False,
feature_vec=True
)
img_features.append(hog_features*255)
# add this to de principal list
features.append(np.concatenate(img_features))
if len(imgs) == 1:
return np.array(features)
return features
print('Features done!')
Now that we can get features from images, we can use this features to train a ML classifier for vehicle detection. 17760 images where used (images from cars (~50%) and images from non-cars (~50%). These features are labeled as "0" (is a car) and "1" (non a car), later passed into a SVM, using SVC. (linearly separates the data).
Here are links to the labeled data for vehicles https://s3.amazonaws.com/udacity-sdc/Vehicle_Tracking/vehicles.zip and non-vehicles https://s3.amazonaws.com/udacity-sdc/Vehicle_Tracking/non-vehicles.zip
Feature values are normalized before being passed in to improve learning. The data was shuffle.
The final test accuracy was ~ 99.2% and took ~ 100 secs.
At the end examples of original image, raw features, normalized features and hog-visualization are plotted.
###########################################################################################
# 2 -> Training a classifier
###########################################################################################
vehicle_imgs = shuffle(glob.glob('./training-data/vehicles/*/*.png'))
nonvehicle_imgs = shuffle(glob.glob('./training-data/non-vehicles/*/*.png'))
n_imgs = len(vehicle_imgs) + len(nonvehicle_imgs)
# take the time
t = time.time()
vehicle_features = extract_features(vehicle_imgs)
nonvehicle_features = extract_features(nonvehicle_imgs)
t2 = time.time()
print('Total extraction time:', round(t2-t, 2), 'secs,', n_imgs, 'image features')
x = np.vstack((vehicle_features, nonvehicle_features)).astype(np.float64)
scaler = StandardScaler().fit(x)
# normalize the features
scaled_x = scaler.transform(x)
# labels vector - 1 is car, 0 is not
y = np.hstack((np.ones(len(vehicle_features)), np.zeros(len(nonvehicle_features))))
min_scaled = np.min(scaled_x)
max_scaled = np.max(scaled_x)
# shuffle data train/test split
rand = random.randint(0, 100)
x_train, x_test, y_train, y_test = train_test_split(scaled_x, y, test_size=0.25, random_state=rand)
# svc training
print('Training the SVC...')
svc = LinearSVC()
t = time.time() # Track the training time
svc.fit(x_train, y_train)
t2 = time.time()
print('Training time:', round(t2-t, 2), 'secs')
# svc accuracy
print('Accuracy of the SVC:', round(svc.score(x_test, y_test), 3))
# Plot an example of raw and scaled features
car_ind = random.randint(0, len(vehicle_imgs))
fig = plt.figure(figsize=(12,4))
plt.subplot(131)
plt.imshow(mpimg.imread(vehicle_imgs[car_ind]))
plt.title('Original Image')
plt.subplot(132)
plt.plot(x[car_ind])
plt.title('Raw Features')
plt.subplot(133)
plt.plot(scaled_x[car_ind])
plt.title('Normalized Features')
fig.tight_layout()
ind = car_ind
# Read in the image
image = mpimg.imread(vehicle_imgs[car_ind])
gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
# Define HOG parameters
orient = 9
pix_per_cell = 8
cell_per_block = 2
# Call our function with vis=True to see an image output
features, hog_image = get_hog_features(gray, orient,
pix_per_cell, cell_per_block,
vis=True, feature_vec=False)
# Plot the examples
fig = plt.figure()
plt.subplot(121)
plt.imshow(image, cmap='gray')
plt.title('Example Car Image')
plt.subplot(122)
plt.imshow(hog_image, cmap='gray')
plt.title('HOG Visualization')
print('Features && SVM done! car_ind =', car_ind)
The SVM was trained on 64x64 pixel images, it's neccesary to find a way to feed it similar input if we want keep the accuracy (~99.2%).
The search was developed using sliding windows. Windows of various sizes will be passed over parts of the image to capture many sub-images, each of these sub-images will then be tested with the classifier to tell us whether the sub-image contains a car or not.
In the output you see the many windows drawn onto the image. A high 75% overlap is used to avoid missing centered images of the vehicles. To reduce computation time, I only search the right half of the image because the pictures are taken from a vehicle in the far-left lane.
###########################################################################################
# 3 -> Finding cars, sliding windows
###########################################################################################
# returns the list of windows to search through
# sizes = the square dimension of the windows
# y_start_stop, x_start_stop = percent of the image where start and stop in each axis
def get_windows(img, sizes=[128, 180, 256], overlap=.75, y_start_stop=(.5, .92), x_start_stop=(.5, 1)):
y_size, x_size = img.shape[:2]
x_start = int(x_start_stop[0]*x_size)
x_stop = int(x_start_stop[1]*x_size)
y_start = int(y_start_stop[0]*y_size)
y_stop = int(y_start_stop[1]*y_size)
# windows will be stored in this array
windows = []
for size in sizes:
step = int(size*(1-overlap))
for y in range(y_start, y_stop-size, step):
for x in range(x_start, x_stop-size, step): # windows from the left
windows.append(((x, y), (x+size, y+size)))
for x in range(x_stop, x_start+size, -step): # windows from the right
windows.append(((x-size, y), (x, y+size)))
return windows
# bounding boxes
def draw_boxes(img, boxes, color=(0, 0, 255), thickness=4):
imcopy = np.copy(img)
for box in boxes:
cv2.rectangle(imcopy, box[0], box[1], color, thickness)
return imcopy
# test 1
image = mpimg.imread('./test_images/test4.jpg')
draw_img = np.copy(image)
# only for demonstration (different windows sizes)
boxes_1 = get_windows(image, sizes=[128], y_start_stop=(.4, .7), x_start_stop=(.1, 1))
boxes_2 = get_windows(image, sizes=[180], y_start_stop=(.4, 1), x_start_stop=(.1, 1))
boxes_3 = get_windows(image, sizes=[256], y_start_stop=(.4, 1), x_start_stop=(.1, 1))
draw_img = draw_boxes(draw_img, boxes_1, color=(0, 255, 0), thickness=3)
draw_img = draw_boxes(draw_img, boxes_2, color=(0, 0, 255), thickness=2)
draw_img = draw_boxes(draw_img, boxes_3, color=(255, 0, 0), thickness=1)
windows = np.concatenate((boxes_1, boxes_2, boxes_3))
print(len(windows), 'windows test 1')
one_plot(draw_img)
# test 2
image = mpimg.imread('./test_images/test4.jpg')
draw_img = np.copy(image)
# only for demonstration (different windows sizes)
boxes_1 = get_windows(image, sizes=[128], y_start_stop=(.5, .7))
boxes_2 = get_windows(image, sizes=[120])
boxes_3 = get_windows(image, sizes=[170])
draw_img = draw_boxes(draw_img, boxes_1, color=(0, 255, 0), thickness=3)
draw_img = draw_boxes(draw_img, boxes_2, color=(0, 0, 255), thickness=2)
draw_img = draw_boxes(draw_img, boxes_3, color=(255, 0, 0), thickness=1)
windows = np.concatenate((boxes_1, boxes_2, boxes_3))
print(len(windows), 'windows test 1')
one_plot(draw_img)
The sliding-window results in > 500 windows, too much! Each of these windows, these sub-images, will be tested by extracting features and passing those features to the classifier using the contains_car method.
To identify a single bounding-box per car a heatmap is used, where windows overlap the regions gets hotter. A heatmap identifies high concentrations of windows. Using the heatmap information allows to draw a single bounding-box around each car. The false positives can be avoided by summing the heatmaps across multiple frames (of a video) and taking a higher threshold. get_aggregate_heatmap along with the stored prev_heatmaps do this work.
###########################################################################################
# 4 -> Boxes around cars, heatmaps
###########################################################################################
# true or false for whether the window of the given image contains a car
def contains_car(img, window):
top_left, bottom_right = window[0], window[1]
sub_img = img[top_left[1]:bottom_right[1], top_left[0]:bottom_right[0]]
features = extract_features([sub_img])
# normalize the features
scaled_features = scaler.transform(features)
# was getting a single HUGE value, so clipping
scaled_features = np.clip(scaled_features, min_scaled, max_scaled)
return svc.predict(scaled_features) == 1
def get_heatmap(img, boxes, thresholded=False):
heatmap = np.zeros_like(img[:,:,0])
for box in boxes:
top_left, bottom_right = box[0], box[1]
heatmap[top_left[1]:bottom_right[1], top_left[0]:bottom_right[0]] += 1
if thresholded: return threshold_heatmap(heatmap)
return heatmap
def threshold_heatmap(heatmap, threshold=2):
heatmap[heatmap < threshold] = 0
return heatmap
def get_aggregate_heatmap(thresholded=True, threshold=4):
global prev_heatmaps
heatmaps = np.array(prev_heatmaps)
if not thresholded: return np.sum(heatmaps, axis=0)
return threshold_heatmap(np.sum(heatmaps, axis=0), threshold=threshold)
def draw_heatmap_boxes(img, heatmap):
label_img, n_items = label(heatmap)
# iterate through all detected cars
for car_number in range(1, n_items+1):
nonzero = (label_img == car_number).nonzero() # find pixels with each car_number label value
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
# bounding box
box = ((np.min(nonzerox), np.min(nonzeroy)), (np.max(nonzerox), np.max(nonzeroy)))
cv2.rectangle(img, box[0], box[1], (0, 0, 255), 6)
return img
# tests
image = mpimg.imread('./test_images/test4.jpg')
draw_img = np.copy(image)
windows_1 = get_windows(image, sizes=[80], overlap=.75, y_start_stop=(.5, .7))
windows_2 = get_windows(image, sizes=[120, 180], overlap=.75)
windows = windows_1 + windows_2
print(len(windows), 'windows')
car_windows = []
for window in windows:
if contains_car(image, window):
car_windows.append(window)
draw_img = draw_boxes(draw_img, car_windows)
heatmap = get_heatmap(image, car_windows, thresholded=False)
heatmap_boxes = draw_heatmap_boxes(np.copy(image), heatmap)
two_plots(draw_img, heatmap_boxes)
cv2.imwrite('./output_images/windows.jpg', cv2.cvtColor(draw_img, cv2.COLOR_RGB2BGR))
cv2.imwrite('./output_images/cars.jpg', cv2.cvtColor(heatmap_boxes, cv2.COLOR_RGB2BGR))
All the steps grouped into a simple pipeline function, find_cars(), given an image the function will return the same imgae with bounding boxes around to detected cars.
###########################################################################################
# 5 -> Basic pipeline
###########################################################################################
prev_heatmaps = []
def find_cars(image, vis=False):
global prev_heatmaps
windows_1 = get_windows(image, sizes=[80], y_start_stop=(.4, .7))
windows_2 = get_windows(image, sizes=[120, 180])
windows = windows_1 + windows_2
car_windows = []
for window in windows:
if contains_car(image, window):
car_windows.append(window)
heatmap = get_heatmap(image, car_windows, thresholded=False)
prev_heatmaps.insert(0, heatmap)
if len(prev_heatmaps) > 5:
prev_heatmaps.pop()
aggregate_heatmap = get_aggregate_heatmap(threshold=4)
if vis == True:
two_plots(heatmap, aggregate_heatmap, 'Heatmap', 'Agg.Heatmap')
return draw_heatmap_boxes(np.copy(image), aggregate_heatmap)
image = mpimg.imread('./test_images/test4.jpg')
one_plot(find_cars(image, True))
It's time to test the pipeline with videos, each frame is passed through the pipeline, using heatmaps and aggregated_heatmaps for the last 5 frames and summed together to avoid false-positives. Finally using all this information, each frame will have the bounding-box over each detected car.
The pipeline produce a new video with the bounding-boxes over each detected car.
The output videos are stored in the directory "output_videos".
###########################################################################################
# 6 -> Processing videos
###########################################################################################
# pipeline of test_video.mp4
prev_heatmaps = []
file_name = 'test_video.mp4'
video = VideoFileClip('./test_videos/{}'.format(file_name)) # get the input video
video = video.fl_image(find_cars) # pipe video into the find_cars pipeline
%time video.write_videofile('./output_videos/{}'.format(file_name), audio=False) # save the new video
# pipeline of project_video.mp4
prev_heatmaps = []
file_name = 'project_video.mp4'
video = VideoFileClip('./test_videos/{}'.format(file_name)) # get the input video
video = video.fl_image(find_cars) # pipe video into the find_cars pipeline
%time video.write_videofile('./output_videos/{}'.format(file_name), audio=False) # save the new video
The final pipeline has one big problem, the computation time, is too slowly, for the project_video.mp4 with 50 second of duration it took ~ 1:21 minutes to process!!!, for a real time application this wouldn't work. I guess that te bottleneck is feature-extraction, all the feature-set need to be extracted in each frame of the ~500 sliding windows. HOG is the most expensive feature. Other important factor in the computation time is the number of sliding windows where the search is performed, that produces a big problem on performance. In the final output video there are some false positives, maybe with a better tunning of heatmaps, this could be solved.
Perhaps exist a better method to detect and track cars on video, replacing the HOG + SVM pipeline. I found a paper on https://arxiv.org/pdf/1512.02325.pdf -> (SSD: Single Shot Multibox Detector) using a deep network to perform the detection. I'll try this on the future.


#CECEFF
Manzona, Dan, Bakari, and I worked on a project together. The challenge was to have a golf ball dropped by 6 feet and have it land in 12 straws. The materials we had for this task were 12 strawws, 18 inches of tape, a scisscors. This task was very difficult to do considering we only had 12 straws and only 18 inches of tape. The second challenge was to build the tallest skyscraper with about 5 pieces of news paper and 12 inches of paper. Ours was between 53 and 58 inches tall. With this project I helped with the planning and building of the project. I helped build the bottom part to the 12 straws. The last thing I helped with is the building of the skyscaper. I helped with the rolling of the papers and the taping of it.