IPv6 in Contiv Network
Contiv Network supports a dual stack with IPv4 and IPv6 interfaces for containers. Docker Swarm APIs do not support IPv6-only containers.
Configuring an IPv6 Network
When configuring a network, you can specify an optional IPv6 subnet pool to allocate IPv6 address for the containers:
$ netctl net create contiv-net --subnet=<IP_ADDRESS>/24 --subnetv6=2001::/100
Running Containers in an IPv6 Network
You run containers in an IPv6 network by assigning them to the named network.
Consider two containers.
First, a container 'web' in Node1:
[vagrant@netplugin-node1 ~]$ docker run -itd --net contiv-net --name web alpine sh
f0b75b2ccf9a7c71d138f74d8092d8535357ebc61feb963f740f1ba593afbc01
Second, a container 'db' in Node2:
[vagrant@netplugin-node2 ~]$ docker run -itd --net contiv-net --name db alpine sh
5ed67c19041dd64c983bab7a21ca3a1ad7e85a6ee981b4140b32068bae6d6e26
IP Address Allocation
The containers are allocated IPv4 and IPv6 address from the corresponding subnet range:
[vagrant@netplugin-node1 ~]$ docker network inspect contiv-net
[
{
"Name": "contiv-net",
"Id": "6c21fa082f4f456730928d076b19148e6db1d88d90fe8f2c0760223af51f4e31",
"Scope": "global",
"Driver": "netplugin",
"IPAM": {
"Driver": "netplugin",
"Config": [
{
"Subnet": "<IP_ADDRESS>/24"
},
{
"Subnet": "2001::/100"
}
]
},
"Containers": {
"455a80938abcaf4a534a08783ff44e10b2a1a5afb3c2690d474b19ed670b3d7a": {
"EndpointID": "d8fcf4dc8d79841073655698ebc692ffe20251abc1c2577cdc6924fb53d7ef62",
"MacAddress": "02:02:14:01:01:02",
"IPv4Address": "<IP_ADDRESS>/24",
"IPv6Address": "2001::2/100"
},
"cb30c9001c118f4ae39b6fda14f6c26c9ccbdcd1110eabded179ce47a7aa63c9": {
"EndpointID": "06e11d75f0bada4b1e1dad848363b3fea323eb3ac175cde0569b3e7f25dff3a0",
"MacAddress": "02:02:14:01:01:03",
"IPv4Address": "<IP_ADDRESS>/24",
"IPv6Address": "2001::3/100"
}
},
"Options": {
"encap": "vxlan",
"pkt-tag": "1",
"tenant": "default"
}
}
]
Container IPv6 Interface
Logging into the two containers shows the following:
Container 'web' in Node1:
[vagrant@netplugin-node1 ~]$ docker exec -it web /bin/sh
/ # ip addr show dev eth0
13: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
link/ether 02:02:14:01:01:03 brd ff:ff:ff:ff:ff:ff
inet <IP_ADDRESS>/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 2001::3/100 scope global
valid_lft forever preferred_lft forever
inet6 fe80::2:14ff:fe01:103/64 scope link
valid_lft forever preferred_lft forever
/ #
Container 'db' in Node2:
[vagrant@netplugin-node2 ~]$ docker exec -it db /bin/sh
/ # ip addr show dev eth0
9: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
link/ether 02:02:14:01:01:04 brd ff:ff:ff:ff:ff:ff
inet <IP_ADDRESS>/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 2001::4/100 scope global
valid_lft forever preferred_lft forever
inet6 fe80::2:14ff:fe01:104/64 scope link
valid_lft forever preferred_lft forever
/ #
Ping6 Between Containers
The following output shows the ability to ping a container's IPv6 address:
[vagrant@netplugin-node1 ~]$ docker exec -it web /bin/sh
/ # ping6 2001::4 -I 2001::3 -c 3
PING 2001::4 (2001::4) from 2001::3: 56 data bytes
64 bytes from 2001::4: seq=0 ttl=64 time=1.689 ms
64 bytes from 2001::4: seq=1 ttl=64 time=2.437 ms
64 bytes from 2001::4: seq=2 ttl=64 time=1.526 ms
--- 2001::4 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.526/1.884/2.437 ms
/ #


Online VAT calculator - exclude or add VAT from / to amount.
Woops! Looks like your internet connection is not working.


Rasanel's Weather Forecasting
"Information you can TRUST!"
☰
Home
Preston
Soda Springs
Fish Haven
Storm Center
Gallery
Preston Idaho
Hero w/ weather Summary
5 Day Forecast
Article
Contact Information


{% blocktrans trimmed %}
You currently allow more than one item per order.
Please note that cancelations will not work correctly with this configuration.
You can change this in your cart settings.
{% endblocktrans %}
{% endif %}
{% if warn_no_swap_groups %}
{% blocktrans trimmed %}
You have set swaps to be allowed, but haven't configured any swap groups.
{% endblocktrans %}
{% endif %}
{% if warn_no_cancel_groups %}
{% blocktrans trimmed %}
You have set cancelations to be allowed, but haven't configured any cancel groups.
{% endblocktrans %}


WITNESS LIST Appropriations-S/C on General Government Committee (Subcommittee on General Government) March 12, 2001-7:00A Budget On: Heald, Charles (Texas Department of Transportation) Hubert, Martin (Texas Department of Agriculture) Pharr, Machelle (Texas Natural Resource Conservation Commission) Purcell, Carolyn (Department of Information Resources) Rath, Diane (Texas Workforce Commission) Saitas, Jeff (Texas Natural Resource Conservation Commission)


Chapter 2 Geographic data in R
Prerequisites
This is the first practical chapter of the book, and therefore it comes with some software requirements. We assume that you have an up-to-date version of R installed and that you are comfortable using software with a command-line interface such as the integrated development environment (IDE) RStudio.
If you are new to R, we recommend reading Chapter 2 of the online book Efficient R Programming by (???) and
learning the basics of the language with reference to resources such as (???) or DataCamp before proceeding.
Organize your work (e.g., with RStudio projects) and give scripts sensible names such as chapter-02.R
to document the code you write as you learn.
The packages used in this chapter can be installed with the following commands:3
install.packages("sf")
install.packages("raster")
install.packages("spData")
devtools::install_github("Nowosad/spDataLarge")
All the packages needed to reproduce the contents of the book can be installed with the following command: devtools::install_github("geocompr/geocompkg")
.
The necessary packages can be ‘loaded’ (technically they are attached) with the library()
function as follows:
library(sf) # classes and functions for vector data
library(raster) # classes and functions for raster data
The other packages that were installed contain data that will be used in the book:
2.1 Introduction
This chapter will provide brief explanations of the fundamental geographic data models: vector and raster. We will introduce the theory behind each data model and the disciplines in which they predominate, before demonstrating their implementation in R.
The vector data model represents the world using points, lines and polygons. These have discrete, well-defined borders, meaning that vector datasets usually have a high level of precision (but not necessarily accuracy as we will see in Section 2.5). The raster data model divides the surface up into cells of constant size. Raster datasets are the basis of background images used in web-mapping and have been a vital source of geographic data since the origins of aerial photography and satellite-based remote sensing devices. Rasters aggregate spatially specific features to a given resolution, meaning that they are consistent over space and scalable (many worldwide raster datasets are available).
Which to use? The answer likely depends on your domain of application:
- Vector data tends to dominate the social sciences because human settlements tend to have discrete borders.
- Raster often dominates in environmental sciences because of the reliance on remote sensing data.
There is much overlap in some fields and raster and vector datasets can be used together: ecologists and demographers, for example, commonly use both vector and raster data. Furthermore, it is possible to convert between the two forms (see Section ??). Whether your work involves more use of vector or raster datasets, it is worth understanding the underlying data model before using them, as discussed in subsequent chapters. This book uses sf and raster packages to work with vector data and raster datasets, respectively.
2.2 Vector data
vector
class (note the monospace
font) in R.
The former is a data model, the latter is an R class just like data.frame
and matrix
.
Still, there is a link between the two: the spatial coordinates which are at the heart of the geographic vector data model can be represented in R using vector
objects.
The geographic vector data model is based on points located within a coordinate reference system (CRS). Points can represent self-standing features (e.g., the location of a bus stop) or they can be linked together to form more complex geometries such as lines and polygons. Most point geometries contain only two dimensions (3-dimensional CRSs contain an additional \(z\) value, typically representing height above sea level).
In this system London, for example, can be represented by the coordinates c(-0.1, 51.5)
.
This means that its location is -0.1 degrees east and 51.5 degrees north of the origin.
The origin in this case is at 0 degrees longitude (the Prime Meridian) and 0 degree latitude (the Equator) in a geographic (‘lon/lat’) CRS (Figure 2.1, left panel).
The same point could also be approximated in a projected CRS with ‘Easting/Northing’ values of c(530000, 180000)
in the British National Grid, meaning that London is located 530 km East and 180 km North of the \(origin\) of the CRS.
This can be verified visually: slightly more than 5 ‘boxes’ — square areas bounded by the gray grid lines 100 km in width — separate the point representing London from the origin (Figure 2.1, right panel).
The location of National Grid’s origin, in the sea beyond South West Peninsular, ensures that most locations in the UK have positive Easting and Northing values.4 There is more to CRSs, as described in Sections 2.4 and ?? but, for the purposes of this section, it is sufficient to know that coordinates consist of two numbers representing distance from an origin, usually in \(x\) then \(y\) dimensions.
sf is a package providing a class system for geographic vector data. Not only does sf supersede sp, it also provides a consistent command-line interface to GEOS and GDAL, superseding rgeos and rgdal (described in Section ??). This section introduces sf classes in preparation for subsequent chapters (Chapters ?? and ?? cover the GEOS and GDAL interface, respectively).
2.2.1 An introduction to simple features
Simple features is an open standard developed and endorsed by the Open Geospatial Consortium (OGC), a not-for-profit organization whose activities we will revisit in a later chapter (in Section ??). Simple Features is a hierarchical data model that represents a wide range of geometry types. Of 17 geometry types supported by the specification, only 7 are used in the vast majority of geographic research (see Figure 2.2); these core geometry types are fully supported by the R package sf (???).5
sf can represent all common vector geometry types (raster data classes are not supported by sf): points, lines, polygons and their respective ‘multi’ versions (which group together features of the same type into a single feature). sf also supports geometry collections, which can contain multiple geometry types in a single object. sf largely supersedes the sp ecosystem, which comprises sp (???), rgdal for data read/write (???) and rgeos for spatial operations (???). The package is well documented, as can be seen on its website and in 6 vignettes, which can be loaded as follows:
vignette(package = "sf") # see which vignettes are available
vignette("sf1") # an introduction to the package
As the first vignette explains, simple feature objects in R are stored in a data frame, with geographic data occupying a special column, usually named ‘geom’ or ‘geometry’.
We will use the world
dataset provided by the spData, loaded at the beginning of this chapter (see nowosad.github.io/spData for a list of datasets loaded by the package).
world
is a spatial object containing spatial and attribute columns, the names of which are returned by the function names()
(the last column contains the geographic information):
## [1] "iso_a2" "name_long" "continent" "region_un" "subregion"
## [6] "type" "area_km2" "pop" "lifeExp" "gdpPercap"
## [11] "geom"
The contents of this geom
column give sf
objects their spatial powers: world$geom
is a ‘list column’ that contains all the coordinates of the country polygons.
The sf package provides a plot()
method for visualizing geographic data:
the following command creates Figure 2.3.
Note that instead of creating a single map, as most GIS programs would, the plot()
command has created multiple maps, one for each variable in the world
datasets.
This behavior can be useful for exploring the spatial distribution of different variables and is discussed further in Section 2.2.3 below.
Being able to treat spatial objects as regular data frames with spatial powers has many advantages, especially if you are already used to working with data frames.
The commonly used summary()
function, for example, provides a useful overview of the variables within the world
object.
## lifeExp geom
## Min. :50.62 MULTIPOLYGON :177
## 1st Qu.:64.96 epsg:4326 : 0
## Median :72.87 +proj=long...: 0
## Mean :70.85
## 3rd Qu.:76.78
## Max. :83.59
## NA's :10
Although we have only selected one variable for the summary
command, it also outputs a report on the geometry.
This demonstrates the ‘sticky’ behavior of the geometry columns of sf objects, meaning the geometry is kept unless the user deliberately removes them, as we’ll see in Section ??.
The result provides a quick summary of both the non-spatial and spatial data contained in world
: the mean average life expectancy is 71 years (ranging from less than 51 to more than 83 years with a median of 73 years) across all countries.
MULTIPOLYGON
in the summary output above refers to the geometry type of features (countries) in the world
object.
This representation is necessary for countries with islands such as Indonesia and Greece.
Other geometry types are described in Section 2.2.5.
It is worth taking a deeper look at the basic behavior and contents of this simple feature object, which can usefully be thought of as a ‘spatial data frame’.
sf
objects are easy to subset.
The code below shows its first two rows and three columns.
The output shows two major differences compared with a regular data.frame
: the inclusion of additional geographic data (geometry type
, dimension
, bbox
and CRS information - epsg (SRID)
, proj4string
), and the presence of a geometry
column, here named geom
:
## Simple feature collection with 2 features and 3 fields
## geometry type: MULTIPOLYGON
## dimension: XY
## bbox: xmin: -180 ymin: -18.28799 xmax: 180 ymax: -0.95
## epsg (SRID): 4326
## proj4string: +proj=longlat +datum=WGS84 +no_defs
## # A tibble: 2 x 4
## iso_a2 name_long continent geom
## <chr> <chr> <chr> <MULTIPOLYGON [°]>
## 1 FJ Fiji Oceania (((180 -16.06713, 180 -16.55522, 179.3641 -16…
## 2 TZ Tanzania Africa (((33.90371 -0.95, 34.07262 -1.05982, 37.6986…
All this may seem rather complex, especially for a class system that is supposed to be simple. However, there are good reasons for organizing things this way and using sf.
Before describing each geometry type that the sf package supports, it is worth taking a step back to understand the building blocks of sf
objects.
Section 2.2.8 shows how simple features objects are data frames, with special geometry columns.
These spatial columns are often called geom
or geometry
: world$geom
refers to the spatial element of the world
object described above.
These geometry columns are ‘list columns’ of class sfc
(see Section 2.2.7).
In turn, sfc
objects are composed of one or more objects of class sfg
: simple feature geometries that we describe in Section 2.2.6.
To understand how the spatial components of simple features work, it is vital to understand simple feature geometries.
For this reason we cover each currently supported simple features geometry type in Section 2.2.5 before moving on to describe how these can be represented in R using sfg
objects, which form the basis of sfc
and eventually full sf
objects.
=
to create a new object called world_mini
in the command world_mini = world[1:2, 1:3]
.
This is called assignment.
An equivalent command to achieve the same result is world_mini <- world[1:2, 1:3]
.
Although ‘arrow assigment’ is more commonly used, we use ‘equals assignment’ because it’s slightly faster to type and easier to teach due to compatibility with commonly used languages such as Python and JavaScript.
Which to use is largely a matter of preference as long as you’re consistent (packages such as styler can be used to change style).
2.2.2 Why simple features?
Simple features is a widely supported data model that underlies data structures in many GIS applications including QGIS and PostGIS. A major advantage of this is that using the data model ensures your work is cross-transferable to other set-ups, for example importing from and exporting to spatial databases.
A more specific question from an R perspective is “why use the sf package when sp is already tried and tested”? There are many reasons (linked to the advantages of the simple features model) including:
- Fast reading and writing of data.
- Enhanced plotting performance.
- sf objects can be treated as data frames in most operations.
- sf functions can be combined using
%>%
operator and works well with the tidyverse collection of R packages. - sf function names are relatively consistent and intuitive (all begin with
st_
).
Due to such advantages, some spatial packages (including tmap, mapview and tidycensus) have added support for sf.
However, it will take many years for most packages to transition and some will never switch.
Fortunately, these can still be used in a workflow based on sf
objects, by converting them to the Spatial
class used in sp:
Spatial
objects can be converted back to sf
in the same way or with st_as_sf()
:
2.2.3 Basic map making
Basic maps are created in sf with plot()
.
By default this creates a multi-panel plot (like sp’s spplot()
), one sub-plot for each variable of the object, as illustrated in the left-hand panel in Figure 2.4.
A legend or ‘key’ with a continuous color is produced if the object to be plotted has a single variable (see the right-hand panel).
Colors can also be set with col =
, although this will not create a continuous palette or a legend.
Plots are added as layers to existing images by setting add = TRUE
.6
To demonstrate this, and to provide a taster of content covered in Chapters ?? and ?? on attribute and spatial data operations, the subsequent code chunk combines countries in Asia:
We can now plot the Asian continent over a map of the world.
Note that the first plot must only have one facet for add = TRUE
to work.
If the first plot has a key, reset = FALSE
must be used (result not shown):
Adding layers in this way can be used to verify the geographic correspondence between layers:
the plot()
function is fast to execute and requires few lines of code, but does not create interactive maps with a wide range of options.
For more advanced map making we recommend using dedicated visualization packages such as tmap (see Chapter ??).
2.2.4 Base plot arguments
There are various ways to modify maps with sf’s plot()
method.
Because sf extends base R plotting methods plot()
’s arguments such as main =
(which specifies the title of the map) work with sf
objects (see ?graphics::plot
and ?par
).7
Figure 2.5 illustrates this flexibility by overlaying circles, whose diameters (set with cex =
) represent country populations, on a map of the world.
A basic version of the map can be created with the following commands (see exercises at the end of this chapter and the script 02-contplot.R
to create Figure 2.5):
plot(world["continent"], reset = FALSE)
cex = sqrt(world$pop) / 10000
world_cents = st_centroid(world, of_largest = TRUE)
plot(st_geometry(world_cents), add = TRUE, cex = cex)
The code above uses the function st_centroid()
to convert one geometry type (polygons) to another (points) (see Chapter ??), the aesthetics of which are varied with the cex
argument.
sf’s plot method also has arguments specific to geographic data. expandBB
, for example, can be used to plot an sf
object in context:
it takes a numeric vector of length four that expands the bounding box of the plot relative to zero in the following order: bottom, left, top, right.
This is used to plot India in the context of its giant Asian neighbors, with an emphasis on China to the east, in the following code chunk, which generates Figure 2.6 (see exercises below on adding text to plots):
india = world[world$name_long == "India", ]
plot(st_geometry(india), expandBB = c(0, 0.2, 0.1, 1), col = "gray", lwd = 3)
plot(world_asia[0], add = TRUE)
Note the use of [0]
to keep only the geometry column and lwd
to emphasize India.
See Section ?? for other visualization techniques for representing a range of geometry types, the subject of the next section.
2.2.5 Geometry types
Geometries are the basic building blocks of simple features.
Simple features in R can take on one of the 17 geometry types supported by the sf package.
In this chapter we will focus on the seven most commonly used types: POINT
, LINESTRING
, POLYGON
, MULTIPOINT
, MULTILINESTRING
, MULTIPOLYGON
and GEOMETRYCOLLECTION
.
Find the whole list of possible feature types in the PostGIS manual.
Generally, well-known binary (WKB) or well-known text (WKT) are the standard encoding for simple feature geometries. WKB representations are usually hexadecimal strings easily readable for computers. This is why GIS and spatial databases use WKB to transfer and store geometry objects. WKT, on the other hand, is a human-readable text markup description of simple features. Both formats are exchangeable, and if we present one, we will naturally choose the WKT representation.
The basis for each geometry type is the point.
A point is simply a coordinate in 2D, 3D or 4D space (see vignette("sf1")
for more information) such as (see left panel in Figure 2.7):
POINT (5 2)
A linestring is a sequence of points with a straight line connecting the points, for example (see middle panel in Figure 2.7):
LINESTRING (1 5, 4 4, 4 1, 2 2, 3 2)
A polygon is a sequence of points that form a closed, non-intersecting ring. Closed means that the first and the last point of a polygon have the same coordinates (see right panel in Figure 2.7).8
- Polygon without a hole:
POLYGON ((1 5, 2 2, 4 1, 4 4, 1 5))
So far we have created geometries with only one geometric entity per feature. However, sf also allows multiple geometries to exist within a single feature (hence the term ‘geometry collection’) using “multi” version of each geometry type:
- Multipoint:
MULTIPOINT (5 2, 1 3, 3 4, 3 2)
- Multilinestring:
MULTILINESTRING ((1 5, 4 4, 4 1, 2 2, 3 2), (1 2, 2 4))
- Multipolygon:
MULTIPOLYGON (((1 5, 2 2, 4 1, 4 4, 1 5), (0 2, 1 2, 1 3, 0 3, 0 2)))
Finally, a geometry collection can contain any combination of geometries including (multi)points and linestrings (see Figure 2.9):
- Geometry collection:
GEOMETRYCOLLECTION (MULTIPOINT (5 2, 1 3, 3 4, 3 2), LINESTRING (1 5, 4 4, 4 1, 2 2, 3 2))
2.2.6 Simple feature geometries (sfg)
The sfg
class represents the different simple feature geometry types in R: point, linestring, polygon (and their ‘multi’ equivalents, such as multipoints) or geometry collection.
Usually you are spared the tedious task of creating geometries on your own since you can simply import an already existing spatial file.
However, there are a set of functions to create simple feature geometry objects (sfg
) from scratch if needed.
The names of these functions are simple and consistent, as they all start with the st_
prefix and end with the name of the geometry type in lowercase letters:
- A point:
st_point()
- A linestring:
st_linestring()
- A polygon:
st_polygon()
- A multipoint:
st_multipoint()
- A multilinestring:
st_multilinestring()
- A multipolygon:
st_multipolygon()
- A geometry collection:
st_geometrycollection()
sfg
objects can be created from three base R data types:
- A numeric vector: a single point
- A matrix: a set of points, where each row represents a point, a multipoint or linestring
- A list: a collection of objects such as matrices, multilinestrings or geometry collections
The function st_point()
creates single points from numeric vectors:
## POINT (5 2)
## POINT Z (5 2 3)
## POINT M (5 2 1)
## POINT ZM (5 2 3 1)
The results show that XY (2D coordinates), XYZ (3D coordinates) and XYZM (3D with an additional variable, typically measurement accuracy) point types are created from vectors of length 2, 3, and 4, respectively.
The XYM type must be specified using the dim
argument (which is short for dimension).
By contrast, use matrices in the case of multipoint (st_multipoint()
) and linestring (st_linestring()
) objects:
# the rbind function simplifies the creation of matrices
## MULTIPOINT
multipoint_matrix = rbind(c(5, 2), c(1, 3), c(3, 4), c(3, 2))
st_multipoint(multipoint_matrix)
## MULTIPOINT (5 2, 1 3, 3 4, 3 2)
## LINESTRING
linestring_matrix = rbind(c(1, 5), c(4, 4), c(4, 1), c(2, 2), c(3, 2))
st_linestring(linestring_matrix)
## LINESTRING (1 5, 4 4, 4 1, 2 2, 3 2)
Finally, use lists for the creation of multilinestrings, (multi-)polygons and geometry collections:
## POLYGON
polygon_list = list(rbind(c(1, 5), c(2, 2), c(4, 1), c(4, 4), c(1, 5)))
st_polygon(polygon_list)
## POLYGON ((1 5, 2 2, 4 1, 4 4, 1 5))
## POLYGON with a hole
polygon_border = rbind(c(1, 5), c(2, 2), c(4, 1), c(4, 4), c(1, 5))
polygon_hole = rbind(c(2, 4), c(3, 4), c(3, 3), c(2, 3), c(2, 4))
polygon_with_hole_list = list(polygon_border, polygon_hole)
st_polygon(polygon_with_hole_list)
## POLYGON ((1 5, 2 2, 4 1, 4 4, 1 5), (2 4, 3 4, 3 3, 2 3, 2 4))
## MULTILINESTRING
multilinestring_list = list(rbind(c(1, 5), c(4, 4), c(4, 1), c(2, 2), c(3, 2)),
rbind(c(1, 2), c(2, 4)))
st_multilinestring((multilinestring_list))
## MULTILINESTRING ((1 5, 4 4, 4 1, 2 2, 3 2), (1 2, 2 4))
## MULTIPOLYGON
multipolygon_list = list(list(rbind(c(1, 5), c(2, 2), c(4, 1), c(4, 4), c(1, 5))),
list(rbind(c(0, 2), c(1, 2), c(1, 3), c(0, 3), c(0, 2))))
st_multipolygon(multipolygon_list)
## MULTIPOLYGON (((1 5, 2 2, 4 1, 4 4, 1 5)), ((0 2, 1 2, 1 3, 0 3, 0 2)))
2.2.7 Simple feature columns (sfc)
One sfg
object contains only a single simple feature geometry.
A simple feature geometry column (sfc
) is a list of sfg
objects, which is additionally able to contain information about the coordinate reference system in use.
For instance, to combine two simple features into one object with two features, we can use the st_sfc()
function.
This is important since sfc
represents the geometry column in sf data frames:
# sfc POINT
point1 = st_point(c(5, 2))
point2 = st_point(c(1, 3))
points_sfc = st_sfc(point1, point2)
points_sfc
## Geometry set for 2 features
## geometry type: POINT
## dimension: XY
## bbox: xmin: 1 ymin: 2 xmax: 5 ymax: 3
## epsg (SRID): NA
## proj4string: NA
## POINT (5 2)
## POINT (1 3)
In most cases, an sfc
object contains objects of the same geometry type.
Therefore, when we convert sfg
objects of type polygon into a simple feature geometry column, we would also end up with an sfc
object of type polygon, which can be verified with st_geometry_type()
.
Equally, a geometry column of multilinestrings would result in an sfc
object of type multilinestring:
# sfc POLYGON
polygon_list1 = list(rbind(c(1, 5), c(2, 2), c(4, 1), c(4, 4), c(1, 5)))
polygon1 = st_polygon(polygon_list1)
polygon_list2 = list(rbind(c(0, 2), c(1, 2), c(1, 3), c(0, 3), c(0, 2)))
polygon2 = st_polygon(polygon_list2)
polygon_sfc = st_sfc(polygon1, polygon2)
st_geometry_type(polygon_sfc)
## [1] POLYGON POLYGON
## 18 Levels: GEOMETRY POINT LINESTRING POLYGON ... TRIANGLE
# sfc MULTILINESTRING
multilinestring_list1 = list(rbind(c(1, 5), c(4, 4), c(4, 1), c(2, 2), c(3, 2)),
rbind(c(1, 2), c(2, 4)))
multilinestring1 = st_multilinestring((multilinestring_list1))
multilinestring_list2 = list(rbind(c(2, 9), c(7, 9), c(5, 6), c(4, 7), c(2, 7)),
rbind(c(1, 7), c(3, 8)))
multilinestring2 = st_multilinestring((multilinestring_list2))
multilinestring_sfc = st_sfc(multilinestring1, multilinestring2)
st_geometry_type(multilinestring_sfc)
## [1] MULTILINESTRING MULTILINESTRING
## 18 Levels: GEOMETRY POINT LINESTRING POLYGON ... TRIANGLE
It is also possible to create an sfc
object from sfg
objects with different geometry types:
# sfc GEOMETRY
point_multilinestring_sfc = st_sfc(point1, multilinestring1)
st_geometry_type(point_multilinestring_sfc)
## [1] POINT MULTILINESTRING
## 18 Levels: GEOMETRY POINT LINESTRING POLYGON ... TRIANGLE
As mentioned before, sfc
objects can additionally store information on the coordinate reference systems (CRS).
To specify a certain CRS, we can use the epsg (SRID)
or proj4string
attributes of an sfc
object.
The default value of epsg (SRID)
and proj4string
is NA
(Not Available), as can be verified with st_crs()
:
## Coordinate Reference System: NA
All geometries in an sfc
object must have the same CRS.
We can add coordinate reference system as a crs
argument of st_sfc()
.
This argument accepts an integer with the epsg
code such as 4326
, which automatically adds the ‘proj4string’ (see Section 2.4):
## Coordinate Reference System:
## EPSG: 4326
## proj4string: "+proj=longlat +datum=WGS84 +no_defs"
It also accepts a raw proj4string (result not shown):
st_crs()
will return a proj4string
but not an epsg
code.
This is because there is no general method to convert from proj4string
to epsg
(see Chapter ??).
2.2.8 The sf class
Sections 2.2.5 to 2.2.7 deal with purely geometric objects, ‘sf geometry’ and ‘sf column’ objects, respectively. These are geographic building blocks of geographic vector data represented as simple features. The final building block is non-geographic attributes, representing the name of the feature or other attributes such as measured values, groups, and other things.
To illustrate attributes, we will represent a temperature of 25°C in London on June 21st, 2017.
This example contains a geometry (the coordinates), and three attributes with three different classes (place name, temperature and date).9
Objects of class sf
represent such data by combining the attributes (data.frame
) with the simple feature geometry column (sfc
).
They are created with st_sf()
as illustrated below, which creates the London example described above:
lnd_point = st_point(c(0.1, 51.5)) # sfg object
lnd_geom = st_sfc(lnd_point, crs = 4326) # sfc object
lnd_attrib = data.frame( # data.frame object
name = "London",
temperature = 25,
date = as.Date("2017-06-21")
)
lnd_sf = st_sf(lnd_attrib, geometry = lnd_geom) # sf object
What just happened? First, the coordinates were used to create the simple feature geometry (sfg
).
Second, the geometry was converted into a simple feature geometry column (sfc
), with a CRS.
Third, attributes were stored in a data.frame
, which was combined with the sfc
object with st_sf()
.
This results in an sf
object, as demonstrated below (some output is ommited):
lnd_sf
#> Simple feature collection with 1 features and 3 fields
#> ...
#> name temperature date geometry
#> 1 London 25 2017-06-21 POINT (0.1 51.5)
## [1] "sf" "data.frame"
The result shows that sf
objects actually have two classes, sf
and data.frame
.
Simple features are simply data frames (square tables), but with spatial attributes stored in a list column, usually called geometry
, as described in Section 2.2.1.
This duality is central to the concept of simple features:
most of the time a sf
can be treated as and behaves like a data.frame
.
Simple features are, in essence, data frames with a spatial extension.
2.3 Raster data
The geographic raster data model usually consists of a raster header and a matrix (with rows and columns) representing equally spaced cells (often also called pixels; Figure 2.10:A).10 The raster header defines the coordinate reference system, the extent and the origin. The origin (or starting point) is frequently the coordinate of the lower-left corner of the matrix (the raster package, however, uses the upper left corner, by default (Figure 2.10:B)). The header defines the extent via the number of columns, the number of rows and the cell size resolution. Hence, starting from the origin, we can easily access and modify each single cell by either using the ID of a cell (Figure 2.10:B) or by explicitly specifying the rows and columns. This matrix representation avoids storing explicitly the coordinates for the four corner points (in fact it only stores one coordinate, namely the origin) of each cell corner as would be the case for rectangular vector polygons. This and map algebra makes raster processing much more efficient and faster than vector data processing. However, in contrast to vector data, the cell of one raster layer can only hold a single value. The value might be numeric or categorical (Figure 2.10:C).
Raster maps usually represent continuous phenomena such as elevation, temperature, population density or spectral data (Figure 2.11). Of course, we can represent discrete features such as soil or land-cover classes also with the help of a raster data model (Figure 2.11). Consequently, the discrete borders of these features become blurred, and depending on the spatial task a vector representation might be more suitable.
2.3.1 An introduction to raster
The raster package supports raster objects in R.
It provides an extensive set of functions to create, read, export, manipulate and process raster datasets.
Aside from general raster data manipulation, raster provides many low-level functions that can form the basis to develop more advanced raster functionality.
raster also lets you work on large raster datasets that are too large to fit into the main memory.
In this case, raster provides the possibility to divide the raster into smaller chunks (rows or blocks), and processes these iteratively instead of loading the whole raster file into RAM (for more information, please refer to vignette("functions", package = "raster")
.
For the illustration of raster concepts, we will use datasets from the spDataLarge (note these packages were loaded at the beginning of the chapter).
It consists of a few raster objects and one vector object covering an area of the Zion National Park (Utah, USA).
For example, srtm.tif
is a digital elevation model of this area (for more details, see its documentation ?srtm
).
First, let’s create a RasterLayer
object named new_raster
:
raster_filepath = system.file("raster/srtm.tif", package = "spDataLarge")
new_raster = raster(raster_filepath)
Typing the name of the raster into the console, will print out the raster header (extent, dimensions, resolution, CRS) and some additional information (class, data source name, summary of the raster values):
new_raster
#> class : RasterLayer
#> dimensions : 457, 465, 212505 (nrow, ncol, ncell)
#> resolution : 0.000833, 0.000833 (x, y)
#> extent : -113, -113, 37.1, 37.5 (xmin, xmax, ymin, ymax)
#> coord. ref. : +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0
#> data source : /home/robin/R/x86_64-pc-linux../3.5/spDataLarge/raster/srtm.tif
#> names : srtm
#> values : 1024, 2892 (min, max)
Dedicated functions report each component: dim(new_raster)
returns the number of rows, columns and layers; the ncell()
function the number of cells (pixels); res()
the raster’s spatial resolution; extent()
its spatial extent; and crs()
its coordinate reference system (raster reprojection is covered in Section ??).
inMemory()
reports whether the raster data is stored in memory (the default) or on disk.
help("raster-package")
returns a full list of all available raster functions.
2.3.2 Basic map making
Similar to the sf package, raster also provides plot()
methods for its own classes.
There are several other approaches for plotting raster data in R that are outside the scope of this section, including:
- Functions such as
spplot()
andlevelplot()
(from the sp and rasterVis packages, respectively) to create facets, a common technique for visualizing change over time. - Packages such as tmap, mapview and leaflet to create interactive maps of raster and vector objects (see Chapter ??).
2.3.3 Raster classes
The RasterLayer
class represents the simplest form of a raster object, and consists of only one layer.
The easiest way to create a raster object in R is to read-in a raster file from disk or from a server.
raster_filepath = system.file("raster/srtm.tif", package = "spDataLarge")
new_raster = raster(raster_filepath)
The raster package supports numerous drivers with the help of rgdal.
To find out which drivers are available on your system, run raster::writeFormats()
and rgdal::gdalDrivers()
.
Rasters can also be created from scratch using the raster()
function.
This is illustrated in the subsequent code chunk, which results in a new RasterLayer
object.
The resulting raster consists of 36 cells (6 columns and 6 rows specified by nrows
and ncols
) centered around the Prime Meridian and the Equator (see xmn
, xmx
, ymn
and ymx
parameters).
The CRS is the default of raster objects: WGS84.
This means the unit of the resolution is in degrees which we set to 0.5 (res
).
Values (vals
) are assigned to each cell: 1 to cell 1, 2 to cell 2, and so on.
Remember: raster()
fills cells row-wise (unlike matrix()
) starting at the upper left corner, meaning the top row contains the values 1 to 6, the second 7 to 12, etc.
new_raster2 = raster(nrows = 6, ncols = 6, res = 0.5,
xmn = -1.5, xmx = 1.5, ymn = -1.5, ymx = 1.5,
vals = 1:36)
For other ways of creating raster objects, see ?raster
.
Aside from RasterLayer
, there are two additional classes: RasterBrick
and RasterStack
.
Both can handle multiple layers, but differ regarding the number of supported file formats, type of internal representation and processing speed.
A RasterBrick
consists of multiple layers, which typically correspond to a single multispectral satellite file or a single multilayer object in memory.
The brick()
function creates a RasterBrick
object.
Usually, you provide it with a filename to a multilayer raster file but might also use another raster object and other spatial objects (see ?brick
for all supported formats).
multi_raster_file = system.file("raster/landsat.tif", package = "spDataLarge")
r_brick = brick(multi_raster_file)
r_brick
#> class : RasterBrick
#> resolution : 30, 30 (x, y)
#> ...
#> names : landsat.1, landsat.2, landsat.3, landsat.4
#> min values : 7550, 6404, 5678, 5252
#> max values : 19071, 22051, 25780, 31961
nlayers()
retrieves the number of layers stored in a Raster*
object:
## [1] 4
A RasterStack
is similar to a RasterBrick
in the sense that it consists also of multiple layers.
However, in contrast to RasterBrick
, RasterStack
allows you to connect several raster objects stored in different files or multiple objects in memory.
More specifically, a RasterStack
is a list of RasterLayer
objects with the same extent and resolution.
Hence, one way to create it is with the help of spatial objects already existing in R’s global environment.
And again, one can simply specify a path to a file stored on disk.
raster_on_disk = raster(r_brick, layer = 1)
raster_in_memory = raster(xmn = 301905, xmx = 335745,
ymn = 4111245, ymx = 4154085,
res = 30)
values(raster_in_memory) = sample(seq_len(ncell(raster_in_memory)))
crs(raster_in_memory) = crs(raster_on_disk)
r_stack = stack(raster_in_memory, raster_on_disk)
r_stack
#> class : RasterStack
#> dimensions : 1428, 1128, 1610784, 2
#> resolution : 30, 30
#> ...
#> names : layer, landsat.1
#> min values : 1, 7550
#> max values : 1610784, 19071
Another difference is that the processing time for RasterBrick
objects is usually shorter than for RasterStack
objects.
Decision on which Raster*
class should be used depends mostly on the character of input data.
Processing of a single mulitilayer file or object is the most effective with RasterBrick
, while RasterStack
allows calculations based on many files, many Raster*
objects, or both.
RasterBrick
and RasterStack
objects will typically return a RasterBrick
.
2.4 Coordinate Reference Systems
Vector and raster spatial data types share concepts intrinsic to spatial data. Perhaps the most fundamental of these is the Coordinate Reference System (CRS), which defines how the spatial elements of the data relate to the surface of the Earth (or other bodies). CRSs are either geographic or projected, as introduced at the beginning of this chapter (see Figure 2.1). This section will explain each type, laying the foundations for Section ?? on CRS transformations.
2.4.1 Geographic coordinate systems
Geographic coordinate systems identify any location on the Earth’s surface using two values — longitude and latitude. Longitude is location in the East-West direction in angular distance from the Prime Meridian plane. Latitude is angular distance North or South of the equatorial plane. Distances in geographic CRSs are therefore not measured in meters. This has important consequences, as demonstrated in Section ??.
The surface of the Earth in geographic coordinate systems is represented by a spherical or ellipsoidal surface. Spherical models assume that the Earth is a perfect sphere of a given radius. Spherical models have the advantage of simplicity but are rarely used because they are inaccurate: the Earth is not a sphere! Ellipsoidal models are defined by two parameters: the equatorial radius and the polar radius. These are suitable because the Earth is compressed: the equatorial radius is around 11.5 km longer than the polar radius (???).11
Ellipsoids are part of a wider component of CRSs: the datum.
This contains information on what ellipsoid to use (with the ellps
parameter in the PROJ CRS library) and the precise relationship between the Cartesian coordinates and location on the Earth’s surface.
These additional details are stored in the towgs84
argument of proj4string notation (see proj.org/usage/projections.html for details).
These allow local variations in Earth’s surface, for example due to large mountain ranges, to be accounted for in a local CRS.
There are two types of datum — local and geocentric.
In a local datum such as NAD83
the ellipsoidal surface is shifted to align with the surface at a particular location.
In a geocentric datum such as WGS84
the center is the Earth’s center of gravity and the accuracy of projections is not optimized for a specific location.
Available datum definitions can be seen by executing st_proj_info(type = "datum")
.
2.4.2 Projected coordinate reference systems
Projected CRSs are based on Cartesian coordinates on an implicitly flat surface. They have an origin, x and y axes, and a linear unit of measurement such as meters. All projected CRSs are based on a geographic CRS, described in the previous section, and rely on map projections to convert the three-dimensional surface of the Earth into Easting and Northing (x and y) values in a projected CRS.
This transition cannot be done without adding some distortion. Therefore, some properties of the Earth’s surface are distorted in this process, such as area, direction, distance, and shape. A projected coordinate system can preserve only one or two of those properties. Projections are often named based on a property they preserve: equal-area preserves area, azimuthal preserve direction, equidistant preserve distance, and conformal preserve local shape.
There are three main groups of projection types - conic, cylindrical, and planar.
In a conic projection, the Earth’s surface is projected onto a cone along a single line of tangency or two lines of tangency.
Distortions are minimized along the tangency lines and rise with the distance from those lines in this projection.
Therefore, it is the best suited for maps of mid-latitude areas.
A cylindrical projection maps the surface onto a cylinder.
This projection could also be created by touching the Earth’s surface along a single line of tangency or two lines of tangency.
Cylindrical projections are used most often when mapping the entire world.
A planar projection projects data onto a flat surface touching the globe at a point or along a line of tangency.
It is typically used in mapping polar regions.
st_proj_info(type = "proj")
gives a list of the available projections supported by the PROJ library.
2.4.3 CRSs in R
Two main ways to describe CRS in R are an epsg
code or a proj4string
definition.
Both of these approaches have advantages and disadvantages.
An epsg
code is usually shorter, and therefore easier to remember.
The code also refers to only one, well-defined coordinate reference system.
On the other hand, a proj4string
definition allows you more flexibility when it comes to specifying different parameters such as the projection type, the datum and the ellipsoid.12
This way you can specify many different projections, and modify existing ones.
This also makes the proj4string
approach more complicated.
epsg
points to exactly one particular CRS.
Spatial R packages support a wide range of CRSs and they use the long-established PROJ library.
Other than searching for EPSG codes online, another quick way to find out about available CRSs is via the rgdal::make_EPSG()
function, which outputs a data frame of available projections.
Before going into more detail, it’s worth learning how to view and filter them inside R, as this could save time trawling the internet.
The following code will show available CRSs interactively, allowing you to filter ones of interest (try filtering for the OSGB CRSs for example):
In sf the CRS of an object can be retrieved using st_crs()
.
For this, we need to read-in a vector dataset:
vector_filepath = system.file("vector/zion.gpkg", package = "spDataLarge")
new_vector = st_read(vector_filepath)
Our new object, new_vector
, is a polygon representing the borders of Zion National Park (?zion
).
st_crs(new_vector) # get CRS
#> Coordinate Reference System:
#> No EPSG code
#> proj4string: "+proj=utm +zone=12 +ellps=GRS80 ... +units=m +no_defs"
In cases when a coordinate reference system (CRS) is missing or the wrong CRS is set, the st_set_crs()
function can be used:
## Warning: st_crs<- : replacing crs does not reproject data; use st_transform
## for that
The warning message informs us that the st_set_crs()
function does not transform data from one CRS to another.
The projection()
function can be used to access CRS information from a Raster*
object:
## [1] "+proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0"
The same function, projection()
, is used to set a CRS for raster objects.
The main difference, compared to vector data, is that raster objects only accept proj4
definitions:
projection(new_raster) = "+proj=utm +zone=12 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0
+units=m +no_defs" # set CRS
We will expand on CRSs and how to project from one CRS to another in much more detail in Chapter ??.
2.5 Units
An important feature of CRSs is that they contain information about spatial units. Clearly, it is vital to know whether a house’s measurements are in feet or meters, and the same applies to maps. It is good cartographic practice to add a scale bar onto maps to demonstrate the relationship between distances on the page or screen and distances on the ground. Likewise, it is important to formally specify the units in which the geometry data or pixels are measured to provide context, and ensure that subsequent calculations are done in context.
A novel feature of geometry data in sf
objects is that they have native support for units.
This means that distance, area and other geometric calculations in sf return values that come with a units
attribute, defined by the units package (???).
This is advantageous, preventing confusion caused by different units (most CRSs use meters, some use feet) and providing information on dimensionality.
This is demonstrated in the code chunk below, which calculates the area of Luxembourg:
##<PHONE_NUMBER> [m^2]
The output is in units of square meters (m2), showing that the result represents two-dimensional space.
This information, stored as an attribute (which interested readers can discover with attributes(st_area(luxembourg))
), can feed into subsequent calculations that use units, such as population density (which is measured in people per unit area, typically per km2).
Reporting units prevents confusion.
To take the Luxembourg example, if the units remained unspecified, one could incorrectly assume that the units were in hectares.
To translate the huge number into a more digestible size, it is tempting to divide the results by a million (the number of square meters in a square kilometer):
## 2416.87 [m^2]
However, the result is incorrectly given again as square meters. The solution is to set the correct units with the units package:
## 2416.87 [km^2]
Units are of equal importance in the case of raster data.
However, so far sf is the only spatial package that supports units, meaning that people working on raster data should approach changes in the units of analysis (for example, converting pixel widths from imperial to decimal units) with care.
The new_raster
object (see above) uses a WGS84 projection with decimal degrees as units.
Consequently, its resolution is also given in decimal degrees but you have to know it, since the res()
function simply returns a numeric vector.
## [1] 0.0008333333 0.0008333333
If we used the UTM projection, the units would change.
## [1] 0.000833 0.000833
Again, the res()
command gives back a numeric vector without any unit, forcing us to know that the unit of the UTM projection is meters.
2.6 Exercises
- Use
summary()
on the geometry column of theworld
data object. What does the output tell us about:- Its geometry type?
- The number of countries?
- Its coordinate reference system (CRS)?
- Run the code that ‘generated’ the map of the world in Figure 2.5 at the end of Section 2.2.4.
Find two similarities and two differences between the image on your computer and that in the book.
- What does the
cex
argument do (see?plot
)? - Why was
cex
set to thesqrt(world$pop) / 10000
? - Bonus: experiment with different ways to visualize the global population.
- What does the
- Use
plot()
to create maps of Nigeria in context (see Section 2.2.4).- Adjust the
lwd
,col
andexpandBB
arguments ofplot()
. - Challenge: read the documentation of
text()
and annotate the map.
- Adjust the
- Create an empty
RasterLayer
object calledmy_raster
with 10 columns and 10 rows. Assign random values between 0 and 10 to the new raster and plot it. - Read-in the
raster/nlcd2011.tif
file from the spDataLarge package. What kind of information can you get about the properties of this file?
Reminder: solutions can be found online at https://geocompr.github.io
spDataLarge is not on CRAN, meaning it must be installed via devtools or with the following command:
install.packages("spDataLarge", repos = "https://nowosad.github.io/drat/", type = "source")
. ↩The origin we are referring to, depicted in blue in Figure 2.1, is in fact the ‘false’ origin. The ‘true’ origin, the location at which distortions are at a minimum, is located at 2° W and 49° N. This was selected by the Ordnance Survey to be roughly in the center of the British landmass longitudinally. ↩
The full OGC standard includes rather exotic geometry types including ‘surface’ and ‘curve’ geometry types, which currently have limited application in real world applications. All 17 types can be represented with the sf package, although (as of summer 2018) plotting only works for the ‘core 7’.↩
plot()
ing of sf objects usessf:::plot.sf()
behind the scenes.plot()
is a generic method that behaves differently depending on the class of object being plotted.↩Note: many plot arguments are ignored in facet maps, when more than one
sf
column is plotted.↩By definition, a polygon has one exterior boundary (outer ring) and can have zero or more interior boundaries (inner rings), also known as holes. A polygon with a hole would be, for example,
POLYGON ((1 5, 2 2, 4 1, 4 4, 1 5), (2 4, 3 4, 3 3, 2 3, 2 4))
↩Other attributes might include an urbanity category (city or village), or a remark if the measurement was made using an automatic station.↩
Depending on the file format the header is part of the actual image data file, e.g., GeoTIFF, or stored in an extra header or world file, e.g., ASCII grid formats. There is also the headerless (flat) binary raster format which should facilitate the import into various software programs. ↩
The degree of compression is often referred to as flattening, defined in terms of the equatorial radius (\(a\)) and polar radius (\(b\)) as follows: \(f = (a - b) / a\). The terms ellipticity and compression can also be used (???). Because \(f\) is a rather small value, digital ellipsoid models use the ‘inverse flattening’ (\(rf = 1/f\)) to define the Earth’s compression. Values of \(a\) and \(rf\) in various ellipsoidal models can be seen by executing
st_proj_info(type = "ellps")
.↩A complete list of the
proj4string
parameters can be found at https://proj.org.↩


New
The next Forgotten Roads MX blog post is now available ...
Dam Ride 2017 - Day 3
by oubiwann
Excerpt
A review of the third and final day of the South Dakota/North Dakota H.O.G. Chapters' 2017 Dam Ride...
Read the rest of
the post here.
Preferences
If you no longer wish to receive email notifications for new FRMX blog posts,
please see the links at the bottom of this email for instructions on how to
unsubscribe.


GPA: 3.88
Courses: Software Development · Artificial Intelligence · Networks and Web Development · Theory of Computing · Statistics · Differential Equations
Courses: Mobile Software Development · Algorithms and Data Structures · Structures and Dynamics of Complex Networks · Graph Theory · Combinatorial Optimization · UI Design
Proficient: Java, Python
Experience: C
Basic: Ruby on Rails, React, JavaScript, HTML, CSS, MySQL, PHP
AWS, Agile Development, Scrum, UX Design, Axure, PC, Mac, and Network troubleshooting, Hardware
Chinese, Cantonese, Malay, English
Apart from coding and exploring my interests in Software Engineering, I do enjoy a wider variety of activities. I am very passionate about cooking, and through the years have developed several personal recipes. I started cooking for my family at 13 years old, and from then the enthusiasm only kept growing.
Besides cooking, I enjoy being outdoors. Biking and Scouting remain favorites, and I cherish every opportunity I get to be out in nature. Biking provides me a sense of freedom and excitement, and Scouting taught me many of life's most valuable lessons. I am a King's Scout, the Malaysian equivalent of an Eagle Scout.
AWS Developer Associate 2018
Applying A Framework For Creating And Analyzing Cybersecurity Questions For Peer Instruction. CCSC-NW 2017, Richland, WA. October 6, 2017
Analyzing And Improving The NCL Ethical Hacking Lab Exercises. CSCE-SAM 2017, Las Vegas, NV. July 17, 2017


IPython Notebook:
Development Lessons
Not your typical intro

We won’t dive into:
- Rich display
- equations, images, sounds, movies, graphs, HTML
- ZeroMQ architecture
- Debugger
- Parallel computing
- Securely sharing a notebook server
- Notebook Viewer
- etc.
My History
PyCon 2012
Installing
Hyping
PyCon 2013
Learning
Learning
Requests
HTTP
Server idiosyncracies
Intrinsic
Germane
Extraneous
Clear Code Interfaces
Interfaces above and below were well-specified
Internal interfaces were in flux
Easy Access to Levels of Abstraction
"Understand one level of abstraction below the one you normally work at."
Corollary
"Have ready access to the layer of abstraction below you."
Fast Iteration
"The power of successive refinement and iterative development. Impossible without continuous aggressive refactoring." - Uncle Bob Martin
Small Pieces, Well Factored
Screen-space constraints forced me to write smaller functions, classes, etc.
Fast iteration helped me mercilessly refactor.
Bonus
Final, working code was there at the end
Why did this succeed?
- Reduced Cognitive Load
- Clear Code Interfaces
- Easy Access to Levels of Abstraction
- Fast Iteration
- Small Pieces, Well Factored
Installation
- Pip
pip install --use-wheel ipython pyzmq jinja2 tornado
- Pre-built Python Distributions


Although it is not compulsory (yet) for devoted readers to shower Mr Key with gifts, it is certainly advisable. I was very pleased to find this lapel accoutrement in my Christmas sock, and would encourage all of you to spend the best part of 2015 considering carefully how you might demonstrate your fealty next time Yuletide comes around.


First Menu
This is the first menu content.
Second Menu
This is the second menu content.
End Menu
This is the end menu content.
Menu
Tap a button below to open a specific menu.
Open First Menu
Open Second Menu
Open End Menu


{% extends 'base.html' %} {% block title %}
It seems we couldn't find what you were looking for. Click below to start a new search.


Sign in
Student ID
Password
Submit
myNSB
Dashboard
power_settings_new
settings
more_vert
dashboard
Dashboard
assignment
Timetable
event_note
Calendar
alarm
Reminders
chrome_reader_mode
The 4U Paper
dns
Classery
home
School Intranet


Select Number of Hand Cards
Fusion
Special Summon
Summon/Activate
Set
Next Phase
Battle Phase
Main
Draw


A Prishtina-based restaurant management company specializing in creating and operating upscale, innovative restaurant concepts, led by Founder and award-winning Chef Ed Gelebus.
Operating since 2005, Flavour is one of the finest restaurants in the Balkans. At Flavour Group, impeccable service is not just our ideal, it is ingrained in everything we do. Rooted deep in our foundation is a selflessness and flexibility that is reflected in an experience wholly centered on others. By embodying our values of adaptability and fluidity, we cultivate a whole new sense of hospitality that is the one thing that makes us the best restaurant in region.


ElectroLens
Intro
Upload a prior configuration file or setup a new configuration.
Upload Configuration File
Setup Configuration
Spatially Resolved Data?
Molecular Data?
Framed Data?
List of Properties for Spatially Resolved Data
x, y, z, rho, property_1, property_2
*The list of properties for spatially resolved data, leave blank if not visualizing spatially resolved data
Density Property:
*The property denotes the density of electron.
Density Lower Limit:
Density Upper Limit:
List of Properties for Molecular Data
x, y, z, atom, property_1, property_2
*The list of properties for molecular data, leave blank if not visualizing molecular data
Property Labeling frame
*The property labeling frame, leave blank if the data is not framed
View 1
System Name:
System Dimensions:
Cubic Cell?
x dimension:
y dimension:
z dimension:
Orthogonal Lattice?
Lattice Vector:
u11:
u12:
u13:
u21:
u22:
u23:
u31:
u32:
u33:
Data File for Spatially Resolved Data:
Number grid point:
x:
y:
z:
Grid spacing:
x:
y:
z:
Data File for Molecular Data:
+ Add view
Submit
Save Config


{# FOR KICKSTART: #} {# THIS WAS IN CONFLICT WITH JQUERY UI ABOVE #}
{{message}}
Default: {{tagbox_default}} #}
{{lex_group}}
More detail about the {{corpusnam}}
Tagging Documentation
Maintainer: Francis Bond< bond@ieee.org>


You can't create a Schedule without first creating a Host for it to use. Please create a host.
Are you sure?


Updates from Sea
I'll be at sea on the R.R.S. James Cook this summer.
The goal of the project, informally known as “Ziploc,” is to study interactions among zinc, iron and phosphate in the subtropical Atlantic.
Phosphate is becoming increasingly scarce in the ocean. Marine microbes, which need phosphate to grow, have developed an entrepreneurial way to acquire phosphate from organic matter, using the enzyme alkaline phosphatase. Alkaline phosphatase requires zinc and/or iron to function, so it is important to consider the distribution of these metals in the ocean, and the associated impact on phytoplankton growth. This will help us to understand and prepare for anthropogenic effects on the ocean. We are going to the North Atlantic gyre to study these complex interactions.
Stay tuned for more background, updates, and photos.


Overview of AC Repair Servicing
Why you choose us?
Trusted & Reliable AC Technicians
Well-equipped & Well-prepared Specialists to Prevent COVID-19
Best pricing & budget-friendly
What's Included?
What's Excluded?
Price of materials or parts
Transportation cost for carrying new materials/parts
Warranty given by manufacturer
Available Services
AC Basic Servicing:
AC Basic service offers primary diagnosis, filter cleaning, test and identify problems by an expert AC technician.
AC Master Service:
AC Master Service offers detail cleaning of the indoor and outdoor units including minor problem-fixing (excluding materials and parts). The service charge varies on your AC amount, height, weight and difficulties.
AC Water Drop Solution:
This service offers identification of the source of dripping water from your AC and fixation water drainage system accordingly. Any additional materials/parts will be charged separately.
AC Jet Wash:
AC Jet Wash offers detailed cleaning of the indoor and outdoor units with Jet Wash Machine including minor problem-fixing (excluding materials and parts). The service charge varies on your AC amount, height, weight, and difficulties.


Use DVRPC’s Data Navigator for Municipal-Level Demographic Info
The Data Navigator allows visitors to find various datasets for a given municipality or county. It automatically provides a map of the geography being queried and a graph of the data. To date, you can view Census information, crash data, population and employment forecasts, and more. New datasets will be added to this interface as they become available, so check back often. To view the application, visit http://www.dvrpc.org/asp/DataNavigator/default.aspx.
Read More


DEV DATA
{% endif %}Running Scrapers
| Org | Group | Name | Start Time | Stop Time | Runtime (hh:mm:ss) | Log Criticals | Log Errors | Log Warnings | Url Error Count |
|---|
Finished Scrapers (History)
| Org | Group | Name | Start Time | Stop Time | Runtime (hh:mm:ss) | Log Criticals | Log Errors | Log Warnings | Ref Data Count | Ref Data Success | Total urls Hit | Url Error Count | Annotation |
|---|


The main configurations file: conf.py¶
Below is the code for conf.py
, the main configuration file that is used for generating this documentation.
This file was originally made by running sphinx-quickstart
.
It was modified and customized such that, hopefully, it has a clearer structure, and is easier to re-modify for the sake of YOUR next project :)
"""
This is the main file in which the configuration for the documentation is made.
"""
# -*- coding: utf-8 -*-
#
# Sphinx185 documentation build configuration file, created by
# sphinx-quickstart on Sat May 12 19:34:31 2018.
# DalyaG: This file was heavily modified from its original build.
# Hope you find this useful :)
# -- Define here your working directory -------------------------------------------------------------------------------
import os
import sys
sys.path.append(os.path.abspath('/Users/dalya/Documents/Sphinx185'))
# -- Some general info about the project -----------------------------------------------------------------------------
project = u'Sphinx185'
copyright = u'2018, DalyaG'
author = u'DalyaG'
# -- A few basic configurations ---------------------------------------------------------------------------------------
# The documentation in this project will be mostly generated from .rst files
# In This project, every auto-documented module/class has its own .rst file, under the main documentation dir,
# which is rendered into an .html page.
# Get more information here: http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
source_suffix = ['.rst']
# This is the name of the main page of the project.
# It means that you need to have an `index.rst` file where you will design the landing page of your project.
# It will be rendered into an .html page that you can find at: `_build/html/index.html`
# (this is a standard name. change it only if you know what you are doing)
master_doc = 'index'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = ['_build']
# List here any paths that contain templates, relative to this directory.
# You can find some not-so-intuitive information here: http://www.sphinx-doc.org/en/master/templating.html
# But the best way to learn is by example, right? :)
# So, for example, in this project, I wanted to change the title of the Table Of Contents in the sidebar.
# So I copied `<Sphinx install dir>/themes/basic/globaltoc.html` into the `_templates` folder,
# and replaced 'Table of Content' with 'Sphinx185'.
templates_path = ['_templates']
# -- Define and configure non-default extensions ----------------------------------------------------------------------
# You can find a list of available extension here: http://www.sphinx-doc.org/en/master/extensions.html
extensions = ['sphinx.ext.todo', 'sphinx.ext.viewcode', 'sphinx.ext.autodoc', 'sphinx.ext.imgmath']
# Above extensions explanation and configurations:
# todo: When you use the syntax ".. todo:: some todo comment" in your docstring,
# it will appear in a highlighted box in the documentation.
# In order for this extension to work, make sure you include the following:
todo_include_todos = True
# viewcode: Next to each function/module in the documentation, you will have an internal link to the source code.
# The source code will have colors defined by the Pygments (syntax highlighting) style.
# You can checkout the available pygments here: https://help.farbox.com/pygments.html
pygments_style = 'native'
# autodoc: The best thing about Sphinx IMHO is autodoc.
# It allows Sphinx to automatically generate documentation for the docstrings in your code.
# Get more info here: http://www.sphinx-doc.org/en/master/ext/autodoc.html
# Some useful configurations:
autoclass_content = "both" # Include both the class's and the init's docstrings.
autodoc_member_order = 'bysource' # In the documentation, keep the same order of members as in the code.
autodoc_default_flags = ['members'] # Default: include the docstrings of all the class/module members.
# imgmath: Sphinx allows use of LaTeX in the html documentation, but not directly. It is first rendered to an image.
# You can add here whatever preamble you are used to adding to your LaTeX document.
imgmath_latex_preamble = r'''
\usepackage{xcolor}
\definecolor{offwhite}{rgb}{238,238,238}
\everymath{\color{offwhite}}
\everydisplay{\color{offwhite}}
'''
# -- Options for HTML output -----------------------------------------------------------------------------------------
# The theme to use for HTML and HTML Help pages.
# You can find available themes here: http://www.sphinx-doc.org/en/master/theming.html
# In this project, I wanted to use a non-default theme, and so I downloaded the `graphite` template from here:
# https://github.com/Cartroo/sphinx-theme-graphite
# Some adjustments I made to graphite:
# - I did not use the pygment configuration, and so removed "pygments_style = graphite.GraphiteStyle" from theme.conf
# and deleted graphite.py
# - In the static folder, I configured several classes both in graphite.css and in html4css1.css,
# you can download the original and compare to find those changes.
html_theme = 'graphite'
# When using a non-built-in theme, define the path to your template code.
html_theme_path = ['.']
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Defining the static path allows me to add my own logo for the project:
# (make sure the theme of your choice support the use of logo.
html_logo = '_static/sphinx_and_dalya.png'
# Custom sidebar templates, must be a dictionary that maps document names to template names.
# In This project I chose to include in the sidebar:
# - Table of Contents: I chose globaltoc as it is less refined,
# and configured the title by editing the globaltoc template (see explanation above, in the templates_path comment)
# - Search box: appears below the TOc, and can be configured by editing css attributes.
html_sidebars = {
'**': [
'globaltoc.html',
'searchbox.html'
]
}
This page was generated using this .rst code:¶
The main configurations file: conf.py
=====================================
|
Below is the code for ``conf.py``, the main configuration file that is used for generating this documentation.
This file was originally made by running ``sphinx-quickstart``.
It was modified and customized such that, hopefully, it has a clearer structure,
and is easier to re-modify for the sake of YOUR next project :)
|
.. literalinclude:: conf.py
|
|
This page was generated using this .rst code:
*********************************************
|
.. literalinclude:: conf.rst
|
|
:ref:`Return Home <mastertoc>`


MIDNIGHT MOVIE MASSACRE (seen at Nolacon II) A film review by Mark R. Leeper Copyright 1988 Mark R. Leeper
Capsule review: Neither competent enough to be fun nor incompetent enough to be funny. Approximately one hundred attempted pieces of humor. About two work. Same fail spectacularly. Rating: -3.
Wade Williams is a well-known fan of 1950s science fiction movies and television. Of late, when you see 1950s science fiction television shows turning up in video stores or on television, they almost invariably show a recent copyright by Wade Williams. Why he has bought up these copyrights I am not sure but rumor had it that he had rereleased the 1950 ROCKETSHIP X-M with new color footage he added and was planning to do the same with HIDEOUS SUN DEMON. The 1988 World Science Fiction Convention featured the world premiere of MIDNIGHT MOVIE MASSACRE, a film produced by Williams.
Blech!
To understand why this film is so bad you need to know a little of how it came to be made. Wade Milliams tried to make a sendup of the old television show SPACE PATROL. When the film was half made, the product was clearly so incompetent that the film would never have been released. Under any circumstances. Never. So here Williams was with only half a film that had two expensive stars and if he finished it, it would be thrown out. The stars? Well, he got Ann Robinson from WAR OF THE WORLDS and Robert Clarke, who finished up a good career by being in a number of cheap, bad science fiction films (though perhaps none so cheap and bad as the first half of SPACE PATROL). It was Williams' bright idea to take his film and dress it up as an *imitation* dead teenager film. Dead teenager films make money. There are millions of teenagers willing to shell out big bucks to see fantasies of others in their age group being carved up like so much poultry. (Think about that if you're waiting for the next generation to come to power and improve things.) How do you make half of SPACE PATROL into a dead teenager film?
The outer film is a satire on 1950s science fiction films, the only films Williams seems to really know well, as well as being a dead teenager film. So the whole outer story is set in 1956. This improves the film within since it suddenly becomes very prophetic as well as pathetic. It predicts 1980s hairstyles for women. When it shows the earth from space, it shows light wispy clouds that were never shown in science fiction films until years later. Then there is the fact that in the Midwest in 1956 (read that virtually none) had midnight shows.
Now, don't get me wrong. I do not mean to imply that the dead teenager portion of the film has much in the way of dead teenagers. Instead, it is taken up mostly by showing everything happening in the audience.
Treat yourself right. Skip this film. This is a -3 film on the -4 to +4 scale.
Mark R. Leeper att!mtgzz!leeper<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


%= stylesheetTag("application.css") %> "> ">
In this state, registering participants and screening encounter creation is closed. You can still edit existing data.
<% } else if (siteStatus == "locked") { %>You can ONLY manage users in this state.
<% } else { %>You can only view the data on the site BUT cannot change anything here.
<% } %>


2nd Layer Contents Listing Test Suite
1.  Get 2nd Depth Contents List (LG Channels)
3. Get 2nd Depth Contents List (CDP)


WITNESS LIST Natural Resources Committee (Subcommittee on Groundwater Districts) April 15, 1999-4:30P SB 635 Registering, but not testifying: For: Matthews, Jim (Cities of Bryan & College Station), Austin Riley, Bill (City of College Station-Manager, Water/Wastewater Utilities), College Station On: Jones, Sam (TNRCC-Manager, District Administration), Austin Knowles, Tommy (TWDB-Deputy Exec. Administrator Planning), Austin Musick, Steve (TNRCC-Manager, Groundwater Assessment Section), Austin SB 700 Registering, but not testifying: On: Jones, Sam (TNRCC-Manager, District Administration), Austin Knowles, Tommy (TWDB-Deputy Executive Admin. Planning), Austin Neighbors, Ronald (Harris-Galveston Coastal Subsidence District-General Manager), Friendswood SB 943 THIS LIST HAS NO WITNESSES. SB 1826 FOR: Byars, George (Blanco County-County Judge), Johnson City Tellez, Victor (Blanco County-Commissioner), Johnson City Registering, but not testifying: On: Jones, Sam (TNRCC-Manager, District Administration), Austin Knowles, Tommy (TWDB-Deputy Exec. Admin. for Planning), Austin Musick, Steve (TNRCC-Manager, Groundwater Assessment Section), Austin SB 1829 FOR: Henry, Linda (McMullen Co. Groundwater Conservation Dist.-County Judge), Tilden McTee, Clifford (Temp. Bd. of Dir. McMullen County Groundwater Cons. District), Tilden Registering, but not testifying: On: Jones, Sam (TNRCC-Manager, District Administration), Austin Knowles, Tommy (TWDB-Deputy Exec. Administrator for Planning), Austin Musick, Steve (TNRCC--Manager, Groundwater Assessment Section), Austin SB 1847 Registering, but not testifying: On: Jones, Sam (TNRCC-Manager, District Administration), Austin Knowles, Tommy (TWDB-Deputy Exec. Admin. for Planning), Austin Musick, Steve (TNRCC-Manager, Groundwater Assessment Section), Austin SB 1848 Registering, but not testifying: On: Knowles, Tommy (TWDB-Deputy Exec. Admin. for Planning), Austin Musick, Steve (TNRCC-Manager, Groundwater Assessment Section), Austin SB 1850 Registering, but not testifying: For: Booth, Michael (City of Victoria, attorney), Austin On: Jones, Sam (TNRCC-Manager, District Administration), Austin Knowles, Tommy (TWDB-Deputy Exec. Admin. for Planning), Austin Musick, Steve (TNRCC-Manager, Groundwater Assessment Section), Austin SB 1852 FOR: Millikin, Jay (Comal County Commissioners' Court-County Commissioner), New Braunfels Registering, but not testifying: For: Salyers, Ellen (Comal County-Asst. District Attorney), New Braunfels On: Jones, Sam (TNRCC-Manager, District Administration), Austin Knowles, Tommy (TWDB-Deputy Exec. Admin. for Planning), Austin Musick, Steve (TNRCC-Manager, Groundwater Assessment Section), Austin


This is a demo of the multisig functionality on bitcore bitcore development guide.
Powered by bitcore.io


Welcome to the Head First Lounge
Join us any evening for refreshing elixirs, conversation and maybe a game or two of Dance Dance Revolution. Wireless access is always provided; BYOWS (Bring Your Own Web Server).
Our Elixirs
-
Green Tea Cooler
Chock full of vitamins and minerals, combines the healthful benefits of green tea with a twist of chamomile blossoms and ginger root.
-
Raspberry Ice Concentration
Combining raspberry juice with lemon grass, citrus peel and rosehips, this icy drink will make your mind feel clear and crisp.
-
Blueberry Bliss Elixir
Blueberries and cherry essence mixed into a base of elderflower herb tea will put you in a relaxed state of bliss in no time.
-
Cranberry Antioxidant Blast
Wake up to the flavors of cranberry and hibiscus in this vitamin C rich elixir.
Come join us!


About Me
- Chief Architect @ ReachForce
- Co-Chair Austin Groovy and Grails User Group
- Co-author of 'Effective Gradle Implementation' Video Series on Packt
- Heavily into DevOps as of late
- Also check out my other talks 'Getting Groovy with Lego Mindstorms EV3'
- ..and getting Groovy and Google Glass and Android Wear!
What We Will Cover
- What is a VPC?
- S3 Storage Service
- ElasticWolf
- sshoogr and gramazon - Basis for Groovy-based Ansible?
- Grails Plugins (Grails 2.x and 3 what works)
- Elastic Load Balancers
- Autoscaling
- Others misc tips
VPC - Virtual Private Cloud
- Proper Architecture (Go Multi-AZ!)
- Routing
- NAT
- ACL Firewall
- Suggested Subnets
- Security Groups
- DHCP Tips
- VPC Flow Logs
- S3 Endpoints
- VPC Peering
- ClassicLink - Lin
- Others misc tips
VPC Overview
- Now required on new AWS accounts
- Very different that EC2 'Classic' which has no private network layer
- Costs nothing extra - you have nothing to lose
VPC - Routing/Subnets
VPC - Routing/Subnets
VPC - Routing/Subnets
adding a 3rd subnet - this gives a helping hand
- auto-scaling instances can access internet through NAT
- Web Servers, Web facing apps can live behind ELB yet be able to run OS updates
- S3 Buckets accessible
- can access your secure subnet as well
VPC - Routing/Subnets - Redundancy
VPC - Routing/Subnets - Redundancy
Now you can add subnets each on a different AZ - You can be 3 9's!
- Create 4 subnets - 2 DMZ's across 2 AZ & 2 secure subnets across AZs too!
- Web Servers, Web facing apps can live behind ELB yet be able to run OS updates
- S3 Buckets accessible
- Can access your secure subnet as well
- Use Route53 Health Checks to failover to backup Region or S3 Static site
- Also can use VPC Peering (Doesn't work across regions though!)
VPC - Routing/Subnets
VPC - Routing/Subnets
VPC - Routing/Subnets
VPC - Routing/Subnets
VPC - Routing/Subnets
Security ACLs
- ACLs like a full firewall - unlike groups
- Protect subnets instead of EC2 instances
- Processed in order of rule #
- ACLs are stateless - responses to inbound bound by outbound rules
- To accommodate various clients and OSs open ephemeral ports 1024-65535 then block malicous ports
VPC - Routing/Subnets
Security Groups
- VPC Security Groups are different than EC2 Groups - use different and descriptive names if both used
- You can use a security group ID (starting with sgXXXXX) in most places where IPs can be entered
- Processed in order of rule #
- ACLs are stateless - responses to inbound bound by outbound rules
- To accommodate various clients and OSs open ephemeral ports 1024-65535 then block malicous ports
VPC - Routing/Subnets
DHCP Options
- Determine What DNS DHCP clients use
- Can do things like set default domain, whether to use internal or external DNS for your VPC
- From Web UI, assign only 1 option set at a time
- Using amazon provided DNS, using naming scheme of 10.0.1.x, default domain ec2.internal
- Can assign your own DNS server names, or even Netbios name servers or NTP servers
- Options are semicolor delimited name value pairs, i.e. 'domain-name=something.com; domain-name-severs=AmazonProvidedDNSs
VPC - Flow Logs
Flow Logs - monitor your traffic!
- Log your VPC Traffic into Cloudwatch Logs
- Pay $.50/GiB to injest logs, $0.03/GiB to archive them
- Cannot log Amazon DNS traffic, Windows License Activation, DHCP traffic
- New as of June 2015 - AWS Partners working on tools
- Can be used for a more complete IDS solution vs simple proxy or host based
- You may need this for security compliance
VPC - Flow Logs
Flow Logs - monitor your traffic!
Let's take a look at a real one
VPC - Peering
Peering - Link VPCs Together
- Link VPCs together across AZs
- Cannot peer in between regions
- Can link VPCs across accounts
- Must not have overlapping CIDR blocks
- Can be used with Placement groups with some limitations
- Private DNS and Security Groups do not work between peers
VPC - Peering
(From AWS Docs)
VPC - Peering Lifecycle
(From AWS Docs)
VPC - S3 Endpoints
S3 Endpoints - Speed Up S3 Access!
- Easy to add - if you are not doing this you should!
- Works with some AWS servicse like Elastic Beanstalk, Opsworks, etc with limits
- Uses IAM policies to control access
- Now S3 traffic doesn't leave AWS network - huge boost!
VPC - IAM Roles
- Don't use root account anymore. Traditional security page is retired
- Use MFA Authentication device for all IAM users on console - hardware or phone app
- Set up your IAM roles for each user and use designated login url they give you
- Use resource level security with IAM to tier access to instances/resources
VPC - Misc EC2 Tips
- Reboot != Stop be careful
- Rebooting a machine from console will keep instance in place and ephemeral storage(OS reboot faster)
- If you issue an 'stop' command you machine will move and lose ephemeral state
VPC - Misc EC2 Tips
- VPN
- Use OpenVPN instead of metered AWS VPN - it runs fine on a tiny instance- save $$
- Use OpenVPN client to leave remote servers connected to VPC - it auto-reconnects
VPC - Misc EC2 Tips - Elastic Load Balancer
- Create internal ELB's inside your vpc for microservices
- Limit 3600s timeout on idle max now
- Can now use sticky source IP with HTTPS!
- Use Haproxy for anything very advanced, can accommodate many options
VPC - Misc EC2 Tips
- EBS Based Instances
- Use EBS backed instances for anything not build for cloud
- If you can build server on the fly with Chef/Puppet like tools, go for ephemeral based
- EBS backed instances have no swap by default - be sure to specify ephemeral disks on launch, use as swap
- EBS backed instances have no ephemeral disks on by default be sure to use them on launch of instance and AMIs
S3 Storage
- Limitations
- Common tools
- Regions with different functionality
- Glacier
- Grails S3 Plugin
S3 Storage
Limitations - Eventual Consistency
- This means after write/update it will EVENTUALLY be consistent
- Make your app retry on read fail - it might not be synced yet
- Different regions have different consistency rules
- US-West and EU Buckets have read after write consistency - but not update or delete (and cost more)
- US-East is so large it cannot handle any kind of consistency after write/update/delete - except patience!
S3 Storage
Limitations - large files
- Files over 5GB supported, but most tools don't handle properly
- S3 Tools must support mime/multipart
- s3cmd(Linux) / CyberDuck 4(Mac/Win32) / S3 Browser(Win32) / Cloudberry Explorer(PRO Win32) / Bucket Explorer
- File > 5GB files work with these tools, it is EXTREMELY slow
S3 Storage
Tools
- s3fs - mount as filesystem - but >5GB files broken, beware of consistency!
- Make sure FUSE is in kernel
- s3cmd is best free command line tools
- Bucket Explorer & CloudBerry Backup are good solid windows clients that parallelize multi-part uploads to ease the pain
- s3 Browser is ok free tool
- For s3fs install on Ubuntu: http://zentraal.com/docs/installing-s3fs-on-ubuntu/
S3 Storage
Different Function Between Different Regions
- US-East-1 Cheapest for full redundancy otherwise to save $ you can use RRS (Or Glacier)
- US-West and EU Buckets have read after write consistency - but not update or delete (and cost more)
- US-East is so large it cannot handle any kind of consistency after write/update/delete - except patience!
- Barring these limitations (and budget!), use the region closed to your VPC instances and regions
S3 Storage
Grails S3 Plugin
- No longer maintained - doesn't work with Grails 2.2.x or 3.x
- Source repo was on codehaus and never moved to github
- Uploads and catalogs assets (will use bucket name you give as base for its bucket name)
- Names files inside bucket with UUIDs to avoid collisions
- Can give each asset a bucket and key pair or global
General AWS Grails Plugins
- Grails AWS Plugin
- AWS SDK Plugin
- DynamoDB GORM Plugin - not covered
- Amazon Flexible Payments - not covered
- SimpleDB GORM Plugin - not convered
- CDN Asset Pipeline plugin - not covered
General AWS Grails Plugins Continued
- AWS App Config Plugin - Store version info in DynamoDB
- Elastic Beanstalk Plugin - help launch on EB
- AWS Instance - Cognito/CloutWatch/SES/SQS/IAM clients
- AWS User-Org - Dynamo DB for User/Group strucutres
- Karman Plugin - Cloud File Karman library plugin
General AWS Grails Plugins - Grails AWS Plugin
Actual Groovy/Grails Code to Manage S3 storage and SES Service (vs Java wrapper)
- Still maintained as of May 2015 (Grails 2.x though)
- cool wrapper for S3 stuff like torrents also SES
- Nifty wrapper for S3 stuff like torrents, public urls
General AWS Grails Plugins - Grails AWS Plugin
Has handy Gant scripts installed - used for SES stats
- Aws-Ses-Get-Send-Quota
- Aws-Sws-Get-Send-Statistics
- Aws-Ses-List-Verified-Emails
- Aws-Ses-Send-Ping-Mail
- Aws-Ses-Verify-Email
General AWS Grails Plugins - Grails AWS Plugin
Aws-Ses-Get-Send-Quota
General AWS Grails Plugins - Grails AWS Plugin
Aws-Sws-Get-Send-Statistics
General AWS Grails Plugins - Grails AWS Plugin
aws-ses-list-verified-emails
General AWS Grails Plugins - Grails AWS Plugin
aws-ses-send-ping-mail
- Script will verify a given email has been 'verified' by Amazon
General AWS Grails Plugins - Grails AWS Plugin
aws-ses-verify-email
- Script will submit an email for verification to amazon, and send an email to that address
General AWS Grails Plugins - Grails AWS Plugin
S3 File Management
- Handy for storing existing assets into S3
-
Convert local File to S3 Storage
def s3file = new File("/tmp/test.txt").s3upload {
path "folder/to/my/file/"
}
-
Upload directly from Stream:
def file = request.getFile('photo')
def uploadedFile = file.inputStream.s3upload(file.originalFilename) {
bucket "file-upload-from-inputstream"
}
General AWS Grails Plugins - Grails AWS Plugin
SES Management
- AFAIK skip this - the mail plugin handles all of this
-
// settings for mail plugin to work with SES
grails {
mail {
host = "email-smtp.us-east-1.amazonaws.com"
port = 25
username = "SESUsername"
password = "SESPassword"
props = ["mail.smtp.starttls.enable":"true",
"mail.smtp.port":"587","mail.debug":"true","mail.smtp.auth":"true"]
// uncomment to force all emails to one address
//<EMAIL_ADDRESS>grails.mail.default.from =<EMAIL_ADDRESS>}
}
General AWS Grails Plugins - AWS SDK Plugin
- Wrapper for AWS Java SDK
-
This is the one to use for most powerful work of services - can access almost all of AWS services that JDK supports
- Uses AWS Web Service API wrapped in a Java Library
- Error handing can be more difficult than using web service api directly
- Works with Grails 3, yay! https://github.com/agorapulse/grails-aws-sdk/tree/grails3
General AWS Grails Plugins - AWS SDK Plugin
Grails 3 AWS SDK Demo
General AWS Grails Plugins - AWS SDK Plugin
Services Supported
- EC2 - ELB,CloudWatch, Elastic Beanstalk
- RDS - Elastic Mapreduce, SDB, Dynamo, Redshift, SimpleDB
- S3/Glacier
- SES (us-east-1,us-west-2,eu-west-1 only)
- SQS, SWF
- CloudFormation,CloudFront
- Elastic Beanstalk, Transcoder, Opsworks(Old Chef)
- Cloudformation, CloudSearch, Elasticache
General AWS Grails Plugins - AWS SDK Plugin
EC2
- Grails service wrapper is 'amazonWebService'
- Call amazonWebService.ec2. - for default region, call amazonWebService.getEc2('region'). for others
- RunInstancesRequest class defines parameters to launch an instance (It seems picky about nulls)
- use amazonWebService.runInstances() to start instances
- Complete docs at: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/
General AWS Grails Plugins - AWS SDK Plugin
RDS
- Not sure why you would spin up entire databases, but you can!
- Most practical use would be to automate snapshots of the database
- If you have a small simple database, service is good
- Cannot start/top RDS instances, and pay premium to use their licenses if using commercial DBs
- Most DB's support 4-6TB disk now, my opinion has changed on RDS
- Has good multi-az failover (but pricey!) with PIOPS EBS Volumes (You can raid your own PIOPS drives!)
General AWS Grails Plugins - AWS SDK Plugin
S3
- manage buckets, upload/download and delete files
- Transfer Manager - handles multipart uploads (big files, batch, fast!)
- full docs here: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/package-summary.html
General AWS Grails Plugins - AWS SDK Plugin
Glacier
- cold storage for data - not immediately available on demand
- vaults - organize your archives and policies (where your files go)
- Upload files via amazonWebService.glacier.uploadArchive(UploadArchiveRequest uploadArchiveRequest) or UploadMultiPart
- Files are retrieved via 'initiateJob'
- Call 'listJobs' until your job is done.
- Can use Amazon SNS to notify you when jobs are complete
- When done get files from your vault via GetJobOutput
General AWS Grails Plugins - AWS SDK Plugin
Glacier
General AWS Grails Plugins - AWS SDK Plugin
SQS
- Simple queue service
- Create / Delete Queues
- Receive Message
- Can also assign permissions and batch messages
- Messages can remain in queue for 12 hours
ElasticWolf
- More powerful than web console gui
- Can control things like auto-scaling
- Created partly for new GovCloud because they had no UI at all initially
- Managed by sales organization instead of IT
- Actively Maintained on Github
sshoogr
- sshoogr - Groovy-based DSL library for working with remote servers through SSH - DSL Allows:
- connecting
- executing remote commands
- copying files and directories
- creating tunnels in a simple and concise way
- gradle plugin also for
- project very active
- sshoogr+gradle+DSL == Groovy Based Ansible???
Repos for these: https://github.com/aestasit/sshoogr and https://github.com/aestasit/gramazon-gradle
gramazon
- gramazon - Groovy based API for Amazon EC2
- interface library that can be used to interact with the Amazon EC2 system and control server resources on demand from your Groovy scripts or from Gradle, using a plug-in.>
- uses gradle to run commands
- maybe could be used as basis for a groovy version of chef or puppet?
- project also very active
- gradle project template available as well as 'gradle-ssh-plugin'
Repos for these: https://github.com/aestasit/sshoogr and https://github.com/aestasit/gramazon-gradle
Autoscaling Overview
- Autoscaling can be used for fault-tolernance (min 1)
- Most efficient user of instance
- Set Policies using templates for how more servers created/terminated
- Control Spot instance bit price
- You app must be able to handle 'sudden death'
- Make sure your debug your AMI BEFORE adding to a autoscale group (debugging failed autoscaled instances is no way to go through life son!)
Autoscaling Setup
- Step 1: Create Launch Config(Just like launching instance but a template)
- Step 2: Create Autoscale Group
- Step 3: Create Policies
- Demo with ElasticWolf (can use web console now too)
Use CodeDeploy with ASG
- Enable app to have Codedeploy payload
- With ASG adds a node, app is auto-deployed!
- Use AMInator or Packer to bake AMIs to feed the ASG
Cloudwatch and Autoscaling - Regular metrics suck!
- Custom metrics are the way to go....http://goo.gl/gGh0jO
- Why? CPU usage and network IO not good enough to scale!
- Use health check plugins - add your own metrics
- If you can mix ELB with ASG's and have ASG monitor ELB 'health'
Elastic Beanstalk
Finally a good solution to push your application to Elastic Beanstalk on Grails!
- Ken Liu now has first class citizen support in Grails for Elastic Beanstalk!
- Easy to set up and configure, just add keys and params to Config.groovy
- Use command 'aws-eb-deploy' and 'aws-eb-stop' - that's it!
- Detailed online manual available at http://kenliu.net/grails-elastic-beanstalk/manual.html
Useful resources
- Elastic Beanstalk plugin by Ken Liu: http://grails.org/plugin/aws-elastic-beanstalk
- Elastic Beanstalk Intro @ Bobby Warner's Blog: http://goo.gl/PQaAxx
- Another Beanstalk: http://goo.gl/0SeYyS
- Sample Beanstalk app: https://github.com/4np/grailsOnAWS
- Oracle, EBS, and other Tips from AWS Architect Tom Laszewski http://cloudconclave.blogspot.com/
Useful resources
- Building an S3 Browser in Grails http://goo.gl/u6lRYB
- Grails AWS SDK Plugin https://goo.gl/I9qrcl
- VPC Deep Dive Slides http://goo.gl/o4g8yj
- Deploying Grails 3 and CDN on Elastic Beanstalk https://goo.gl/x163Fv
- AWS Global Interactive Map https://goo.gl/qpHR09


for booking {{serviceName|capitalize}} service.
click here to check your quotes status
{{contextualText}}


Fried chicken salad is the perfect blend of crispy chicken and fresh veggies tossed in your favorite dressing.Make it even quicker with frozen chicken tenders or pick up a few in your deli section
Shrimp Spaghetti is such a comforting pasta dish, perfect for busy weeknights! Plump, garlicky prawns nestle on a bed of spaghetti, bejeweled with bright red baby tomatoes and vibrant green spinach. It’s simple but loaded with flavor from the seasoned seafood broth. Bright, easy, filling, and so delicious! Don’t like shrimps? Make it with any other protein like chicken or salmon chunks.
Pepperoni is characteristically soft, slightly smoky, and bright red in color. Thinly sliced pepperoni is a popular pizza topping in American pizzerias.
Elongated, crispy, crunchy and intensely fragrant, churros consist of deep-fried yeast dough encrusted with sugar.Although some may argue against consuming these sweet treats, warning others about the dangerous effects of sugar and fat on human bodies, the popularity of churros throughout the world doesn't seem to wane.
Fish and chips is a hot dish consisting of fried fish in batter served with chips. The dish originated in England and is an example of culinary fusion, combining Iberian Jewish fried fish with Belgian fried potatoes..
Buffalo wings, also called hot wings, chicken wings, or simply wings, deep-fried, unbreaded chicken wings or drums coated with a vinegar-and-cayenne-pepper hot sauce mixed with butter. They are commonly served with celery and a blue cheese dipping sauce, which acts as a cooling agent for the mouth


Thank you for using Movie Seeker. If your already a member, click the button below to sign in.
As streaming becomes the method of choice for media consumption, we all face a big problem. How do we decide what to watch? With so much to choose between (from which platform to watch with, to the genre of movie) and broad categories that don’t really help us decide, it can take more time to choose what movie you want to watch than it will to actually watch the movie! That’s why we created Movie Seeker. Movie Seeker makes the browsing process both streamlined and customizable by using your input to find you exactly the right movie at exactly the right time.
Creating an account is fast, easy, and free!
On the following page, login's for various streaming services will be available
Check out some features below!
Username:
Password:
Email:
Phone (optional):
Movie Seeker saves you time and energy by eliminating the need to search through hundreds of titles and summaries. Let Movie Seeker do the searching for you! Input the characteristics of exactly the type of movie you want to watch and Movie Seeker will present you with the best match in a flash.
Do you have specific taste in movies? Are you ever in the mood for something that doesn’t fit cleanly in a generic movie genre? Make customized genres and lists to help Movie Seeker find better matches to suit your preferences!
It can be a pain to search for a specific title across multiple streaming platforms. You can sometimes look between a few and don’t even find the movie you want. With Movie Seeker that information can be found all in one place as it compares what movies are available where. No more switching and searching!
If you have ever tried to watch a movie with a group, you know how hard it is to pick a movie that will please everyone. Movie Seeker can solve that problem! Just adjust your criteria to include a little of what everyone likes and get suggestions that will please your whole party!
What are you waiting for? Click the button to see if your streaming service is available.