D Learning Check Solutions
D.1 Chapter 2 Solutions
library(dplyr)
library(ggplot2)
library(nycflights13)
(LC2.1) What does any ONE row in this flights
dataset refer to?
- A. Data on an airline
- B. Data on a flight
- C. Data on an airport
- D. Data on multiple flights
Solution: This is data on a flight. Not a flight path! Example:
- a flight path would be United 1545 to Houston
- a flight would be United 1545 to Houston at a specific date/time. For example: 2013/1/1 at 5:15am.
(LC2.2) What are some examples in this dataset of categorical variables? What makes them different than quantitative variables?
Solution: Hint: Type ?flights
in the console to see what all the variables mean!
- Categorical:
carrier
the companydest
the destinationflight
the flight number. Even though this is a number, its simply a label. Example United 1545 is not less than United 1714
- Quantitative:
distance
the distance in milestime_hour
time
(LC2.3) What does int
, dbl
, and chr
mean in the output above?
Solution:
int
: integer. Used to count things i.e. a discrete value. Ex: the # of cars parked in a lotdbl
: double. Used to measure things. i.e. a continuous value. Ex: your height in incheschr
: character. i.e. text
D.2 Chapter 3 Solutions
library(nycflights13)
library(ggplot2)
library(dplyr)
(LC3.1) Take a look at both the flights
and alaska_flights
data frames by running View(flights)
and View(alaska_flights)
in the console. In what respect do these data frames differ?
Solution: flights
contains all flight data, while alaska_flights
contains only data from Alaskan carrier “AS”. We can see that flights has 336776 rows while alaska_flights
has only 714
(LC3.2) What are some practical reasons why dep_delay
and arr_delay
have a positive relationship?
Solution: The later a plane departs, typically the later it will arrive.
(LC3.3) What variables (not necessarily in the flights
data frame) would you expect to have a negative correlation (i.e. a negative relationship) with dep_delay
? Why? Remember that we are focusing on numerical variables here.
Solution: An example in the weather
dataset is visibility
, which measure visibility in miles. As visibility increases, we would expect departure delays to decrease.
(LC3.4) Why do you believe there is a cluster of points near (0, 0)? What does (0, 0) correspond to in terms of the Alaskan flights?
Solution: The point (0,0) means no delay in departure nor arrival. From the point of view of Alaska airlines, this means the flight was on time. It seems most flights are at least close to being on time.
(LC3.5) What are some other features of the plot that stand out to you?
Solution: Different people will answer this one differently. One answer is most flights depart and arrive less than an hour late.
(LC3.6) Create a new scatterplot using different variables in the alaska_flights
data frame by modifying the example above.
Solution: Many possibilities for this one, see the plot below. Is there a pattern in departure delay depending on when the flight is scheduled to depart? Interestingly, there seems to be only two blocks of time where flights depart.
ggplot(data = alaska_flights, mapping = aes(x = dep_time, y = dep_delay)) +
geom_point()
(LC3.7) Why is setting the alpha
argument value useful with scatterplots? What further information does it give you that a regular scatterplot cannot?
Solution: Why is setting the alpha
argument value useful with scatterplots? What further information does it give you that a regular scatterplot cannot? It thins out the points so we address overplotting. But more importantly it hints at the (statistical) density and distribution of the points: where are the points concentrated, where do they occur. We will see more about densities and distributions in Chapter 6 when we switch gears to statistical topics.
(LC3.8) After viewing the Figure 3.4 above, give an approximate range of arrival delays and departure delays that occur the most frequently. How has that region changed compared to when you observed the same plot without the alpha = 0.2
set in Figure 3.2?
Solution: After viewing the Figure 3.4 above, give a range of arrival delays and departure delays that occur most frequently? How has that region changed compared to when you observed the same plot without the alpha = 0.2
set in Figure 3.2? The lower plot suggests that most Alaska flights from NYC depart between 12 minutes early and on time and arrive between 50 minutes early and on time.
(LC3.9) Take a look at both the weather
and early_january_weather
data frames by running View(weather)
and View(early_january_weather)
in the console. In what respect do these data frames differ?
Solution: Take a look at both the weather
and early_january_weather
data frames by running View(weather)
and View(early_january_weather)
in the console. In what respect do these data frames differ? The rows of early_january_weather
are a subset of weather
.
(LC3.10) View()
the flights
data frame again. Why does the time_hour
variable uniquely identify the hour of the measurement whereas the hour
variable does not?
Solution: View()
the flights
data frame again. Why does the time_hour
variable correctly identify the hour of the measurement whereas the hour
variable does not? Because to uniquely identify an hour, we need the year
/month
/day
/hour
sequence, whereas there are only 24 possible hour
’s.
(LC3.11) Why should linegraphs be avoided when there is not a clear ordering of the horizontal axis?
Solution: Why should linegraphs be avoided when there is not a clear ordering of the horizontal axis? Because lines suggest connectedness and ordering.
(LC3.12) Why are linegraphs frequently used when time is the explanatory variable?
Solution: Why are linegraphs frequently used when time is the explanatory variable? Because time is sequential: subsequent observations are closely related to each other.
(LC3.13) Plot a time series of a variable other than temp
for Newark Airport in the first 15 days of January 2013.
Solution: Plot a time series of a variable other than temp
for Newark Airport in the first 15 days of January 2013. Humidity is a good one to look at, since this very closely related to the cycles of a day.
ggplot(data = early_january_weather, mapping = aes(x = time_hour, y = humid)) +
geom_line()
(LC3.14) What does changing the number of bins from 30 to 60 tell us about the distribution of temperatures?
Solution: The distribution doesn’t change much. But by refining the bin width, we see that the temperature data has a high degree of accuracy. What do I mean by accuracy? Looking at the temp
variabile by View(weather)
, we see that the precision of each temperature recording is 2 decimal places.
(LC3.15) Would you classify the distribution of temperatures as symmetric or skewed?
Solution: It is rather symmetric, i.e. there are no long tails on only one side of the distribution
(LC3.16) What would you guess is the “center” value in this distribution? Why did you make that choice?
Solution: The center is around 55.26°F. By running the summary()
command, we see that the mean and median are very similar. In fact, when the distribution is symmetric the mean equals the median.
(LC3.17) Is this data spread out greatly from the center or is it close? Why?
Solution: This can only be answered relatively speaking! Let’s pick things to be relative to Seattle, WA temperatures:
While, it appears that Seattle weather has a similar center of 55°F, its temperatures are almost entirely between 35°F and 75°F for a range of about 40°F. Seattle temperatures are much less spread out than New York i.e. much more consistent over the year. New York on the other hand has much colder days in the winter and much hotter days in the summer. Expressed differently, the middle 50% of values, as delineated by the interquartile range is 30°F:
(LC3.18) What other things do you notice about the faceted plot above? How does a faceted plot help us see relationships between two variables?
Solution:
- Certain months have much more consistent weather (August in particular), while others have crazy variability like January and October, representing changes in the seasons.
- Because we see
temp
recordings split bymonth
, we are considering the relationship between these two variables. For example, for example for summer months, temperatures tend to be higher.
(LC3.19) What do the numbers 1-12 correspond to in the plot above? What about 25, 50, 75, 100?
Solution:
- While month is technically a number between 1-12, we’re viewing it as a categorical variable here. Specifically an ordinal categorical variable since there is a ordering to the categories
- 25, 50, 75, 100 are temperatures
(LC3.20) For which types of data-sets would these types of faceted plots not work well in comparing relationships between variables? Give an example describing the nature of these variables and other important characteristics.
Solution:
- We’d have 365 facets to look at. Way to many.
- We don’t really care about day-to-day fluctuation in weather so much, but maybe more week-to-week variation. We’d like to focus on seasonal trends.
(LC3.21) Does the temp
variable in the weather
data-set have a lot of variability? Why do you say that?
Solution: Again, like in LC (LC3.17), this is a relative question. I would say yes, because in New York City, you have 4 clear seasons with different weather. Whereas in Seattle WA and Portland OR, you have two seasons: summer and rain!
(LC3.22) What does the dot at the bottom of the plot for May correspond to? Explain what might have occurred in May to produce this point.
Solution: It appears to be an outlier. Let’s revisit the use of the filter
command to hone in on it. We want all data points where the month
is 5 and temp<25
weather %>%
filter(month==5 & temp < 25)
# A tibble: 1 x 15
origin year month day hour temp dewp humid wind_dir wind_speed wind_gust
<chr> <dbl> <dbl> <int> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 JFK 2013 5 8 22 13.1 12.0 95.3 80 8.06 NA
# ... with 4 more variables: precip <dbl>, pressure <dbl>, visib <dbl>,
# time_hour <dttm>
There appears to be only one hour and only at JFK that recorded 13.1 F (-10.5 C) in the month of May. This is probably a data entry mistake! Why wasn’t the weather at least similar at EWR (Newark) and LGA (La Guardia)?
(LC3.23) Which months have the highest variability in temperature? What reasons do you think this is?
Solution: We are now interested in the spread of the data. One measure some of you may have seen previously is the standard deviation. But in this plot we can read off the Interquartile Range (IQR):
- The distance from the 1st to the 3rd quartiles i.e. the length of the boxes
- You can also think of this as the spread of the middle 50% of the data
Just from eyeballing it, it seems
- November has the biggest IQR, i.e. the widest box, so has the most variation in temperature
- August has the smallest IQR, i.e. the narrowest box, so is the most consistent temperature-wise
Here’s how we compute the exact IQR values for each month (we’ll see this more in depth Chapter 5 of the text):
group
the observations bymonth
then- for each
group
, i.e.month
,summarize
it by applying the summary statistic functionIQR()
, while making sure to skip over missing data viana.rm=TRUE
then arrange
the table indesc
ending order ofIQR
weather %>%
group_by(month) %>%
summarize(IQR = IQR(temp, na.rm=TRUE)) %>%
arrange(desc(IQR))
| month | IQR |
|---|---|
| 11 | 16.02 |
| 12 | 14.04 |
| 1 | 13.77 |
| 9 | 12.06 |
| 4 | 12.06 |
| 5 | 11.88 |
| 6 | 10.98 |
| 10 | 10.98 |
| 2 | 10.08 |
| 7 | 9.18 |
| 3 | 9.00 |
| 8 | 7.02 |
(LC3.24) We looked at the distribution of a numerical variable over a categorical variable here with this boxplot. Why can’t we look at the distribution of one numerical variable over the distribution of another numerical variable? Say, temperature across pressure, for example?
Solution: Because we need a way to group many numerical observations together, say by grouping by month. For pressure, we have near unique values for pressure, i.e. no groups, so we can’t make boxplots.
(LC3.25) Boxplots provide a simple way to identify outliers. Why may outliers be easier to identify when looking at a boxplot instead of a faceted histogram?
Solution: In a histogram, the bin corresponding to where an outlier lies may not by high enough for us to see. In a boxplot, they are explicitly labelled separately.
(LC3.26) Why are histograms inappropriate for visualizing categorical variables?
Solution: Histograms are for numerical variables i.e. the horizontal part of each histogram bar represents an interval, whereas for a categorical variable each bar represents only one level of the categorical variable.
(LC3.27) What is the difference between histograms and barplots?
Solution: See above.
(LC3.28) How many Envoy Air flights departed NYC in 2013?
Solution: Envoy Air is carrier code MQ
and thus 26397 flights departed NYC in 2013.
(LC3.29) What was the seventh highest airline in terms of departed flights from NYC in 2013? How could we better present the table to get this answer quickly?
Solution: What a pain! We’ll see in Chapter 5 on Data Wrangling that applying arrange(desc(n))
will sort this table in descending order of n
!
(LC3.30) Why should pie charts be avoided and replaced by barplots?
Solution: In our opinion, comparisons using horizontal lines are easier than comparing angles and areas of circles.
(LC3.31) What is your opinion as to why pie charts continue to be used?
Solution: Legacy?
(LC3.32) What kinds of questions are not easily answered by looking at the above figure?
Solution: Because the red, green, and blue bars don’t all start at 0 (only red does), it makes comparing counts hard.
(LC3.33) What can you say, if anything, about the relationship between airline and airport in NYC in 2013 in regards to the number of departing flights?
Solution: The different airlines prefer different airports. For example, United is mostly a Newark carrier and JetBlue is a JFK carrier. If airlines didn’t prefer airports, each color would be roughly one third of each bar.}
(LC3.34) Why might the side-by-side (AKA dodged) barplot be preferable to a stacked barplot in this case?
Solution: We can easily compare the different aiports for a given carrier using a single comparison line i.e. things are lined up
(LC3.35) What are the disadvantages of using a side-by-side (AKA dodged) barplot, in general?
Solution: It is hard to get totals for each airline.
(LC3.36) Why is the faceted barplot preferred to the side-by-side and stacked barplots in this case?
Solution: Not that different than using side-by-side; depends on how you want to organize your presentation.
(LC3.37) What information about the different carriers at different airports is more easily seen in the faceted barplot?
Solution: Now we can also compare the different carriers within a particular airport easily too. For example, we can read off who the top carrier for each airport is easily using a single horizontal line.
D.3 Chapter 4 Solutions
library(dplyr)
library(ggplot2)
library(nycflights13)
library(tidyr)
library(readr)
(LC4.1) Consider the following data frame of average number of servings of beer, spirits, and wine consumption in three countries as reported in the FiveThirtyEight article Dear Mona Followup: Where Do People Drink The Most Beer, Wine And Spirits?
# A tibble: 3 x 4
country beer_servings spirit_servings wine_servings
<chr> <int> <int> <int>
1 Canada 240 122 100
2 South Korea 140 16 9
3 USA 249 158 84
This data frame is not in tidy format. What would it look like if it were?
Solution: There are three variables of information included: country, alcohol type, and number of servings. In tidy format, each of these variables of information are included in their own column.
# A tibble: 9 x 3
country `alcohol type` servings
<chr> <chr> <int>
1 Canada beer 240
2 Canada spirit 122
3 Canada wine 100
4 South Korea beer 140
5 South Korea spirit 16
6 South Korea wine 9
7 USA beer 249
8 USA spirit 158
9 USA wine 84
Note that how the rows are sorted is inconsequential in whether or not the data frame is in tidy format. In other words, the following data frame sorted by alcohol type instead of country is equally in tidy format.
# A tibble: 9 x 3
country `alcohol type` servings
<chr> <chr> <int>
1 Canada beer 240
2 South Korea beer 140
3 USA beer 249
4 Canada spirit 122
5 South Korea spirit 16
6 USA spirit 158
7 Canada wine 100
8 South Korea wine 9
9 USA wine 84
(LC4.2) What properties of the observational unit do each of lat
, lon
, alt
, tz
, dst
, and tzone
describe for the airports
data frame? Note that you may want to use ?airports
to get more information.
Solution: lat
long
represent the airport geographic coordinates, alt
is the altitude above sea level of the airport (Run airports %>% filter(faa == "DEN")
to see the altitude of Denver International Airport), tz
is the time zone difference with respect to GMT in London UK, dst
is the daylight savings time zone, and tzone
is the time zone label.
(LC4.3) Provide the names of variables in a data frame with at least three variables in which one of them is an identification variable and the other two are not. In other words, create your own tidy dataset that matches these conditions.
Solution:
- In the
weather
example in LC3.8, the combination oforigin
,year
,month
,day
,hour
are identification variables as they identify the observation in question. - Anything else pertains to observations:
temp
,humid
,wind_speed
, etc.
(LC4.4) Convert the dem_score
data frame into a tidy data frame and assign the name of dem_score_tidy
to the resulting long-formatted data frame.
Solution: Running the following in the console:
dem_score_tidy <- gather(data = dem_score, key = year, value = democracy_score, - country)
Let’s now compare the dem_score
and dem_score_tidy
. dem_score
has democracy score information for each year in columns, whereas in dem_score_tidy
there are explicit variables year
and democracy_score
. While both representations of the data contain the same information, we can only use ggplot()
to create plots using the dem_score_tidy
data frame.
dem_score
# A tibble: 96 x 10
country `1952` `1957` `1962` `1967` `1972` `1977` `1982` `1987` `1992`
<chr> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 Albania -9 -9 -9 -9 -9 -9 -9 -9 5
2 Argentina -9 -1 -1 -9 -9 -9 -8 8 7
3 Armenia -9 -7 -7 -7 -7 -7 -7 -7 7
4 Australia 10 10 10 10 10 10 10 10 10
5 Austria 10 10 10 10 10 10 10 10 10
6 Azerbaijan -9 -7 -7 -7 -7 -7 -7 -7 1
7 Belarus -9 -7 -7 -7 -7 -7 -7 -7 7
8 Belgium 10 10 10 10 10 10 10 10 10
9 Bhutan -10 -10 -10 -10 -10 -10 -10 -10 -10
10 Bolivia -4 -3 -3 -4 -7 -7 8 9 9
# ... with 86 more rows
dem_score_tidy
# A tibble: 864 x 3
country year democracy_score
<chr> <chr> <int>
1 Albania 1952 -9
2 Argentina 1952 -9
3 Armenia 1952 -9
4 Australia 1952 10
5 Austria 1952 10
6 Azerbaijan 1952 -9
7 Belarus 1952 -9
8 Belgium 1952 10
9 Bhutan 1952 -10
10 Bolivia 1952 -4
# ... with 854 more rows
(LC4.5) Read in the life expectancy data stored at https://moderndive.com/data/le_mess.csv and convert it to a tidy data frame.
Solution: The code is similar
life_expectancy <- read_csv('https://moderndive.com/data/le_mess.csv')
life_expectancy_tidy <- gather(data = life_expectancy, key = year, value = life_expectancy, -country)
We observe the same construct structure with respect to year
in life_expectancy
vs life_expectancy_tidy
as we did in dem_score
vs dem_score_tidy
:
life_expectancy
# A tibble: 202 x 67
country `1951` `1952` `1953` `1954` `1955` `1956` `1957` `1958` `1959` `1960`
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Afghan… 27.1 27.7 28.2 28.7 29.3 29.8 30.3 30.9 31.4 31.9
2 Albania 54.7 55.2 55.8 56.6 57.4 58.4 59.5 60.6 61.8 62.9
3 Algeria 43.0 43.5 44.0 44.4 44.9 45.4 45.9 46.4 47.0 47.5
4 Angola 31.0 31.6 32.1 32.7 33.2 33.8 34.3 34.9 35.4 36.0
5 Antigu… 58.3 58.8 59.3 59.9 60.4 60.9 61.4 62.0 62.5 63.0
6 Argent… 61.9 62.5 63.1 63.6 64.0 64.4 64.7 65 65.2 65.4
7 Armenia 62.7 63.1 63.6 64.1 64.5 65 65.4 65.9 66.4 66.9
8 Aruba 59.0 60.0 61.0 61.9 62.7 63.4 64.1 64.7 65.2 65.7
9 Austra… 68.7 69.1 69.7 69.8 70.2 70.0 70.3 70.9 70.4 70.9
10 Austria 65.2 66.8 67.3 67.3 67.6 67.7 67.5 68.5 68.4 68.8
# ... with 192 more rows, and 56 more variables: `1961` <dbl>, `1962` <dbl>,
# `1963` <dbl>, `1964` <dbl>, `1965` <dbl>, `1966` <dbl>, `1967` <dbl>,
# `1968` <dbl>, `1969` <dbl>, `1970` <dbl>, `1971` <dbl>, `1972` <dbl>,
# `1973` <dbl>, `1974` <dbl>, `1975` <dbl>, `1976` <dbl>, `1977` <dbl>,
# `1978` <dbl>, `1979` <dbl>, `1980` <dbl>, `1981` <dbl>, `1982` <dbl>,
# `1983` <dbl>, `1984` <dbl>, `1985` <dbl>, `1986` <dbl>, `1987` <dbl>,
# `1988` <dbl>, `1989` <dbl>, `1990` <dbl>, `1991` <dbl>, `1992` <dbl>,
# `1993` <dbl>, `1994` <dbl>, `1995` <dbl>, `1996` <dbl>, `1997` <dbl>,
# `1998` <dbl>, `1999` <dbl>, `2000` <dbl>, `2001` <dbl>, `2002` <dbl>,
# `2003` <dbl>, `2004` <dbl>, `2005` <dbl>, `2006` <dbl>, `2007` <dbl>,
# `2008` <dbl>, `2009` <dbl>, `2010` <dbl>, `2011` <dbl>, `2012` <dbl>,
# `2013` <dbl>, `2014` <dbl>, `2015` <dbl>, `2016` <dbl>
life_expectancy_tidy
# A tibble: 13,332 x 3
country year life_expectancy
<chr> <chr> <dbl>
1 Afghanistan 1951 27.1
2 Albania 1951 54.7
3 Algeria 1951 43.0
4 Angola 1951 31.0
5 Antigua and Barbuda 1951 58.3
6 Argentina 1951 61.9
7 Armenia 1951 62.7
8 Aruba 1951 59.0
9 Australia 1951 68.7
10 Austria 1951 65.2
# ... with 13,322 more rows
(LC4.6) What are common characteristics of “tidy” datasets?
Solution: Rows correspond to observations, while columns correspond to variables.
(LC4.7) What makes “tidy” datasets useful for organizing data?
Solution: Tidy datasets are an organized way of viewing data. We’ll see later that this format is required for the ggplot2
and dplyr
packages for data visualization and wrangling.
(LC4.8) What are some advantages of data in normal forms? What are some disadvantages?
Solution: When datasets are in normal form, we can easily _join
them with other datasets! For example, can we join the flights
data with the planes
data? We’ll see this more in Chapter 5!
D.4 Chapter 5 Solutions
library(dplyr)
library(ggplot2)
library(nycflights13)
(LC5.1) What’s another way using the “not” operator !
we could filter only the rows that are not going to Burlington, VT nor Seattle, WA in the flights
data frame? Test this out using the code above.
Solution:
# Original in book
not_BTV_SEA <- flights %>%
filter(!(dest == "BTV" | dest == "SEA"))
# Alternative way
not_BTV_SEA <- flights %>%
filter(!dest == "BTV" & !dest == "SEA")
# Yet another way
not_BTV_SEA <- flights %>%
filter(dest != "BTV" & dest != "SEA")
(LC5.2) Say a doctor is studying the effect of smoking on lung cancer for a large number of patients who have records measured at five year intervals. She notices that a large number of patients have missing data points because the patient has died, so she chooses to ignore these patients in her analysis. What is wrong with this doctor’s approach?
Solution: The missing patients may have died of lung cancer! So to ignore them might seriously bias your results! It is very important to think of what the consequences on your analysis are of ignoring missing data! Ask yourself:
- There is a systematic reasons why certain values are missing? If so, you might be biasing your results!
- If there isn’t, then it might be ok to “sweep missing values under the rug.”
(LC5.3) Modify the above summarize
function to create summary_temp
to also use the n()
summary function: summarize(count = n())
. What does the returned value correspond to?
Solution: It corresponds to a count of the number of observations/rows:
weather %>%
summarize(count = n())
# A tibble: 1 x 1
count
<int>
1 26115
(LC5.4) Why doesn’t the following code work? Run the code line by line instead of all at once, and then look at the data. In other words, run summary_temp <- weather %>% summarize(mean = mean(temp, na.rm = TRUE))
first.
summary_temp <- weather %>%
summarize(mean = mean(temp, na.rm = TRUE)) %>%
summarize(std_dev = sd(temp, na.rm = TRUE))
Solution: Consider the output of only running the first two lines:
weather %>%
summarize(mean = mean(temp, na.rm = TRUE))
# A tibble: 1 x 1
mean
<dbl>
1 55.3
Because after the first summarize()
, the variable temp
disappears as it has been collapsed to the value mean
. So when we try to run the second summarize()
, it can’t find the variable temp
to compute the standard deviation of.
(LC5.5) Recall from Chapter 3 when we looked at plots of temperatures by months in NYC. What does the standard deviation column in the summary_monthly_temp
data frame tell us about temperatures in New York City throughout the year?
Solution:
| month | mean | std_dev |
|---|---|---|
| 1 | 35.6 | 10.22 |
| 2 | 34.3 | 6.98 |
| 3 | 39.9 | 6.25 |
| 4 | 51.7 | 8.79 |
| 5 | 61.8 | 9.68 |
| 6 | 72.2 | 7.55 |
| 7 | 80.1 | 7.12 |
| 8 | 74.5 | 5.19 |
| 9 | 67.4 | 8.47 |
| 10 | 60.1 | 8.85 |
| 11 | 45.0 | 10.44 |
| 12 | 38.4 | 9.98 |
The standard deviation is a quantification of spread and variability. We see that the period in November, December, and January has the most variation in weather, so you can expect very different temperatures on different days.
(LC5.6) What code would be required to get the mean and standard deviation temperature for each day in 2013 for NYC?
Solution:
summary_temp_by_day <- weather %>%
group_by(year, month, day) %>%
summarize(
mean = mean(temp, na.rm = TRUE),
std_dev = sd(temp, na.rm = TRUE)
)
summary_temp_by_day
# A tibble: 364 x 5
# Groups: year, month [?]
year month day mean std_dev
<dbl> <dbl> <int> <dbl> <dbl>
1 2013 1 1 37.0 4.00
2 2013 1 2 28.7 3.45
3 2013 1 3 30.0 2.58
4 2013 1 4 34.9 2.45
5 2013 1 5 37.2 4.01
6 2013 1 6 40.1 4.40
7 2013 1 7 40.6 3.68
8 2013 1 8 40.1 5.77
9 2013 1 9 43.2 5.40
10 2013 1 10 43.8 2.95
# ... with 354 more rows
Note: group_by(day)
is not enough, because day
is a value between 1-31. We need to group_by(year, month, day)
library(dplyr)
library(nycflights13)
summary_temp_by_month <- weather %>%
group_by(month) %>%
summarize(
mean = mean(temp, na.rm = TRUE),
std_dev = sd(temp, na.rm = TRUE)
)
(LC5.7) Recreate by_monthly_origin
, but instead of grouping via group_by(origin, month)
, group variables in a different order group_by(month, origin)
. What differs in the resulting dataset?
Solution:
by_monthly_origin <- flights %>%
group_by(month, origin) %>%
summarize(count = n())
by_monthly_origin
| month | origin | count |
|---|---|---|
| 1 | EWR | 9893 |
| 1 | JFK | 9161 |
| 1 | LGA | 7950 |
| 2 | EWR | 9107 |
| 2 | JFK | 8421 |
| 2 | LGA | 7423 |
| 3 | EWR | 10420 |
| 3 | JFK | 9697 |
| 3 | LGA | 8717 |
| 4 | EWR | 10531 |
| 4 | JFK | 9218 |
| 4 | LGA | 8581 |
| 5 | EWR | 10592 |
| 5 | JFK | 9397 |
| 5 | LGA | 8807 |
| 6 | EWR | 10175 |
| 6 | JFK | 9472 |
| 6 | LGA | 8596 |
| 7 | EWR | 10475 |
| 7 | JFK | 10023 |
| 7 | LGA | 8927 |
| 8 | EWR | 10359 |
| 8 | JFK | 9983 |
| 8 | LGA | 8985 |
| 9 | EWR | 9550 |
| 9 | JFK | 8908 |
| 9 | LGA | 9116 |
| 10 | EWR | 10104 |
| 10 | JFK | 9143 |
| 10 | LGA | 9642 |
| 11 | EWR | 9707 |
| 11 | JFK | 8710 |
| 11 | LGA | 8851 |
| 12 | EWR | 9922 |
| 12 | JFK | 9146 |
| 12 | LGA | 9067 |
The difference is they are organized/sorted by month
first, then origin
(LC5.8) How could we identify how many flights left each of the three airports for each carrier
?
Solution: We could summarize the count from each airport using the n()
function, which counts rows.
count_flights_by_airport <- flights %>%
group_by(origin, carrier) %>%
summarize(count=n())
count_flights_by_airport
| origin | carrier | count |
|---|---|---|
| EWR | 9E | 1268 |
| EWR | AA | 3487 |
| EWR | AS | 714 |
| EWR | B6 | 6557 |
| EWR | DL | 4342 |
| EWR | EV | 43939 |
| EWR | MQ | 2276 |
| EWR | OO | 6 |
| EWR | UA | 46087 |
| EWR | US | 4405 |
| EWR | VX | 1566 |
| EWR | WN | 6188 |
| JFK | 9E | 14651 |
| JFK | AA | 13783 |
| JFK | B6 | 42076 |
| JFK | DL | 20701 |
| JFK | EV | 1408 |
| JFK | HA | 342 |
| JFK | MQ | 7193 |
| JFK | UA | 4534 |
| JFK | US | 2995 |
| JFK | VX | 3596 |
| LGA | 9E | 2541 |
| LGA | AA | 15459 |
| LGA | B6 | 6002 |
| LGA | DL | 23067 |
| LGA | EV | 8826 |
| LGA | F9 | 685 |
| LGA | FL | 3260 |
| LGA | MQ | 16928 |
| LGA | OO | 26 |
| LGA | UA | 8044 |
| LGA | US | 13136 |
| LGA | WN | 6087 |
| LGA | YV | 601 |
All remarkably similar! Note: the n()
function counts rows, whereas the sum(VARIABLE_NAME)
funciton sums all values of a certain numerical variable VARIABLE_NAME
.
(LC5.9) How does the filter
operation differ from a group_by
followed by a summarize
?
Solution:
filter
picks out rows from the original dataset without modifying them, whereasgroup_by %>% summarize
computes summaries of numerical variables, and hence reports new values.
(LC5.10) What do positive values of the gain
variable in flights
correspond to? What about negative values? And what about a zero value?
Solution:
- Say a flight departed 20 minutes late, i.e.
dep_delay = 20
- Then arrived 10 minutes late, i.e.
arr_delay = 10
. - Then
gain = dep_delay - arr_delay = 20 - 10 = 10
is positive, so it “made up/gained time in the air.” - 0 means the departure and arrival time were the same, so no time was made up in the air. We see in most cases that the
gain
is near 0 minutes. - I never understood this. If the pilot says “we’re going make up time in the air” because of delay by flying faster, why don’t you always just fly faster to begin with?
(LC5.11) Could we create the dep_delay
and arr_delay
columns by simply subtracting dep_time
from sched_dep_time
and similarly for arrivals? Try the code out and explain any differences between the result and what actually appears in flights
.
Solution: No because you can’t do direct arithmetic on times. The difference in time between 12:03 and 11:59 is 4 minutes, but 1203-1159 = 44
(LC5.12) What can we say about the distribution of gain
? Describe it in a few sentences using the plot and the gain_summary
data frame values.
Solution: Most of the time the gain is a little under zero, most of the time the gain is between -50 and 50 minutes. There are some extreme cases however!
(LC5.13) Looking at Figure 5.7, when joining flights
and weather
(or, in other words, matching the hourly weather values with each flight), why do we need to join by all of year
, month
, day
, hour
, and origin
, and not just hour
?
Solution: Because hour
is simply a value between 0 and 23; to identify a specific hour, we need to know which year, month, day and at which airport.
(LC5.14) What surprises you about the top 10 destinations from NYC in 2013?
Solution: This question is subjective! What surprises me is the high number of flights to Boston. Wouldn’t it be easier and quicker to take the train?
(LC5.15) What are some ways to select all three of the dest
, air_time
, and distance
variables from flights
? Give the code showing how to do this in at least three different ways.
Solution:
# The regular way:
flights %>%
select(dest, air_time, distance)
# A tibble: 336,776 x 3
dest air_time distance
<chr> <dbl> <dbl>
1 IAH 227 1400
2 IAH 227 1416
3 MIA 160 1089
4 BQN 183 1576
5 ATL 116 762
6 ORD 150 719
7 FLL 158 1065
8 IAD 53 229
9 MCO 140 944
10 ORD 138 733
# ... with 336,766 more rows
# Since they are sequential columns in the dataset
flights %>%
select(dest:distance)
# A tibble: 336,776 x 3
dest air_time distance
<chr> <dbl> <dbl>
1 IAH 227 1400
2 IAH 227 1416
3 MIA 160 1089
4 BQN 183 1576
5 ATL 116 762
6 ORD 150 719
7 FLL 158 1065
8 IAD 53 229
9 MCO 140 944
10 ORD 138 733
# ... with 336,766 more rows
# Not as effective, by removing everything else
flights %>%
select(-year, -month, -day, -dep_time, -sched_dep_time, -dep_delay, -arr_time,
-sched_arr_time, -arr_delay, -carrier, -flight, -tailnum, -origin,
-hour, -minute, -time_hour)
# A tibble: 336,776 x 6
dest air_time distance gain hours gain_per_hour
<chr> <dbl> <dbl> <dbl> <dbl> <dbl>
1 IAH 227 1400 -9 3.78 -2.38
2 IAH 227 1416 -16 3.78 -4.23
3 MIA 160 1089 -31 2.67 -11.6
4 BQN 183 1576 17 3.05 5.57
5 ATL 116 762 19 1.93 9.83
6 ORD 150 719 -16 2.5 -6.4
7 FLL 158 1065 -24 2.63 -9.11
8 IAD 53 229 11 0.883 12.5
9 MCO 140 944 5 2.33 2.14
10 ORD 138 733 -10 2.3 -4.35
# ... with 336,766 more rows
(LC5.16) How could one use starts_with
, ends_with
, and contains
to select columns from the flights
data frame? Provide three different examples in total: one for starts_with
, one for ends_with
, and one for contains
.
Solution:
# Anything that starts with "d"
flights %>%
select(starts_with("d"))
# A tibble: 336,776 x 5
day dep_time dep_delay dest distance
<int> <int> <dbl> <chr> <dbl>
1 1 517 2 IAH 1400
2 1 533 4 IAH 1416
3 1 542 2 MIA 1089
4 1 544 -1 BQN 1576
5 1 554 -6 ATL 762
6 1 554 -4 ORD 719
7 1 555 -5 FLL 1065
8 1 557 -3 IAD 229
9 1 557 -3 MCO 944
10 1 558 -2 ORD 733
# ... with 336,766 more rows
# Anything related to delays:
flights %>%
select(ends_with("delay"))
# A tibble: 336,776 x 2
dep_delay arr_delay
<dbl> <dbl>
1 2 11
2 4 20
3 2 33
4 -1 -18
5 -6 -25
6 -4 12
7 -5 19
8 -3 -14
9 -3 -8
10 -2 8
# ... with 336,766 more rows
# Anything related to departures:
flights %>%
select(contains("dep"))
# A tibble: 336,776 x 3
dep_time sched_dep_time dep_delay
<int> <int> <dbl>
1 517 515 2
2 533 529 4
3 542 540 2
4 544 545 -1
5 554 600 -6
6 554 558 -4
7 555 600 -5
8 557 600 -3
9 557 600 -3
10 558 600 -2
# ... with 336,766 more rows
(LC5.17) Why might we want to use the select()
function on a data frame?
Solution: To narrow down the data frame, to make it easier to look at. Using View()
for example.
(LC5.18) Create a new data frame that shows the top 5 airports with the largest arrival delays from NYC in 2013.
Solution:
top_five <- flights %>%
group_by(dest) %>%
summarize(avg_delay = mean(arr_delay, na.rm = TRUE)) %>%
arrange(desc(avg_delay)) %>%
top_n(n = 5)
top_five
# A tibble: 5 x 2
dest avg_delay
<chr> <dbl>
1 CAE 41.8
2 TUL 33.7
3 OKC 30.6
4 JAC 28.1
5 TYS 24.1
(LC5.19) Using the datasets included in the nycflights13
package, compute the available seat miles for each airline sorted in descending order. After completing all the necessary data wrangling steps, the resulting data frame should have 16 rows (one for each airline) and 2 columns (airline name and available seat miles). Here are some hints:
- Crucial: Unless you are very confident in what you are doing, it is worthwhile to not starting coding right away, but rather first sketch out on paper all the necessary data wrangling steps not using exact code, but rather high-level pseudocode that is informal yet detailed enough to articulate what you are doing. This way you won’t confuse what you are trying to do (the algorithm) with how you are going to do it (writing
dplyr
code). - Take a close look at all the datasets using the
View()
function:flights
,weather
,planes
,airports
, andairlines
to identify which variables are necessary to compute available seat miles. - Figure 5.7 above showing how the various datasets can be joined will also be useful.
- Consider the data wrangling verbs in Table 5.1 as your toolbox!
Solution: Here are some examples of student-written pseudocode. Based on our own pseudocode, let’s first display the entire solution.
flights %>%
inner_join(planes, by = "tailnum") %>%
select(carrier, seats, distance) %>%
mutate(ASM = seats * distance) %>%
group_by(carrier) %>%
summarize(ASM = sum(ASM, na.rm = TRUE)) %>%
arrange(desc(ASM))
# A tibble: 16 x 2
carrier ASM
<chr> <dbl>
1 UA<PHONE_NUMBER>6
2 DL<PHONE_NUMBER>1
3 B6<PHONE_NUMBER>
4 AA<PHONE_NUMBER>
5 US<PHONE_NUMBER>
6 VX<PHONE_NUMBER>
7 EV<PHONE_NUMBER>
8 WN<PHONE_NUMBER>
9 9E 776970310
10 HA 642478122
11 AS 314104736
12 FL 219628520
13 F9 184832280
14 YV 20163632
15 MQ 7162420
16 OO 1299835
Let’s now break this down step-by-step. To compute the available seat miles for a given flight, we need the distance
variable from the flights
data frame and the seats
variable from the planes
data frame, necessitating a join by the key variable tailnum
as illustrated in Figure 5.7. To keep the resulting data frame easy to view, we’ll select()
only these two variables and carrier
:
flights %>%
inner_join(planes, by = "tailnum") %>%
select(carrier, seats, distance)
# A tibble: 284,170 x 3
carrier seats distance
<chr> <int> <dbl>
1 UA 149 1400
2 UA 149 1416
3 AA 178 1089
4 B6 200 1576
5 DL 178 762
6 UA 191 719
7 B6 200 1065
8 EV 55 229
9 B6 200 944
10 B6 200 1028
# ... with 284,160 more rows
Now for each flight we can compute the available seat miles ASM
by multiplying the number of seats by the distance via a mutate()
:
flights %>%
inner_join(planes, by = "tailnum") %>%
select(carrier, seats, distance) %>%
# Added:
mutate(ASM = seats * distance)
# A tibble: 284,170 x 4
carrier seats distance ASM
<chr> <int> <dbl> <dbl>
1 UA 149 1400 208600
2 UA 149 1416 210984
3 AA 178 1089 193842
4 B6 200 1576 315200
5 DL<PHONE_NUMBER>36
6 UA<PHONE_NUMBER>29
7 B6 200 1065 213000
8 EV 55 229 12595
9 B6<PHONE_NUMBER>00
10 B6 200 1028 205600
# ... with 284,160 more rows
Next we want to sum the ASM
for each carrier. We achieve this by first grouping by carrier
and then summarizing using the sum()
function:
flights %>%
inner_join(planes, by = "tailnum") %>%
select(carrier, seats, distance) %>%
mutate(ASM = seats * distance) %>%
# Added:
group_by(carrier) %>%
summarize(ASM = sum(ASM))
# A tibble: 16 x 2
carrier ASM
<chr> <dbl>
1 9E 776970310
2 AA<PHONE_NUMBER>
3 AS 314104736
4 B6<PHONE_NUMBER>
5 DL<PHONE_NUMBER>1
6 EV<PHONE_NUMBER>
7 F9 184832280
8 FL 219628520
9 HA 642478122
10 MQ 7162420
11 OO 1299835
12 UA<PHONE_NUMBER>6
13 US<PHONE_NUMBER>
14 VX<PHONE_NUMBER>
15 WN<PHONE_NUMBER>
16 YV 20163632
However, because for certain carriers certain flights have missing NA
values, the resulting table also returns NA
’s. We can eliminate these by adding a na.rm = TRUE
argument to sum()
, telling R that we want to remove the NA
’s in the sum. We saw this in Section (summarize):
flights %>%
inner_join(planes, by = "tailnum") %>%
select(carrier, seats, distance) %>%
mutate(ASM = seats * distance) %>%
group_by(carrier) %>%
# Modified:
summarize(ASM = sum(ASM, na.rm = TRUE))
# A tibble: 16 x 2
carrier ASM
<chr> <dbl>
1 9E 776970310
2 AA<PHONE_NUMBER>
3 AS 314104736
4 B6<PHONE_NUMBER>
5 DL<PHONE_NUMBER>1
6 EV<PHONE_NUMBER>
7 F9 184832280
8 FL 219628520
9 HA 642478122
10 MQ 7162420
11 OO 1299835
12 UA<PHONE_NUMBER>6
13 US<PHONE_NUMBER>
14 VX<PHONE_NUMBER>
15 WN<PHONE_NUMBER>
16 YV 20163632
Finally, we arrange()
the data in desc()
ending order of ASM
.
flights %>%
inner_join(planes, by = "tailnum") %>%
select(carrier, seats, distance) %>%
mutate(ASM = seats * distance) %>%
group_by(carrier) %>%
summarize(ASM = sum(ASM, na.rm = TRUE)) %>%
# Added:
arrange(desc(ASM))
# A tibble: 16 x 2
carrier ASM
<chr> <dbl>
1 UA<PHONE_NUMBER>6
2 DL<PHONE_NUMBER>1
3 B6<PHONE_NUMBER>
4 AA<PHONE_NUMBER>
5 US<PHONE_NUMBER>
6 VX<PHONE_NUMBER>
7 EV<PHONE_NUMBER>
8 WN<PHONE_NUMBER>
9 9E 776970310
10 HA 642478122
11 AS 314104736
12 FL 219628520
13 F9 184832280
14 YV 20163632
15 MQ 7162420
16 OO 1299835
While the above data frame is correct, the IATA carrier
code is not always useful. For example, what carrier is WN
? We can address this by joining with the airlines
dataset using carrier
is the key variable. While this step is not absolutely required, it goes a long way to making the table easier to make sense of. It is important to be empathetic with the ultimate consumers of your presented data!
flights %>%
inner_join(planes, by = "tailnum") %>%
select(carrier, seats, distance) %>%
mutate(ASM = seats * distance) %>%
group_by(carrier) %>%
summarize(ASM = sum(ASM, na.rm = TRUE)) %>%
arrange(desc(ASM)) %>%
# Added:
inner_join(airlines, by = "carrier")
# A tibble: 16 x 3
carrier ASM name
<chr> <dbl> <chr>
1 UA<PHONE_NUMBER>6 United Air Lines Inc.
2 DL<PHONE_NUMBER>1 Delta Air Lines Inc.
3 B6<PHONE_NUMBER> JetBlue Airways
4 AA<PHONE_NUMBER> American Airlines Inc.
5 US<PHONE_NUMBER> US Airways Inc.
6 VX<PHONE_NUMBER> Virgin America
7 EV<PHONE_NUMBER> ExpressJet Airlines Inc.
8 WN<PHONE_NUMBER> Southwest Airlines Co.
9 9E 776970310 Endeavor Air Inc.
10 HA 642478122 Hawaiian Airlines Inc.
11 AS 314104736 Alaska Airlines Inc.
12 FL 219628520 AirTran Airways Corporation
13 F9 184832280 Frontier Airlines Inc.
14 YV 20163632 Mesa Airlines Inc.
15 MQ 7162420 Envoy Air
16 OO 1299835 SkyWest Airlines Inc.
D.5 Chapter 6 Solutions
library(ggplot2)
library(dplyr)
library(moderndive)
library(gapminder)
library(skimr)


Secure work from home
With an increasing number of users working from home, there will be an increase in IT attacks targeting home users.
The intention of this site is to provide simple, understandable, best practices for regular people working from home -
temporarily or permanently. Some steps might not apply to you.
All information is provided "as is".
Liability disclaimer.
Your workplace most certainly have an IT policy. You should follow that policy at all times. If you find yourself in a situation you don't how to handle, reach out to your IT department. If any steps outlined in this guide contradicts that policy, follow the policy of your organisation.
If your organisation provide a VPN solution, use it at all times when working from home. It might be slow or block your favourite cute kitten page, but it’s definitely more secure.
Keep your computer updated - don’t postpone updates when prompted, install them as soon as possible. Your employer want you to install them even if it affects your productivity for half an hour. Get a cup of coffee while you wait.
Keep your personal computer updated as well.
Make sure you have a Wi-Fi password consisting of both letters and numbers. Change your routers default login credentials, and install the latest updates for your router. Click here to learn more.
Set up a guest Wi-Fi network and use that for your work computer. Most fairly modern routers have support for a second WiFi network for guests. Guests can not communicate with other devices in your home, and your home devices can not communicate with them. This protects your work computer from potentially infected devices in your home. Here's instructions on how to set up a guest network.
Check both your work and personal email for known password leaks and update your passwords. Don’t use the same password for different services. Use a password manager like 1password (paid) or the free, open source Keepass.
Keep your work related passwords and private passwords separated. Ask your organisations IT department about company guidelines regarding passwords and password managers.
Use Two-factor authentication (2FA) whenever a service provides it.
Keep yourself up to date with your organisation. Communicate with your colleagues daily. You might not need them but they might need you.
Be extra careful before you open any files and links emailed to you. If you get a suspicious email from a known contact, reach out to them either by phone or by creating a new email (do not respond on the one you received) and ask them if they sent the email and what it is.
Watch out for spear phishing! Spear phishing is highly sophisticated attacks which target you. Spear phishing occur by email, social media, phone and regular mail. Spear phishing attacks often try to make you act out of stress. Read, breathe and think before acting! Your employer puts their trust in you. Don’t be an easy target.
Don’t use your work computer for private matters. Reading the news is one thing, while logging in to social networks and other services opens up for potential attack vectors. Follow your organisations IT guidelines.
Don’t use your personal computer for work unless you're working at a bring-your-own-device (BYOD) workplace. If you really, really need to print just one work document using your personal computer, you still do not do it. Do not use USB memory sticks or USB hard drives unless instructed to by your IT department.
It might seem like a good idea to let your kids watch Netflix or browse the internet on your work computer at times. However, this puts your company's data at risk. Lock your computer when you leave the room.
Don’t print out sensitive information unless you really need to, even if you are allowed to. Keep your work related assets in one place and don’t scatter it around your home.
Troy Hunt, who created haveibeenpwned.com, have a great collection of security related links here.


Learning Vector Quantization 🐧✌
START
Random
Circle
ExclusiveOr
Gaussian
Spiral
3-Circle
Browser not supported for Canvas. Get a real browser.


You run tests more than once
It makes sense to automate anything you repeat 10+ times.
Taurus improves experience of JMeter, Selenium and others.
Step 1: Install It
pip install bzt
More about installing it...
Step 2: Create a Config
More about config syntax...
Step 3: Run The Test
More about running it...
Step 4: Go Deeper
Read from the docs how to get
interactive reports,
create
complex scenarios,
set
failure criteria.
We also have a
knowledge base with articles and videos for easy learning.
Feel free to ask questions on our
support forums.


Winter storm warning for SE Wyoming, NE Colorado, W Nebraska
DENVER (AP) - The National Weather Service has issued winter storm warnings for northeastern Colorado, southeastern Wyoming and western Nebraska, saying heavy snow in the region could hamper travel during Thanksgiving week.
The weather service says storms Monday through Tuesday afternoon could bring about 15 inches (38 centimeters) of snow to Cheyenne, 11 inches (28 centimeters) of snow to Denver and similar amounts to western Nebraska.
Meteorologist Kyle Fredin said Sunday the Denver area is forecast to see 6 to 8 hours of heavy snowfall starting late Monday that will cause significant travel impacts and possibly temporary road closures.
The agency says there’s a chance travel could become nearly impossible Monday night and Tuesday morning across the Interstate 25 urban corridor in northeastern Colorado and into Wyoming.
United Airlines announced Sunday it would waive the change fee for re-booking flights scheduled to leave Denver International Airport on Tuesday.
Sign up for Daily Newsletters
Copyright © 2019 The Washington Times, LLC.
Please read our comment policy before commenting.


Notebooks
Story: Story of |kua ka khumm
Story of |kua ka khumm
Title
Story of |kua ka khumm
Collection
Summary
... a cunning woman of this [Early] race, who lived alone in a house underground, deceived and robbed /kwámmana [in this story he is called |kua ka khumm]. This was told in the Katkop dialect by !kwéiten ta //kën (L VI.-2. 4034-4053). An old woman asks |kua ka khumm to build her a shelter, even though she already has one under the ground. She takes his things and disappears into her house.
Comments
1) Lloyd took down this story twice as she did not understand it properly the first time she took it down (see p.4042v); the two versions are included together here, 2) p.4033v: the location of the old woman's house in the pan; the story was told to !kweiten ta ||ken by her mother (≠kamme-an); Dia!kwain says that |kua is short for |kwammana, 3) p.4036v: |kua's things come of their own accord when he summons them, 4) This story is found in Book VI-2
Contributors
!kweiten ta ||ken (Rachel) (VI)
Date
6 January 1875 (started)
Categories
History (Early Race), |kaggen (the Mantis)
Keywords
old woman (of Early Race) , old woman (who lives on a pan) , old woman (her house) , old woman (is cunning) , old woman (and |kua ka khumm) , shelter (the making of) , shelter (a house of bushes) , shelter (for old woman on pan) , shelter (and |kua ka khumm's story) , |kua ka khumm (or |kwammana) , |kua ka khumm (does not understand) , |kua ka khumm (is foolish) , |kua ka khumm (is advised) , |kua ka khumm (commands his things) , |kua ka khumm (the old woman takes his things) , |kua ka khumm (his story) , |kua ka khumm (and old woman who lives on a pan) , |kua ka khumm (calls for his things) , |kua ka khumm (is deceived by old woman) , Early Race (the story of |kua ka khumm) , Early Race (the First Bushmen or !khwe |na ss'e !k'e) , house (or shelter) , house (making of) , house (of the old woman who lived on a pan) , house (and |kua ka khumm's story) , |kwammana (called |kua ka khumm) , |kwammana (and |kua ka khumm's story) , |kwammana (is deceived by the old woman who lives on a pan) , |kwammana (another name for) , name (another for |kwammana) , name (|kwammana called |kua ka khumm)
Story Pages
4034-4053


| 1275 | (A) | he wexe a Marchant amys when he that Monaye tuke |
| 1269 | (C) | He wax a marchaunt a-mysse . whenne he þe moneye toke |
| 1263 | (D) | he was a marchaunt a-mys whan he þat money fongid |
| 1239 | (E) | he was a marchaunt amys . whan he mony fongyd |
| (Ex) | ---- no text ---- | |
| 1305 | (L) | he wexe marchaunte amys þat þe money fenged |
| (P) | ---- no text ---- | |
| 1246 | (U) | he was marchant amysse / that the moneye fonge |
| 340 | (V) | he wax marchaunt of mysse . whanne he money fonged |


This package contains components that exert forces and torques
between two frame connectors, e.g., between two parts.
| Model |
Description |
| WorldForce |
External force acting at the frame to which this component is
connected and defined by 3 input signals, that are interpreted as
one vector resolved in frame world, frame_b or frame_resolve.
|
| WorldTorque |
External torque acting at the frame to which this component is
connected and defined by 3 input signals, that are interpreted as
one vector resolved in frame world, frame_b or frame_resolve.
|
|
WorldForceAndTorque |
External force and external torque acting at the frame to which
this component is connected and defined by 3+3 input signals, that
are interpreted as a force and as a torque vector resolved in frame
world, frame_b or frame_resolve.
|
| Force |
Force acting between two frames defined by 3 input signals
resolved in frame world, frame_a, frame_b or in
frame_resolve.
|
| Torque |
Torque acting between two frames defined by 3 input signals
resolved in frame world, frame_a, frame_b or in
frame_resolve.
|
| ForceAndTorque |
Force and torque acting between two frames defined by 3+3 input
signals resolved in frame world, frame_a, frame_b or in
frame_resolve.
|
| LineForceWithMass |
General line force component with an optional point mass on the
connection line. The force law can be defined by a component of
Modelica.Mechanics.Translational
|
|
LineForceWithTwoMasses |
General line force component with two optional point masses on
the connection line. The force law can be defined by a component of
Modelica.Mechanics.Translational
|
| Spring |
Linear translational spring with optional mass
|
| Damper |
Linear (velocity dependent) damper
|
|
SpringDamperParallel |
Linear spring and damper in parallel connection |
|
SpringDamperSeries |
Linear spring and damper in series connection |
| Name |
Description |
| WorldForce |
External force acting at frame_b, defined by 3 input signals
and resolved in frame world, frame_b or frame_resolve |
| WorldTorque |
External torque acting at frame_b, defined by 3 input signals
and resolved in frame world, frame_b or frame_resolve |
| WorldForceAndTorque |
External force and torque acting at frame_b, defined by 3+3
input signals and resolved in frame world, frame_b or in
frame_resolve |
| Force |
Force acting between two frames, defined by 3 input signals and
resolved in frame world, frame_a, frame_b or frame_resolve |
| Torque |
Torque acting between two frames, defined by 3 input signals
and resolved in frame world, frame_a, frame_b or frame_resolve |
| ForceAndTorque |
Force and torque acting between two frames, defined by 3+3
input signals and resolved in frame world, frame_a, frame_b or
frame_resolve |
| LineForceWithMass |
General line force component with an optional point mass on the
connection line |
| LineForceWithTwoMasses |
General line force component with two optional point masses on
the connection line |
| Spring |
Linear translational spring with optional mass |
| Damper |
Linear (velocity dependent) damper |
| SpringDamperParallel |
Linear spring and linear damper in parallel |
| SpringDamperSeries |
Linear spring and linear damper in series connection |
| Internal |
Internal package, should not be used by user |


If you are using a default build environment as documented in the Google APIs Client Library for C++ Installation Guide then:
service_apis
directory in
the build root.
make
from the root directory
To use the specialized service library, consumers may either:
INCLUDE
path.
CMakeLists.txt
file, add the following:INCLUDE_DIRECTORIES(${GOOGLEAPIS_SERVICE_REPOSITORY_DIR}/{{ api.module.name })
target_link_libraries
for your
build target my_build_target:target_link_libraries(my_build_target
#include "/.h"
The provided CMakeLists.txt
file is a CMake file compatible with
those in the Google APIs Client Library for C++ SDK.
See Getting Started with the Google APIs Client Library for C++ for more information on how to use these service API libraries.


Enter your email address :
Enter your password :
First enter your WebEx site URL.
We also found a site that requires you to
sign in through your corporate website


When I began to study Neural Networks during my coursework, I realized the complexity involved in representing those large layers of neurons in code. But when I read about Keras and started experimenting with it, everything was super easy to understand and perfectly made sense.
To all those who want to actually write some code to build a Deep Neural Network, but don’t know where to begin, I highly suggest you to visit Keras website as well as it’s github page.
Update: As Python2 faces end of life, the below code only supports Python3.
In this post, we will learn the simple 6 steps with which we can create our first deep neural network using Keras and Python. Let’s get started!
1
2
3
4
5
6
7
* How to setup environment to create Deep Neural Nets?
* How to analyze and understand a training dataset?
* How to load and work with .csv file in Python?
* How to work with Keras, NumPy and Python for Deep Learning?
* How to create a Deep Neural Net in less than 5 minutes?
* How to split training and testing data for evaluating our model?
* How to display metrics to analyze performance of our model?
Deep Learning Environment Setup
Before getting into concept and code, we need some libraries to get started with Deep Learning in Python. Copy and paste the below commands line-by-line to install all the dependencies needed for Deep Learning using Keras in Linux. I used Ubuntu 14.04 LTS 64-bit architecture as my OS and I didn’t use any GPU to speed up computations for this tutorial.
1
2
3
4
5
6
7
8
9
10
sudo pip install numpy
sudo pip install scipy
sudo pip install matplotlib
sudo pip install seaborn
sudo pip install scikit-learn
sudo pip install pillow
sudo pip install h5py
sudo pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git
sudo pip install tensorflow
sudo pip install keras
Keras needs Theano or TensorFlow as its backend to perform numerical computations. TensorFlow won’t work in Ubuntu 32-bit architecture (at the time of writing this tutorial). So, better have a 64-bit OS to properly install Keras with both TensorFlow and Theano backend.
If you need to work with Deep Learning on a Windows machine, please visit my post on environment setup for Windows here.
Deep Learning flowchart
Let’s talk about the classic Supervised Learning problem.
Supervised Learning means we have a set of training data along with its outcomes (classes or real values). We train a deep learning model with the training data so that the model will be in a position to predict the outcome (class or real value) of future unseen data (or test data).
This problem has two sub-divisions namely Classification and Regression.
- Classification - If the output variable to be predicted by our model is a label or a category, then it is a Classification problem. Ex: Predicting the name of a flower species.
- Regression - If the output variable to be predicted by our model is a real or continuous value (integer, float), then it is a Regression problem. Ex: Predicting the stock price of a company.
We will concentrate on a Supervised Learning Classification problem and learn how to implement a Deep Neural Network in code using Keras.
Note: To learn more about Deep Learning theory, I highly suggest you to register in Andrew NG's machine learning course and deep learning course at Coursera or visit Stanford University's awesome website.
Download the below image for your future reference. Any complex problem related to Supervised Learning can be solved using this flowchart. We will walk through each step one-by-one in detail.
Analyse the Dataset
Deep Learning is all about Data and Computational Power.
You read it right! The first step in any Deep Learning problem is to collect more data to work with, analyse the data thoroughly and understand the various parameters/attributes in it. Attributes (also called as features) to look at in any dataset are as follows.
- Dataset characteristics - Multivariate, Univariate, Sequential, Time-series, Text.
- Attribute characteristics - Real, Integers, Strings, Float, Binary.
- Number of Instances - Total number of rows.
- Number of Attributes - Total number of columns.
- Associated Tasks - Classification, Regression, Clustering, Other problem.
For this tutorial, we will use the Pima Indian Diabetes Dataset from Kaggle. This dataset contains the patient medical record data for Pima Indians and tell us whether they had an onset of diabetes within 5 years or not (last column in the dataset). It is typically a binary classification problem where
- 1 = yes! the patient had an onset of diabetes in 5 years.
- 0 = no! the patient had no onset of diabetes in 5 years.
In this dataset, there are 8 attributes (i.e 8 columns) that describes each patient (i.e a single row) and a total of 768 instances (i.e total number of rows or number of patients).
Go to this link, register/login, download the dataset, save it inside a folder named pima-indians-diabetes and rename it as dataset.csv. Below is the folder structure to follow.
1
2
3
|--pima-indians-diabetes
|--|--dataset.csv
|--|--train.py
Prepare the dataset
Data must be represented in a structured way for computers to understand.
Representing our analyzed data is the next step to do in Deep Learning. Data will be represented as an n-dimensional matrix in most of the cases (whether it is numerical or images or videos). Some of the common file-formats to store matrices are csv, cPickle and h5py. If you have millions of data (say millions of images), h5py is the file-format of choice. We will stick with CSV file-format in this tutorial.
CSV (Comma Separated Values) file formats can easily be loaded in Python in two ways. One using NumPy and other using Pandas. We will use NumPy.
Below is the code to import all the necessary libraries and load the pima-indians diabetes dataset.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# organize imports
from keras.models import Sequential
from keras.models import Dense
from sklearn.model_selection import train_test_split
import numpy as np
# seed for reproducing same results
seed = 9
np.random.seed(seed)
# load pima indians dataset
dataset = np.loadtxt('dataset.csv', delimiter=',', skiprows=1)
# split into input and output variables
X = dataset[:,0:8]
Y = dataset[:,8]
# split the data into training (67%) and testing (33%)
(X_train, X_test, Y_train, Y_test) = train_test_split(X, Y, test_size=0.33, random_state=seed)
- Line (1-3) handles the imports to build our deep learning model using Keras.
- Line (6-7) fix a seed for reproducing same results if we wish to train and evaluate our network more than once.
- Line (10) loads the dataset from the .csv file saved on disk. It uses an argument delimiter to inform NumPy that all attributes are separated by , in the .csv file.
- Line (13-14) splits the dataset into input and output variables.
- X has all the 8 attributes of the patients
- Y has whether they have an onset of diabetes or not.
- Line (17) splits the dataset into training (67%) and testing (33%) using the scikit-learn’s train_test_split function. By performing this split, we can easily verify the performance of our model in an unseen test dataset.
Create the Model
In Keras, a model is created using Sequential. You may wanna recall that Neural Networks holds large number of neurons residing inside several sequential layers.
We will create a model that has fully connected layers, which means all the neurons are connected from one layer to its next layer. This is achieved in Keras with the help of Dense function.
We will use the above Deep Neural Network architecture which has a single input layer, 2 hidden layers and a single output layer.
The input data which is of size 8 is sent to the first hidden layer that has randomly initialized 8 neurons. This is a very useful approach, if we don’t have any clue about the no.of.neurons to specify at the very first attempt. From here, we can easily perform trial-and-error procedure to increase the network architecture to produce good results. The next hidden layer has 6 neurons and the final output layer has 1 neuron that outputs whether the patient has an onset of diabetes or not.
Figure 4 shows our Deep Neural Network with an input layer, a hidden layer having 8 neurons, a hidden layer having 6 neurons and an output layer with a single neuron.
Below are the four lines of code to create the above architecture.
1
2
3
4
5
# create the model
model = Sequential()
model.add(Dense(8, input_dim=8, init='uniform', activation='relu'))
model.add(Dense(6, init='uniform', activation='relu'))
model.add(Dense(1, init='uniform', activation='sigmoid'))
- Line (1) creates a Sequential model to add layers one at a time.
- Line (2), the first layer expects four arguments:
- 8: No.of.neurons present in that layer.
- input_dim: specify the dimension of the input data.
- init: specify whether uniform or normal distribution of weights to be initialized.
- activation: specify whether relu or sigmoid or tanh activation function to be used for each neuron in that layer.
- Line (3), the next hidden layer has 6 neurons with an uniform initialization of weights and relu activation function
- Line (4), the output layer has only one neuron as this is a binary classification problem. The activation function at output is sigmoid because it outputs a probability in the range 0 and 1 so that we could easily discriminate output by assigning a threshold.
Compile the model
After creating the model, three parameters are needed to compile the model in Keras.
- loss: This is used to evaluate a set of weights. It is needed to reduce the error between actual output and expected output. It could be binary_crossentropy or categorical_crossentropy depending on the problem. As we are dealing with a binary classification problem, we need to pick binary_crossentropy. Here is the list of loss functions available in Keras.
- optimizer: This is used to search through different weights for the network. It could be adam or rmsprop depending on the problem. Here is the list of optimizers available in Keras.
- metrics: This is used to collect the report during training. Normally, we pick accuracy as our performance metric. Here is the list of metrics available in Keras.
These parameters are to be tuned according to the problem as our model needs some optimization in the background (which is taken care by Theano or TensorFlow) so that it learns from the data during each epoch (which means reducing the error between actual output and predicted output).
epoch is the term used to denote the number of iterations involved during the training process of a neural network.
1
2
# compile the model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
- Line (2) chooses a binary_crossentropy loss function and the famous Stochastic Gradient Descent (SGD) optimizer adam. It also collects the accuracy metric for training.
Fit the model
After compiling the model, the dataset must be fitted with the model. The fit() function in Keras expects five arguments -
- X_train: the input training data.
- Y_train: the output training classes.
- validation_data: Tuple of testing or validation data used to check the performance of our network.
- nb_epoch: how much iterations should the training process take place.
- batch_size: No.of.instances that are evaluated before performing a weight update in the network.
1
2
# fit the model
model.fit(X_train, Y_train, validation_data=(X_test, Y_test), nb_epoch=100, batch_size=5)
- Line (2) chooses 100 iterations to be performed by the deep neural network with a batch_size of 5.
Evaluate the model
After fitting the dataset to the model, the model needs to be evaluated. Evaluating the trained model with an unseen test dataset shows how our model predicts output on unseen data. The evaluate() function in Keras expects two arguments.
- X - the input data.
- Y - the output data.
1
2
3
# evaluate the model
scores = model.evaluate(X_test, Y_test)
print("Accuracy: %.2f%%" %(scores[1]*100))
Complete code
Excluding comments and empty new lines, we have combined all the 6 steps together and created our first Deep Neural Network model using 18 lines of code. Below is the entire code of the model we have just built.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# organize imports
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import train_test_split
import numpy as np
# seed for reproducing same results
seed = 9
np.random.seed(seed)
# load pima indians dataset
dataset = np.loadtxt('dataset.csv', delimiter=',', skiprows=1)
# split into input and output variables
X = dataset[:,0:8]
Y = dataset[:,8]
# split the data into training (67%) and testing (33%)
(X_train, X_test, Y_train, Y_test) = train_test_split(X, Y, test_size=0.33, random_state=seed)
# create the model
model = Sequential()
model.add(Dense(8, input_dim=8, init='uniform', activation='relu'))
model.add(Dense(6, init='uniform', activation='relu'))
model.add(Dense(1, init='uniform', activation='sigmoid'))
# compile the model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# fit the model
model.fit(X_train, Y_train, validation_data=(X_test, Y_test), nb_epoch=200, batch_size=5, verbose=0)
# evaluate the model
scores = model.evaluate(X_test, Y_test)
print("Accuracy: %.2f%%" % (scores[1]*100))
- Save the code with a filename train.py in the same folder as the dataset.
- Open up a command prompt and go to that folder.
- Type python train.py
I got an accuracy of 72.83% which is pretty good in our first try without any hyperparameter tuning or changing the network architecture.
Kindly post your results in the comments after tuning around with nb_epoch, batch_size, activation, init, optimizer and loss. You can also increase the number of hidden layers and the number of neurons in each hidden layer.
Summary
Thus, we have built our first Deep Neural Network (Multi-layer Perceptron) using Keras and Python in a matter of minutes. Note that we haven’t even touched any math involved behind these Deep Neural Networks as it needs a separate post to understand. We have strictly focused on how to solve a supervised learning classification problem using Keras and Python.
In case if you found something useful to add to this article or you found a bug in the code or would like to improve some points mentioned, feel free to write it down in the comments. Hope you found something useful here.


Take the security of your data into your own hands
Cryptomator is a simple tool for digital self-defense. It allows you to protect your cloud data by yourself and independently.
Cryptomator is a simple tool for digital self-defense. It allows you to protect your cloud data by yourself and independently.
Cryptomator's technology meets the latest standards and encrypts both files and filenames with AES and 256 bit key length.
You don't have to trust Cryptomator blindly, because it is open source software. For you as a user, this means that everyone can see the code.
In addition to independent security audits, the software is continuously and publicly tested in an automated way and has a measurable code quality and test coverage that is far above the industry average.
When using Cryptomator and a cloud service with an appropriate DPA, you can synchronize data GDPR-compliant over the cloud with an entire team.


Bleeding Edge Web: June 2017
Housekeeping
News From the Bleeding Edge
— Corey Butler (@goldglovecb)
Node & Arduino: A fun Introduction
— Noah Glaser
Additional Announcements
Housekeeping
Parking Validation ($5 instead of $30)
Buffalo Billiards After-Meetup
Thanks to our food & drink sponsor:
News from the
Bleeding Edge
Browser Pop Quiz
Last check — May 24, 2017:
Browser Pop Quiz
As of June 28, 2017:
Chrome 59
Released June 20
Chrome 59 Devtools: Code Coverage
Chrome 59 Devtools: Screenshots
Chrome 59 Devtools: Blocking
Firefox 54
Released June 13
- Date.UTC() conforms to ES 2017
- WebRTC TCP ICE candidate support enabled by default.
Firefox 54 Devtools: Requests
- Now includes data transfer size
Edge 15 (16215)
once
and passive
Event Listener support
position: sticky;
support!
- ES2017 Shared Memory & Atomics support
Libraries, Tools &
Frameworks
Framework News
- V8 6.0 (rest/spread properties on objects)
- N-API for Node (experimental)
- npm 5
- ES Lint v4
- Heads Up: Angular 5 due Sept 19, 2017
More Perspective
Parse Times: 1MB of JS (courtesy FluentConf)
Pitch competition for AI, ML, Comp Vision, AR/VR.
Thanks!
←
→
/
#


Education
Achievements
Skills
- Programming languages
Python, Java, C, MATLAB, SQL, R, JavaScript, Bash
- Web development
HTML, CSS, JQuery, Spring, Flask, Pyramid
- Databases
Postgres, MySQL
- Technical
Machine learning (TensorFlow), statistics (R and Python), embedded system design
- General
Git, subversion, LaTeX, Linux, software best practices, testing frameworks (pytest, JUnit), encryption methods, documentation
- Languages
English (native), German (intermediate)
Extracurricular
Hobbies


Next Show:
{{nextShow[0].date_text}}
@ {{nextShow[0].venue}}

open_in_new Launch New Game
-OR-
games Join Game
Game {{game.num}} ({{game.date_abrev}})
About 5 for 5
Game: Pick 5 songs you think Phish will play tonight.
Scoring is realtime (hopefully) during a show. Highlighted songs have been played.
Directions: Start a new game, make your picks, and then send the link your friends. Use this button on the picks page to copy the link.
Games can be created 24 hrs before the show starts.
Win: Picking rare songs scores more points. Score the highest and try to go 5 for 5!Disclaimer: Not affiliated with Phish or their properties. No personal info is being collected. Google Analytics used for Anonymous Usage Statistics. No revenue is being generated by the content of this site. Whoooooooo-roch Inc. 2016


http://www.theprojectspot.com/tutorial-post/introduction-to-artificial-neural-networks-part-1/7
percepton training visualization (initial version provided as a courtesy of Jed Borovik)
Interactive classifier using deep neural network
layer_defs = [
{type:'input', out_sx:1, out_sy:1, out_depth:2},
{type:'fc', num_neurons:6, activation: 'tanh'},
{type:'fc', num_neurons:2, activation: 'tanh'},
{type:'softmax', num_classes:2}];
net = new convnetjs.Net();
net.makeLayers(layer_defs);
trainer = new convnetjs.Trainer(net);
var point = new convnetjs.Vol(1,1,2); // needs to match input layer
point.w = [3.0, 4.0];
var prediction = net.forward(point);
// probability of classes in .w
if(prediction.w[0] > prediction.w[1]) // red;
else // green;
Predictions will also be painted as background colors
var data = [[-0.4326, 1.1909], [3.0, 4.0], [1.8133, 1.0139 ]];
var labels = [1, 1, 0];
var N = labels.length;
var avloss = 0.0;
for (var iter=0; iter < 20; iter++) {
for (var ix=0; ix < N; ix++) {
var point = new convnetjs.Vol(1,1,2);
var label = labels[ix];
point.w = data[ix];
var stats = trainer.train(point, label);
avloss += stats.loss;
}
}
// make this as small as possible
avloss /= N*iters;
Convnetjs uses stochastic gradient descent by default
Just one hidden layer, number of perceptrons variable
1, 2, 3, 5
Does not classify, but tries to find a coninuous function that goes through all data points
Quick Quiz: How many neurons in hidden layer?
layer_defs = [
{type:'input', out_sx:1, out_sy:1, out_depth:1},
{type:'fc', num_neurons:5, activation:'sigmoid'},
{type:'regression', num_neurons:1}];
uses convolutional hidden layers for filtering
Tiny images in 10 classes, 6000 per class in training set
Udacity Course 730, Deep Learning (L3 Convolutional Neural Networks > Convolutional Networks)
layer_defs = [
{type:'input', out_sx:32, out_sy:32, out_depth:3},
{type:'conv', sx:5, filters:16, stride:1, pad:2, activation:'relu'},
{type:'pool', sx:2, stride:2}),
{type:'conv', sx:5, filters:20, stride:1, pad:2, activation:'relu'}.
{type:'pool', sx:2, stride:2},
{type:'conv', sx:5, filters:20, stride:1, pad:2, activation:'relu'},
{type:'pool', sx:2, stride:2},
{type:'softmax', num_classes:10}];
// 16 5x5 filters will be convolved with the input
// output again 32x32 as stride (step width) is 1
// 2 pixels on each edge will be padded with 0 matching 5x5 pixel filter
{type:'conv', sx:5, filters:16, stride:1, pad:2, activation:'relu'}
convolutional layers remove noise, add semantics
// perform max pooling in 2x2 non-overlapping neighborhoods
// output will be 16x16 as stride (step width) is 2
{type:'pool', sx:2, stride:2}),
pooling layers in between provide translation invariance
{type:'softmax', num_classes:10}
assigns a probability to each of the 10 categories
By AI strategy (gray)
http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html


--- layout: default title: Designing Your Design Process permalink: /events/2014/design-process/ --- {% assign currentEvent = page.title %}


{% extends 'base.html' %} {% block extra_head %}
If you don't want to receive e-mail messages to {{ email }} ever again, confirm by clicking the Never email me again button. If anyone tries to use this email address for a sender or recipient, we will silently discard the email.
You might still receive Happiness Packets that were already sent before you blocked your email address.
{% endblock content %}


Trump to host vaping meeting on Friday, White House says
U.S. President Donald Trump will host a meeting on vaping at the White House on Friday with a range of groups, including industry and public health representatives, a White House spokesman said on Wednesday.
WASHINGTON: U.S. President Donald Trump will host a meeting on vaping at the White House on Friday with a range of groups, including industry and public health representatives, a White House spokesman said on Wednesday.
"The policy making process is not stalled," White House spokesman Judd Deere said in a statement.
(Reporting by Jeff Mason; Writing by Susan Heavey; Editing by Chris Reese)


Look at this ship's cook prepare a meal in heavy seas
by Mark FrauenfelderThe Avontuur's cook, Lasse Scharz, is accustomed to making food under less-than-ideal conditions. In this video, the camera is fixed to the ship, so we don't see it rock. Instead we see Scharz comically leaning as plates fly off the counter as if enchanted.
Image: YouTube


| pgpool-II 4.0.3 Documentation | |||
|---|---|---|---|
| Prev | Up | Chapter 6. Client Authentication | Next |
The following subsections describe the authentication methods in more detail.
When trust authentication is specified, Pgpool-II assumes that anyone who can connect to the server is authorized to access connect with whatever database user name they specify.
This authentication method is the password-based authentication methods in which MD-5-hashed password is sent by client. Since Pgpool-II does not has the visibility of PostgreSQL's database user password and client application only sends the MD5-hash of the password, so md5 authentication in Pgpool-II is supported using the pool_passwd authentication file.
Note: If Pgpool-II is operated in raw mode or there's only 1 backend configured, you don't need to setup pool_passwd.
To use the md5 authentication pool_passwd authentication file must contain the user password in either plain text md5 or AES encrypted format.
The pool_passwd file should contain lines in the following format:
"username:plain_text_passwd"
"username:encrypted_passwd"
here are the steps to enable md5 authentication:
1- Login as the database's operating system user and type "pg_md5 --md5auth --username=username password" user name and md5 encrypted password are registered into pool_passwd. If pool_passwd does not exist yet, pg_md5 command will automatically create it for you.
Note: user name and password must be identical to those registered in PostgreSQL server.
2- Add an appropriate md5 entry to pool_hba.conf. See Section 6.1 for more details.
3- After changing md5 password (in both pool_passwd and PostgreSQL of course), reload the pgpool configurations.
This authentication method also known as SCRAM is a challenge-response based authentication that prevents the password sniffing on untrusted connections. Since Pgpool-II does not has the visibility of PostgreSQL's database user password, so SCRAM authentication is supported using the pool_passwd authentication file.
To use the SCRAM authentication pool_passwd authentication file must contain the user password in either plain text or AES encrypted format.
"username:plain_text_passwd"
"username:AES_encrypted_passwd"
Note: md5 type user passwords in pool_passwd file can't be used for scram authentication
Here are the steps to enable scram-sha-256 authentication:
1- Create pool_passwd file entry for database user and password in plain text or AES encrypted format. The pg_enc utility that comes with Pgpool-II can be used to create the AES encrypted password entries in the pool_passwd file.
Note: User name and password must be identical to those registered in the PostgreSQL server.
2- Add an appropriate scram-sha-256 entry to pool_hba.conf. See Section 6.1 for more details.
3- After changing SCRAM password (in both pool_passwd and PostgreSQL of course), reload the Pgpool-II configuration.
This authentication method uses SSL client certificates to perform authentication. It is therefore only available for SSL connections. When using this authentication method, the Pgpool-II will require that the client provide a valid certificate. No password prompt will be sent to the client. The cn (Common Name) attribute of the certificate will be compared to the requested database user name, and if they match the login will be allowed.
Note: The certificate authentication works between client and Pgpool-II, for the backend authentication you can use any other authentication method
This authentication method uses PAM (Pluggable Authentication Modules) as the authentication mechanism. The default PAM service name is pgpool. PAM authentication is supported using user information on the host where Pgpool-II is executed. For more information about PAM, please read the Linux-PAM Page.
To enable PAM authentication, you need to create a service-configuration file for Pgpool-II in the system's PAM configuration directory (which is usually at "/etc/pam.d"). A sample service-configuration file is installed as "share/pgpool-II/pgpool.pam" under the install directory.
Note: To enable PAM support the Pgpool-II must be configured with "--with-pam"


{%extends "crowdcop_web/error_template.html"%} {%block title%}404- Page Not Found{%endblock%} {%block img%} {%endblock%} {%block error-title%}Oops!{%endblock%} {%block error-msg%}
*Only approved tips will contribute to new or existing campaigns.
{%endblock%}


Today’s local and national sports
by Las Vegas Review-Journal
Here are all of the sports, locally and nationally, on television and radio today.
SUNDAY
BASKETBALL: COLLEGE
Marquette at Wisconsin 10 a.m. FS1
New Mexico State at Arizona 11 a.m. PAC12
Florida at Connecticut noon ESPN
Women, Valparaiso at North Dakota noon FCSC
Women, Northwestern at Duke noon ACCN
Women, Arizona State at Minnesota 1 p.m. BTN
Seton Hall at Saint Louis 1 p.m. ESPNU
Idaho State at Washington State 1 p.m. PAC12
WNIT final, Missouri State at Oregon State 1:30 p.m. CBSSN
North Carolina Central at Louisville 2 p.m. ACCN
Wake Forest at UNC-Charlotte 3 p.m. ESPNU
Rider at Arizona State 3 p.m. PAC12
Georgia State at Georgetown 4:30 p.m. FS1
Texas-Arlington at Oregon 5 p.m. PAC12
BASKETBALL: NBA G LEAGUE
Raptors 905 at Maine 10 a.m. NBAtv
BASKETBALL: NBA
Atlanta at L.A. Lakers 6:30 p.m. SPTSN
FOOTBALL: NFL
Dallas at Detroit 10 a.m. KVVU-5
Houston at Baltimore 10 a.m. KLAS-8
Jacksonville at Indianapolis 10 a.m. KWWN (1100 AM, 100.9 FM)
Cincinnati at Oakland 1:25 p.m. KLAS-8, KYMT-FM (93.1)
Chicago at L.A. Rams 5:20 p.m. KSNV-3, KBAD-AM (920), KWWN (1100 AM, 100.9 FM)
FOOTBALL: CANADIAN LEAGUE CONFERENCE FINALS
Edmonton at Hamilton 10 a.m. ESPNews
Winnipeg at Saskatchewan 1:30 p.m. ESPN2
GOLF
PGA, Mayakoba Classic 11 a.m. TGC
HOCKEY: WESTERN LEAGUE
Moose Jaw at Saskatoon 12:30 p.m. NHLN
HOCKEY: NHL
Calgary at Vegas 4 p.m. ATTSN, NHLN, KRLV (1340 AM, 98.9 FM)
HORSE RACING
Quarters: Los Alamitos 5 p.m. TVG
MOTOR SPORTS
F1, Brazilian Grand Prix 9:05 a.m. ESPN2
NASCAR Cup, Ford EcoBoost 400 noon KSNV-3, KBAD-AM (920)
NHRA, Auto Club Finals 1 p.m. FS1
RUGBY: EUROPEAN CHAMPIONS CUP
Lyon at Northampton 5 a.m. NBCSN
Saracens at Racing 92 7 a.m. NBCSN
SOCCER: UEFA EURO 2020 QUALIFIER
Portugal at Luxembourg 5:50 a.m. ESPNews
SOCCER: COLLEGE CONFERENCE CHAMPIONSHIPS
ACC, Virginia vs. Clemson 9 a.m. ESPNU
Big East, Providence at Georgetown 9 a.m. FS2
Big Ten, Michigan vs. Indiana 11 a.m. BTN
SOCCER: FIFA U-17 WORLD CUP CHAMPIONSHIP
Mexico vs. Brazil 1:50 p.m. FS2
SOCCER: USL CHAMPIONSHIP
Real Monarchs at Louisville City FC 4:30 p.m. ESPN2
TENNIS
ATP Finals doubles final 7:30 a.m. TTC
USTA, JSM Challenger singles final 9:30 a.m. TTC
ATP Finals singles final 10 a.m. ESPN
WRESTLING: COLLEGE
Virginia Tech at Ohio State 9 a.m. BTN
LOCALLY
HOCKEY
Golden Knights
vs. Calgary Flames, T-Mobile Arena, 4 p.m. Sunday; vs. Toronto Maple Leafs, T-Mobile Arena, 7 p.m. Tuesday; vs. San Jose Sharks, T-Mobile Arena, 7 p.m. Thursday; vs. Edmonton Oilers, T-Mobile Arena, 7 p.m. Saturday
MOTOR SPORTS
Drag Racing
Street Car Super Nationals 15, The Strip at Las Vegas Motor Speedway, Thursday-Nov. 24
Stock Car Racing
West Coast Short Track Championships, The Bullring at Las Vegas Motor Speedway, Saturday
RUNNING
Humana
Rock ’n’ Roll Las Vegas Marathon, Las Vegas Boulevard, 4 p.m. Sunday
UNLV
Men’s Basketball
vs. Abilene Christian, Thomas & Mack Center, 7 p.m. Monday; vs. Texas State, Thomas & Mack Center, 7 p.m. Wednesday; vs. Southern Methodist, Thomas & Mack Center, 7:30 p.m. Saturday
Women’s Basketball
at Southern Utah, Cedar City, Utah, 6 p.m. Tuesday; vs. South Carolina State, Cox Pavilion, 5:30 p.m. Saturday
Football
vs. San Jose State, Sam Boyd Stadium, 1 p.m. Saturday
Swimming and Diving
at Northwestern Invitational, Evanston, Ill., Friday-Nov. 24
Volleyball
at New Mexico, Albuquerque, N.M., 6 p.m. Tuesday; at Air Force, Colorado Springs, Colo., noon Saturday


** ** :------------------------------------------: ** : : ** : IMPORTANT : ** : : ** : A new version of this function must be : ** : produced whenever a new leap second is : ** : announced. There are four items to : ** : change on each such occasion: : ** : : ** : 1) A new line must be added to the set : ** : of statements that initialize the : ** : array "changes". : ** : : ** : 2) The parameter IYV must be set to : ** : the current year. : ** : : ** : 3) The "Latest leap second" comment : ** : below must be set to the new leap : ** : second date. : ** : : ** : 4) The "This revision" comment, later, : ** : must be set to the current date. : ** : : ** : Change (2) must also be carried out : ** : whenever the function is re-issued, : ** : even if no leap seconds have been : ** : added. : ** : : ** : Latest leap second: 2012 June 30 : ** : : ** :__________________________________________: ** ** Status: support function. ** ** Given: ** iy int UTC: year (Notes 1 and 2) ** im int month (Note 2) ** id int day (Notes 2 and 3) ** fd double fraction of day (Note 4) ** ** Returned: ** deltat double TAI minus UTC, seconds ** ** Returned (function value): ** int status (Note 5): ** 1 = dubious year (Note 1) ** 0 = OK ** -1 = bad year ** -2 = bad month ** -3 = bad day (Note 3) ** -4 = bad fraction (Note 4) ** ** Notes: ** ** 1) UTC began at 1960 January 1.0 (JD 2436934.5) and it is improper ** to call the function with an earlier date. If this is attempted, ** zero is returned together with a warning status. ** ** Because leap seconds cannot, in principle, be predicted in ** advance, a reliable check for dates beyond the valid range is ** impossible. To guard against gross errors, a year five or more ** after the release year of the present function (see parameter ** IYV) is considered dubious. In this case a warning status is ** returned but the result is computed in the normal way. ** ** For both too-early and too-late years, the warning status is ** j=+1. This is distinct from the error status j=-1, which ** signifies a year so early that JD could not be computed. ** ** 2) If the specified date is for a day which ends with a leap second, ** the UTC-TAI value returned is for the period leading up to the ** leap second. If the date is for a day which begins as a leap ** second ends, the UTC-TAI returned is for the period following the ** leap second. ** ** 3) The day number must be in the normal calendar range, for example ** 1 through 30 for April. The "almanac" convention of allowing ** such dates as January 0 and December 32 is not supported in this ** function, in order to avoid confusion near leap seconds. ** ** 4) The fraction of day is used only for dates before the ** introduction of leap seconds, the first of which occurred at the ** end of 1971. It is tested for validity (0 to 1 is the valid ** range) even if not used; if invalid, zero is used and status ** j=-4 is returned. For many applications, setting fd to zero is ** acceptable; the resulting error is always less than 3 ms (and ** occurs only pre-1972). ** ** 5) The status value returned in the case where there are multiple ** errors refers to the first error detected. For example, if the ** month and day are 13 and 32 respectively, j=-2 (bad month) ** will be returned. ** ** 6) In cases where a valid result is not available, zero is returned. ** ** References: ** ** 1) For dates from 1961 January 1 onwards, the expressions from the ** file ftp://maia.usno.navy.mil/ser7/tai-utc.dat are used. ** ** 2) The 5ms timestep at 1961 January 1 is taken from 2.58.1 (p87) of ** the 1992 Explanatory Supplement. ** ** Called: ** iauCal2jd Gregorian calendar to Julian Day number ** ** This revision: 2012 June 14 ** ** Original version 2012-03-01 ** ** Copyright (C) 2013 Naoki Arita. See notes at end. */ { /* Release year for this version of iauDat */ #define IYV (2012) /* Reference dates (MJD) and drift rates (s/day), pre leap seconds */ static const double drift[][2] = { { 37300.0, 0.0012960 }, { 37300.0, 0.0012960 }, { 37300.0, 0.0012960 }, { 37665.0, 0.0011232 }, { 37665.0, 0.0011232 }, { 38761.0, 0.0012960 }, { 38761.0, 0.0012960 }, { 38761.0, 0.0012960 }, { 38761.0, 0.0012960 }, { 38761.0, 0.0012960 }, { 38761.0, 0.0012960 }, { 38761.0, 0.0012960 }, { 39126.0, 0.0025920 }, { 39126.0, 0.0025920 } }; /* Number of Delta(AT) expressions before leap seconds were introduced */ #define NERA1 ((int) (sizeof drift / sizeof (double) / 2)) /* Dates and Delta(AT)s */ static const struct { int iyear, month; double delat; } changes[] = { { 1960, 1, 1.4178180 }, { 1961, 1, 1.4228180 }, { 1961, 8, 1.3728180 }, { 1962, 1, 1.8458580 }, { 1963, 11, 1.9458580 }, { 1964, 1, 3.2401300 }, { 1964, 4, 3.3401300 }, { 1964, 9, 3.4401300 }, { 1965, 1, 3.5401300 }, { 1965, 3, 3.6401300 }, { 1965, 7, 3.7401300 }, { 1965, 9, 3.8401300 }, { 1966, 1, 4.3131700 }, { 1968, 2, 4.2131700 }, { 1972, 1, 10.0 }, { 1972, 7, 11.0 }, { 1973, 1, 12.0 }, { 1974, 1, 13.0 }, { 1975, 1, 14.0 }, { 1976, 1, 15.0 }, { 1977, 1, 16.0 }, { 1978, 1, 17.0 }, { 1979, 1, 18.0 }, { 1980, 1, 19.0 }, { 1981, 7, 20.0 }, { 1982, 7, 21.0 }, { 1983, 7, 22.0 }, { 1985, 7, 23.0 }, { 1988, 1, 24.0 }, { 1990, 1, 25.0 }, { 1991, 1, 26.0 }, { 1992, 7, 27.0 }, { 1993, 7, 28.0 }, { 1994, 7, 29.0 }, { 1996, 1, 30.0 }, { 1997, 7, 31.0 }, { 1999, 1, 32.0 }, { 2006, 1, 33.0 }, { 2009, 1, 34.0 }, { 2012, 7, 35.0 } }; /* Number of Delta(AT) changes */ const int NDAT = sizeof changes / sizeof changes[0]; /* Miscellaneous local variables */ int j, i, m; double da, djm0, djm; /* Initialize the result to zero. */ *deltat = da = 0.0; /* If invalid fraction of a day, set error status and give up. */ if (fd < 0.0 || fd > 1.0) return -4; /* Convert the date into an MJD. */ j = iauCal2jd(iy, im, id, &djm0, &djm); /* If invalid year, month, or day, give up. */ if (j < 0) return j; /* If pre-UTC year, set warning status and give up. */ if (iy < changes[0].iyear) return 1; /* If suspiciously late year, set warning status but proceed. */ if (iy > IYV + 5) j = 1; /* Combine year and month to form a date-ordered integer... */ m = 12*iy + im; /* ...and use it to find the preceding table entry. */ for (i = NDAT-1; i >=0; i--) { if (m >= (12 * changes[i].iyear + changes[i].month)) break; } /* Get the Delta(AT). */ da = changes[i].delat; /* If pre-1972, adjust for drift. */ if (i < NERA1) da += (djm + fd - drift[i][0]) * drift[i][1]; /* Return the Delta(AT) value. */ *deltat = da; /* Return the status. */ return j; /*---------------------------------------------------------------------- ** ** Celes is a wrapper of the SOFA Library for Ruby. ** ** This file is redistributed and relicensed in accordance with ** the SOFA Software License (http://www.iausofa.org/tandc.html). ** ** The original library is available from IAU Standards of ** Fundamental Astronomy (http://www.iausofa.org/). ** ** ** ** ** ** Copyright (C) 2013, Naoki Arita ** All rights reserved. ** ** Redistribution and use in source and binary forms, with or without ** modification, are permitted provided that the following conditions ** are met: ** ** 1 Redistributions of source code must retain the above copyright ** notice, this list of conditions and the following disclaimer. ** ** 2 Redistributions in binary form must reproduce the above copyright ** notice, this list of conditions and the following disclaimer in ** the documentation and/or other materials provided with the ** distribution. ** ** 3 Neither the name of the Standards Of Fundamental Astronomy Board, ** the International Astronomical Union nor the names of its ** contributors may be used to endorse or promote products derived ** from this software without specific prior written permission.   ** **--------------------------------------------------------------------*/ }


The congress.datalog.utility
Module¶
-
class
congress.datalog.utility.
BagGraph
(graph=None)¶ Bases:
congress.datalog.utility.Graph
A graph data structure with bag semantics for nodes and edges.
Keeps track of the number of times each node/edge has been inserted. A node/edge is removed from the graph only once it has been deleted the same number of times it was inserted. Deletions when no node/edge already exist are ignored.
-
add_edge
(val1, val2, label=None)¶ Add edge from VAL1 to VAL2 with label LABEL to graph.
Also adds the nodes VAL1 and VAL2 (important for refcounting).
-
add_node
(val)¶ Add node VAL to graph.
-
delete_edge
(val1, val2, label=None)¶ Delete edge from VAL1 to VAL2 with label LABEL.
LABEL must match (even if None). Also deletes nodes whenever the edge exists.
-
delete_node
(val)¶ Delete node VAL from graph (but leave all edges).
-
edge_count
(val1, val2, label=None)¶
-
edge_in
(val1, val2, label=None)¶
-
node_count
(node)¶
-
node_in
(val)¶
-
-
class
congress.datalog.utility.
Cycle
¶ Bases:
frozenset
An immutable set of 2-tuples to represent a directed cycle
Extends frozenset, adding a list_repr method to represent a cycle as an ordered list of nodes.
The set representation facilicates identity of cycles regardless of order. The list representation is much more readable.
-
list_repr
()¶ Return list-of-nodes representation of cycle
-
-
class
congress.datalog.utility.
Graph
(graph=None)¶ Bases:
object
A standard graph data structure.
With routines applicable to analysis of policy.
-
add_edge
(val1, val2, label=None)¶ Add edge from VAL1 to VAL2 with label LABEL to graph.
Also adds the nodes.
-
add_node
(val)¶ Add node VAL to graph.
-
cycles
()¶ Return list of cycles. None indicates unknown.
-
delete_edge
(val1, val2, label=None)¶ Delete edge from VAL1 to VAL2 with label LABEL.
LABEL must match (even if None). Does not delete nodes.
-
delete_node
(val)¶ Delete node VAL from graph and all edges.
-
dependencies
(node)¶ Returns collection of node names reachable from NODE.
If NODE does not exist in graph, returns None.
-
depth_first_search
(roots=None)¶ Run depth first search on the graph.
Also modify self.nodes, self.counter, and self.cycle. Use all nodes if @roots param is None or unspecified
-
dfs
(node, target=None, dfs_stack=None)¶ DFS implementation.
Assumes data structures have been properly prepared. Creates start/begin times on nodes. Adds paths from node to target to self.__target_paths
-
class
dfs_data
(begin=None, end=None)¶ Bases:
object
Data for each node in graph during depth-first-search.
-
class
Graph.
edge_data
(node=None, label=None)¶ Bases:
object
Data for each edge in graph.
-
Graph.
edge_in
(val1, val2, label=None)¶
-
Graph.
find_dependent_nodes
(nodes)¶ Return all nodes dependent on @nodes.
Node T is dependent on node T. Node T is dependent on node R if there is an edge from node S to T,
and S is dependent on R.Note that node T is dependent on node T even if T is not in the graph
-
Graph.
find_reachable_nodes
(roots)¶ Return all nodes reachable from @roots.
-
Graph.
has_cycle
()¶ Checks if there are cycles.
Run depth_first_search only if it has not already been run.
-
Graph.
next_counter
()¶ Return next counter value and increment the counter.
-
Graph.
node_in
(val)¶
-
Graph.
reset
(roots=None)¶ Return nodes to pristine state.
-
Graph.
reset_nodes
()¶
-
Graph.
roots
()¶ Return list of nodes with no incoming edges.
-
Graph.
stratification
(labels)¶ Return the stratification result.
Return mapping of node name to integer indicating the stratum to which that node is assigned. LABELS is the list of edge labels that dictate a change in strata.
-
-
class
congress.datalog.utility.
OrderedSet
(iterable=None)¶ Bases:
_abcoll.MutableSet
Provide sequence capabilities with rapid membership checks.
Mostly lifted from the activestate recipe[1] linked at Python’s collections documentation[2]. Some modifications, such as returning True or False from add(key) and discard(key) if a change is made.
[1] - http://code.activestate.com/recipes/576694/ [2] - https://docs.python.org/2/library/collections.html
-
add
(key)¶
-
discard
(key)¶
-
pop
(last=True)¶
-
-
class
congress.datalog.utility.
iterstr
(iterable)¶ Bases:
object
Lazily provides informal string representation of iterables.
Calling __str__ directly on iterables returns a string containing the formal representation of the elements. This class wraps the iterable and instead returns the informal representation of the elements.