2 Tidy Data
Tidying up data is often where most of the data work happens. It is known by many names–cleaning, wrangling, tidying–but it all ultimately is to make the data in a format that we can analyze it.
2.1 Define Tidy Data
Tidy data are discussed in many resources (e.g., R for Data Science). “Tidy datasets are all alike but every messy dataset is messy in its own way. Tidy datasets provide a standardized way to link the structure of a dataset (its physical layout) with its semantics (its meaning). In this section, I’ll provide some standard vocabulary for describing the structure and semantics of a dataset, and then use those definitions to define tidy data,” (https://vita.had.co.nz/papers/tidy-data.pdf).
2.2 Tidying (Cleaning) Data
For examples, we are going to import some data from a .csv
file.
<- rio::import("data_csv.csv") data
2.3 Changing Variable Type
Importing data into R can cause your variables to change types. For example, lets see what the race variable class is now that the file has been imported into R.
class(data$Race)
## [1] "character"
As you see, its a character. We want it to be labeled as a factor. We can do this by using the as.factor() function in baseR. Make sure you save it as another variable in your dataset. If you don’t assign it to another variable, it will only temporarily show your variable as a factor but not actually change it to a factor for later analyses.
$RaceF <- as.factor(data$Race)
data$SmokingF <- as.factor(data$smoking) data
Now lets see what the class of our RaceF variable is.
class(data$RaceF)
## [1] "factor"
Please note- when importing from a SAV file, some factors may be “haven-labelled.” Use this technique to change haven-labelled variables into factors as well.
2.4 Changing a whole dataset to a certain class
If you want your whole dataset to be a certain class, you can easily do that in R using map_df(). Let’s say we want the whole dataset to be numeric.
library(tidyverse)
<- map_df(data, as.numeric)
data_num ::glimpse(data) tibble
## Rows: 100
## Columns: 16
## $ V1 <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ ID <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ Educ <int> 14, 15, 23, 14, 22, 15, 23, 21, 21, 20, 13, 12, 22, 12, 2…
## $ Age <int> 58, 32, 47, 35, 51, 73, 46, 66, 38, 53, 19, 52, 24, 69, 5…
## $ Sex <chr> "female", "male", "male", "female", "male", "male", "male…
## $ Race <chr> "other", "black", "other", "other", "white", "white", "la…
## $ Married <int> 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, …
## $ Income_1 <int> 42504, 45341, 69261, 41654, 66149, 46009, 68791, 62362, 6…
## $ INCOME_2 <int> 56418, 59810, 92139, 56264, 88635, 59918, 91662, 83720, 8…
## $ Depression_1 <int> 23, 20, 16, 14, 10, 14, 13, 6, 20, 11, 24, 21, 12, 10, 10…
## $ Depression_2 <int> 26, 25, 20, 19, 13, 17, 17, 5, 26, 13, 28, 24, 14, 12, 12…
## $ Anx_1 <int> 21, 20, 15, 15, 8, 14, 16, 5, 17, 11, 26, 23, 12, 10, 12,…
## $ Anx_2 <int> 22, 23, 19, 10, 7, 16, 17, 4, 19, 10, 25, 22, 9, 12, 12, …
## $ smoking <chr> "Non-smoker", "Smoker", "Smoker", "99", "Non-smoker", "No…
## $ RaceF <fct> other, black, other, other, white, white, latinx, white, …
## $ SmokingF <fct> Non-smoker, Smoker, Smoker, 99, Non-smoker, Non-smoker, N…
As you can see, the character and factor variables that do not contain numbers are unable to be numeric. We will go over how to change that later. You can also change the whole dataset to be factors using the same code, but changing as.numeric to as.factor.
2.5 Janitor Package
The janitor package has useful functions that make it easy to clean your dataset. Use the clean_names() function to make variable names consistent.
library(janitor)
<- clean_names(data)
clean_df
::glimpse(clean_df) tibble
## Rows: 100
## Columns: 16
## $ v1 <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ id <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ educ <int> 14, 15, 23, 14, 22, 15, 23, 21, 21, 20, 13, 12, 22, 12, 2…
## $ age <int> 58, 32, 47, 35, 51, 73, 46, 66, 38, 53, 19, 52, 24, 69, 5…
## $ sex <chr> "female", "male", "male", "female", "male", "male", "male…
## $ race <chr> "other", "black", "other", "other", "white", "white", "la…
## $ married <int> 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, …
## $ income_1 <int> 42504, 45341, 69261, 41654, 66149, 46009, 68791, 62362, 6…
## $ income_2 <int> 56418, 59810, 92139, 56264, 88635, 59918, 91662, 83720, 8…
## $ depression_1 <int> 23, 20, 16, 14, 10, 14, 13, 6, 20, 11, 24, 21, 12, 10, 10…
## $ depression_2 <int> 26, 25, 20, 19, 13, 17, 17, 5, 26, 13, 28, 24, 14, 12, 12…
## $ anx_1 <int> 21, 20, 15, 15, 8, 14, 16, 5, 17, 11, 26, 23, 12, 10, 12,…
## $ anx_2 <int> 22, 23, 19, 10, 7, 16, 17, 4, 19, 10, 25, 22, 9, 12, 12, …
## $ smoking <chr> "Non-smoker", "Smoker", "Smoker", "99", "Non-smoker", "No…
## $ race_f <fct> other, black, other, other, white, white, latinx, white, …
## $ smoking_f <fct> Non-smoker, Smoker, Smoker, 99, Non-smoker, Non-smoker, N…
Now, all of our variables are lowercase and consistent.
You can also remove columns and/or rows that are completely missing. We do not have any missing rows or columns in our real dataset, so we will make one up to show you how it works.
<- data.frame(var1 = c(1, NA, 6, 8, 3),
df var2 = c("a", NA, "c", "s", "y"),
var3 = c(NA, NA, NA, NA, NA))
df
## var1 var2 var3
## 1 1 a NA
## 2 NA <NA> NA
## 3 6 c NA
## 4 8 s NA
## 5 3 y NA
remove_empty(df, "rows")
## var1 var2 var3
## 1 1 a NA
## 3 6 c NA
## 4 8 s NA
## 5 3 y NA
remove_empty(df, "cols")
## var1 var2
## 1 1 a
## 2 NA <NA>
## 3 6 c
## 4 8 s
## 5 3 y
remove_empty(df, c("rows", "cols"))
## var1 var2
## 1 1 a
## 3 6 c
## 4 8 s
## 5 3 y
2.6 Washer function in the Furniture package
Note the use of head()
to only print 6 rows (instead of all of them).
library(furniture)
washer(clean_df, 99, value = NA) %>%
head()
## v1 id educ age sex race married income_1 income_2 depression_1
## 1 1 1 14 58 female other 0 42504 56418 23
## 2 2 2 15 32 male black 0 45341 59810 20
## 3 3 3 23 47 male other 0 69261 92139 16
## 4 4 4 14 35 female other 1 41654 56264 14
## 5 5 5 22 51 male white 0 66149 88635 10
## 6 6 6 15 73 male white 1 46009 59918 14
## depression_2 anx_1 anx_2 smoking race_f smoking_f
## 1 26 21 22 Non-smoker other Non-smoker
## 2 25 20 23 Smoker black Smoker
## 3 20 15 19 Smoker other Smoker
## 4 19 15 10 <NA> other <NA>
## 5 13 8 7 Non-smoker white Non-smoker
## 6 17 14 16 Non-smoker white Non-smoker
2.7 Tidyverse
2.7.1 Pipes
me %>%
wake_up("8:00am") %>%
exercise(30, units = "minutes") %>%
shower(15, units = "minutes") %>%
eat_breakfast("toast") %>%
go_to_work("basement")
Piping takes what is on the left-hand side and puts it in the right hand side’s function. If you look at the code below, you’ll see how code looks without a pipe.
summary(clean_df)
## v1 id educ age
## Min. : 1.00 Min. : 1.00 Min. :10.00 Min. :18.00
## 1st Qu.: 25.75 1st Qu.: 25.75 1st Qu.:15.00 1st Qu.:35.75
## Median : 50.50 Median : 50.50 Median :20.00 Median :49.00
## Mean : 50.50 Mean : 50.50 Mean :18.65 Mean :48.24
## 3rd Qu.: 75.25 3rd Qu.: 75.25 3rd Qu.:22.00 3rd Qu.:59.25
## Max. :100.00 Max. :100.00 Max. :25.00 Max. :75.00
## sex race married income_1
## Length:100 Length:100 Min. :0.00 Min. :29329
## Class :character Class :character 1st Qu.:0.00 1st Qu.:45348
## Mode :character Mode :character Median :0.00 Median :59840
## Mean :0.43 Mean :56053
## 3rd Qu.:1.00 3rd Qu.:66275
## Max. :1.00 Max. :75987
## income_2 depression_1 depression_2 anx_1
## Min. : 39557 Min. : 4.00 Min. : 4.00 Min. :-1.00
## 1st Qu.: 59985 1st Qu.: 9.00 1st Qu.:11.00 1st Qu.: 8.00
## Median : 79867 Median :13.00 Median :17.00 Median :13.50
## Mean : 74689 Mean :13.68 Mean :16.47 Mean :13.26
## 3rd Qu.: 88179 3rd Qu.:18.00 3rd Qu.:22.00 3rd Qu.:18.00
## Max. :100518 Max. :25.00 Max. :31.00 Max. :26.00
## anx_2 smoking race_f smoking_f
## Min. : 1.0 Length:100 black :20 99 : 8
## 1st Qu.: 8.0 Class :character latinx:33 Non-smoker:70
## Median :13.0 Mode :character other :14 Smoker :14
## Mean :13.3 white :33 NA's : 8
## 3rd Qu.:18.0
## Max. :27.0
Now, here is what the code looks like using a pipe.
%>%
clean_df summary()
## v1 id educ age
## Min. : 1.00 Min. : 1.00 Min. :10.00 Min. :18.00
## 1st Qu.: 25.75 1st Qu.: 25.75 1st Qu.:15.00 1st Qu.:35.75
## Median : 50.50 Median : 50.50 Median :20.00 Median :49.00
## Mean : 50.50 Mean : 50.50 Mean :18.65 Mean :48.24
## 3rd Qu.: 75.25 3rd Qu.: 75.25 3rd Qu.:22.00 3rd Qu.:59.25
## Max. :100.00 Max. :100.00 Max. :25.00 Max. :75.00
## sex race married income_1
## Length:100 Length:100 Min. :0.00 Min. :29329
## Class :character Class :character 1st Qu.:0.00 1st Qu.:45348
## Mode :character Mode :character Median :0.00 Median :59840
## Mean :0.43 Mean :56053
## 3rd Qu.:1.00 3rd Qu.:66275
## Max. :1.00 Max. :75987
## income_2 depression_1 depression_2 anx_1
## Min. : 39557 Min. : 4.00 Min. : 4.00 Min. :-1.00
## 1st Qu.: 59985 1st Qu.: 9.00 1st Qu.:11.00 1st Qu.: 8.00
## Median : 79867 Median :13.00 Median :17.00 Median :13.50
## Mean : 74689 Mean :13.68 Mean :16.47 Mean :13.26
## 3rd Qu.: 88179 3rd Qu.:18.00 3rd Qu.:22.00 3rd Qu.:18.00
## Max. :100518 Max. :25.00 Max. :31.00 Max. :26.00
## anx_2 smoking race_f smoking_f
## Min. : 1.0 Length:100 black :20 99 : 8
## 1st Qu.: 8.0 Class :character latinx:33 Non-smoker:70
## Median :13.0 Mode :character other :14 Smoker :14
## Mean :13.3 white :33 NA's : 8
## 3rd Qu.:18.0
## Max. :27.0
It may not seem to make the code more readable, but the more complex your code is the easier it will be to read it using pipes. It also allows you to connect multiple lines of code without having to overwrite your dataset multiple times. As we go through more data cleaning and wrangling, you’ll begin to see how piping makes a difference when coding.
2.7.2 Subset Data
<IP_ADDRESS> Select function
The select function allows you to pull specific information out of our dataset. You can do this using baseR commands, or with the select function from tidyverse.
head(clean_df[, c("id", "educ", "age")])
## id educ age
## 1 1 14 58
## 2 2 15 32
## 3 3 23 47
## 4 4 14 35
## 5 5 22 51
## 6 6 15 73
head(clean_df[, c(1:3)])
## v1 id educ
## 1 1 1 14
## 2 2 2 15
## 3 3 3 23
## 4 4 4 14
## 5 5 5 22
## 6 6 6 15
%>%
clean_df select(id, educ, age) %>%
head()
## id educ age
## 1 1 14 58
## 2 2 15 32
## 3 3 23 47
## 4 4 14 35
## 5 5 22 51
## 6 6 15 73
%>%
clean_df select(1:3) %>%
head()
## v1 id educ
## 1 1 1 14
## 2 2 2 15
## 3 3 3 23
## 4 4 4 14
## 5 5 5 22
## 6 6 6 15
Each code chunk does the same thing. The first two, using [, is the “base R” way of selecting variables. The last two, using the pipe, is the tidyverse way. Both work great so the choice is yours.
<IP_ADDRESS> starts_with(), ends_with(), contains()
When you use tidyverse, you can pull variables out that have common features. Here are a few examples. Note that we use the select function as well as another function to pull out information.
This line of code pulls out the variables that start with the word “income.”
%>%
clean_df select(starts_with("income")) %>%
head()
## income_1 income_2
## 1 42504 56418
## 2 45341 59810
## 3 69261 92139
## 4 41654 56264
## 5 66149 88635
## 6 46009 59918
This code allows you to select all of the variables in your dataset that end with _1 and _2.
%>%
clean_df select(ends_with(c("_1", "_2"))) %>%
head()
## income_1 depression_1 anx_1 income_2 depression_2 anx_2
## 1 42504 23 21 56418 26 22
## 2 45341 20 20 59810 25 23
## 3 69261 16 15 92139 20 19
## 4 41654 14 15 56264 19 10
## 5 66149 10 8 88635 13 7
## 6 46009 14 14 59918 17 16
With this code you can pull out any variables that contain “anx” anywhere in the variable name.
%>%
clean_df select(contains("anx")) %>%
head()
## anx_1 anx_2
## 1 21 22
## 2 20 23
## 3 15 19
## 4 15 10
## 5 8 7
## 6 14 16
2.7.3 Filter, equality/inequality, and/or, within
$anx_1 > 20,] clean_df[clean_df
## v1 id educ age sex race married income_1 income_2 depression_1
## 1 1 1 14 58 female other 0 42504 56418 23
## 11 11 11 13 19 male latinx 0 38609 51231 24
## 12 12 12 12 52 male latinx 0 35765 48350 21
## 27 27 27 11 32 female latinx 1 31930 43994 25
## 31 31 31 12 32 female black 0 35574 48605 20
## 32 32 32 16 32 female other 1 48240 64209 20
## 39 39 39 23 53 male white 0 69101 90538 24
## 41 41 41 24 41 female white 0 71409 96501 25
## 42 42 42 21 48 male white 0 63792 84229 22
## 56 56 56 20 64 male other 0 59826 80546 22
## 72 72 72 21 52 female black 1 62825 84191 25
## 77 77 77 17 50 female white 0 51535 67826 23
## 86 86 86 16 56 female black 0 49602 64341 21
## 94 94 94 15 68 male latinx 0 44938 60644 23
## 95 95 95 20 23 male latinx 1 59629 79479 25
## 96 96 96 19 60 female white 1 56833 75545 20
## depression_2 anx_1 anx_2 smoking race_f smoking_f
## 1 26 21 22 Non-smoker other Non-smoker
## 11 28 26 25 <NA> latinx <NA>
## 12 24 23 22 Non-smoker latinx Non-smoker
## 27 31 25 27 <NA> latinx <NA>
## 31 24 21 23 Non-smoker black Non-smoker
## 32 24 25 25 Non-smoker other Non-smoker
## 39 29 25 24 Non-smoker white Non-smoker
## 41 31 26 21 Non-smoker white Non-smoker
## 42 25 22 25 Non-smoker white Non-smoker
## 56 25 25 23 99 other 99
## 72 30 25 24 Non-smoker black Non-smoker
## 77 27 22 24 Non-smoker white Non-smoker
## 86 25 23 25 Non-smoker black Non-smoker
## 94 26 24 27 Non-smoker latinx Non-smoker
## 95 29 23 23 Smoker latinx Smoker
## 96 25 21 20 Non-smoker white Non-smoker
%>%
clean_df filter(anx_1 > 20) %>%
head()
## v1 id educ age sex race married income_1 income_2 depression_1
## 1 1 1 14 58 female other 0 42504 56418 23
## 2 11 11 13 19 male latinx 0 38609 51231 24
## 3 12 12 12 52 male latinx 0 35765 48350 21
## 4 27 27 11 32 female latinx 1 31930 43994 25
## 5 31 31 12 32 female black 0 35574 48605 20
## 6 32 32 16 32 female other 1 48240 64209 20
## depression_2 anx_1 anx_2 smoking race_f smoking_f
## 1 26 21 22 Non-smoker other Non-smoker
## 2 28 26 25 <NA> latinx <NA>
## 3 24 23 22 Non-smoker latinx Non-smoker
## 4 31 25 27 <NA> latinx <NA>
## 5 24 21 23 Non-smoker black Non-smoker
## 6 24 25 25 Non-smoker other Non-smoker
%>%
clean_df filter(sex == "female") %>%
head()
## v1 id educ age sex race married income_1 income_2 depression_1
## 1 1 1 14 58 female other 0 42504 56418 23
## 2 4 4 14 35 female other 1 41654 56264 14
## 3 8 8 21 66 female white 1 62362 83720 6
## 4 9 9 21 38 female other 0 62716 84363 20
## 5 10 10 20 53 female white 1 60309 80255 11
## 6 15 15 22 57 female latinx 1 66286 87292 10
## depression_2 anx_1 anx_2 smoking race_f smoking_f
## 1 26 21 22 Non-smoker other Non-smoker
## 2 19 15 10 99 other 99
## 3 5 5 4 Non-smoker white Non-smoker
## 4 26 17 19 Non-smoker other Non-smoker
## 5 13 11 10 Smoker white Smoker
## 6 12 12 12 Non-smoker latinx Non-smoker
%>%
clean_df filter(sex == "female" & race == "black") %>%
head()
## v1 id educ age sex race married income_1 income_2 depression_1
## 1 20 20 21 56 female black 0 62751 84363 5
## 2 31 31 12 32 female black 0 35574 48605 20
## 3 43 43 25 49 female black 0 74573 99489 16
## 4 44 44 21 33 female black 0 63295 84883 8
## 5 60 60 22 25 female black 0 65956 87883 18
## 6 64 64 25 21 female black 0 75987 100466 4
## depression_2 anx_1 anx_2 smoking race_f smoking_f
## 1 7 3 3 Non-smoker black Non-smoker
## 2 24 21 23 Non-smoker black Non-smoker
## 3 18 15 16 Non-smoker black Non-smoker
## 4 10 7 7 Non-smoker black Non-smoker
## 5 21 16 16 Non-smoker black Non-smoker
## 6 5 1 2 Non-smoker black Non-smoker
%>%
clean_df filter(sex == "female" | educ == 20) %>%
head()
## v1 id educ age sex race married income_1 income_2 depression_1
## 1 1 1 14 58 female other 0 42504 56418 23
## 2 4 4 14 35 female other 1 41654 56264 14
## 3 8 8 21 66 female white 1 62362 83720 6
## 4 9 9 21 38 female other 0 62716 84363 20
## 5 10 10 20 53 female white 1 60309 80255 11
## 6 15 15 22 57 female latinx 1 66286 87292 10
## depression_2 anx_1 anx_2 smoking race_f smoking_f
## 1 26 21 22 Non-smoker other Non-smoker
## 2 19 15 10 99 other 99
## 3 5 5 4 Non-smoker white Non-smoker
## 4 26 17 19 Non-smoker other Non-smoker
## 5 13 11 10 Smoker white Smoker
## 6 12 12 12 Non-smoker latinx Non-smoker
Can also have it match multiple things
%>%
clean_df filter(race %in% c("black", "latinx")) %>%
head()
## v1 id educ age sex race married income_1 income_2 depression_1
## 1 2 2 15 32 male black 0 45341 59810 20
## 2 7 7 23 46 male latinx 1 68791 91662 13
## 3 11 11 13 19 male latinx 0 38609 51231 24
## 4 12 12 12 52 male latinx 0 35765 48350 21
## 5 13 13 22 24 male latinx 0 65359 88375 12
## 6 14 14 12 69 male latinx 1 35638 47970 10
## depression_2 anx_1 anx_2 smoking race_f smoking_f
## 1 25 20 23 Smoker black Smoker
## 2 17 16 17 Non-smoker latinx Non-smoker
## 3 28 26 25 <NA> latinx <NA>
## 4 24 23 22 Non-smoker latinx Non-smoker
## 5 14 12 9 Non-smoker latinx Non-smoker
## 6 12 10 12 Non-smoker latinx Non-smoker
2.7.4 Mutate
Anytime you see mutate() it means you are adding a new variable or modifying an existing one. For example, lets take a look at what class our sex variable is.
class(clean_df$sex)
## [1] "character"
When we imported the data, R did not know sex was a factor. So, we need to change it to be factor. Using the mutate function allows us to modify existing variables, or create new variables. In this case, I make a new variable called “sex_f” and change sex to be a factor.
<- clean_df %>%
clean_df mutate(sex_f = factor(sex))
class(clean_df$sex_f)
## [1] "factor"
There are loads of useful functions for mutating a variable! We will present a few here.
<IP_ADDRESS> Case_when
When you want to modify variables based on various possible conditions, you can use the case_when() function. For example, when a categorical variable is dummy-coded to be 0,1,2.. you can change it to have actual name labels. In this example, our married variable i supposed to be factor of 0 = not married and 1 = married. Lets look at our married variable before we use mutate and case_when.
summary(clean_df$married)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.00 0.00 0.00 0.43 1.00 1.00
The next code chunk will show a new variable, called “married_f” that has levels “married” and “not married” and is a factor. To do this, we need to use mutate to create the new married_f variable, and then tell r that when the married variable = 1, then label married, and if married = 0, label it not married. Note that we need to use == to specify that it is equal to something. We also need to put our labels in "" when they are words. The last line of code turns married_f into a factor, rather than a character variable.
<- clean_df %>%
clean_df mutate(married_f = case_when(
== 1 ~ "married",
married == 0 ~ "not married"
married %>% as.factor())
)
glimpse(clean_df)
## Rows: 100
## Columns: 18
## $ v1 <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ id <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ educ <int> 14, 15, 23, 14, 22, 15, 23, 21, 21, 20, 13, 12, 22, 12, 2…
## $ age <int> 58, 32, 47, 35, 51, 73, 46, 66, 38, 53, 19, 52, 24, 69, 5…
## $ sex <chr> "female", "male", "male", "female", "male", "male", "male…
## $ race <chr> "other", "black", "other", "other", "white", "white", "la…
## $ married <int> 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, …
## $ income_1 <int> 42504, 45341, 69261, 41654, 66149, 46009, 68791, 62362, 6…
## $ income_2 <int> 56418, 59810, 92139, 56264, 88635, 59918, 91662, 83720, 8…
## $ depression_1 <int> 23, 20, 16, 14, 10, 14, 13, 6, 20, 11, 24, 21, 12, 10, 10…
## $ depression_2 <int> 26, 25, 20, 19, 13, 17, 17, 5, 26, 13, 28, 24, 14, 12, 12…
## $ anx_1 <int> 21, 20, 15, 15, 8, 14, 16, 5, 17, 11, 26, 23, 12, 10, 12,…
## $ anx_2 <int> 22, 23, 19, 10, 7, 16, 17, 4, 19, 10, 25, 22, 9, 12, 12, …
## $ smoking <chr> "Non-smoker", "Smoker", "Smoker", "99", "Non-smoker", "No…
## $ race_f <fct> other, black, other, other, white, white, latinx, white, …
## $ smoking_f <fct> Non-smoker, Smoker, Smoker, 99, Non-smoker, Non-smoker, N…
## $ sex_f <fct> female, male, male, female, male, male, male, female, fem…
## $ married_f <fct> not married, not married, not married, married, not marri…
Here is another example.
<- clean_df %>%
clean_df mutate(income_cat = case_when(
> mean(income_1, na.rm=TRUE) + sd(income_1, na.rm=TRUE) ~ "High Earner",
income_1 > mean(income_1, na.rm=TRUE) ~ "Mid-High Earner",
income_1 <= mean(income_1, na.rm=TRUE) - sd(income_1, na.rm=TRUE) ~ "Low Earner",
income_1 <= mean(income_1, na.rm=TRUE) ~ "Mid-Low Earner",
income_1 %>% as.factor())
)
glimpse(clean_df)
## Rows: 100
## Columns: 19
## $ v1 <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ id <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ educ <int> 14, 15, 23, 14, 22, 15, 23, 21, 21, 20, 13, 12, 22, 12, 2…
## $ age <int> 58, 32, 47, 35, 51, 73, 46, 66, 38, 53, 19, 52, 24, 69, 5…
## $ sex <chr> "female", "male", "male", "female", "male", "male", "male…
## $ race <chr> "other", "black", "other", "other", "white", "white", "la…
## $ married <int> 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, …
## $ income_1 <int> 42504, 45341, 69261, 41654, 66149, 46009, 68791, 62362, 6…
## $ income_2 <int> 56418, 59810, 92139, 56264, 88635, 59918, 91662, 83720, 8…
## $ depression_1 <int> 23, 20, 16, 14, 10, 14, 13, 6, 20, 11, 24, 21, 12, 10, 10…
## $ depression_2 <int> 26, 25, 20, 19, 13, 17, 17, 5, 26, 13, 28, 24, 14, 12, 12…
## $ anx_1 <int> 21, 20, 15, 15, 8, 14, 16, 5, 17, 11, 26, 23, 12, 10, 12,…
## $ anx_2 <int> 22, 23, 19, 10, 7, 16, 17, 4, 19, 10, 25, 22, 9, 12, 12, …
## $ smoking <chr> "Non-smoker", "Smoker", "Smoker", "99", "Non-smoker", "No…
## $ race_f <fct> other, black, other, other, white, white, latinx, white, …
## $ smoking_f <fct> Non-smoker, Smoker, Smoker, 99, Non-smoker, Non-smoker, N…
## $ sex_f <fct> female, male, male, female, male, male, male, female, fem…
## $ married_f <fct> not married, not married, not married, married, not marri…
## $ income_cat <fct> Mid-Low Earner, Mid-Low Earner, Mid-High Earner, Low Earn…
This last one is a complex example but it shows the utility of case_when()
. Any guesses what this is doing?
%>%
data rowwise() %>%
mutate(
var1 =
case_when(
sum(
> 2,
item1 > 2,
item2 > 2,
item3 > 2,
item4 > 2,
item5 na.rm=TRUE
>= 6 &
) sum(
> 3,
perf1 > 3,
perf2 > 3,
perf3 na.rm=TRUE
> 0 ~ 1,
) TRUE ~ 0
)%>%
) ungroup() %>%
mutate(var1 = factor(var1))
<IP_ADDRESS> Rowsums and Rowmeans
<- clean_df %>%
clean_df mutate(dep_v1 = furniture::rowmeans(depression_1, depression_2))
<- clean_df %>%
clean_df rowwise() %>%
mutate(dep_v2 = mean(c(depression_1, depression_2))) %>%
ungroup()
glimpse(clean_df)
## Rows: 100
## Columns: 21
## $ v1 <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ id <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
## $ educ <int> 14, 15, 23, 14, 22, 15, 23, 21, 21, 20, 13, 12, 22, 12, 2…
## $ age <int> 58, 32, 47, 35, 51, 73, 46, 66, 38, 53, 19, 52, 24, 69, 5…
## $ sex <chr> "female", "male", "male", "female", "male", "male", "male…
## $ race <chr> "other", "black", "other", "other", "white", "white", "la…
## $ married <int> 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, …
## $ income_1 <int> 42504, 45341, 69261, 41654, 66149, 46009, 68791, 62362, 6…
## $ income_2 <int> 56418, 59810, 92139, 56264, 88635, 59918, 91662, 83720, 8…
## $ depression_1 <int> 23, 20, 16, 14, 10, 14, 13, 6, 20, 11, 24, 21, 12, 10, 10…
## $ depression_2 <int> 26, 25, 20, 19, 13, 17, 17, 5, 26, 13, 28, 24, 14, 12, 12…
## $ anx_1 <int> 21, 20, 15, 15, 8, 14, 16, 5, 17, 11, 26, 23, 12, 10, 12,…
## $ anx_2 <int> 22, 23, 19, 10, 7, 16, 17, 4, 19, 10, 25, 22, 9, 12, 12, …
## $ smoking <chr> "Non-smoker", "Smoker", "Smoker", "99", "Non-smoker", "No…
## $ race_f <fct> other, black, other, other, white, white, latinx, white, …
## $ smoking_f <fct> Non-smoker, Smoker, Smoker, 99, Non-smoker, Non-smoker, N…
## $ sex_f <fct> female, male, male, female, male, male, male, female, fem…
## $ married_f <fct> not married, not married, not married, married, not marri…
## $ income_cat <fct> Mid-Low Earner, Mid-Low Earner, Mid-High Earner, Low Earn…
## $ dep_v1 <dbl> 24.5, 22.5, 18.0, 16.5, 11.5, 15.5, 15.0, 5.5, 23.0, 12.0…
## $ dep_v2 <dbl> 24.5, 22.5, 18.0, 16.5, 11.5, 15.5, 15.0, 5.5, 23.0, 12.0…
2.7.5 Forcats Package
library(forcats) # forcats is also in tidyverse so if you downloaded tidyverse you don't need to download this as well
<IP_ADDRESS> Relevel
You can manually reorder factor levels
%>%
clean_df mutate(race_f = fct_relevel(race_f, "white", "black", "latinx", "other"))
## # A tibble: 100 × 21
## v1 id educ age sex race married income_1 income_2 depression_1
## <int> <int> <int> <int> <chr> <chr> <int> <int> <int> <int>
## 1 1 1 14 58 female other 0 42504 56418 23
## 2 2 2 15 32 male black 0 45341 59810 20
## 3 3 3 23 47 male other 0 69261 92139 16
## 4 4 4 14 35 female other 1 41654 56264 14
## 5 5 5 22 51 male white 0 66149 88635 10
## 6 6 6 15 73 male white 1 46009 59918 14
## 7 7 7 23 46 male latinx 1 68791 91662 13
## 8 8 8 21 66 female white 1 62362 83720 6
## 9 9 9 21 38 female other 0 62716 84363 20
## 10 10 10 20 53 female white 1 60309 80255 11
## # … with 90 more rows, and 11 more variables: depression_2 <int>, anx_1 <int>,
## # anx_2 <int>, smoking <chr>, race_f <fct>, smoking_f <fct>, sex_f <fct>,
## # married_f <fct>, income_cat <fct>, dep_v1 <dbl>, dep_v2 <dbl>
<IP_ADDRESS> Reorder
Reorder a factor by its levels’ frequency
%>%
clean_df mutate(race_f = fct_infreq(race_f))
## # A tibble: 100 × 21
## v1 id educ age sex race married income_1 income_2 depression_1
## <int> <int> <int> <int> <chr> <chr> <int> <int> <int> <int>
## 1 1 1 14 58 female other 0 42504 56418 23
## 2 2 2 15 32 male black 0 45341 59810 20
## 3 3 3 23 47 male other 0 69261 92139 16
## 4 4 4 14 35 female other 1 41654 56264 14
## 5 5 5 22 51 male white 0 66149 88635 10
## 6 6 6 15 73 male white 1 46009 59918 14
## 7 7 7 23 46 male latinx 1 68791 91662 13
## 8 8 8 21 66 female white 1 62362 83720 6
## 9 9 9 21 38 female other 0 62716 84363 20
## 10 10 10 20 53 female white 1 60309 80255 11
## # … with 90 more rows, and 11 more variables: depression_2 <int>, anx_1 <int>,
## # anx_2 <int>, smoking <chr>, race_f <fct>, smoking_f <fct>, sex_f <fct>,
## # married_f <fct>, income_cat <fct>, dep_v1 <dbl>, dep_v2 <dbl>
Reorder a factor by a numeric variable
%>%
clean_df mutate(race_f = fct_reorder(race_f, income_1))
## # A tibble: 100 × 21
## v1 id educ age sex race married income_1 income_2 depression_1
## <int> <int> <int> <int> <chr> <chr> <int> <int> <int> <int>
## 1 1 1 14 58 female other 0 42504 56418 23
## 2 2 2 15 32 male black 0 45341 59810 20
## 3 3 3 23 47 male other 0 69261 92139 16
## 4 4 4 14 35 female other 1 41654 56264 14
## 5 5 5 22 51 male white 0 66149 88635 10
## 6 6 6 15 73 male white 1 46009 59918 14
## 7 7 7 23 46 male latinx 1 68791 91662 13
## 8 8 8 21 66 female white 1 62362 83720 6
## 9 9 9 21 38 female other 0 62716 84363 20
## 10 10 10 20 53 female white 1 60309 80255 11
## # … with 90 more rows, and 11 more variables: depression_2 <int>, anx_1 <int>,
## # anx_2 <int>, smoking <chr>, race_f <fct>, smoking_f <fct>, sex_f <fct>,
## # married_f <fct>, income_cat <fct>, dep_v1 <dbl>, dep_v2 <dbl>
<IP_ADDRESS> Lump
When you have a factor with too many levels, you can lump all of the infrequent ones into one factor, “other.” You can specify how many levels you want to include. the argument “n” is the number of levels you want to keep, besides the “other” category that will be made from the lump function. You can also include the argument other_level = "" if you would like to change the name from other to something more specific.
%>%
clean_df mutate(race_f = fct_lump(race_f, n = 2))
## # A tibble: 100 × 21
## v1 id educ age sex race married income_1 income_2 depression_1
## <int> <int> <int> <int> <chr> <chr> <int> <int> <int> <int>
## 1 1 1 14 58 female other 0 42504 56418 23
## 2 2 2 15 32 male black 0 45341 59810 20
## 3 3 3 23 47 male other 0 69261 92139 16
## 4 4 4 14 35 female other 1 41654 56264 14
## 5 5 5 22 51 male white 0 66149 88635 10
## 6 6 6 15 73 male white 1 46009 59918 14
## 7 7 7 23 46 male latinx 1 68791 91662 13
## 8 8 8 21 66 female white 1 62362 83720 6
## 9 9 9 21 38 female other 0 62716 84363 20
## 10 10 10 20 53 female white 1 60309 80255 11
## # … with 90 more rows, and 11 more variables: depression_2 <int>, anx_1 <int>,
## # anx_2 <int>, smoking <chr>, race_f <fct>, smoking_f <fct>, sex_f <fct>,
## # married_f <fct>, income_cat <fct>, dep_v1 <dbl>, dep_v2 <dbl>
In this example, we previously had four levels of the race variable: white, black, latinx, and other. When we specified n = 2, we lumped the other and black categories (the most infrequent two) into one “Other” category. It kept our two most frequent categories: white and latinx.
<IP_ADDRESS> Other Useful Functions for Factors
2.7.6 Group_by
The group_by() function allows for us to group our data by a specific variable and compare groups with descriptive statistics.
%>%
clean_df group_by(race_f) %>%
summarize(N = n(),
mean = mean(depression_1),
sd = sd(depression_1))
## # A tibble: 4 × 4
## race_f N mean sd
## <fct> <int> <dbl> <dbl>
## 1 black 20 12.9 6.93
## 2 latinx 33 13.5 5.75
## 3 other 14 15.4 6.10
## 4 white 33 13.6 6.09
This example groups the dataset by race_f and summarizes it by the number per race (the n() argument), and provides the mean (mean()) and standard deviation (sd()) for depression scores by race.


Courses we offer in George Brown College
Computer Science Engineering (CSE) encompasses a variety of topics that relates to computation, like analysis of algorithms, programming languages, program design, software, and computer hardware.
Take A CourseThis stream of engineering deals with analog transmission, basic electronics, microprocessors, solid state devices, digital and analog communication, analog integrated circuits, microwave engineering, satellite communication, antennae and wave progression. It also deals with the manufacturing of electronic devices, circuits, and communications equipment.
Take A CourseElectrical engineering is an engineering discipline concerned with the study, design and application of equipment, devices and systems which use electricity, electronics, and electromagnetism. It emerged as an identifiable occupation in the latter half of the 19th century after commercialization of the electric telegraph, the telephone, and electrical power generation, distribution and use.
Take A CourseChemical engineering is a branch of engineering that uses principles of chemistry, physics, mathematics, biology, to efficiently use, produce, design, and transform energy and materials. The work of chemical engineers can range from the utilisation of nano-technology and nano-materials in the laboratory to large-scale industrial processes that convert chemicals, raw materials, living cells, microorganisms, and energy into useful forms and products.
Take A CourseFee details for the various course plans


 The person taking the photo may have to move if there's not.
Don't worry about cropping the photo – we’ll do that for you when you submit your application.


HDFS Users Guide
Purpose
This document is a starting point for users working with Hadoop Distributed File System (HDFS) either as a part of a Hadoop cluster or as a stand-alone general purpose distributed file system. While HDFS is designed to "just work" in many environments, a working knowledge of HDFS helps greatly with configuration improvements and diagnostics on a specific cluster.
Overview
HDFS is the primary distributed storage used by Hadoop applications. A HDFS cluster primarily consists of a NameNode that manages the file system metadata and DataNodes that store the actual data. The HDFS Architecture Guide describes HDFS in detail. This user guide primarily deals with the interaction of users and administrators with HDFS clusters. The HDFS architecture diagram depicts basic interactions among NameNode, the DataNodes, and the clients. Clients contact NameNode for file metadata or file modifications and perform actual file I/O directly with the DataNodes.
The following are some of the salient features that could be of interest to many users.
- Hadoop, including HDFS, is well suited for distributed storage and distributed processing using commodity hardware. It is fault tolerant, scalable, and extremely simple to expand. MapReduce, well known for its simplicity and applicability for large set of distributed applications, is an integral part of Hadoop.
- HDFS is highly configurable with a default configuration well suited for many installations. Most of the time, configuration needs to be tuned only for very large clusters.
- Hadoop is written in Java and is supported on all major platforms.
- Hadoop supports shell-like commands to interact with HDFS directly.
- The NameNode and Datanodes have built in web servers that makes it easy to check current status of the cluster.
-
New features and improvements are regularly implemented in HDFS.
The following is a subset of useful features in HDFS:
- File permissions and authentication.
- Rack awareness: to take a node's physical location into account while scheduling tasks and allocating storage.
- Safemode: an administrative mode for maintenance.
- fsck: a utility to diagnose health of the file system, to find missing files or blocks.
- fetchdt: a utility to fetch DelegationToken and store it in a file on the local system.
- Rebalancer: tool to balance the cluster when the data is unevenly distributed among DataNodes.
- Upgrade and rollback: after a software upgrade, it is possible to rollback to HDFS' state before the upgrade in case of unexpected problems.
- Secondary NameNode: performs periodic checkpoints of the namespace and helps keep the size of file containing log of HDFS modifications within certain limits at the NameNode.
Prerequisites
The following documents describe how to install and set up a Hadoop cluster:
- Single Node Setup for first-time users.
- Cluster Setup for large, distributed clusters.
The rest of this document assumes the user is able to set up and run a HDFS with at least one DataNode. For the purpose of this document, both the NameNode and DataNode could be running on the same physical machine.
Web Interface
NameNode and DataNode each run an internal web server in order to display basic information about the current status of the cluster. With the default configuration, the NameNode front page is at http://namenode-name:50070/. It lists the DataNodes in the cluster and basic statistics of the cluster. The web interface can also be used to browse the file system (using "Browse the file system" link on the NameNode front page).
Shell Commands
Hadoop includes various shell-like commands that directly interact with HDFS and other file systems that Hadoop supports. The command bin/hdfs dfs -help lists the commands supported by Hadoop shell. Furthermore, the command bin/hdfs dfs -help command-name displays more detailed help for a command. These commands support most of the normal files system operations like copying files, changing file permissions, etc. It also supports a few HDFS specific operations like changing replication of files. For more information see File System Shell Guide.
DFSAdmin Command
The bin/hadoop dfsadmin command supports a few HDFS administration related operations. The bin/hadoop dfsadmin -help command lists all the commands currently supported. For e.g.:
- -report : reports basic statistics of HDFS. Some of this information is also available on the NameNode front page.
- -safemode : though usually not required, an administrator can manually enter or leave Safemode.
- -finalizeUpgrade : removes previous backup of the cluster made during last upgrade.
- -refreshNodes : Updates the set of hosts allowed to connect to namenode. Re-reads the config file to update values defined by dfs.hosts and dfs.host.exclude and reads the entires (hostnames) in those files. Each entry not defined in dfs.hosts but in dfs.hosts.exclude is decommissioned. Each entry defined in dfs.hosts and also in dfs.host.exclude is stopped from decommissioning if it has aleady been marked for decommission. Entires not present in both the lists are decommissioned.
For command usage, see dfsadmin.
Secondary NameNode
The NameNode stores modifications to the file system as a log appended to a native file system file, edits. When a NameNode starts up, it reads HDFS state from an image file, fsimage, and then applies edits from the edits log file. It then writes new HDFS state to the fsimage and starts normal operation with an empty edits file. Since NameNode merges fsimage and edits files only during start up, the edits log file could get very large over time on a busy cluster. Another side effect of a larger edits file is that next restart of NameNode takes longer.
The secondary NameNode merges the fsimage and the edits log files periodically and keeps edits log size within a limit. It is usually run on a different machine than the primary NameNode since its memory requirements are on the same order as the primary NameNode. The secondary NameNode is started by bin/start-dfs.sh on the nodes specified in conf/masters file.
The start of the checkpoint process on the secondary NameNode is controlled by two configuration parameters.
- fs.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints, and
- fs.checkpoint.size, set to 64MB by default, defines the size of the edits log file that forces an urgent checkpoint even if the maximum checkpoint delay is not reached.
The secondary NameNode stores the latest checkpoint in a directory which is structured the same way as the primary NameNode's directory. So that the check pointed image is always ready to be read by the primary NameNode if necessary.
For command usage, see secondarynamenode.
Import Checkpoint
The latest checkpoint can be imported to the NameNode if all other copies of the image and the edits files are lost. In order to do that one should:
- Create an empty directory specified in the dfs.name.dir configuration variable;
- Specify the location of the checkpoint directory in the configuration variable fs.checkpoint.dir;
- and start the NameNode with -importCheckpoint option.
The NameNode will upload the checkpoint from the fs.checkpoint.dir directory and then save it to the NameNode directory(s) set in dfs.name.dir. The NameNode will fail if a legal image is contained in dfs.name.dir. The NameNode verifies that the image in fs.checkpoint.dir is consistent, but does not modify it in any way.
For command usage, see namenode.
Rebalancer
HDFS data might not always be be placed uniformly across the DataNode. One common reason is addition of new DataNodes to an existing cluster. While placing new blocks (data for a file is stored as a series of blocks), NameNode considers various parameters before choosing the DataNodes to receive these blocks. Some of the considerations are:
- Policy to keep one of the replicas of a block on the same node as the node that is writing the block.
- Need to spread different replicas of a block across the racks so that cluster can survive loss of whole rack.
- One of the replicas is usually placed on the same rack as the node writing to the file so that cross-rack network I/O is reduced.
- Spread HDFS data uniformly across the DataNodes in the cluster.
Due to multiple competing considerations, data might not be uniformly placed across the DataNodes. HDFS provides a tool for administrators that analyzes block placement and rebalanaces data across the DataNode. A brief administrator's guide for rebalancer as a PDF is attached to HADOOP-1652.
For command usage, see balancer.
Rack Awareness
Typically large Hadoop clusters are arranged in racks and network traffic between different nodes with in the same rack is much more desirable than network traffic across the racks. In addition NameNode tries to place replicas of block on multiple racks for improved fault tolerance. Hadoop lets the cluster administrators decide which rack a node belongs to through configuration variable dfs.network.script. When this script is configured, each node runs the script to determine its rack id. A default installation assumes all the nodes belong to the same rack. This feature and configuration is further described in PDF attached to HADOOP-692.
Safemode
During start up the NameNode loads the file system state from the fsimage and the edits log file. It then waits for DataNodes to report their blocks so that it does not prematurely start replicating the blocks though enough replicas already exist in the cluster. During this time NameNode stays in Safemode. Safemode for the NameNode is essentially a read-only mode for the HDFS cluster, where it does not allow any modifications to file system or blocks. Normally the NameNode leaves Safemode automatically after the DataNodes have reported that most file system blocks are available. If required, HDFS could be placed in Safemode explicitly using 'bin/hadoop dfsadmin -safemode' command. NameNode front page shows whether Safemode is on or off. A more detailed description and configuration is maintained as JavaDoc for NameNode.setSafeMode().
fsck
HDFS supports the fsck command to check for various inconsistencies. It it is designed for reporting problems with various files, for example, missing blocks for a file or under-replicated blocks. Unlike a traditional fsck utility for native file systems, this command does not correct the errors it detects. Normally NameNode automatically corrects most of the recoverable failures. By default fsck ignores open files but provides an option to select all files during reporting. The HDFS fsck command is not a Hadoop shell command. It can be run as 'bin/hadoop fsck'. For command usage, see fsck. fsck can be run on the whole file system or on a subset of files.
fetchdt
HDFS supports the fetchdt command to fetch Delegation Token and store it in a file on the local system. This token can be later used to access secure server (NameNode for example) from a non secure client. Utility uses either RPC or HTTPS (over Kerberos) to get the token, and thus requires kerberos tickets to be present before the run (run kinit to get the tickets). The HDFS fetchdt command is not a Hadoop shell command. It can be run as 'bin/hadoop fetchdt DTfile '. After you got the token you can run an HDFS command without having Kerberos tickets, by pointing HADOOP_TOKEN_FILE_LOCATION environmental variable to the delegation token file. For command usage, see fetchdt command.
Recovery Mode
Typically, you will configure multiple metadata storage locations. Then, if one storage location is corrupt, you can read the metadata from one of the other storage locations.
However, what can you do if the only storage locations available are corrupt? In this case, there is a special NameNode startup mode called Recovery mode that may allow you to recover most of your data.
You can start the NameNode in recovery mode like so: namenode -recover
When in recovery mode, the NameNode will interactively prompt you at the command line about possible courses of action you can take to recover your data.
If you don't want to be prompted, you can give the -force option. This option will force recovery mode to always select the first choice. Normally, this will be the most reasonable choice.
Because Recovery mode can cause you to lose data, you should always back up your edit log and fsimage before using it.
Upgrade and Rollback
When Hadoop is upgraded on an existing cluster, as with any software upgrade, it is possible there are new bugs or incompatible changes that affect existing applications and were not discovered earlier. In any non-trivial HDFS installation, it is not an option to loose any data, let alone to restart HDFS from scratch. HDFS allows administrators to go back to earlier version of Hadoop and rollback the cluster to the state it was in before the upgrade. HDFS upgrade is described in more detail in Hadoop Upgrade Wiki page. HDFS can have one such backup at a time. Before upgrading, administrators need to remove existing backup using bin/hadoop dfsadmin -finalizeUpgrade command. The following briefly describes the typical upgrade procedure:
- Before upgrading Hadoop software, finalize if there an existing backup. dfsadmin -upgradeProgress status can tell if the cluster needs to be finalized.
- Stop the cluster and distribute new version of Hadoop.
- Run the new version with -upgrade option (bin/start-dfs.sh -upgrade).
- Most of the time, cluster works just fine. Once the new HDFS is considered working well (may be after a few days of operation), finalize the upgrade. Note that until the cluster is finalized, deleting the files that existed before the upgrade does not free up real disk space on the DataNodes.
-
If there is a need to move back to the old version,
- stop the cluster and distribute earlier version of Hadoop.
- start the cluster with rollback option. (bin/start-dfs.h -rollback).
File Permissions and Security
The file permissions are designed to be similar to file permissions on other familiar platforms like Linux. Currently, security is limited to simple file permissions. The user that starts NameNode is treated as the superuser for HDFS. Future versions of HDFS will support network authentication protocols like Kerberos for user authentication and encryption of data transfers. The details are discussed in the Permissions Guide.
Scalability
Hadoop currently runs on clusters with thousands of nodes. The PoweredBy Wiki page lists some of the organizations that deploy Hadoop on large clusters. HDFS has one NameNode for each cluster. Currently the total memory available on NameNode is the primary scalability limitation. On very large clusters, increasing average size of files stored in HDFS helps with increasing cluster size without increasing memory requirements on NameNode. The default configuration may not suite very large clustes. The FAQ Wiki page lists suggested configuration improvements for large Hadoop clusters.
Related Documentation
This user guide is a good starting point for working with HDFS. While the user guide continues to improve, there is a large wealth of documentation about Hadoop and HDFS. The following list is a starting point for further exploration:
- Hadoop Site: The home page for the Apache Hadoop site.
- Hadoop Wiki: The home page (FrontPage) for the Hadoop Wiki. Unlike the released documentation, which is part of Hadoop source tree, Hadoop Wiki is regularly edited by Hadoop Community.
- FAQ: The FAQ Wiki page.
- Hadoop JavaDoc API.
- Hadoop User Mailing List : core-user[at]hadoop.apache.org.
- Explore src/hdfs/hdfs-default.xml. It includes brief description of most of the configuration variables available.
- Hadoop Commands Guide: Hadoop commands usage.


@(flags: List[(String, Int)])(implicit context: service.AppContext[AnyContent]) @main("Dashboard for Contributors"){
| Flag | Count |
|---|---|
| @f._1 | @f._2 |
The following exams need some work. We require content, review, improvement so that all questions become good quality.


Status:
New feature! Read mainstream media chess articles here!
©<EMAIL_ADDRESS>Home| Tactics| Audio Chess| Blind Chess


| Package | Description |
|---|---|
| com.opengamma.strata.product.cms | |
| com.opengamma.strata.product.swap |
Entity objects describing a swap.
|
| Modifier and Type | Method and Description |
|---|---|
SwapIndex |
ResolvedCmsLeg.getIndex()
Gets the swap index of the leg.
|
SwapIndex |
CmsPeriod.getIndex()
Gets the swap index.
|
SwapIndex |
CmsLeg.getIndex()
Gets the swap index.
|
| Modifier and Type | Method and Description |
|---|---|
MetaProperty<SwapIndex> |
CmsPeriod.Meta.index()
The meta-property for the
index property. |
MetaProperty<SwapIndex> |
CmsLeg.Meta.index()
The meta-property for the
index property. |
| Modifier and Type | Method and Description |
|---|---|
CmsPeriod.Builder |
CmsPeriod.Builder.index(SwapIndex index)
Sets the swap index.
|
CmsLeg.Builder |
CmsLeg.Builder.index(SwapIndex index)
Sets the swap index.
|
| Modifier and Type | Class and Description |
|---|---|
class |
ImmutableSwapIndex
A swap index implementation based on an immutable set of rules.
|
| Modifier and Type | Field and Description |
|---|---|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_10Y
EUR Rates 1100 for tenor of 10 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_12Y
EUR Rates 1100 for tenor of 12 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_15Y
EUR Rates 1100 for tenor of 15 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_1Y
EUR Rates 1100 for tenor of 1 year.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_20Y
EUR Rates 1100 for tenor of 20 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_25Y
EUR Rates 1100 for tenor of 25 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_2Y
EUR Rates 1100 for tenor of 2 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_30Y
EUR Rates 1100 for tenor of 30 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_3Y
EUR Rates 1100 for tenor of 3 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_4Y
EUR Rates 1100 for tenor of 4 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_5Y
EUR Rates 1100 for tenor of 5 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_6Y
EUR Rates 1100 for tenor of 6 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_7Y
EUR Rates 1100 for tenor of 7 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_8Y
EUR Rates 1100 for tenor of 8 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1100_9Y
EUR Rates 1100 for tenor of 9 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_10Y
EUR Rates 1200 for tenor of 10 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_12Y
EUR Rates 1200 for tenor of 12 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_15Y
EUR Rates 1200 for tenor of 15 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_1Y
EUR Rates 1200 for tenor of 1 year.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_20Y
EUR Rates 1200 for tenor of 20 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_25Y
EUR Rates 1200 for tenor of 25 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_2Y
EUR Rates 1200 for tenor of 2 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_30Y
EUR Rates 1200 for tenor of 30 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_3Y
EUR Rates 1200 for tenor of 3 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_4Y
EUR Rates 1200 for tenor of 4 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_5Y
EUR Rates 1200 for tenor of 5 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_6Y
EUR Rates 1200 for tenor of 6 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_7Y
EUR Rates 1200 for tenor of 7 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_8Y
EUR Rates 1200 for tenor of 8 years.
|
static SwapIndex |
SwapIndices.EUR_EURIBOR_1200_9Y
EUR Rates 1200 for tenor of 9 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_10Y
GBP Rates 1100 for tenor of 10 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_12Y
GBP Rates 1100 for tenor of 12 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_15Y
GBP Rates 1100 for tenor of 15 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_1Y
GBP Rates 1100 for tenor of 1 year.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_20Y
GBP Rates 1100 for tenor of 20 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_25Y
GBP Rates 1100 for tenor of 25 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_2Y
GBP Rates 1100 for tenor of 2 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_30Y
GBP Rates 1100 for tenor of 30 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_3Y
GBP Rates 1100 for tenor of 3 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_4Y
GBP Rates 1100 for tenor of 4 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_5Y
GBP Rates 1100 for tenor of 5 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_6Y
GBP Rates 1100 for tenor of 6 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_7Y
GBP Rates 1100 for tenor of 7 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_8Y
GBP Rates 1100 for tenor of 8 years.
|
static SwapIndex |
SwapIndices.GBP_LIBOR_1100_9Y
GBP Rates 1100 for tenor of 9 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_10Y
USD Rates 1100 for tenor of 10 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_15Y
USD Rates 1100 for tenor of 15 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_1Y
USD Rates 1100 for tenor of 1 year.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_20Y
USD Rates 1100 for tenor of 20 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_2Y
USD Rates 1100 for tenor of 2 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_30Y
USD Rates 1100 for tenor of 30 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_3Y
USD Rates 1100 for tenor of 3 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_4Y
USD Rates 1100 for tenor of 4 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_5Y
USD Rates 1100 for tenor of 5 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_6Y
USD Rates 1100 for tenor of 6 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_7Y
USD Rates 1100 for tenor of 7 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_8Y
USD Rates 1100 for tenor of 8 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1100_9Y
USD Rates 1100 for tenor of 9 years.
|
static SwapIndex |
SwapIndices.USD_LIBOR_1500_1Y
USD Rates 1500 for tenor of 1 year.
|
| Modifier and Type | Method and Description |
|---|---|
static SwapIndex |
SwapIndex.of(String uniqueName)
Obtains an instance from the specified unique name.
|
| Modifier and Type | Method and Description |
|---|---|
static ExtendedEnum<SwapIndex> |
SwapIndex.extendedEnum()
Gets the extended enum helper.
|
Copyright 2009-Present by OpenGamma Inc. and individual contributors
Apache v2 licensed
Additional documentation can be found at opengamma.github.io/StrataDocs.


This ced-capstone is to show how an external remote service can be governed by blockchain smart contracts. A series of service and state contracts govern the interactions, while a remote service bridge initiates the remote service and returns the results to the smart contract.
For now, we created a very simple version that shows the current block number.


--- layout: game-page title: "Raelum Skywatchers" header-img: "img/games/raelum-skywatchers/game_banner.jpg" meta_description: "Play Raelum Skywatchers! A cat flying physics game with tons of cool upgrades. That's right, cats can fly too!" ---
Raelum, a distant planet civilized by cats, is facing a crisis! Guide Taron through his flight-filled journey to find his father and save his planet! Can you help our hero achieve the ranks of a Raelum Skywatcher? Show off your love for kitties and physics through fun-filled stages in the game. There are plenty of equipment and upgrades to try out. Be sure to unlock every achievement!


--- layout: page description: "Welcome" ---
My name is Matt DePero, and I am currently was a computer science student at Miami University. I am dedicated to obtaining the knowledge and experience necessary to achieve a meaningful career in the field of information technology. I have always had a passion for computing, and through coursework, real world experience, a problem solving mindset, and high work ethic, I hope to leave a lasting impact on the industry.
This website is designed to create a profile of my projects and experience while a student at Miami.


Toggle navigation
John Smith
My Account
Log Out
Action Center
Donate
John Smith
Fairfax, VA
100
Calls Attempted
9
Calls Completed
19
Petitions Signed
Top Priority
BENGHAZI
Say You’ll Watch 13 Hours: The Secret Soldiers of Benghazi
Survey
Current Call Surveys
Election 2016
What Questions Do You Want Asked During the Democrat Debate?
Petition
Survey
Quick 1 Question Survey
Survey
Latest Petitions
Religious Freedom
Tell Obama to Defend Religious Freedom
Petition
Veterans Affairs Scandal
Stand With Our Veterans
Petition
All
Surveys
Petitions
2nd Amendment
Stand up for the Second Amendment
Petition
Terrorism
Obama’s Wrong on Syrian Refugees
Petition
Barack Obama
Grade Obama’s Presidency
Survey
Terrorirsm
Keep Terrorists Out of My Backyard
Petition
Benghazi
The Secret Soldiers of Benghazi
Survey
2nd Amendment
Stand up for the Second Amendment
Petitiion
Terrorism
Obama’s Wrong on Syrian Refugees
Petition
Barack Obama
Lorem ipsum dolor sit amet, cons ectetur adipi sicing elit. Incidunt sdfsf
Survey
Load More


Default validation rules
Selected option must be
Mr
Please select your title
Selected value must be Mr
Select title
Mr
Mrs
Miss
Ms
Minimum length must be
3 characters
Please provide your first name
Length must be at least 3 characters
Validation triggered by ajax response
Minimum length must be
3 characters
Please provide your last name
Length must be at least 3 characters
Value must be a valid
email address
Please provide your email address
Invalid email address
Value must be a
password matching regex
and confirmed
Please choose your password
Length must be at least 6 characters, one capital letter and one number
Passwords do not match
Value must be a
password matching regex
and match password field
Please confirm your password
Length must be at least 5 characters
This field will be ignored because it does not have a valid attribute
One option must be selected
You must select one option
Option 1
Option 2
Option 3
At least one option must be selected
You must select at least one option
Option 1
Option 2
Option 3
Exactly one option must be selected
You must select exactly one option
You must select exactly one option
Option 1
Option 2
Option 3
Checkbox must be checked
You must agree to our newsletter and conditions
I agree to the terms and conditions
PROCESSING
Custom action and serialize attribute
Selected option must be
Mr
Please select your title
Selected value must be Mr
Select title
Mr
Mrs
Miss
Ms
PROCESSING
Custom regex
The string must contain only capital letters
^[A-Z]+$
Please select your title
Invalid format
PROCESSING
CKEDITOR
Please provide a body
Please provide a body
PROCESSING


Nature Blogs is a science blog aggregator. Since May 1st 2013 API keys are no longer necessary.
| ALM Name | nature |
| ALM Configuration | staleness: [ 1.day, 1.month, 1.month, 1.month] requests_per_hour: 200 |
| ALM Core Attributes | id url date (as published_at) |
ALM Other Attributes | title blog title blog url |
| Protocol | REST |
| Format | JSON |
| Rate-limiting | 2/sec 5,000/day |
| Authentication | no |
| Restriction by IP Address | no |
| API URL | http://blogs.nature.com/posts.json?doi=DOI&api_key=API_KEY |
| License | CC-BY-NC |
Example Response
[
{
post: {
percent_complex_words: 12.9994,
popularity: 0,
created_at: "2009-02-05T17:12:41Z",
title: "A new link in the chain of whale evolution",
body: "Paleontologists have found a new fossil of a whale ancestor - and its announced just after I finish watching my preview DVD of Nat Geo's Morphed on whale evolution. I smell fate.Anyhow, the new whale predecessor was unveiled in a PLoS One article ...",
updated_at: "2009-09-22T23:09:56Z",
flesch: -21.5166,
url: "feedproxy.google.com/~r/observationsofanerd/~3/aMcNB2CZW_Y/new-link-in-chain-of-whale-evolution.html",
blog_id: 558,
id: 29082,
hashed_id: "a8b538120801f6735681c4ce346e3d27f4c1739f3bf349eec7f206d34cdba95b",
blog: {
percent_complex_words: 14.0934,
popularity: 2,
created_at: "2008-11-05T16:42:48Z",
title: "Observations of a Nerd",
niche_tag: "life_sciences",
updated_at: "2012-07-28T01:13:27Z",
flesch: -39.0166,
feed_url: "scienceblogs.com/observations/index.xml",
url: "http://scienceblogs.com/observations/",
decided_on: "2008-11-06T19:34:40Z",
rank: 426,
num_posts_per_week: 1.754,
id: 558,
outgoing_bloglove: 2,
num_words: 5852.28,
ip: "bogus address",
niche_rank: 141,
header_title: "BLOGS",
fog: 46.4802,
description: "A blog about anything and everything that piques the interest of a biologist",
status: "accepted"
},
links_to_doi: [
"10.1371/journal.pone.0004366"
],
num_words: 6708,
published_at: "2009-02-05T17:12:41Z",
fog: 40.0465
}
},


{{app.name}}, Version {{app.version}}
License
[2014] - [2022] Rapidfacture GmbH. All Rights Reserved.
All information, intellectual and technical concepts contained herein are
property of Bernhard Römer and Felix Furtmayr, the main owners of Rapidfacture GmbH and are protected by trade secret or copyright law.
Dissemination of this information or reproduction of this material is strictly
forbidden unless prior written permission is obtained from Rapidfacture GmbH.
info@rapidfacture.com


You can invite up to 2 additional users on the Free plan. There is no limit on team members for the Premium plan.
You can invite up to 2 additional users on the Free plan. There is no limit on team members for the Premium plan.
No more than 2GB. All files in your account must fit your alloted storage space.
Click “Forgot password” from the login page or “Change password” from your profile page. A reset link will be emailed to you.
Yes! Send us a message and we’ll process your request no questions asked.
Chat and email support is available 24/7. Phone lines are open during normal business hours.


|
ELENA Standard Library 2.0 |
system'Object | +--system'io'BinaryReader
A binary reader prototype
| Method Summary | |
close |
Closes the reader and its stream
|
read &bytearray:bytearray
&length:int
= int
|
Reads the specified number of bytes from the stream
|
read &bytearray:bytearray
&length:int
&out'length:int
|
Reads the specified number of bytes from the stream
|
read &out'byte:byte
|
Reads a byte
|
read &out'char:char
|
Reads a character
|
read &out'int:int
|
Reads an integer
|
read &out'long:long
|
Reads a long integer
|
read &out'real:real
|
Reads a real number
|
read &out'short:short
|
Reads a short integer
|
stream = stream
|
Returns the owner stream
|
system'Object | +--system'io'BinaryWriter
A binary writer prototype
| Method Summary | |
close |
Closes the reader and its stream
|
stream = stream
|
Returns the owner stream
|
write &byte:byte
|
Writes a byte
|
write &bytearray:bytearray
&length:int
|
Writes the specified number of bytes into the stream
|
write &char:char
|
Writes a character
|
write &int:int
|
Writes an integer
|
write &long:long
|
Writes a long integer
|
write &real:real
|
Writes real number
|
write &short:short
|
Writes a short integer
|
system'Object | +--system'io'Stream | +--system'io'FileStreamAll Implemented Protocols:
A file stream
| Constructor Summary | |
openForRead &path:literal
|
Opens a file stream for reading
|
openForReWrite &path:literal
|
Opens a file stream for writing
|
| Method Summary | |
close |
Closes the current stream and releases any resources associated with the current stream
|
read &bytearray:bytearray
&length:int
&out'int:int
|
Reads a sequence of bytes from the current stream
|
write &bytearray:bytearray
&length:int
|
Writes a sequence of bytes into the current stream
|
system'Object | +--system'Exception | +--system'io'IOException
A i/o exception
| Constructor Summary | |
new |
Creates the exception
|
new &message:literal
|
Creates an exception with a custom error message
|
| Method Summary | |
if &IOError:function
|
Executes the exception handling code
|
system'Object | +--system'io'StreamAll Implemented Protocols:
A stream prototype
| Method Summary | |
close |
Closes the current stream and releases any resources associated with the current stream
|
stream = stream
|
Returns itself
|
read &bytearray:bytearray
&length:int
&out'int:int
|
Reads a sequence of bytes from the current stream
|
write &bytearray:bytearray
&length:int
|
Writes a sequence of bytes into the current stream
|
system'Object | +--system'io'TextWriter | +--system'io'StreamWriter
A text file writer
| Constructor Summary | |
new &path:literal
|
Creates a new Ansi file
|
new &stream:stream
&encoder:encoder
|
Creates a text writer for the provided stream and encoding
|
| Method Summary | |
close |
Closes the reader and its stream
|
encoder = encoder
|
Returns the current encoder
|
stream = stream
|
Returns the owner stream
|
write &chararray:chararray
&length:int
|
Writes the specified number of characters into the stream
|
write &literal:literal
&length:int
|
Writes the specified number of characters into the stream
|
system'Object | +--system'io'TextReader | +--system'io'StreamReader
A text file reader
| Constructor Summary | |
new &path:literal
|
Opens an existing Ansi file
|
new &stream:stream
&encoder:encoder
|
Creates a text reader for the provided stream and encoding
|
| Method Summary | |
close |
Closes the reader and its stream
|
encoder = encoder
|
Returns the current encoder
|
stream = stream
|
Returns the owner stream
|
read &chararray:chararray
&length:int
&out'length:int
|
Reads the specified number of characters from the stream
|
system'Object | +--system'io'TextReader
A text reader prototype
| Method Summary | |
close |
Closes the reader and its stream
|
encoder = encoder
|
Returns the current encoder
|
read &chararray:chararray
&length:int
&out'length:int
|
Reads the specified number of characters from the stream
|
readLine = literal
|
Reads a line of characters from the current stream and returns the data as a string.
|
stream = stream
|
Returns the owner stream
|
system'Object | +--system'io'TextWriter
A text writer prototype
| Method Summary | |
close |
Closes the reader and its stream
|
encoder = encoder
|
Returns the current encoder
|
stream = stream
|
Returns the owner stream
|
write &chararray:chararray
&length:int
|
Writes the specified number of characters into the stream
|
write &literal:literal
|
Writes the literal into the stream
|
write &literal:literal
&length:int
|
Writes the specified number of characters into the stream
|
writeLine : literal
|
Writes a literal into the current stream and a new line symbols.
|
|
ELENA Standard Library 2.0 |


--- layout: post title: Virtual Sprawl On Wall Street date: '2009-12-02T23:03:19-08:00' tags: tumblr_url: http://alexhwilliams.tumblr.com/post/267394617/virtual-sprawl-on-wall-street ---
Having implemented virtualization software to conquer “server sprawl” (or, more specifically, to perform more work on fewer servers and thus minimize the proliferation of servers), many Wall Street firms are now in the throes of “virtual sprawl.” This panicky term refers to the condition in which IT managers and, in some cases, even end users have been installing virtual machines all over the organization, creating a chaotic mess that is hard to manage.


Projects
Polymapper 📍
A simple web-based tool to generate coordinate arrays from polylines. You can either paste coords or choose an area on the map.
SixthBoro Ⓜ
The iOS application that better helps display PATH train times in a fast and feature-packed way. Built with my friend Avi.
Walkmore 🚶♂
This project is aimed at helping users to be more active. Walkmore will help a user plan a day trip. Walkmore helps its users create a walking path. The path a user take will ultimately get the user from point A to point B, but it will take the user by historic landmarks (of the user’s choosing) that are along the way. Built at Flatiron School in 2016.
Bolt Learning ⚡
Worked on an iOS client along with Jack and Phil. Bolt leverages innovative adaptive learning algorithms to provide students with a smarter way to prepare for the SAT®.
Open Source
Brocade 📦
Swift library for the free and open product database.
News API v2 🗞
Swift wrapper for the News API v2.
WhaleAlert 🐋
Swift implementation of the WhaleAlert API.
PopcornNotify 🍿
Send simple emails and text messages in Swift from PopcornNotify.
Jumprock ✉
Send static forms in Swift via Jumprock’s API.
ApproveAPI (deprecated) ✅
ApproveAPI in Swift.
Savvy (deprecated) 💱
Savvy Wallet cryptocurrency tools, formerly PayBear.
PayBear (deprecated) 🐻
Paybear API on iOS.


--- layout: example permalink: /examples/easymde/ exampleId: easymde stylesheets: - /libs/easymde/dist/easymde.min.css -<EMAIL_ADDRESS>- /examples/easymde/easymde.css overview: |- This example demonstrates editing of markdown with styling applied in line using the Easy Markdown Editor. The Easy Markdown Editor uses CodeMirror under the hood, so we used several of our CodeMirror bindings to implement this integration with very little code. ---


Why is my time infinite?
This happens when the UV Index value is zero. If the sun has set, you can stay as long as you want outside.
error_outline DisclaimerThis happens when the UV Index value is zero. If the sun has set, you can stay as long as you want outside.
error_outline DisclaimerOften times when new users use this app as soon as they add a sunscreen they will notice that their exposure time will skyrocket. The app may even tell you you can stay several hours, if not days in the sun. So is this real? In theory yes, if you had a constantly perfect sunscreen application that would be the case. In reality though, sunscreen fades off with time due to sweating, water, friction, etc... So the answer is ONLY IF YOU REAPPLY SUNSCREEN EVERY 2 HOURS.
error_outline DisclaimerSunscreens are used to protect the skin from the harmful effects of the sun. They help to prevent sunburn and premature aging (such as wrinkles, leathery skin). Sunscreens also help to decrease the risk of skin cancer and also of sunburn-like skin reactions (sun sensitivity) caused by some medications (including tetracyclines, sulfa drugs, phenothiazines such as chlorpromazine).
language SourceSunscreen should never be used to prolong the duration of sun exposure. - WHO
Wearing sunscreen does not mean that you can stay out longer in the sun. Sunscreens cannot protect against all of the sun's radiation. - WebMD
language SourceMost people wear 25-50% of the recommended amount, which is why undesirable sunburns and tanning can occur despite sunscreen application. To achieve the Sun Protection Factor (SPF, which protects against the sun’s UVB radiation) reflected on a bottle of sunscreen, you should use approximately two milligrams of sunscreen per square centimeter of skin. In practice, this means applying the equivalent of a shot glass.
language SourceApply it 30 minutes before sunbathing. Repply it every 2 hours. If you have undergone an activity that has potentially removed it (sweating, swimming, rubbing on a towel) reapply it immediately.
language SourceYou can still get a tan with sunscreen on, because while sun creams block many damaging skin-burning rays, they're not as great at filtering out the milder rays, which penetrate deeper into the skin.
language SourceSunscreens are commonly rated and labeled with a sun protection factor (SPF) that measures the fraction of sunburn-producing UV rays that reach the skin. For example, "SPF 15" means that 1⁄15 of the burning radiation reaches the skin through the recommended thickness of sunscreen.
language SourceUVA: the longest waves (315-400nm) penetrate deep into the skin causing skin aging. They don’t make you tan.
UVB: the medium waves (280-315nm) can make you tan, but can also be responsible for skin cancer.
UVC: the shortest waves (100-280nm) are absorbed in the atmosphere.
Every intense sunbathing episode brings you closer to skin aging and overexposure can lead to skin cancer.
language Source"There is no healthy way to tan. Any tan can increase your risk of developing skin cancer. Getting a tan does very little to protect your skin from the harmful effects of the sun. The idea that there is such a thing as a healthy tan is a myth. Any tan can increase the risk of skin cancer. The best thing to avoid skin cancer is to avoid getting tanned." - NHS
language SourceClouds do affect UV radiation but they don’t always decrease it. Depending on factors such as the type of cloud and the thickness of the cloud.
- UV radiation can penetrate through a thin cloud.
- Patchy clouds could also slightly intensify UV levels because radiation can be reflected off the cloud’s edge.
- Thicker clouds block part of the UV radiation, but not all.
language SourceNo. Every material has an albedo (reflectivity). Generally you should be ok if you’re not in direct sunlight unless you’re next to snow or on the beach. Snow is highly reflective and the sea is considered reflective at high angles of incidence. You can get a noticeable sunburn from sitting under an umbrella in the grass in the middle of the day (sunny or cloudy) after just 35 minutes.
language SourceFor people who have a history or high risk of skin cancer, genetic diseases such as albinism or xeroderma pigmentosum or certain immune disorders, SPF 50 may not be enough. You should avoid exposing yourself to the sun as much as possible.
language SourceSome UV radiation is essential to the body as it stimulates the production of vitamin D. Vitamin D has an important function in increasing calcium and phosphorus absorption from food and plays a crucial role in skeletal development, immune function and blood cell formation. There is no doubt that a little sunlight is good for you! But 5 to 15 minutes of casual sun exposure of hands, face and arms two to three times a week during the summer months is sufficient to keep your vitamin D levels high.
language SourceYes, you do. Compared to fair-skinned people, dark-skinned people have a much lower risk of developing melanoma or non-melanoma skin cancers. They do not normally have to apply sunscreens and can safely tolerate relatively high levels of UV radiation without getting burnt. But – regardless of skin colour - the risk of eye damage and of harmful effects on the immune system remains.
language SourceIt’s important not to rely on high-SPF sunscreens alone. No single method of sun defense can protect you perfectly. Sunscreen is just one vital part of a strategy that should also include seeking shade and covering up with clothing, including wide-brimmed hats and UV-blocking sunglasses.
language Source


This is the first question.
This is the second question.
This is the third question.
This is the fourth question.
This is the fifth question.
This is the sixth question.
This is another question.
This is another question.
This is another question.
This is another question.
This is another question.
This is another question.
This is another question.
This is another question.
This is another question.


Dear Western Union Customer,
Western Union Deparment has recently reviewed your account, and suspect that your account may have been accessed from an unauthorised computer or by a third party. This may be due to changes in your IP address or location. Protecting the security of your account and the Western Union network is our primary concern.
Western Union Online Department has temporarily locked your account and recommends you to follow the link below to verify your identity and unlock your account.
https://www.westernunion.com/WUCOMWEB/unlockMid.do?
To protect your account, please keep in mind these instructions:
* Do not share your password with other users.
* Log off and close the internet explorer window after using your online account, especially if you are in a public place.
We apologise for any inconvenience this may cause, and appreciate your assistance in helping us maintaing the integrity of the entire Western Union system
Regards,
Western Union Security Team


| Production |
Increasing temperature extremes and variability, and changes in precipitation patterns and extreme weather events can cause damage to agricultural crops, reduce productivity, and decrease yield [
37,
38,
39,
47,
48]
|
| In the Prairies, groundwater quality and quantity may be reduced as summer rainfall decreases, which can increase irrigation needs for agricultural crops [
49]
|
| Water resources will face increasing demand and competition from other users (i.e., oil and gas industry) [
49] as well as domestic demand [
50], causing further impacts on water availability
|
| Sea level rise could cause inundation of agricultural lands in coastal regions, damaging crops, and creating unsuitable conditions for agricultural production and result in salt water intrusion of aquifers, reducing the quality of irrigation water [
38]
|
| Increasing temperatures and changes in precipitation patterns may create more favorable conditions for pests, invasive species and plant diseases [
36,
37,
48]
|
| Rising temperatures and increased concentrations of atmospheric CO
2 may decrease the effectiveness of some herbicides used for pest control [
8]
|
| Increasing ozone pollution, a by-product of fossil fuel combustion, can inhibit photosynthesis in plants, reducing the quality and productivity of the crop [
37,
48]
|
| Temperature extremes can adversely impact livestock health and decrease productivity [
48]
|
| Extreme weather events may reduce land available for livestock pasture and foraging [
36]
|
| The distribution and productivity of natural and farmed fish will change as ocean and freshwater temperatures and ocean acidification increase [
8,
38]
|
| Rising temperatures may create favorable conditions for aquatic disease and invasive species [
51]
|
| Increasing temperatures and changing precipitation patterns are changing the quality and distribution of populations of traditionally harvested species in Canada (i.e., Caribou) [
14]
|
| Processing |
Increasing temperatures and extreme temperature events may increase the risk of food spoilage or contamination while in storage at processing facilities [
40]
|
| Traditional food storage and preparation practices may be at risk, for example, permafrost melt may have implications for the stability and safety of traditional in-ground freezers used by Indigenous communities in the Canadian North [
14]
|
| Reduced or variable water availability may challenge food processing operations, which require significant amounts of potable water [
38]
|
| The physical infrastructure of processing facilities may be damaged by extreme weather events (i.e., flooding) which can disrupt operations [
40]
|
| Extreme weather events may cause disruption to energy supplies, labor availability and technological infrastructure critical to processing operations [
40]
|
| Availability, quality, and the cost of raw materials and inputs may be variable as a result of climate change impacts in the food production sector, from both international and domestic sources [
7,
41]
|
| Distribution |
Extreme weather events can cause damage and disruption to transportation infrastructure including road, rail, marine, and air transport infrastructure as well as urban and public transportation systems [
42]
|
| Temperature extremes, permafrost melt, changes in precipitation patterns, and freeze-thaw cycles can compromise the integrity of road, rail, marine, and air transport infrastructure [
42]
|
| Extreme weather events can cause physical damage to distribution facilities (i.e., grocery stores and food banks) and also disrupt energy supplies, labor availability, and technological infrastructure critical for distribution site operations [
40,
41]
|
| Preparation and Consumption |
Increasing temperatures, changes in precipitation patterns, and extreme weather events can create favorable conditions for the transmission, survival, and growth of many common foodborne pathogens [
40,
52,
53]
|
| Increasing temperatures may result in a rise of activities where food preparation and consumption occurs outdoors (i.e., barbeques, picnics), potentially increasing the risk of exposure to foodborne illness [
40,
54]
|
| Extreme weather events may facilitate chemical contamination at food production sites (i.e., contaminated flood waters inundating agricultural crops) [
40]
|
| An increase in ocean temperature and changes in salinity may increase the risk of pathogens that are known to contaminate seafood (i.e.,
Vibrio) [
40,
52,
55]
|
| In the Canadian North, climate change may allow for the emergence of new pathogens, viruses, and parasites that affect wildlife harvested as part of traditional and country food systems [
14]
|
| A changing climate may increase the need for the use of pesticides which can lead to increased pesticide residues in the food supply [
6]
|
| Increasing concentrations of atmospheric CO
2 can alter the nutritional content of some agricultural crops, with studies indicating decreased concentration of protein, iron, zinc, and key minerals [
8,
40,
56,
57]
|


Coming later this month is the inaugural edition of the Kai Studios Cup, a one-day single-elimination bracket to see who takes home the bragging rights and the cash.
Kai Studios is a New Zealand based casting organisation focussed on providing a professional, polished product maximising the viewer's experience. Founded by Zac Neill in 2016, Kai Studios arose out of a need to both grow the scene and to create a sports-broadcast feel to all aspects of the production process. While the base will always be in Dota 2, Kai Studios is open to any and all esports. For any casting or business enquiries please contact admin@kaistudios.nz.


What is Boilersuit
Boilersuit Light is a front-end application to extract, prepare, transform and analysestructured data. There is no need to set up a database connection to start working - Boilersuit automatically creates a local SQLite database file. This works astonishingly well with data up to a couple of gigabytes (in theory almost unlimited).
Boilerspeech and SQL short-hand notation
Transformation logic is defined in a special script language which is a superset of SQL. That means you can use native SQL if you like:
CREATE TABLE germanclients AS SELECT id, name, location FROM clients WHERE location='Germany' ORDER BY name ASC;The shorthand BS like notation would be
germanclients := clients(id, name{ASC}, location{='Germany'});More handy short notations can be found here. Apart from the handy short notations for frequently used SQL queries
Pre-Processing Functions
As an additional feature, BS provides pre-processing functions for attributes, e.g. to 'suck' out parts of an attribute based on regular expressions, or to conveniently format numbers and dates, and others like so:
germanclients2 := germanclients(id, SUCK(name, [A-Za-z]+, 1) AS firstname, SUCK(name, [A-Za-z]+, 1) AS lastname);More on this here.
Boilerspeech
Short Notation
SELECT
Transformation logic is defined in a special script language which is a superset of SQL. That means you can use native SQL if you like:
CREATE TABLE germanclients AS SELECT id, name, location FROM clients WHERE location='Germany' ORDER BY name ASC;The shorthand BS like notation would be
germanclients := clients(id, name{ASC}, location{='Germany'});
COUNT
#table; #mytable(attr1 like 'some value'); #cars(serialnumber = 12345);
LEFT OUTER JOIN
-- SELECT * FROM ourcompanies oc LEFT OUTER JOIN companylist cl ON oc.name = cl.name exactmatch := ourcompanies(name)->companylist(name);
FREQUENCY TABLES
Often we have a code, or an id, in two tables and we would like to count how often they occur in any of both tables (if at all), and also see which keys exist only in the one but not in the other, and vice versa.
exactmatch := ourcompanies(name)->companylist(name);
CREATE / DROP TABLES
Creating/Dropping a table in Boilersuit is as simple as typing
+mytable(*id, attribute1, attribute2); -- create -mytable; -- drop
Preprocessing Functions
A range of preprocessing functions are available:| Function | Purpose |
|---|---|
| bsfMagicDate | Tries to recognize a date expression from various formats into the BS default format MM.dd.yyyy.
Works even if the formats in the source data vary from row
to row. For unrecognized expressions, a warning will be shown.
-- checking against common formats transformed := source(, magicDate(MYDATE) AS STANDARDIZED_MY_DATE);If a preferred set of allowed formats should be used, they can be added as arguments: -- checking against y-m-d and y.m.d: transformed := source(, magicDate(MYDATE, y-m-d, y.m.d[,...]) AS STANDARDIZED_MY_DATE); |
| bsfFormatNumber | Constructs a string using a pre-existing number in a certain way often useful to construct artificial IDs
for things. Lets assume we have a table with id's
id ----- 1 239 4000Then, we may want to construct id's of uniform length like so: result := table(id, formatNumber(id, A-DDDD) AS NEW_ID); -- Result will be as follows: id NEW_ID ---- ----------- 1 A-0001 239 A-0239 4000 A-4000 |
| bsfSuck | "Sucks" out specific patterns from the source data based on a regular expression |
| bsfHash | Hashes the source data using the String.hashCode method (probably depends on JVM used to run BS). If the hash value is > 0, an 'X' is added to the hash as a prefix, otherwise and 'Y' - this explains the Xxxxxx resp. Yxxxxx format of the hash result. |
How to get it
Boilersuit comes in 3 pieces:| Component | What is it | Get it |
|---|---|---|
| boilersuit-core | This is the engine behind BS containing what you need to connect to databases, interpretation logic needed to run scripts etc. It is a pre-requisite of any Boilersuit application and it's a java library. | Open Source, at GitHub |
| boilersuit-light | The free version of BoilerSuit consisting of a simple command-line like interface to manipulate and view data, to play around with it all and to run your BS scripts. It contains the boiler-suit core library and won't run without it | Open Source, at GitHub |
| boilersuit-professional | The professional version of Boilersuit contains special features targeted at various business needs such as data migration, reconciliation, predicitve modelling. | Contact us |


Welcome To The Beautiful Fyre Resort
Beachfront Accommodations
Island Activities
Travel Arrangements from Our Partners
Media
Contact Us!


Waypoint App
Features
- * SASS support including sourceMaps
- * Minimal CSS styling of the view
- * Gulp watch, build and local server tasks
- * Responsive navigation
- * Owl slider directive
- * Responsive navigation
- * localStorage service for set, get, remove data
- * queryService $http wrapper to handle calls
- * clear folder structure
- * less than 10 request in build version
- * minified CSS and JS build files
- * google analytics snippet


iPhone 4
iPad 2
iPod touch
iPod classic
iPod shuffle
iPod nano
MacBook Air
MacBook Pro
Mac mini
Mac Pro


Builds and returns an HTMLElement for the given result.
the result to build an HTMLElement from.
Builds and returns an array of HTMLElement with the given result set.
the result set to build an array of HTMLElement from.
Return the debug info about this component.
Disable the component. Normally this means that the component will not execute handlers for the framework events (query events, for example). Component are enabled by default on creation.
Executes a query to fetch new results. After the query returns, renders the new results.
Asserts that there are more results to display by verifying whether the last query has returned as many results as requested.
Asserts that the ResultList
is not currently fetching results.
The number of results to fetch and display.
Enable the component. Normally this means that the component will execute handlers for the framework events (query events, for example). Components are enabled by default on creation.
Get the fields needed to be automatically included in the query for this result list.
Return the bindings, or environment, for the current component.
Gets the list of currently displayed result.
Gets the list of currently displayed result HTMLElement.
Empties the current result list content and appends the given array of HTMLElement.
Can append to existing elements in the result list, or replace them.
Triggers the newResultsDisplayed
and newResultDisplayed
events.
Get the bound component to the given HTMLElement. Throws an assert if the HTMLElement has no component bound, unless using the noThrow argument.
If there is multiple component bound to the current HTMLElement, you must specify the component class.
HTMLElement for which to get the bound component.
Optional component class. If the HTMLElement has multiple components bound, you must specify which one you are targeting.
Boolean option to tell the method to not throw on error.
Allows the component to bind events and execute them only when it is enabled.
Contains the state of options for different components. Mainly used by ResultLink.
Contains the state of different components (enabled vs disabled). Allows to get/set values. Triggers component state event when modified. Each component can listen to those events.
A disabled component will not participate in the query, or listen to ComponentEvents.
Allows component to log in the dev console.
Contains the singleton that allows to trigger queries.
Contains the state of the query. Allows to get/set values. Trigger query state event when modified. Each component can listen to those events.
A reference to the root HTMLElement (the SearchInterface).
A reference to the root of every component, the SearchInterface.
The static ID that each component need to be identified.
For example, SearchButton -> static ID : SearchButton -> className : CoveoSearchButton
A reference to the Analytics.client.
Whether the ResultList
should scan its result templates to discover which fields it must request to
be able to render all results.
Setting this option to true
ensures that the Coveo Search API does not return fields that are unnecessary for
the UI to function.
Notes:
true
.true
in the Coveo for Sitecore integration.data-auto-select-fields-to-include='true' data-auto-select-fields-to-include='false'
Whether to automatically retrieve an additional page of results and append it to the
results that the ResultList
is currently displaying when the user scrolls down to the bottom of the
infiniteScrollContainer
.
See also the infiniteScrollPageSize
and
enableInfiniteScrollWaitingAnimation
options.
It is important to specify the infiniteScrollContainer
option manually if you want the scrolling element to be
something else than the default window
element. Otherwise, you might find yourself in a strange state where the
framework rapidly triggers multiple successive query.
data-enable-infinite-scroll='true' data-enable-infinite-scroll='false'
Whether to display the waitingAnimation
while fetching additional
results.
data-enable-infinite-scroll-waiting-animation='true' data-enable-infinite-scroll-waiting-animation='false'
Whether to scroll back to the top of the page when the end-user interacts with a facet.
Note: Setting this option to false
has no effect on dynamic facets. To disable this behavior on a DynamicFacet
component, you must set its own enableScrollToTop
option to false
.
data-enable-scroll-to-top='true' data-enable-scroll-to-top='false'
A list of fields to include in the query results.
If you set the autoSelectFieldsToInclude
option to
true
, the Coveo Search API returns the fields you specify for this option (if those fields are available) in
addition to the fields which the ResultList
automatically requests.
Otherwise, the Coveo Search API only returns the fields you specify for this option (if those fields are available), unless you leave this option undefined, in which case the Coveo Search API returns all available fields.
The element that triggers fetching additional results when the end user scrolls down to its bottom.
You can change the container by specifying its selector (e.g.,
data-infinite-scroll-container-selector='#someCssSelector'
).
By default, the framework uses the first vertically scrollable parent element it finds, starting from the
ResultList
element itself. A vertically scrollable element is an element whose CSS overflow-y
attribute is
scroll
.
This implies that if the framework can find no scrollable parent, it uses the window
itself as a scrollable
container.
This heuristic is not perfect, for technical reasons. There are always some corner case CSS combination which the framework will not be able to correctly detect as 'scrollable'.
It is highly recommended that you manually set this option if you wish something else than the window
to be the
scrollable element.
The number of additional results to fetch when the user scrolls down to the bottom of the
infiniteScrollContainer
.
data-infinite-scroll-page-size='5'
Specifies the layout to use when displaying results in this ResultList
(see
Result Layouts). Specifying a value for this option automatically
populates a [ResultLayout
]{@link ResultLayout} component with a switcher for the layout.
For example, if there are two ResultList
components in the page, one with its layout
set to list
and the
other with the same option set to card
, then the ResultLayout
component will render two buttons respectively
entitled List and Card.
See the ValidLayout
type for the list of possible values.
data-layout='card'
The element inside which to insert the rendered result templates.
Performing a new query clears the content of this element.
You can change the container by specifying its selector (e.g.,
data-result-container-selector='#someCssSelector'
).
If you specify no value for this option, a div
element will be dynamically created and appended to the result
list. This element will then be used as a result container.
The type of animation to display while waiting for a query to return.
The possible values are:
fade
: Fades out the current list of results while the query is executing.spinner
: Shows a spinning animation while the query is executing.none
: Use no animation during queries.See also the waitAnimationContainer
option.
data-wait-animation='spinner'
The element inside which to display the waitAnimation
.
You can change this by specifying a CSS selector (e.g.,
data-wait-animation-container-selector='#someCssSelector'
).
Defaults to the value of the resultsContainer
option.
Creates a new ResultList
component. Binds various event related to queries (e.g., on querySuccess ->
renderResults). Binds scroll event if the enableInfiniteScroll
option is true
.
The HTMLElement on which to instantiate the component.
The options for the ResultList
component.
The bindings that the component requires to function normally. If not set, these will be automatically resolved (with a slower execution time).
The class that this component should instantiate. Components that extend the base ResultList
use this. Default value is CoveoResultList
.
The
ResultList
component is responsible for displaying query results by applying one or several result templates (see Result Templates).It is possible to include multiple
ResultList
components along with a singleResultLayout
component in a search page to provide different result layouts (see Result Layouts).This component supports infinite scrolling (see the
enableInfiniteScroll
option).


Ming Fang |
|
Master of Science in Computer Science Research Assistant Carnegie Mellon Database Group School of Computer Science Carnegie Mellon University Email: mingf AT cs DOT cmu DOT edu Github: https://github.com/mindbergh |
I am a master student in School of Computer Science at Carnegie Mellon University. My interest is in building systems for managing and processing Big Data.
Research Projects:
Peloton
Worked with a team to build an experimental distributed main memory database designed for supporting both OLTP and OLAP workloads.- Ported PostgreSQL to C++11 to provide a clean and modern code base.
- Implemented a number of components in Peloton, including query plan mapper from PostgreSQL, query execution engine and multi-version concurrency control.
- Led team effort on profiling and optimizing the performance of the system.
- Maintained a continuous integration server for the project using Jenkins, including setting up building and testing framework, adding supports for code coverage and documentation.
- Deployed Spark Streaming and evaluated its capabilities of performing incremental computation for a number of applications.
Weiss
Developed a system-initiative, plan-based dialogue system that can discusses subjective information with users.- Built a daily processing pipeline in Python for scraping, transforming and loading human discussions from multiple sources.
- Designed and implemented the dialogue system as a web-based service with mobile application APIs using Django.
- Conducted user study and evaluated the efficacy of the system.
cuSpark
http://www.yaomuyang.com/cuspark/
Built a GPU-accelerated, functional parallel data processing framework, inspired by Spark.- Designed and implemented a number of abstract functional primitives for data processing.
- Developed GPU auto-partitioning mechanisms to allow larger than memory datasets.
- Implemented logistic regression and k-mean using cuSpark and achieved comparable performance with Thrust implementations.
- Won the 3rd Place out of 74 teams in the 2015 Parallelism Competition at CMU.
Selective Coursework:
Summer 2014
- 15-513 Introduction to Computer Systems
Fall 2014
- 15-641 Computer Networks
- 10-601 Machine Learning
- 18-342 Fundamentals of Embedded Systems
- 15-619 Cloud Computing
Spring 2015
- 15-618 Parallel Computer Architecture and Programming
- 15-615 Database Applications
- 15-640 Distributed Systems
- 15-210 Parallel and Sequential Data Structures and Algorithms
Fall 2015
- 15-605 Operating System Design and Implementation
- 15-412 Information Security & Privacy
Course Projects:
Pebbles - A Linux-like Operating System Kernel
Implemented a multi-process, multi-threaded, pre-emptible kernel for X86 32-bit processor that has the following components:- Virtual memory module using fixed 4K page size,
- Full sets of system calls including fork(), thread_fork(), exec(), vanish(), wait(), swexn(), deschedule(), yield(), new_pages(), remove_pages(), etc,
- Pluggable task scheduler with round-robin, randomized scheduling policy,
- User level thread library built on top of the system calls with full synchronization primitives supports, such as mutex, semaphore, read-write lock and condition variable
- User space hardware driver API
- Simple interprocess communication library
Twitter Analytics
Deployed a web service on Amazon AWS that responses a number of analytical queries for Twitter data using Undertow web framework. Processed 1TB of twitter data using Hadoop MapReduce and HBase with Amazon AWS.Liso - An HTTP/1.1 web server
Designed and implemented a HTTP 1.1 web server in C. which supports:- Concurrent connections based on select().
- Persistent HTTP/HTTPS connections via TLS.
- CGI interactions.
Congestion Control with BitTorrent
Implemented a p2p file transfer application on top of UDP with the following features:- Congestion control similar to TCP (slow start and congestion avoidance)
- Simultaneously download/upload different chunks of a file from/to different peers
- Discover available chunks via SHA
- Timeout for potentially crashed peers
Video CDN
Implemented a video content distribution network and ran simulations on different network topologies. The CDN includes:- A proxy that is able to adaptively select bit-rate according to network throughput.
- A simple DNS server supporting load balancing. The algorithms for load balancing includes Round Robin and OSPF.
- An accompanying DNS resolution library
Gravel - Real-Time Operating System Kernel for ARM processor
Implemented a multitask RTOS Kernel with the following features:- basic system calls (read, write, exit, sleep, time, mutex_lock, mutex_unlock, etc).
- context switching and rate monotonic task scheduling.
- timer and interrupt controller drivers.
- mutexes for concurrency control.
- priority inheritance using Highest Locker Priority Protocol.
- a Snake game based on system calls of the implemented RTOS.