For the last 3 months I’ve been working on Porting the Comment Alter module to Drupal 8 as my GSoC’16 project under mentorship of boobaa and czigor. This blog is an excerpt of the work I did during this time period. Weekly blog posts for the past 12 weeks can be accessed here.
Creating schema for the module: Implemented hook_schema()
to store the old and new entity revision IDs along with the parent entity type as altered by a comment. The revision IDs are used to show the differences over comments. The parent entity type is used to delete the entries from the comment_alter table when any revision of the parent entity is deleted, because in Drupal 8 we can have same revision IDs from different entity types. So to remove entries of particular entity type we need the entity type.
Using ThirdPartySettings to alter field config edit form - Implemented hook_form_FORM_ID_alter()
for field_config_edit_form
. This provides an interface to:
Make any field comment alterable - Makes it possible to select any field we want to be altered by the comment. Currently all the Drupal core provided fields works well with the module.
Hide alteration from diff - If the comment alterable option is enabled, then this option hides the differences shown over the comments. Instead of the differences, a link to the revision comparison is displayed for nodes. For the rest of the entities a “Changes are hidden” message is shown.
Use latest revision - When a module like Workbench makes the current revision of an entity not the latest one, this option forces the Comment Alter module to use the latest revision instead of the current one. This option is present on the comment field settings.
Adds Diff link on comments - Adds a Diff link on comments which takes us to the comparison page of two revisions, generated by the Diff module.
Comment altering while replying to a comment - By default comment alterable fields can not be altered while replying to a comment. This option allows altering of fields even while replying to comments.
Adding pseudo fields: Implemented hook_entity_extra_field_info()
to add one pseudo field for each comment alterable field on respective comment form display, and one pseudo field on comment display to show the changes made at the comment. Using these pseudo fields the position of the comment alterable fields can be re-ordered in the comment form. This gives site-builders flexibility to alter the position of any alterable fields.
Attaching comment alterable fields’ widgets on comment form: Comment alterable field widgets are retrieved from the form-display of the parent entity and they are attached to the comment form only after ensuring that there are no side effects. To support same name fields on both comment and parent entity, #parent
property is provided so that the submitted field values for our alterable field widgets appears at a different location, not at the top level of $form_state->getValues()
. All these added fields are re-orderable. Column informations and old values are also stored to the form at this stage, to later check if there were any changes made on the comment alterable fields.
Adding submit and validation callback for the altered comment form: First the submitted values are checked against the old values to see if the values of the alterable field changed at all or not. If they changed, then the parent entity values are updated and this is done by building the parent entity form display and copying the altered field values into it. Then the form is saved. In case the parent entity doesn’t support revisions, then do nothing else just save the parent entity with altered values. Otherwise create a revision and store the comment ID, old and new revision IDs and parent entity type in the comment alter database table, which is used to show the differences on comments using the Diff module.
Showing differences on comments: Using the Diff module and comment alter database table, the differences are shown over a particular comment. Only possible if the parent entity supports revisions. Diff module is used to get the differences between the two revisions and then those differences are rendered on comments in table format along with some custom styling.
Adding PHPUnit tests: Added automated unit tests to check the functionality of the module for different field types and widgets. The tests are written for EntityTestRev entity types to keep them as generic as possible. This was the toughest part for me as I was stuck at some places in tests for quite a while as thses tests took lot of time to run and debugging them really is hard. But in the end I’m happy that I was able to complete all the tests.
Screencast/Demo video: Created a demo video showing how the Comment Alter module works, along with a simple use case.
My mentors asked me to skip the Rules Integration part because the Rules module doesn’t have a stable or a beta release yet, only their first alpha release is there. So, the Rules Integration part is postponed till we get a stable or beta release.
My GitHub repo contains all the commits which I made during this project
The Comment Alter module’s project page on drupal.org
Thank you!


What is your age?
What is your favorite color?
What is your favorite animal?
What is your favorit gaming console?
What is the best time of the year?
Based on the information you provided you should start your programming language in !
Would you like to try again?


Tesla now does sedans, SUVs, sports cars, semis, and pickups — here are 5 more segments for the company to attack, Business Insider - Business Insider Singapore
by Matthew DeBord, Business Insider US- Tesla unveiled its much-anticipated Cybertruck last week. Reaction was … mixed.
- But the arrival of the Cybertruck suggests some new, untapped vehicle segments that Tesla could move into.
- These include full-size SUVs, convertibles, and even motorsports.
- Sign up for Business Insider’s transportation newsletter, Shifting Gears, to get more stories like this in your inbox.
- Visit Business Insider’s homepage for more stories.
The Tesla Cybertruck has arrived!
Regardless of what you think of it – love, hate, fear, whatever – the Cybertruck proves Tesla can add yet another vehicle platform to its current mix of two sedans, two SUVs, a sports car, and a semi truck.
Here are five possibilities for segments that Tesla could take on next:


The Faculty (1998)
A Film Review by Mark O'Hara
Visit Online Film Critics Society at http://www.ofcs.org
What do you get when you combine literary elements with a "B" storyline? A film that's hard to brag about but difficult to dislike. 'The Faculty.'
Kevin Williamson, the writer of 'Scream,' has churned out another post-modern horror flick, a yarn targeted toward teenagers, in which teenagers are targets. Into the formulaic plot he has mixed a pinch of motifs, a smidgen of imagery, a dash of foreshadowing. The result is a movie full of flavors, another self-reflexive romp that does not stay with you for long.
At Herrington High School in Ohio, most of the kids are punks and most of the teachers are burnt out. There's Mrs. Drake the principal (Bebe Neuwirth), who plays the part of a frank realist. She tells Mrs. Olsen the drama teacher (Piper Laurie) that no, plays will not receive the proper funding - but football will. "Have you seen the stands on Friday nights? The whole town is there," she quips. There's also an alcoholic teacher who does not even know what chapter the class is supposed to be covering. Finally we meet the foul-mouthed football coach, played wonderfully by Robert Patrick with his smooth face and squinty eyes. It's Coach Williams who is first infected, or rather invaded, by the alien species that eventually takes over the faculty, most of the student body, and ostensibly, at one of those Friday games under the lights, the entire town.
It takes awhile for all this to happen, and along the way we watch as Williamson and director Robert Rodriguez sketch in the traits that help us to tell the six teen heroes apart. Casey (Elijah Wood) is the photographer for the school paper, a computer geek who has not yet raised a whisker and who is continually harassed by various badasses of the school. Zeke is played by Josh Hartnett (a dead ringer for a young Tommy Lee Jones); his character is a dealer of anything illicit. From the trunk of his muscle car Zeke peddles pills, cherry-flavored condoms, and a drug of his own concoction, a white powder that can be sniffed from the barrel of a ball point. The final male is Stan (Shawn Wayne Hatosy), the star quarterback who decides to quit the team so he might focus on his studies.
Stan's decision poses problems for his girlfriend, the shallow Delilah (Jordana Brewster), a stunning cheerleader and editor of the school paper. She implies she'll drop him but quick. Delilah also has problems with a girl named Stokely (Clea DuVall), calling her a violent lesbian. The third female is Marybeth, a slim blonde from Atlanta, an unassuming new kid in school who tries to befriend everyone. Thus the stereotypes are set.
Although the mild-mannered Casey and his editor witness, through the slats in a closet door, a couple of the alien-possessed teachers graphically recruit the school nurse, they cannot get their friends to believe them. It takes a scene of full confrontation with the erstwhile science teacher. He corners the six and says, "This won't take long" - as if he expects the teens to line up compliantly for the grotesque inoculation. What follows - a turning point - is one of the strongest scenes in the film.
Soon afterwards, we see one of the weakest. Yes, the film is derivative, and for the most part the script tries to be witty about its borrowing; but the scene in which suspicion raises its Gorgon head smacks too much of Kurt Russell and the crew from John Carpenter's 'The Thing.' Here even the acting lapses into a congeries of wooden, menacing expressions.
Many of the other allusions work well.  She makes use of her extensive reading of science fiction - especially Robert Heinlein - to guide her harried school mates in their quest to escape and, hopefully, to conquer the aliens. We also glean elements of 'Invasion of the Body Snatchers,' 'The Puppet Masters,' and, when one of the youngsters is compared to 'Ripley,' the 'Alien' movies. There's a neat scene near the end when Casey is running from the thing, and we see shadowed cross-hatching on the floor of the locker room - reminiscent of the lighting in 'Aliens.'
Water imagery is remarkable as well. Rodriguez comes close to being heavy-handed here, placing water in several key scenes: there's the sprinkler system on the football field (a sure sign that academics are being neglected!); the tank in the science room, where an embryonic alien is revived; a rainstorm into which the thing inhabiting the coach extends its thin tentacles; bottled water and the faculty room cooler; and the indoor pool in which one of the creatures does an instantaneous shape-shift into the fanged lobster-snake hybrid chasing Casey. All these images work, though, lending unity and even a certain comedy to the plot. Cool special effects, too.
What's gratuitous in this late '90's teen treatment? For one, the language goes too far, probably in the service of sending up real problems in many American high schools. The drug use gives a tawdry tone, though figuring prominently in the resolution. Most noticeably, the behavior of the teachers, after they are taken over, is inexplicably bold and sexual. Bebe Neuwirth as Principal Drake dresses in skimpy outfits, flaunting her skinny curves. Another teacher, Miss Burke (Famke Janssen), threatens and curses out Zeke, and later speaks provocatively before attempting to, well, impregnate him. Again and again Rodriguez and Williamson cater to young viewers, working sex and screams into the same sentence.
'The Faculty' knows it is smart. Though it is not packed as full of self-reflexivity as the 'Scream' movies, the clever and engaging tones are there. Fans of science fiction and horror, teenagers 16 and over, teachers seeking easy laughs, go to see or rent this film. Hokey, yes, but light entertainment blended with just a touch of literature.
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Chapter 3 Explore Data with R
3.1 Getting Started
We will be using these three datasets for the analysis today.. Please download the following files:
In this lesson, we are going to start building figures and table from our data.
We are going to do so inside an R - notebook, so we can write ourself a little tutorial/report as what we are finding as we go!!
To write a report, we will make use of R-Markdown syntax. A cheatsheet for R Markdown Syntax is here.
Add a new chunk by clicking the Insert Chunk button on the toolbar or by pressing Ctrl+Alt+I.
We are also going to make heavy use of the “tidyverse” suite of packages. These packages include:
- ggplot2: for plotting (ggplot cheat sheet)
- dplyr & tidyr: for manipulating dataframes (data Wrangling cheat sheet)
- knitr: for adding tables to our reports
For more info on programming with the tidyverse I highly recommend the online book R for data science by Garrett Grolemund & Hadley Wickham.
3.1.1 The packages for today
library(tidyverse)
## ── Attaching packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
## ✔ ggplot2 3.1.1 ✔ purrr 0.3.2
## ✔ tibble 2.1.1 ✔ dplyr 0.8.1
## ✔ tidyr 0.8.3 ✔ stringr 1.4.0
## ✔ readr 1.3.1 ✔ forcats 0.4.0
## ── Conflicts ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
3.1.2 reading in the data
demo_df <- read_csv("~/Desktop/messy_demographic.csv")
cog_df <- read_csv("~/Desktop/messy_cognitive.csv")
gene_df <- read_csv("~/Desktop/messy_genotype.csv")
3.1.3 copy and paste the cleaning code
These are all the things we learned to do in Intro to R.
We are going to put them in one big chunk here.
library(stringr)
demo_df[demo_df==""] <- NA
demo_df[demo_df=="missing"] <- NA
demo_df[demo_df=="9999"] <- NA
demo_df <- demo_df %>%
mutate(age = as.numeric(age),
ethnicity = factor(ethnicity),
sex = factor(sex, levels = c(0,1),
labels = c("Male", "Female")),
dx = factor(dx, levels = c(0,1),
labels = c("Control", "Case")))
cog_df[cog_df==""] <- NA
cog_df[cog_df=="missing"] <- NA
cog_df[cog_df=="9999"] <- NA
cog_df <- cog_df %>%
mutate(cog1 = as.numeric(cog1),
cog2 = as.numeric(cog2),
cog3 = as.numeric(cog3),
subject_ID = str_replace(subID, "subject", "SUB_")) %>%
select(subject_ID, cog1:cog3)
gene_df[gene_df==""] <- NA
gene_df[gene_df=="missing"] <- NA
gene_df[gene_df=="9999"] <- NA
gene_df <- gene_df %>%
mutate(genotype = factor(genotype,
levels=c(0,1,2),
labels=c("AA","AG","GG")),
subject_ID = str_replace(subID, "subject", "SUB_")) %>%
select(-subID)
alldata <- demo_df %>%
inner_join(cog_df, by="subject_ID") %>%
inner_join(gene_df, by="subject_ID")
3.1.4 Let’s see what we have here
We can use the summary function (from base R) to get an idea of what is in our dataset.
summary
will print some summary statistics for numeric variables are counts for our factors.
summary(alldata)
## subject_ID age sex ethnicity dx
## Length:350 Min. :22.00 Male :130 AA : 70 Control:166
## Class :character 1st Qu.:43.00 Female:216 As : 42 Case :178
## Mode :character Median :49.00 NA's : 4 Cauc :196 NA's : 6
## Mean :50.38 In : 19
## 3rd Qu.:58.00 Other: 14
## Max. :89.00 NA's : 9
## NA's :8
## cog1 cog2 cog3 genotype
## Min. : 2.868 Min. :20.23 Min. : 0.05796 AA :103
## 1st Qu.: 8.783 1st Qu.:29.22 1st Qu.: 8.82092 AG :145
## Median :10.939 Median :31.36 Median : 19.79222 GG : 94
## Mean :11.087 Mean :31.66 Mean : 25.04099 NA's: 8
## 3rd Qu.:12.964 3rd Qu.:34.03 3rd Qu.: 35.55916
## Max. :22.189 Max. :44.74 Max. :104.21326
## NA's :5 NA's :5 NA's :9
3.2 Using tableone to make Table 1 of your paper
Table one is a cool package that creates the demogaphics table.
It takes four useful arguments + data: the data to plot + vars: the variables (from your data) to include in your table + factorVars: a list of which variables (in vars) should be treated as factors + strata: the name of a variable to split the table by.
library(tableone)
CreateTableOne(alldata,
vars = c("age", "sex",
"genotype","ethnicity",
"cog1", "cog2", "cog3"),
factorVars = c("sex",
"genotype","ethnicity"),
strata = "dx")
## Stratified by dx
## Control Case p test
## n 166 178
## age (mean (SD)) 50.42 (11.95) 50.57 (11.06) 0.905
## sex = Female (%) 127 (77.4) 86 (48.9) <0.001
## genotype (%) 0.371
## AA 44 (27.2) 58 (33.1)
## AG 68 (42.0) 73 (41.7)
## GG 50 (30.9) 44 (25.1)
## ethnicity (%) 0.523
## AA 36 (22.4) 32 (18.4)
## As 17 (10.6) 25 (14.4)
## Cauc 92 (57.1) 100 (57.5)
## In 11 ( 6.8) 8 ( 4.6)
## Other 5 ( 3.1) 9 ( 5.2)
## cog1 (mean (SD)) 9.94 (2.91) 12.13 (3.44) <0.001
## cog2 (mean (SD)) 31.55 (3.88) 31.78 (3.79) 0.573
## cog3 (mean (SD)) 25.20 (21.76) 25.25 (20.39) 0.982
3.3 Research Question 1 (two group comparison)
<IP_ADDRESS> Is performance on Cognitive Scale One (cog1) associated with Diagnosis (Dx)
To test this statistically, we are going to run an independant samples t-test, using the base t.test
function.
When we call the t.test function, we are going to use “formula” notation. we’re our dependant variable goes on the left side of a ~
and the predictors go to the right
i.e. y ~ x
.
t.test(cog1 ~ dx, data = alldata)
##
## Welch Two Sample t-test
##
## data: cog1 by dx
## t = -6.347, df = 334.92, p-value = 7.133e-10
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -2.871388 -1.512676
## sample estimates:
## mean in group Control mean in group Case
## 9.940047 12.132079
One cool thing to remember about everything you do in R is that they generate useful objects we can save.
my_ttest_result <- t.test(cog1 ~ dx, data = alldata)
my_ttest_result$p.value
## [1] 7.13299e-10
3.3.1 Plotting our result
library(ggplot2)
ggplot(data = alldata, aes(x = dx, y = cog1)) +
geom_boxplot()
#### removing NA rows before plotting
We can use tidyr’s drop_na
to remove rows with NA’s in the dx
column
alldata %>%
drop_na(dx) %>%
ggplot(aes(x = dx, y = cog1, fill = dx)) +
geom_boxplot()
<IP_ADDRESS> Introducing the the dotplot geom
geom_dotplot
has to be one of my favs.. Let’s use it in combination with geom_boxplot
.
Let’s also use the labs()
call to relabel our axes with more descriptive titles.
Note: by setting labs(x = NULL)
we are removing the word “dx” from the bottom of the plot
alldata %>%
drop_na(dx) %>%
ggplot(aes(x = dx, y = cog1, fill = dx)) +
geom_dotplot(binaxis = "y", stackdir = "center") +
geom_boxplot(alpha = 0.5) + # using "alpha" to make the box-plot semi-transparent
labs(x = NULL, #remove dx from the bottom
y = "Cognitive Score 1", #add more desciptive title to y-axis
fill = "Diagnosis") # change dx to Diagnosis in the legend
3.4 pretty tables with dplyr
In the section below we use two powerful tools from dplyr:
summarise
: will calculate summary statistics, we are going to ask for three stats here:- n_age: the number of valid (i.e. not NA) values for age
- mean_age: the mean for age (after removing NA values using the na.rm argument)
- sd_age: the standard deviation for age after removing the NA values
group_by
: will split the data by the “grouping” variable(s) you input- in this case we are asking for separate summary statistics for each diagnostic group, for each genotype
After calculating our summary table we are going to call kable
, a function in the knitr
package. That make the table appear nicer in our “knitted” report.
Note: kable has some kewl optional arguments you can use to format you table further, such as col.names
, digits
, and align
. Check out ?kable
for more info..
table_2 <- alldata %>%
drop_na(dx) %>%
group_by(dx,genotype) %>%
summarise(n_age = sum(!is.na(age)), #the total number of observations that are NOT NA
mean_age = mean(age, na.rm = T),
sd_age = sd(age, na.rm = T))
## Warning: Factor `genotype` contains implicit NA, consider using
## `forcats::fct_explicit_na`
library(knitr)
kable(table_2)
| dx | genotype | n_age | mean_age | sd_age |
|---|---|---|---|---|
| Control | AA | 43 | 52.00000 | 10.574002 |
| Control | AG | 66 | 49.92424 | 11.024882 |
| Control | GG | 50 | 49.94000 | 14.217566 |
| Control | NA | 4 | 47.50000 | 12.476645 |
| Case | AA | 57 | 55.08772 | 10.554756 |
| Case | AG | 71 | 48.61972 | 10.355628 |
| Case | GG | 43 | 48.37209 | 11.176129 |
| Case | NA | 2 | 38.00000 | 9.899495 |
3.5 use forcats to label risk_carriers
When working with a genotype of interest, it’s very common group participants into “carriers” and “non-carriers” of a risk-allele. In this dataset, our risk allele is the “G” genotype. We will create a new “risk_carrier” variable by combining levels of the genotype variable.
Working with factors in R can be…annoying. Especially when we want to create new factors out of old factors. For this reason, the tidyverse includes a forcats
package. That includes many function (all starting with fct_
) for doing stuff to factors. We will use fct_collapse
to combine the “GG” and “AG” genotype factors into a new “carrier” level in a new factor.
library(forcats)
alldata <- alldata %>%
mutate(risk_carrier = fct_collapse(genotype,
carrier = c("GG", "AG"),
non_carrier = "AA"))
3.6 Make a simple scatter plot
Let’s use geom_point()
and geom_smooth
, together to create a scatter plot with a trendline.
Note: to make the trendline straight, we use “method = lm” when calling geom_smooth().
alldata %>%
ggplot(aes(x = age, y = cog1)) +
geom_point() +
geom_smooth(method = "lm")
3.7 Use plotting to show the age by risk carrier interaction
When we add a new mapping of color = risk_carrier
to the top line of our plot this mapping applies to both the points and the trendline.
Note: we are using drop_na
to remove the NA as a category so that it does not occur as a third color in our plot.
alldata %>%
drop_na(risk_carrier) %>%
ggplot(aes(x = age, y = cog1, color = risk_carrier)) +
geom_point() +
geom_smooth(method = "lm")
We can use faceting to add and extra dimension - separate plots for male and female subjects.
alldata %>%
drop_na(risk_carrier,sex) %>%
ggplot(aes(x = age, y = cog1, color = risk_carrier)) +
geom_point() +
geom_smooth(method = "lm") +
facet_wrap(~sex)
facet_grid
will allow us to facet by TWO variables. In this case sex and dx.
alldata %>%
dplyr::filter(!is.na(risk_carrier), !is.na(sex), !is.na(dx)) %>%
ggplot(aes(x = age, y = cog1, color = risk_carrier)) +
geom_point() +
geom_smooth(method = "lm") +
facet_grid(dx~sex)
3.8 BONUS section - gather cognitive scores into one plot
One things we want to do sometimes is to stack data from multiple columns in one column. The reason why this is useful, will hopefully become apparent in the next example.
This “stacking” task (also refered to as “melting”) is accomplished using gather
from the tidyr
package. gather
creates two new columns from your data: “key” and “value”.
+ key: a new column that will hold the old variable names from the gathered data
+ value: a new column that will hold the data values
After the key and value arguments. We can tell gather
what colums we want to stack using the same syntax used by dplyr’s select()
. In this example, we tell it we want to gather our three cognitive scales using starts_with()
.
library(tidyr)
alldata %>%
gather(cog_scale, cognitive_score, starts_with("cog"))
## # A tibble: 1,050 x 9
## subject_ID age sex ethnicity dx genotype risk_carrier cog_scale
## <chr> <dbl> <fct> <fct> <fct> <fct> <fct> <chr>
## 1 SUB_1 43 Male Cauc Cont… GG carrier cog1
## 2 SUB_2 47 Fema… Cauc Case AG carrier cog1
## 3 SUB_3 69 Fema… Cauc Case AA non_carrier cog1
## 4 SUB_4 51 Male Cauc Case GG carrier cog1
## 5 SUB_5 52 Fema… Cauc Cont… AA non_carrier cog1
## 6 SUB_6 71 Male AA Case AA non_carrier cog1
## 7 SUB_7 56 Fema… Cauc Case AA non_carrier cog1
## 8 SUB_8 35 Fema… <NA> Cont… GG carrier cog1
## 9 SUB_9 42 Fema… Cauc Cont… AG carrier cog1
## 10 SUB_10 45 Fema… Other Case <NA> <NA> cog1
## # … with 1,040 more rows, and 1 more variable: cognitive_score <dbl>
The beauty of gather is that it can be combined with other the rest of the tidyverse using the pipe. Let’s feed out gathered result to ggplot.
Note: using scales = "free"
when faceting will force ggplot to draw a separate axis for each subplot (by default, scales = "fixed"
, meaning that the same axes are drawn for each plot).
alldata %>%
gather(cog_scale, cognitive_score, starts_with("cog")) %>%
drop_na(risk_carrier, dx) %>%
ggplot(aes(x = age, y = cognitive_score, color = risk_carrier)) +
geom_point() +
geom_smooth(method = "lm") +
facet_wrap(~cog_scale, scales = "free")


ARJ(1) Arj Software ARJ(1) NAME arj - Archiver for .arj files SYNOPSIS arj command [-switch[-|+|option]] archive[.arj] [base directory] [!list name|path name|wildcard name] DESCRIPTION arj is a compression and file archiving utility. It was invented by Robert K. Jung. ARJ stands for Archived by Robert Jung. COMMANDS ac Add Chapter to chapter archive. cc Convert archive to Chapter archive. dc Delete last Chapter from archive. a Add files to archive. b execute Batch or dos command. c Comment archive files. d Delete files from archive. e Extract files from archive. f Freshen files in archive. g Garble files in archive. i check Integrity of the arj program. j Join archives to archive k remove obsolete bacKup files. l List contents of archive. m Move files to archive. n reName files in archive. o Order files in archive. p Print files to standard output. q recover damaged arj file. r Remove paths from filenames. s Sample files to screen with pause. t Test integrity of archive. u Update files to archive. v Verbosely list contents of archive. w Where are text strings in archive. x eXtract files with full pathname. y copY archive with new options. SWITCHES c skip time-stamp Check e Exclude paths from names f Freshen existing files g Garble with password i with no progress Indicator m with Method 0, 1, 2, 3, 4 n only New files (not exist) r Recurse subdirectories u Update files (new and newer) v enable multiple Volumes w assign Work directory x eXclude selected files y assume Yes on all queries hk enable ARJ-PROTECT damage protection DETAILED SWITCHES - Disables switch char + Inhibits ARJ_SW usage +var: set environment variable @ Set 1 token per response file line ! Set list char (!) & Set batch critical error handler # Select files by number $ Add/extract volume label $A: add/extract label to drive A a Allow any file Attribute a1: any files, directories and UNIX special files b Backup changed files b1: Backup changed, reset archive bits b2: only reset archive bits b3: reset archive bit during restore b4: do not restore bits, reset arc b5: do not restore any file attributes c Skip time-stamp Check d Delete added files asks permission before deleting d1: Delete without permission (same as arj m) d2: Truncate instead of deleting e Exclude paths from names e1: Exclude base dir from names f Freshen existing files f1: Freshen with OLDER files f2: Freshen with DIFFERENT files f3: Freshen with CRC mismatched files g Garble with password gstew: garble with password stew g?: prompt for password i Show no progress Indicator i1: show bar graph Indicator i2: show percentage and bar graph i3: total progress percentage i4: total progress bar graph i5: total percentage and bar graph i6: total percentage and file graph k Keep a .bak of arj archive l Create List_name file lnames.list: create names.list m With Method 0, 1, 2, 3, 4 m0: store (no compression) m1: good compression (default) m2: less memory and compression m3: FAST! less compression m4: FASTEST! least compression n Only New files (not exist) o On or after YYYYMMDDHHMMSS o: on today o19901225: on/after 12/25/1990 ob: Before YYYYMMDDHHMMSS ob: before today ob19901225: before 12/25/1990 od: no older than N Days od5: five or less days old oa: After YYYYMMDDHHMMSS (last access) oa: on today oa19901225: after 12/25/1990 oab: Before YYYYMMDDHHMMSS (last access) oab: before today oab19901225: before 12/25/1990 oc: After YYYYMMDDHHMMSS (created) oc: on today oc19901225: after 12/25/1990 ocb: Before YYYYMMDDHHMMSS ocb: before today ocb19901225: before 12/25/1990 p Match using full Pathnames p1: match Pathname with subdirs q Query on each file r Recurse subdirectories s Set archive time-Stamp to newest s1: save original time-Stamp s2: set archive time-Stamp s3: save both original time-Stamps t Set file Type (default 0) t0: set binary file type t1: set C text file type t1f: force C text file type t1g: set C text with graphics u Update files (new + newer) u1: Update files (new + OLDER) u2: Update files (new + different) u3: Update files (new + CRC mismatch) v Enable multiple Volumes v360: build 362000 byte volumes v50K: build 50000 byte volumes va: auto-detect space available vd/tmp/*: delete /tmp/* ve: use alternate volume naming vi: inhibit diskette change test vr50K: reserve 50000 bytes of space on first volume vs: provide command prompt vscmd: execute cmd before each vol vv: beep between volumes vw: keep Whole files in volumes vz: provide command with no echo v360,v720,v1200,v1440: abbrevs Volume options may be in any order except s and z which must be last w Assign Work directory w/tmp: use /tmp as work directory x eXclude selected files x*.sh: exclude *.sh files x!names: exclude files in names multiple exclusions are allowed y Assume Yes on all queries except diskette volume prompts Use this switch for batch mode z Supply archive comment file _ Convert filenames to lower case zarc.cmt: use arc.cmt for comments z/dev/null: use to strip comments SHIFTED SWITCHES h# Append date string to name h#1: append time string to name h#2: append DHHMMSS to name (custom: h#{Y,M,D,N,h,m,s}) h#YYYY-MMDD: append YYYY-MMDD h$ Try to use longnames (Win2000) h2 Obsolete synonym for -2 group of options ha Ignore readonly Attribute hb Select by file attribute/type a - archive bit set b - archive bit clear r - readonly s - system h - hidden d - directory n - normal file, not d,h,r,s,c,l c - chapter label l - volume label f - abbrev for n,d,r,s,h u - UNIX special files w - win95 long filenames hb: select all types hbsh: select hidden, system only hbd: select dirs only hc Execute Command at arj start hcclear: execute clear he Skip test of security Envelope he1: set error on security Envelope hf Set short filenames only hf1: set longnames in file comment or extract with w95lname.nnn hf2: use long filenames in DOS hf3: set longnames in file comment using shortname wildcards hf4: set all files archived in Windows as Windows long filenames hg Specify garble module filename hgarjcrypt.so: set arjcrypt.so as name of garble module hg!: specify use of 40 bit encryption hi Detailed display in Index files hk Build data protection record hk Default amount of recovery data hk1: more recovery data hk9: maximum amount of recovery data Disable ARJ-PROTECT when used with the "y" command hl Return error for Listfile error hm Increase file list capacity hm!: redirect -hm temp file hm! : move temp file to XMS mem hm!/tmp : move temp file to /tmp hn Specify non-existent filespec hn####: set #### as filespec ho Only extract files with matches ho1: Only extract files without matches hq Ignore open access error on adds hq1: ignore open not found error on adds hq2: ignore access and not found on adds hr Try to ignore archive errors hs Disable file Sharing ht Set Target directory ht/destdir hu Allow Update of volume archives hv Check for arj version hw Scroll save search filename display hw1: display only filenames with matches hx Set default archive extensions hx.arj.sdn hy set arj32 to use ANSI codepage set arj to extract ANSI codepage archive hy1: force arj32 to use OEM codepage hy2: force arj32 to use ANSI codepage hz Create an ARJ-SECURITY envelope j# Store only CRC and not the file j#1: store only date, size, attributes j#2: restore file attributes j#3: restore file dates j#4: restore file attributes and dates j$ Do not process file DTA and DTC ja Set P command to display ANSI ja1: inhibit the display of comments jb Set chapter backup processing jb0: clear chapter archive status jb*: select all chapter backup files jc Exit after Count of files jc5: exit after 5 files jc nam1 nam2: exit after 2 files jd Ensure free Disk space e -jd50K: skip file if<50000 free l -jd1000: set error if<1000 free je Create self-extracting archive je1: create SFXJR archive jf Store/use Full specified path jf1: store/use path minus drive jg [OBSOLETE] jg1: [OBSOLETE] jh Set Huffman buffer size jh65535: set to 65535 bytes (max) jh2048: set to 2048 bytes (min) ji Create Index file jiindex.file: create index.file jk Keep temp archive on error jl Display only filespecs jm Set Maximum compression mode jm1: set faster Maximum compression jn Restart volumes at fileName jn/bin/test: restart at /bin/test jn: get restart information from previously written index file jo Query when updating archive file jo: extract to unique Output names jo1: extract to unique Output names keeping file extension jp Pause after each screenful jp50: pause, set page size to 50 jq Set string parameter jqstring: set parameter to string jr Recover broken archive files jr1: Recover badly broken archive files js Store archives by suffix (default is arj, arc, lzh, pak, rar, zip, zoo, j, and uc2). js.zoo: store only .zoo files jt Test temporary archive by CRC jt1: Test CRC and file contents jt2: Test only added file contents jt3: Test only file date-time and size ju Translate UNIX style paths jv Set Verbose display jv1: set special verbose list mode jw Set extract output filename jwnew.file: output to new.file jx Start at eXtended position jx10000: start at position 10000 jy Suppress queries assuming Yes a - skip append query c - skip create directory query d - skip delete files query k - skip disk space available query n - skip new filename prompt o - skip overwrite file query r - erase all type-ahead before query s - skip scanned enough text query v - skip proceed to next volume query y - accept single character Y/N/A/Q jycny: skip create, new name queries in single character input mode jz Supply file for file comment jzfile.cmt: use file.cmt for comments jz/dev/null: use to strip file comments 2a Serialize names on extraction 2b [UNIX] restrict to Block devices 2b or 2b.: current device only 2b-/usr: all except where /usr is located 2c [OS/2, NT] handle Crit. EAs only 2d arj/DOS compatibility mode 2d1: (OBSOLETE) comment compatibility 2e [OS/2, NT] filter Extended attrs 2e.*: pack/unpack system EAs only 2e.ICON?: pack/unpack icons only 2e: disable EAs handling 2f Disable comment serialization across multiple volumes 2h [UNIX] Disable hardlink handling 2h1: Extract hardlinks as symlinks 2k Skip century display in dates 2k1: use a non-ambiguous date display format 2l [OS/2, NT] treat .LONGNAME EAs as filenames when possible 2o [UNIX] store/restore file ownership 2o1: Store numeric UID/GID values 2o2: Same as -2o but saves group as well 2p Specify process Priority class OS/2, NT: 2p1...2p4 Linux, FreeBSD: 2p1...2p41 2r Revert directory Recursion order 2s [UNIX] preserve Symlink properties 2x [OS/2, NT]: eXclude extended attrs 2x.ICON?: ignore icons DEFAULTS Save path information in archive Save long filenames Save/restore extended attributes Save/restore file date-time created and accessed in Win95 and OS/2 Save timestamps in the UNIX format under UNIX-like operating systems Don't save drive and root in path information Display comment ANSI sequences with built-in display handler Prompt before overwriting output files Use method 1 compression Use binary mode Use ! as list file symbol The "ac" command is an abbreviation for "u -u2 -jb" The "cc" command is an abbreviation for "y -jb" The "dc" command is an abbreviation for "d -jb" Under Win32, arj uses OEM codepage by default. The "-hy" option sets arj to use the ANSI codepage which is the Windows graphical environment default. arj/Win32 self-extracting archives should be built using the "-hy" option. To ensure compatibility with previous versions of arj, the "-2d" parame‐ ter has to be specified when archiving under UNIX. USER PROMPT RESPONSES Yes yes No no Quit abort out of arj Always always assume yes for current type of query Skip always assume no for current type of query Global always assume yes for all queries except diskette volume prompts Command prompt for and execute one system command ENVIRONMENT ARJ_SW = -jyry -jv -i1 ARJ_SW = /etc/arj.cfg EXIT STATUS 0 Success 1 Warning (specified file to add to archive not found, specified file to list, extract, etc., not found, or answering negatively to "OK to proceed to next volume..." prompt) 2 Fatal error 3 CRC error (header or file CRC error or bad password) 4 ARJ-SECURITY error or attempt to update an ARJ-SECURED archive 5 Disk full or write error 6 Cannot open archive or file 7 Simple user error (bad parameters) 8 Not enough memory 9 Not an arj archive 10 [DOS] XMS memory error (read or write) 11 User control break 12 Too many chapters (over 250) EXAMPLES Add files minus directory structure: arj a -e archive * Add two files to archive: arj a archive name1 name2 Add files to archive with verification: arj a -jt archive * Add files with maximum compression: arj a -jm archive * Add several directories of files: arj a archive ... dir1 dir2 dir3 Add thousands of files to an archive: arj a -hm1 -r archive * Add a chapter to an archive: arj ac archive /foo/dir/* -r Comment archive header only: arj c archive -zcmt.fil Convert standard archive to chapter: arj cc archive Convert chapter archive to standard: arj y archive -jb0 Delete the last chapter from an archive: arj dc archive * Strip archive comment only: arj c archive -z/dev/null Extract files from archive: arj e archive Extract maintaining directory structure: arj x archive Extract new and newer files without query: arj e archive -u -y Extract subdirectory from archive: arj e archive subdir/* -p1 Extract files to directory: arj x archive destdir/ * Extract files to directory: arj x archive * -htdestdir Extract files to directory with space: arj x archive "Some Dir/ " Extract files to directory with space: arj x archive "-htSome Dir" Extract a Win95 archive to plain DOS: arj x archive -hf2 List files in archive: arj l archive Move files to archive: arj m archive *.tex Move files from archive: arj e -d archive *.tex Recover files from a damaged archive: arj x archive -jr Test integrity of files in archive: arj t archive Test multiple arj archives: arj t *.arj -hr Add files to a multiple volume archive: arj a -va /floppy/archive * Create up to 999 archive volumes: arj a -va /floppy/archive.001 Extract from a multiple volume archive: arj x -v /floppy/archive Create a multiple volume self-extractor: arj a -va -je /floppy/archive * Convert archive to self-extractor: arj y -je1 archive Backup dir /foo to dir /floppy: arj a /floppy/backup /foo -a1 -b2 -r -vvas -w/foo -js -jt -hk Restore backup on dir /floppy to dir /foo: arj x /floppy/backup /foo -vv -y SELF-EXTRACTOR ARCHIVES If the first two characters of an arj comment are "))" in a self-extrac‐ tor, then the self-extractor will accept the appended line as its command line. ")) -o" will force the self-extractor to overwrite existing files. SEE ALSO rearj(1), unarj(1), arj-register(1). AUTHOR This manual page was written by Ola Lundqvist<EMAIL_ADDRESS>in pod format, then converted by Guillem Jover<EMAIL_ADDRESS>to nroff format. 3.10 2004-10-18 ARJ(1)


The Department of Statistics & Applied Probability and the
Center for Financial Mathematics and Actuarial Research are
partnering with Hull
Tactical to run a data science contest on predicting the
S&P 500 returns. Undergraduate and Masters students may
participate in teams of up to 4 members, to compete for prizes
of up to $2500 in two categories: (i) Best Prediction
and (ii) Most Creative.
Timeline
- Team registration opens February 15th.
- Blair Hull
of Hull Tactical will make a special presentation
on February 21st.
- The first submission deadline for the Best Prediction
category is April 8th.
- The live portion of the prediction contest begins
April 8th and ends May 8th.
- Submissions for the Most Creative category are
due on May 8th.
- The contest ends May 8th with winners announced soon
after.
- Winning teams present their work
and are awarded prizes on May 16th.
Registration
Register your team here
or search for like-minded teammates here.
Registration ends on March 15th.
Rules
The Best Prediction category of the contest will have two
evaluation phases. Phase I will involve an out-of-sample test of
your prediction algorithm based on some period in 2018 (as a
result the data set you are given does not include 2018). This
phase will be completed on April 8th. Phase II involves a live
contest that will run from April 8th to May 8th. It will
evaluate your prediction algorithms on real time S&P 500 returns
and will include a live leaderboard.
The Most Creative category will expand the judgement
criteria to topics other than just prediction. This could be
an explanation of some market phenomena, a novel trading
strategy or an insightful analysis. The submitted entries in the
form of a typewritten report will be judged by a panel of
experts consisting of practitioners and academics.
A team is allowed to participate in one or both of the contest categories.
You are encouraged to participate in both!
Office Hours
Our PSTAT PhD candidate Aditya Maheshwari will be available
during office hours to provide help with your progress in either
of the contest categories. Aditya's office hours will be
announced soon! Send all questions to
<EMAIL_ADDRESS>Data
The ERP contest is about prediction of the stock market. The
objective is to build a predictive model for weekly S&P 500
returns. For this purpose Hull Tactical is providing you with
this data set
of variables that may aid this task.
There are numerous ways to build a prediction model, ranging from a
simple linear regression to more sophisticated methods from
machine learning. The research team at Hull Tactical has
described some of their experiences with S&P 500 prediction in the
following articles:
Return Predictability and Market-Timing: A One-Month Model
Seasonal Effects and Other Anomalies
A Practitioner's Defense of Return Predictability
The data file above contains various financial variables over a
span of 1952-2017. A brief description of each of these variables may be
found here. To get started on building an intuition as to why
these may hold predictive power, you may look at the
Baltic
Dry Index (BDIY), the
Volatility Index (VIX),
calendar
effect (SIM, TOM) and
sentiment
index (Hulbert).
You can find all of these variables in the description files
linked above.
Please do not use the Hull Tactical data set for any purpose
other than the ERP contest.
Contact
For contest related inquiries you may email<EMAIL_ADDRESS>Please do NOT email the PSTAT staff or faculty with any contest related questions.


Git, GitHub, and version control walk into a bar...
Attack of the clones, forks, and requests to Push!
10/21/2015
Greetings
Greetings and welcome to a beginner's orientation on the concept of Git. With a few powerful tools, you will be able to create and upload web files for all the world to see. As you dive further into this topic later on, you may discover that the workflow might not come as easily as you had previously hoped. Oftentimes you might find yourself banging your head against a wall only to notice a door a few feet away. Patience and research are key during this time. For now, let us take a look at the general operation of Git.
The Command Line
Think about all the actions you do on your computer using its graphic interface. You can use your mouse to open files, create documents, drag folders around, etc. Now imagine doing the majority of commands in a fraction of the time by typing in a few keywords. This is essentially what a command line interface allows you to do.
The actions you commit via the command line terminal and the actions you take by clicking around your machine's interface are just different ways of interacting with the same system. Keep this in mind as you teach yourself the basic commands. For example, when you type in a command like mkdir (make directory), think about how you would typically interact with your machine to perform the same function. As you continue to teach yourself more commands, you will begin to appreciate the speed and agility with which you can navigate your system, move folders, or create files.
What's important to keep in mind when writing actions on the command line is that with great ease comes great opportunities to make computer-altering mistakes. Rarely will commands you type in be greeted with a warning. If a warning or error message presents itself, take the time to try to understand what exactly your computer it attempting to say to you. Attention to detail is key here. Make sure you are always targeting the right directory or file before going any further.
The command line terminal is not only for interacting with your machine. You can use it to push projects you have made into the world for all to see! This will be addressed in the next two sections.
Why version control?
No one gives birth to an adult. (Thank goodness, right?) Maybe you think the first version of your code got everything right. No bugs? Never going to add features later? Will it continue to work forever, as people upgrade their operating systems or their browsers? Things can get complicated pretty quickly, and if you want any help, you'll have multiple people contributing to the code at once. You need some system for keeping track of what changed in each version, and a way to go back to old versions or have multiple versions in the works at once.
Git is a version control system for coding projects and software development. Before we can talk about Git and its relation to the command line terminal, we have to talk about what a version control system is.
Essentially a version control system is the means by which every revision of a project's components are tracked for every contributor to see. Imagine an author keeping various manuscripts of one work, each version detailing where a specific change was made through time. A version control system for software development is a step above that, explicitly outlining every change, showing the content of a previous version, and even requiring review by team members before it can go live.
There are several reasons for why version control is crucial in development. As noted before, many contributors can be working on multiple parts of a single project at once. Many of these components hinge on each other to keep the whole system going. Imagine if suddenly most parts of the project stopped working and the root cause had to be tracked down. With a version control system, a teammate can see what recent changes were made in the program’s blueprint, figure out how it is impacting the overall system, and make any necessary modifications. It is a means by which developers can quickly get a diagnosis on what is causing the symptoms to be treated. Furthermore, version control often requires peer review before changes go live. It is essentially a checkpoint to catch anything that needs to be changed.
Git is a version control system specific to software development. In relation to your command line terminal, it is a program that gives you the power to make and track changes on a project with a few simple commands. Git keeps track of the changes you make by pinpointing when and where they were done. This history of your work can be accessed through your command line terminal using specific Git commands. For simplicity's sake, it is best to use Git bash, the simple terminal you can download along with the Git program.
How git helps...
All the code for your project gets stored in a directory, and git keeps track of what changes you make to files in that directory. You can create code branches to try something out without messing up your master copy. Git lets you switch seemlessly between them, without having to manage all those copies of your code yourself.What about GitHub?
As the name suggests, GitHub can function as a central online repository for your code and history of your code changes, and those of other people contributing to the same code. It can also be how you release your code to the public, or as a backup in case you drop your laptop into lava.Github: where version control and group projects come together
Earlier in the previous section I described how version control was essential to projects as it established work history and its impact. Github is where the tools of Git can manifest themselves to establish accountability for actions within a project. While the tools of Git live locally in your computer, Github is the means by which your teammates or the general public can see the changes you have made. Github also operates as the platform through which your teammates can review changes you want to make to software.
I apologize if you have read the past few sentences over and over again. Do not worry. This can be made a bit clearer by reestablishing the function of each of our essential platforms.
Command line terminal: the interface through which you can navigate through and make changes to files in your computer.
Git: The system by which certain changes are logged for reference.
Github: The online platform through which certain changes that were logged by git are made visible to others.
And how are all three connected? Using specific Git commands through the command line terminal, you can make and track changes to a file on your machine and then push them through to Github.
Git resides in your machine. Github resides on the internet for all to see. In order to have your teammates see the changes you have made, you must explicitly make a connection to Github through your terminal using specific Git commands.
In Sum
The command line terminal is the interface by which you run certain operations on your machine. Git is the version control software through which particular operations can be historically tracked and pushed into existence. Git is used through specific git commands in the command line terminal. Github is the cloud-based system (or simply put, website) through which changes to a file or project can be pushed through for others to see. Those changes can be pushed by making an explicit connection to the site through the terminal using Git commands.
Keep these general concepts of the systems when working through the smaller steps. Remember that the overall goal is to create and make changes in software while sharing them with teammates. These components all work together to make that possible.


Your Green Gene Dashboard
Green Score
84
keyboard_arrow_up
3.5%
Avg. Local Score
82
keyboard_arrow_up
5.5%
Calories Remaining
947
keyboard_arrow_down
8%
Body Mass Index
26.6
keyboard_arrow_down
3%
Green Score Hisory
84
keyboard_arrow_up
3.5%
Users
32
keyboard_arrow_up
23%
CTR
0.20%
keyboard_arrow_up
4%
| Name | Green Score | Age | Start date | Current progress |
|---|---|---|---|---|
| Yoda | 91 | 900 | 2/27/18 | +18% |
| Luke Skywalker | 89 | 23 | 1/19/17 | +2% |
| Obi-Wan Kenobi | 88 | 43 | 7/5/17 | +9% |
| Darth Vader | 82 | 57 | 8/2/18 | -4% |
| Han Solo | 81 | 33 | 5/27/17 | -12% |
| C-3PO | 78 | 27 | 3/22/16 | +1% |
| Darth Maul | 75 | 22 | 9/23/17 | +2% |
| Chewbacca | 73 | 39 | 8/5/18 | +8% |
| Boba Fett | 71 | 43 | 9/21/17 | +6% |
| R2-D2 | 69 | 27 | 4/27/18 | +17% |
Food Ate Today


--- layout: page title: All About Me... description: This is what I do. background: '/img/bg-about.jpg' ---
Hi, I'm Jayesh and I create all the stuff you see on the site. I'm a(mostly) self-taugh software engineer, who loves coding and helping people to code.
I'm obessed with: my camera, wife, son and building art with JavaScript.
Writing blogs on technologies I am learning has completly changed my life!


Enter the amount of the bill, including tax and tip, as well as the number of people in the group who are splitting the bill. We do all the math!
Enter the email addresses of your friends, family, colleagues, or random meal buddies. We'll send them all a reminder about the bill.
When the other group members check their email, everyone will have a unique link to pay their share of the bill for the meal. No one needs to worry about getting cash or sending money through another app - Split the Bill lets you handle everything in one place!


Benefits
At CREA you will be able to prosper in personal life and career in a friendly environment with a great team.
Your happiness is important to us. Besides interesting and challenging projects we greatly value family and personal time.
We offer work on international projects with large scale impact, access to expert knowledge worldwide, use of new and innovative technology.
We offer organic fruit and veggies to keep you focused throughout the day, while free gym membership is available for you to stay energized.
... and more
New, spacious, creativity boosting HeadQuarters, with a balcony exit from every office. Excellent access from all directions and plenty of free parking spaces for your space ships.
We are a diverse group of talented people, always supportive of each other and always pushing forward. A team that is always ready to help and step up.
Work on interesting and always evolving projects that will challenge you every single day and push your skills to the next level.


class: center, middle, inverse, title-slide # ETC1010: Data Modelling and Computing ## Lecture 5A: Missing Data ### Dr. Nicholas Tierney & Professor Di Cook ### EBS, Monash U. ### 2019-08-30 --- class: bg-blue center middle .vvhuge.white[ While the song is playing... ] .vhuge.white[ Draw a mental model / concept map of last lectures content on data visualisation. ] --- # recap .vhuge[ - Joins - advanced data vis ] --- # Motivation .vvhuge[ Why would I want to deal with missing data? ] ??? I was handed a dataset in my PhD that had more than 65% of the data missing. We had to spend quite a bit of time exploring the missingness relationships and the whole process was very frustrating, this inspired me to develop methods for exploring missing data, which ended up being a substantial part of my thesis But today we are going to talk about some example weather data from Melbourne. --- class: center, middle # Example ### San Francisco weather data ### || Date | Wind | Temp || ### Using the R package: [`GSODR`](https://github.com/ropensci/gsodr) ### [(Global Surface Summary of the Day)](https://data.noaa.gov/dataset/global-surface-summary-of-the-day-gsod) ### Written by Adam Sparks ### github.com/ropensci/GSODR ---  --- class: middle, center # Why data was missing  ???f So here we can create a graphic where we can see the data that are present are in blue, and the data that are missing are in red. What we see here then is that the data that are missing for air temperature are in red here, but are actually measured only beyond gust speeds of 8 kpmh So in cases of strong winds, the air temperature measurements break. Done. Dusted. --- class: inverse, center, middle .vvhuge[ Wait, What? ] ??? But, like all good explanations, this one is simple - but the process to get there, where we describe it, probably was not. In order to get to a position where you could generate this graphic, and show this data, you probably had to spend more time than you would have liked developing exploratory data analyses and models It might seem obvious to point out the precise mechanism for generating the missingness, but this is kind of difficult when there is a lot of missing data, or when there are many variables. --- class: inverse, center, middle .vvhuge[ What dealing with missing data looks like in a paper ] --- background-image: url(https://njtierney.updog.co/gifs/narnia-doppel.jpg) background-size: contain background-position: 50% 50% class: center, bottom, inverse --- background-image: url(https://njtierney.updog.co/gifs/narnia-golf-beer.gif) background-color: #000000 background-size: contain background-position: 50% 50% class: center, bottom, inverse --- class: inverse, center, middle .vvhuge[ What dealing with missing data actually looks like ] --- background-image: url(https://njtierney.updog.co/gifs/narnia-dart-throw.gif) background-size: contain background-color: #000000 background-position: 50% 50% class: center, bottom, inverse --- class: inverse, center, middle .vvhuge[ What I want dealing with missing data to be like ] --- background-image: url(https://njtierney.updog.co/gifs/njt-ideal-missing-data.gif) background-size: contain background-color: #000000 background-position: 50% 50% class: center, bottom, inverse --- class: inverse, middle, center # Learn more  .pull-left[ .center[ .huge[ visdat.njtierney.com ] ] ] .pull-right[ .center[ .huge[ naniar.njtierney.com ] ] ] --- class: bg-main1 # Overview .huge[ 1. What even are missing values 1. How to start looking at missing data 1. How to start exploring missing data 1. How to impute (fill in) Missing values ] --- class: bg-main1 # What are missing values? .vhuge[ > Missing values are values that should have been recorded but were not. `NA` = **N**ot **A**vailable. ] ??? Before we get started, we need to define missing values Missing values are values that should have been recorded, but were not. Think of it this way: You might accidentally not record seeing a bird - this is a missing value. This is different to recording that there were no birds observed. R stores missing values as `NA`, which stands for **N**ot **A**vailable. --- class: bg-main1 # How do I check if I have missing values? ```r x <- c(1, NA, 3, NA, NA, 5) ``` ```r any_na(x) ``` ``` [1] TRUE ``` ```r are_na(x) ``` ``` [1] FALSE TRUE FALSE TRUE TRUE FALSE ``` ```r n_miss(x) ``` ``` [1] 3 ``` ```r prop_miss(x) ``` ``` [1] 0.5 ``` ??? Missing values don't jump out and scream "I'm here!". They're usually hidden, like a needle in a haystack. To detect missing values use `any_na`, which returns TRUE if there are any missings, and FALSE if there are none. `are_na` asks "are these NA?" and returns TRUE/FALSE for each value `are_na` shows us 3 TRUE values - 3 missing values. To avoid counting each TRUE yourself, `n_miss` counts the number of missings And `prop_miss` gives the proportion of missings, which gives important context: 50% of data is missing! --- class: bg-main1 # Working with missing data .huge[ `NA` + [anything] = `NA` ] ```r heights ``` ``` Sophie Dan Fred 165 177 NA ``` ```r sum(heights) ``` ``` [1] NA ``` --- class: bg-main1 # Working with missing data .vhuge[ `na.rm = TRUE` will removes missings ] ```r sum(heights, na.rm = TRUE) ``` ``` [1] 342 ``` -- .vhuge[ Use this power `responsibly`! ] ??? So what happens when we mix missing values with our calculations? We need to know what happens, so we can be primed to find these cases. The general rule is this: Calculations with `NA` return `NA`. Say you have the height of three friends: Sophie, Dan, and Fred. The sum of their heights returns `NA`, This is because we don't know the sum of a number and NA. --- class: bg-main5 # Your turn: .vlarge[ - Open rstudio.cloud - go to `exercise-5a-intro-missing.Rmd` - If you want to use R / Rstudio on your laptop: - Install R + Rstudio (see ) - open R - type the following: ```r # install.packages("usethis") library(usethis) use_course("dmac.netlify.com/lectures/lecture5a/exercise/exercise-5a.zip") ``` ] --- class: bg-main1 # Introduction to missingness summaries .huge[ Basic summaries of missingness: - `n_miss` - `n_complete` ] -- .huge[ Dataframe summaries of missingness: - `miss_var_summary` - `miss_case_summary` These functions work with `group_by` ] ??? Now that you understand what missing values are, how to count them, and how they operate, let's scale these up to more detailed summaries of missingness. We need to summarise missing data to identify variables, cases, or patterns of missingness, as these can bias our data analysis. There are two main summaries: basic, and dataframe summaries. Basic summaries return a single number, like the number of missing or complete values using `n_miss` or `n_complete`. However, you will need more detailed missingness summaries to help you on your journey through a data analysis. This lesson introduces you to missing data summaries. `naniar` provides a family of functions all starting with `miss_`., which each provide different summaries of missingness, and return a dataframe. This allows us to see features that can be difficult to articulate, or time consuming to calculate. For example, `miss_var_summary` and `miss_case_summary` return the number and percentage of missings in each variable or case. These summaries work with `dplyr`''s `group_by`, so you can fluidly explore missingness by each groups. --- class: bg-main1 # Missing data summaries: Variables ```r miss_var_summary(dat_sf_clean) ``` ``` ## # A tibble: 6 x 3 ## variable n_miss pct_miss ## <chr> <int> <dbl> ## 1 temp_min 70 17.3 ## 2 temp_max 70 17.3 ## 3 temp_avg 70 17.3 ## 4 wind_speed_max 23 5.68 ## 5 date 0 0 ## 6 month 0 0 ``` ??? Use `miss_var_summary` to summarise the number of missings in each variable. This returns a dataframe where each row is a variable. It also includes summaries of the number and percentage of missings for each variable in the dataset, and is sorted by the number of missings. For example, Ozone has 37 missing values, and is about 24.2 percent missing. --- class: bg-main1 # Missing data summaries: Cases ```r miss_case_summary(dat_sf_clean) ``` ``` ## # A tibble: 405 x 3 ## case n_miss pct_miss ## <int> <int> <dbl> ## 1 89 4 66.7 ## 2 182 4 66.7 ## 3 188 4 66.7 ## 4 271 4 66.7 ## 5 6 3 50 ## 6 7 3 50 ## 7 10 3 50 ## 8 29 3 50 ## 9 37 3 50 ## 10 39 3 50 ## # … with 395 more rows ``` ??? Similar to `miss_var_summary`, `miss_case_summary` returns a summary dataframe, where each case represents a dataset row number. Here, case 5 - the fifth row in the dataset - has 2 missing values, which means 33% of that case is missing. --- class: bg-main1 # Missing data tabulations: variables ```r miss_var_table(dat_sf_clean) ``` ``` ## # A tibble: 3 x 3 ## n_miss_in_var n_vars pct_vars ## <int> <int> <dbl> ## 1 0 2 33.3 ## 2 23 1 16.7 ## 3 70 3 50 ``` --- class: bg-main1 # Missing data tabulations: cases ```r miss_case_table(dat_sf_clean) ``` ``` ## # A tibble: 4 x 3 ## n_miss_in_case n_cases pct_cases ## <int> <int> <dbl> ## 1 0 316 78.0 ## 2 1 19 4.69 ## 3 3 66 16.3 ## 4 4 4 0.988 ``` ??? Tabulation of missingness counts the number of times there are 0, 1, 2, 3, and so on, missings. They are very useful, compact summaries that reveal interesting structure. `miss_var_table` returns a dataframe with the number of missings in a variable, and the number and percentage of variables affected. For example, there are four variables with no missings detected, which corresponds to 66.7 percent of variables, and there was 1 variable with 7 missings, and 1 variable with 37 missings. Similarly, `miss_case_table` returns the same information, but for cases. --- class: bg-main1 # Using summaries with `group_by` ```r dat_sf_clean %>% group_by(month) %>% miss_var_summary() ``` ``` ## # A tibble: 60 x 4 ## month variable n_miss pct_miss ## <dbl> <chr> <int> <dbl> ## 1 1 temp_min 7 11.5 ## 2 1 temp_max 7 11.5 ## 3 1 temp_avg 7 11.5 ## 4 1 wind_speed_max 4 6.56 ## 5 1 date 0 0 ## 6 2 temp_min 5 12.8 ## 7 2 temp_max 5 12.8 ## 8 2 temp_avg 5 12.8 ## 9 2 wind_speed_max 4 10.3 ## 10 2 date 0 0 ## # … with 50 more rows ``` ??? Sometimes you are interested in missingness for groups in the data. Each missingness summary function can be calculated by group, using `group_by` from `dplyr`. For example, we can look at the missingness by Month in the airquality dataset. Here we see that Month 5 for Ozone there are 5 missings, but for Month 6 Ozone has 21 missings. --- class: bg-main5 # Your Turn .huge[ - Open exercise-5a-summarise-missings.Rmd ] --- class: bg-main1 # Introduction to missing data visualisations in naniar .huge[ - Visualisation can quickly capture an idea or thought. - `naniar` provides a friendly family of missing data visualization functions. - Each visualization corresponds to a data summary. - Visualisations help you operate closer to the speed of thought. ] ??? We cover how to get a bird's eye view of the data, how to look at missings in the variables and cases, and how to generate visualizations for missing spans and across groups in the data. We now know what missing values are, how they work, how to count and summarise them - now let's look at some of the built-in visualisations that come with `naniar`. Data summaries are very useful, but sometimes an idea or a thought can be quickly captured with a visualisation. `naniar` provides a friendly family of missing data visualisation functions, each presenting different visualisations missingness summaries. In fact, each of these visualisations is a nice compact shorthand for the data summaries. While you could create similar and more complex visualisations using the summary information from the previous lesson, this can be repetitive. The visualisations in `naniar` reduce repetition and increase iteration, so you can operate closer to the speed of thought. --- class: bg-main1 # Get a bird's eye view of the missing data ```r vis_miss(dat_sf_clean) ```  ??? When you first get a dataset, it can be difficult to get a visceral sense of **where** the missings are. To get an overview of the amount of missingness, use the `vis_miss` function from the `visdat` package. `vis_miss` produces a "heatmap" of the missingness - like as if the plot corresponded to the dataset as a giant spreadsheet, with values coloured black for missing, and grey for present. `vis_miss` also provides missingness summary statistics, showing the overall percentage of missingness in the legend, and the amount of missings in each variable. These can be turned off in its options, described in the helpfile. --- class: bg-main1 # Get a bird's eye view of the missing data ```r vis_miss(dat_sf_clean, cluster = TRUE) ```  ??? `vis_miss` also allows for clustering of the missing data by setting `cluster = TRUE`: this orders the rows by missingness to identify common co-occurrences. --- class: bg-main1 # Look at missings in cases ```r gg_miss_var(dat_sf_clean) ```  --- class: bg-main1 # Look at missings in cases ```r gg_miss_case(dat_sf_clean) ```  ??? To quickly show the missingness in variables and cases, we visualise them using `gg_miss_var` and `gg_miss_case`. Note that these are visual analogues of the `miss_var_summary` and `miss_case_summary` functions. These plots show the amount of missingness on the x axis, and for `gg_miss_var`, each point represents the amount of missingness in that variable, and for `gg_miss_case`, each line represents the amount of missingness in that case. Note that these visualisations are ordered so that the most missing is at the top. The ordsering in `gg_miss_case` can be turned off with option, `order_cases = FALSE`. --- class: bg-main1 # Look at missings in variables ```r gg_miss_var(dat_sf_clean, facet = month) ```  ??? `gg_miss_var` and `gg_miss_case` also allow for facetting by one variable. This means you can explore missingness in cases and variables across the levels of another group. This plot is facetted by month, showing the number of missings in each variable for each month. Here we see that Ozone in Month 6 has the most missings. --- # Visualizing missingness patterns ```r gg_miss_upset(dat_sf_clean) ```  ??? To visualise the common combinations of missingness - which variables and cases go missing together, use `gg_miss_upset`. This powerful visualisation shows the number of combinations of missing values that co-occur. An upset plot of the `airquality` dataset shows there are only missing values in Ozone and Solar.R, with 35 in only Ozone, 5 in Solar.R, and in both Ozone and Solar.R, there are 2 missing cases. --- # Visualizing factors of missingness ```r gg_miss_fct(x = dat_sf_clean, fct = month) ```  ??? To explore how missingness in each variable changes across a factor, use `gg_miss_fct`. This displays a heatmap visualisation showing the factors on the x axis, each other variable on the y axis, and the amount of missingness coloured from dark purple to yellow. `gg_miss_fct` does not support facetting. --- class: bg-main1 # Your turn .huge[ - complete exercise-5a-visualise-missings.Rmd ] --- class: bg-main1, middle .pull-left[ # `miss_*` # `miss_var_*` # `miss_case_*` ] -- .pull-right[ # `gg_miss_*` # `gg_miss_var` # `gg_miss_case` ] --- class: bg-black .vvhuge.white[ Principles of Tidy _Missing_ Data ] --- class: bg-main1 .vvhuge[ The Shadowlands ] .huge[ Representing Missing values ] --- class: bg-white # Tidy Data .huge.pull-left[ Variables in columns Observations in Rows One value per cell ] .pull-right[  ] --- class: bg-white # Data Shadow .huge.pull-left[ Variable ends in NA Values are missing (NA) or not (!NA) ] .pull-right[  ] --- class: bg-main1 center middle # Tidy Missing Data .huge[ `bind_shadow(data)` ]  --- class: bg-main1 # `bind_shadow()` ```r bind_shadow(dat_sf_clean) %>% glimpse() ``` ``` ## Observations: 405 ## Variables: 12 ## $ date <date> 2017-01-02, 2017-01-03, 2017-01-04, 2017-01-05, 2017-01-06, … ## $ month <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1… ## $ temp_min <dbl> 5.9, 8.4, 10.4, 5.8, 4.4, NA, NA, 11.9, 11.9, NA, 6.7, 5.5, 6… ## $ temp_max <dbl> 10.5, 11.1, 14.5, 9.4, 9.1, NA, NA, 16.0, 14.6, NA, 11.2, 11.… ## $ temp_avg <dbl> 8.4, 9.7, 12.4, 8.1, 6.4, NA, NA, 13.5, 13.2, NA, 8.5, 7.7, 8… ## $ wind_speed_max <dbl> 5.1, 5.1, 6.7, 5.1, 5.1, 8.8, 8.2, 7.2, 7.7, 8.2, 5.1, 4.1, 4… ## $ date_NA <fct> !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !… ## $ month_NA <fct> !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !… ## $ temp_min_NA <fct> !NA, !NA, !NA, !NA, !NA, NA, NA, !NA, !NA, NA, !NA, !NA, !NA,… ## $ temp_max_NA <fct> !NA, !NA, !NA, !NA, !NA, NA, NA, !NA, !NA, NA, !NA, !NA, !NA,… ## $ temp_avg_NA <fct> !NA, !NA, !NA, !NA, !NA, NA, NA, !NA, !NA, NA, !NA, !NA, !NA,… ## $ wind_speed_max_NA <fct> !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !NA, !… ``` --- class: bg-main1 # Shadows In Practice: Explore one variable ```r dat_sf_clean %>% ggplot(aes(x = wind_speed_max)) + geom_density() ```  --- # Shadows In Practice: Explore one variable ```r dat_sf_clean %>% *bind_shadow() %>% ggplot(aes(x = wind_speed_max, * colour = temp_avg_NA)) + geom_density() ```  ??? - Provides a consistent way to set the existing location of missing values. - This has additional great benefits when combined with imputation - Also allows for additional visualisations --- class: bg-main1 # In Practice: Explore two variables .left-code[ ```r ggplot(dat_sf_clean, aes(x = temp_avg, y = wind_speed_max)) + geom_point() ``` ] .right-plot[  ] --- class: bg-main1 # Impute shadow values into our realm .pull-left[ ``` ## # A tibble: 7 x 2 ## temp_avg temp_avg_NA ## <dbl> <fct> ## 1 8.4 !NA ## 2 9.7 !NA ## 3 12.4 !NA ## 4 8.1 !NA ## 5 6.4 !NA ## 6 NA NA ## 7 NA NA ``` ] -- .pull-right[ ``` ## # A tibble: 7 x 2 ## temp_avg temp_avg_NA ## <dbl> <fct> ## 1 8.4 !NA ## 2 9.7 !NA ## 3 12.4 !NA ## 4 8.1 !NA ## 5 6.4 !NA ## 6 5.66 NA ## 7 5.69 NA ``` ] ??? One approach, (ggobi) is to **shift** missing values below the minimum value This then means that they can be plotted on the same axis. Typically, when exploring this data, you would do something like this: The problem with this is that ggplot does not handle missings be default, and removes the missing values. This makes it hard to explore the missing values. --- class: bg-main1 # `impute_below()` # Impute missing values from the shadows into our realm ```r dat_sf_clean %>% slice(5:10) %>% * mutate(temp_avg_shift = impute_below(temp_avg)) %>% select(temp_avg, temp_avg_shift) ``` ``` ## # A tibble: 6 x 2 ## temp_avg temp_avg_shift ## <dbl> <dbl> ## 1 6.4 6.4 ## 2 NA 5.73 ## 3 NA 5.74 ## 4 13.5 13.5 ## 5 13.2 13.2 ## 6 NA 5.53 ``` --- class: bg-main1 # `geom_miss_point()` .left-code[ ```r ggplot(dat_sf_clean, aes(x = wind_speed_max, y = temp_avg)) + * geom_miss_point() ``` ] .right-plot[  ] ??? Instead, we have create a ggplot2 geom, `geom_missing_point()`. This `geom` allows for missing values to be displayed, and also works with the rest of ggplot2 - themes, and facets as well. --- class: center # Facets!  --- class: bg-main1 .vhuge[ Exploring imputed values ] -- .huge[ - Imputation is the process of filling in missing values with some other estimate ] --- # What about this imputation thing? ```r dat_sf_clean %>% * simputation::impute_lm(temp_avg ~ wind_speed_max) %>% ggplot(aes(x = date, y = temp_avg)) + geom_point() ```  --- class: bg-main1 .vhuge[ They are .yello[invisible!] Where are the imputed values? ] --- # Tidy Missing Data reveals the imputations! ```r *bind_shadow(dat_sf_clean) %>% simputation::impute_lm(temp_avg ~ wind_speed_max + date) %>% ggplot(aes(x = date, y = temp_avg, * colour = temp_avg_NA)) + geom_point() + scale_colour_brewer(palette = "Dark2") ```  --- # Shadows make things clearer! ```r *bind_shadow(dat_sf_clean) %>% simputation::impute_lm(temp_avg ~ wind_speed_max) %>% ggplot(aes(x = temp_avg, * fill = wind_speed_max_NA)) + geom_density(alpha = 0.5) ```  --- class: bg-main1 # Example data: oceanbuoys ```r oceanbuoys ``` ``` ## # A tibble: 736 x 8 ## year latitude longitude sea_temp_c air_temp_c humidity wind_ew wind_ns ## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> ## 1 1997 0 -110 27.6 27.1 79.6 -6.40 5.40 ## 2 1997 0 -110 27.5 27.0 75.8 -5.30 5.30 ## 3 1997 0 -110 27.6 27 76.5 -5.10 4.5 ## 4 1997 0 -110 27.6 26.9 76.2 -4.90 2.5 ## 5 1997 0 -110 27.6 26.8 76.4 -3.5 4.10 ## 6 1997 0 -110 27.8 26.9 76.7 -4.40 1.60 ## 7 1997 0 -110 28.0 27.0 76.5 -2 3.5 ## 8 1997 0 -110 28.0 27.1 78.3 -3.70 4.5 ## 9 1997 0 -110 28.0 27.2 78.6 -4.20 5 ## 10 1997 0 -110 28.0 27.2 76.9 -3.60 3.5 ## # … with 726 more rows ``` --- # Start looking at missing values ```r vis_dat(oceanbuoys) ```  --- # Start looking at missing values ```r vis_miss(oceanbuoys) ```  ??? West Pacific Tropical Atmosphere Ocean Data, 1993 & 1997, for improved detection, understanding and prediction of El Nino and La Nina, collected from [http://www.pmel.noaa.gov/tao/index.shtml](http://www.pmel.noaa.gov/tao/index.shtml) The data has 736 observations, and 8 variables. All of the variables are numeric (including year). ??? What do we learn? - Two variables, air temperature and humidity, have a large number of missings. - Year, latitude and longitude have no missings - Sea temperature has a couple of missings - Some rows have a lot of missings Overall statistics, 3% of possible values are missing. That's not much. BUT, both air temperature and humidity have more than 10% missing which is too much to ignore. This type of display is called a "heatmap", displays the data table, with cells coloured according to some other information. In this case it is type of variable, and missingness status. What do we learn? - Most of the variables are `character` (text) variables, or `integer` variables. - Missing values are located only in the counts, but it is in blocks, so perhaps corresponds to some category levels of the other variables. --- class: bg-main1 # Missing value patterns ```r gg_miss_upset(oceanbuoys) ```  ??? --- # Missings Tend to get ignored by most software .left-code[ ```r ggplot(oceanbuoys, aes(x = sea_temp_c, y = humidity)) + geom_point() + theme(aspect.ratio = 1) ``` ] .right-plot[  ??? and this is a problem, because results computed on data with missing values might be biased. for example, `ggplot` ignores them, but at least tells you its ignoring them: --- # Add missings to plot with `geom_miss_point()` .left-code[ ```r ggplot(oceanbuoys, aes(x = sea_temp_c, y = humidity)) + scale_colour_brewer(palette="Dark2") + geom_miss_point() + theme(aspect.ratio=1) ``` ] .right-plot[  --- # Facet By year ```r ggplot(oceanbuoys, aes(x = sea_temp_c, y = humidity)) + geom_miss_point() + scale_colour_brewer(palette = "Dark2") + facet_wrap(~year) + theme(aspect.ratio=1) ```  ] --- # Understanding missing dependencies .left-code[ ```r ggplot(oceanbuoys, aes(x = sea_temp_c, y = air_temp_c)) + geom_miss_point() + scale_colour_brewer(palette="Dark2") + facet_wrap(~year) + theme(aspect.ratio=1) ``` ] .right-plot[  ] ??? What do we learn? - There is a different missingness pattern in each of the years - Year needs to be accounted for in finding good substitute values. --- # Strategies for working with missing values .huge[ - Small fraction of cases have several missings (around 5%) - explore data, and possibly drop the cases - A variable or two, out of many, have a lot of missings, drop the variables ] --- # Strategies for working with missing values .huge[ - If missings are small in number, but located in many cases and variables, you need to impute these values, to do most analyses - Designing the imputation should take into account dependencies that you have seen between missingness and existing variables. - For the ocean buoys data this means imputation needs to be done separately by year ] --- ## Common ways to impute values .huge[ - (Usually bad) Simple parametric: use the mean or median of the complete cases for each variable - (Better) More complex: use models to predict missing values - (Best) Multiple imputation: Use a statistical distribution, e.g. normal model and simulate a value (or set of values, hot deck imputation) for the missings. ] --- class: bg-main1 # Setup for missings ```r tao_shadow <- bind_shadow(oceanbuoys) tao_shadow ``` ``` ## # A tibble: 736 x 16 ## year latitude longitude sea_temp_c air_temp_c humidity wind_ew wind_ns year_NA ## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <fct> ## 1 1997 0 -110 27.6 27.1 79.6 -6.40 5.40 !NA ## 2 1997 0 -110 27.5 27.0 75.8 -5.30 5.30 !NA ## 3 1997 0 -110 27.6 27 76.5 -5.10 4.5 !NA ## 4 1997 0 -110 27.6 26.9 76.2 -4.90 2.5 !NA ## 5 1997 0 -110 27.6 26.8 76.4 -3.5 4.10 !NA ## 6 1997 0 -110 27.8 26.9 76.7 -4.40 1.60 !NA ## 7 1997 0 -110 28.0 27.0 76.5 -2 3.5 !NA ## 8 1997 0 -110 28.0 27.1 78.3 -3.70 4.5 !NA ## 9 1997 0 -110 28.0 27.2 78.6 -4.20 5 !NA ## 10 1997 0 -110 28.0 27.2 76.9 -3.60 3.5 !NA ## # … with 726 more rows, and 7 more variables: latitude_NA <fct>, longitude_NA <fct>, ## # sea_temp_c_NA <fct>, air_temp_c_NA <fct>, humidity_NA <fct>, wind_ew_NA <fct>, ## # wind_ns_NA <fct> ``` --- class: bg-main1 # Imputing the Mean (ignoring year). ```r tao_imp_mean <- tao_shadow %>% mutate(sea_temp_c = impute_mean(sea_temp_c), air_temp_c = impute_mean(air_temp_c)) tao_shadow ``` ``` ## # A tibble: 736 x 16 ## year latitude longitude sea_temp_c air_temp_c humidity wind_ew wind_ns year_NA ## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <fct> ## 1 1997 0 -110 27.6 27.1 79.6 -6.40 5.40 !NA ## 2 1997 0 -110 27.5 27.0 75.8 -5.30 5.30 !NA ## 3 1997 0 -110 27.6 27 76.5 -5.10 4.5 !NA ## 4 1997 0 -110 27.6 26.9 76.2 -4.90 2.5 !NA ## 5 1997 0 -110 27.6 26.8 76.4 -3.5 4.10 !NA ## 6 1997 0 -110 27.8 26.9 76.7 -4.40 1.60 !NA ## 7 1997 0 -110 28.0 27.0 76.5 -2 3.5 !NA ## 8 1997 0 -110 28.0 27.1 78.3 -3.70 4.5 !NA ## 9 1997 0 -110 28.0 27.2 78.6 -4.20 5 !NA ## 10 1997 0 -110 28.0 27.2 76.9 -3.60 3.5 !NA ## # … with 726 more rows, and 7 more variables: latitude_NA <fct>, longitude_NA <fct>, ## # sea_temp_c_NA <fct>, air_temp_c_NA <fct>, humidity_NA <fct>, wind_ew_NA <fct>, ## # wind_ns_NA <fct> ``` --- class: bg-main1 # Imputing the Mean (ignoring year). .left-code[ ```r ggplot(tao_imp_mean, aes(x = sea_temp_c, y = air_temp_c, colour = air_temp_c_NA)) + geom_point(alpha = 0.7) + facet_wrap(~year) + scale_colour_brewer(palette = "Dark2") + theme(aspect.ratio = 1) ``` ] .right-plot[  ] ??? What do we learn? - Oh, this is so wrong! - The imputed values are nothing like the complete case values --- # Impute Mean by year ```r tao_shadow <- tao_shadow %>% group_by(year) %>% mutate(sea_temp_c = impute_mean(sea_temp_c), air_temp_c = impute_mean(air_temp_c)) ``` --- # by year .left-code[ ```r ggplot(tao_shadow, aes(x = sea_temp_c, y = air_temp_c, colour=air_temp_c_NA)) + geom_point(alpha=0.7) + facet_wrap(~year) + scale_colour_brewer(palette="Dark2") + theme(aspect.ratio=1) ``` ] .right-plot[  ] ??? What do we learn? - The imputed values are closer to the complete case values - However, they form a rigid line, mismatching the variation - and they extend outside the range of complete values. This is a problem in that the imputed air temperature value for these high sea temperature cases is lower than we would expect, and thus possibly impeding good model fitting ### Two minute challenge - Change the code to plot sea temperature against humidity, with colour representing missing humidity values. What do you learn about the imputations? ### Relationship with other variables ```r ggplot(data = tao_shadow, aes(x = wind_ew, y=wind_ns, colour=air_temp_c_NA)) + scale_colour_brewer(palette="Dark2") + geom_point(alpha=0.7) + theme(aspect.ratio=1) ```  What do we learn? - The lowest values of east-west winds have no missing values. Maybe it is less likely to have air temperature missing values when there are light east-west winds? ### Two minute challenge - Generate the shadow matrix and make a plot of the winds, coloured by missingness on humidity. --- # Your Turn: .huge[ - lab quiz open (requires answering questions from Lab exercise) - go to rstudio.cloud and finish final exercise - If you want to use R / Rstudio on your laptop: - Install R + Rstudio (see ) - open R - type the following: ```r # install.packages("usethis") library(usethis) use_course("dmac.netlify.com/lectures/lecture4b/exercise/exercise-5a.zip") ``` ] --- # Resources .huge[ - [R-miss-Tastic](https://rmisstastic.netlify.com) - [naniar](http://naniar.njtierney.com/) - [visdat](http://visdat.njtierney.com/) ] --- # Share and share alike
This work is licensed under a Creative Commons Attribution 4.0 International License.


Classes | |
| class | ckBuf< T > |
| A memory pool for the specified type. More... | |
| class | ckFlag< D, T > |
| A set of bits as a data structure. More... | |
| class | ckID |
| An unique value to identify objects. More... | |
| class | ckList< T > |
| A sequence of elements stored in a linked list. More... | |
| class | ckMap< K, D > |
| A sorted associative container that contains unique key/value pairs. More... | |
| class | ckMsg< N > |
| A general purpose message with fixed number parameters. More... | |
| class | ckStr< T, N > |
| A string with fixed length. More... | |
| class | ckTree< T > |
| A tree container which can have a parent and children. More... | |
| class | ckType< D, T > |
| An enum type with specified size. More... |


{% extends "base.html" %} {% block title %}Index{% endblock %} {% block head %} {{ super() }} {% endblock %} {% block content %}
Upload multiple files with the file dialog or by dragging and dropping images onto the dashed region


{% extends "mturk/experiments/filter_content_inst_content.html" %} {% block instructions_content_text %}
Planar is NOT the same as flat. It is a related but different concept. Please read the instructions below to see the difference.
You will be shown a collection of images with objects outlined in red. Click on the objects that are planar -- regions with a single flat surface.
Planar shapes can have some variation, such as patterns on wood, or grooves between bricks, or fuzz on a carpet. For example, a brick or tile wall is still considered planar.
However, two very different planar surfaces (such as two perpendicular walls in the same red shape) do not count and should be rejected.
To explain what "planar" means, consider these examples. They are side views of objects.


✔ Task created.
An error has occured.Please make sure that you've filled the task name and that the board exists and is writable by you.
Connection error has occured.Cannot communicate with the KanbanTool server. Please make sure that you are connected to the Internet.
Task will be added to the board. Click here to select different one.


These are the 10 best companies to work for in South Africa
by Staff WriterThe Top Employers Institute has released its the ranking of the top employers in South Africa for 2020.
The ranking is based on employers that provide excellent employee conditions, nurture and develop talent throughout all levels of the organisation, and which strive to continuously optimise employment practices.
The companies are evaluated across a number of key areas including:
- Talent strategy;
- Workforce planning;
- On-boarding;
- Learning & development;
- Performance management;
- Leadership development;
- Career & succession management;
- Compensation & benefits;
- Culture.
The Top Employers Institute then evaluates the implementation of these practices and reviews how they are supported through strategy, ownership, practices, measurement and technology. It must also be noted that the Top Employers certification is by the application.
Microsoft was named as the top South African employer in 2020, moving up from third place in 2019. Telecommunications company Vodacom took second place (2019: 6th), while professional services company Accenture took third place (2019: 2nd),
Tata Consultancy Services and Unilver rounded up the top five for 2020.
The 10 best companies are outlined below:
1. Microsoft
2. Vodacom
3. Accenture
4. Tata Consultancy Services
5. Unilever
6. EY
7. Nestle
8. JTI
9. SAP
10. Sasol


--- layout: archive title: Zulip Chat Archive permalink: /stream/116395-maths/topic/new.20is_group_hom.html ---
Recently is_group_hom
became a class. Previously we had
def is_group_anti_hom (f : α → β) : Prop := ∀ a b : α, f (a * b) = f b * f a namespace is_group_anti_hom variables {f : α → β} (H : is_group_anti_hom f) include H theorem inv (a : α) : (f a)⁻¹ = f a⁻¹ := ...
and I could then write
variables {α : Type} [group α] def conj (a b : α) := a*b*a⁻¹ lemma inv_conj : (conj b a)⁻¹ = conj b (a⁻¹) := conj_is_mph.inv a
Is there such a compact way to write the proof of inv_conj with the new is_group_hom
? I'm not talking about the LHS/RHS switch. I'm talking about the ability to use projection notation instead of writing (is_group_hom.inv (conj b) a).symm
I already complained about this :P
I'm not really complaining
right, but the answer is still you can't use projection
I slightly prefer the old way in this case but I'm open to learning about advantages of the new way
Now you do
is_group_hom.one f
I think
You should be using inv_conj
in your proofs anyways, so this is a one-time cost
but what is conj_is_mph
?
Hm, I didn't realize it was almost exactly the same question (I was vaguely aware of Kenny asking something about this change but didn't check)
also you should probably swap inv_conj
for the same reason is_group_hom.inv
was swapped
Indeed my only use of this lemma was a rw [<-invconj]
I'm trying (once more...) to get back to old stuff because I hope I'll have time to resume
I tried to work on the pi instance PR. But I failed in the preparatory work
I wanted to properly dispatch the content of Johannes' prod_module.lean
But trying to move stuff always gets Lean to go crazy
everything gets frozen in VSCode and I only end up with a broken mathlib
Maybe I should PR everything into the prod_module
black-hole and then let you or Johannes decide when this becomes a problem
import algebra.group variables {α : Type} [group α] (a b : α) def conj := a * b * a⁻¹ instance conj.is_group_hom : is_group_hom (conj a) := λ x y, by simp [conj, mul_assoc] lemma inv_conj : conj a (b⁻¹) = (conj a b)⁻¹ := is_group_hom.inv (conj a) b
@Patrick Massot c'est c'que tu veux?
I know how to do this (except I went with by finish [is_group_hom, conj]
in the instance proof. I was only asking how to properly use the new interface
right, which I demonstrated
{% endraw %}Last updated: Apr 24 2020 at 19:02 UTC


| ADD_BOT |
data class ADD_BOT |
| AVAILABLE_ROOMS |
data class AVAILABLE_ROOMS |
| AllowedCardSelection |
enum class AllowedCardSelection |
| AvailableRoom |
data class AvailableRoom |
| CARD_PLAYED |
data class CARD_PLAYED |
| CHANGE_NAME |
data class CHANGE_NAME |
| CHAT_MESSAGE |
data class CHAT_MESSAGE |
| CONNECT |
data class CONNECT |
| CONNECTED |
data class CONNECTED |
| CONNECTING |
data class CONNECTING |
| CONNECTION_CONFIRMATION |
data class CONNECTION_CONFIRMATION |
| CONNECT_FAILED |
data class CONNECT_FAILED |
| CREATE_ROOM |
data class CREATE_ROOM |
| Card |
data class Card : ICard |
| CardTransformation |
data class CardTransformation
Is used for passing info about that how card should be transformed |
| CardValueChange |
data class CardValueChange : ICard
Tells whether card changed it's suit and rank |
| DEAL_DECK |
data class DEAL_DECK |
| DECK_DEALT |
data class DECK_DEALT |
| DISCONNECT |
data class DISCONNECT |
| DISCONNECTED |
data class DISCONNECTED |
| DO_PASS_HANDOVER |
data class DO_PASS_HANDOVER |
| DO_TAKE_HANDOVER |
data class DO_TAKE_HANDOVER |
| GAME_CARD_SELECTION_CHANGED |
data class GAME_CARD_SELECTION_CHANGED |
| GAME_ENDED |
data class GAME_ENDED |
| GAME_GRILL_SELECTION_CHANGED |
data class GAME_GRILL_SELECTION_CHANGED |
| GAME_HANDOVER_SELECTION_CHANGED |
data class GAME_HANDOVER_SELECTION_CHANGED |
| GAME_LEAVE_SERVER |
data class GAME_LEAVE_SERVER |
| GAME_NAME_TOO_LONG |
data class GAME_NAME_TOO_LONG |
| GAME_PLAY_CARD |
data class GAME_PLAY_CARD |
| GAME_SET_STAGE |
data class GAME_SET_STAGE |
| GAME_STARTED |
data class GAME_STARTED |
| GET_ROOMS |
data class GET_ROOMS |
| GRILL_CARDS_BUTTON |
data class GRILL_CARDS_BUTTON |
| GRILL_PLAYED |
data class GRILL_PLAYED |
| HAND_OVER_PLAYED |
data class HAND_OVER_PLAYED |
| HAND_OVER_TAKEN |
data class HAND_OVER_TAKEN |
| ICard |
interface ICard |
| I_AM_READY |
data class I_AM_READY |
| I_WANT_NEW_GAME |
data class I_WANT_NEW_GAME |
| JOIN_ROOM |
data class JOIN_ROOM |
| LEAVE_ROOM |
data class LEAVE_ROOM |
| LastPlayerScore |
data class LastPlayerScore |
| NAME_TAKEN |
data class NAME_TAKEN |
| NEXT_ROUND |
data class NEXT_ROUND |
| NEXT_TURN |
data class NEXT_TURN |
| PASS_CARDS_BUTTON |
data class PASS_CARDS_BUTTON |
| PLAYER_JOINED |
data class PLAYER_JOINED |
| PLAYER_REPLACED_WITH_BOT |
data class PLAYER_REPLACED_WITH_BOT |
| PLAY_CARD |
data class PLAY_CARD |
| PLAY_GRILL |
data class PLAY_GRILL |
| PLAY_HAND_OVER |
data class PLAY_HAND_OVER |
| PlayerSide |
enum class PlayerSide |
| READY_BUTTON |
data class READY_BUTTON |
| REGISTER |
data class REGISTER |
| REGISTERED |
data class REGISTERED |
| ROOM_CREATED |
data class ROOM_CREATED |
| ROOM_FULL |
data class ROOM_FULL |
| ROOM_JOINED |
data class ROOM_JOINED |
| ROOM_LEFT |
data class ROOM_LEFT |
| ROOM_NAME_TAKEN |
data class ROOM_NAME_TAKEN |
| Rank |
enum class Rank |
| SEND_CHAT_MESSAGE |
data class SEND_CHAT_MESSAGE |
| SERVER_FULL |
data class SERVER_FULL |
| SORT_HAND_BUTTON |
data class SORT_HAND_BUTTON |
| State |
data class State
Shared game state |
| Suit |
enum class Suit |
| TAKE_ALL_BUTTON |
data class TAKE_ALL_BUTTON |
| TAKE_CARDS_BUTTON |
data class TAKE_CARDS_BUTTON |
| TAKE_HANDOVER |
data class TAKE_HANDOVER |
| TAKE_NOTHING_BUTTON |
data class TAKE_NOTHING_BUTTON |
| TRICK_COLLECTED |
data class TRICK_COLLECTED |
| TRICK_FINISHED |
data class TRICK_FINISHED |
| UPDATED_SCORES |
data class UPDATED_SCORES |
| Address |
typealias Address = String |
| BottomHand |
typealias BottomHand = MutableList<Card> |
| Grill |
typealias Grill = List<Card> |
| HandOver |
typealias HandOver = List<Card> |
| Id |
typealias Id = String |
| Name |
typealias Name = String |
| Number |
typealias Number = Int |
| theBlackCatCard |
val theBlackCatCard: Card |


My MP3 player is refusing to wake up. And after I bought three new albums this weekend.
8/18/2009, 2:28:31 PM
Favs: 0
Retweets: 0
link--- layout: tweet ---
My MP3 player is refusing to wake up. And after I bought three new albums this weekend.
8/18/2009, 2:28:31 PM
Favs: 0
Retweets: 0
link


Installation and usage instructions are available on GitHub.
All images in the examples are given width="100%"
property
to make them responsive.
On CSS level, the component supports standard pseudo selectors, such
as
:focus
and :hover
. This could be taken
into advantage when tuning the component's behavior.
Some shadows can be used in order to give divider and arrows more contrast on bright background.
Beware: shadows negatively affect component's rendering performance.
Resize the window to see it in action.
Add slot="handle"
attribute to any tag inside
<img-comparison-slider>
and it will become a
handle. In this example we use <svg>
.
The combination of component's :hover
and
:focus
pseudo-classes, CSS, and arbitrary HTML for a
handle provides plenty of opportunities for the component styling.
Slots first
and second
should not
necessarily be of <img>
tag. They could be of
arbitrary HTML. Let's use it to our advantage and create
good-looking semantic-friendly labels.
The divider position could be specified in percents with the help of
value
property.


To-Do Lists Aren't Your Thing
You're in high demand. You are in charge of a lot of people including yourself and your to-do list proves it. You've tried to write it down in the past, but it just falls by the wayside. Your list keeps getting longer and before you know it you're missing the important taks.
Overload
The truth is, many people are overwhelmed by the sheer amount of items on their list. Unable to figure out where to start some put down the list and pick up their phone to reward themselves for writing all of that. Their list for today has 30 items including cleaning gutters, washing the car, recording a demo cd, and taking the kids to soccer. An admirable list indeed, but at the end of the day if the kids made it to soccer, had a balanced home cooked meal, and they spent 30 minutes practicing guitar, how will they perceive that day? If they look at a list of 30 items and say I completed less than 1% they're going to feel bad about what was actually a productive and fulfilling day.
Quality over Quantity
Instead of writing yourself a list of 10 items each day keep it to no more than 3. Make them tasks that you can really commit to doing. If you are only able to get to two items then you've been a success and you'll see those items start ticking off the big list. When you are choosing items for your today-list try prioritizing those items with deadlines approaching first. For more help narrowing down your list check out The ABC's of Priorities or meet with one of our coaches.
Too Much Too Do
If you're like most of us today your to-do list has about 50 items and that's just what you've written down. It can be an overwhelming task to look at the larger list and determine what you should start with. There's an easy trick to categorize your list and make it easier to select which of them should be on your 3 Items or Less List. If you learned the alphabet then you're golden. If not, rest easy, I'm only going to make you learn 4 of the 26 letters. Whew! We will categorize items into either A, B, C, or D. Bear with me I promise there's a treat.
Category A -High Priority
Category A is reserved for items where you will have a negative consequence if you do not complete them. For example, if you do not pay your bill on time you will get a late fee.  Forgetting to call your sister on her birthday may result in being shut out. These are your top priority items. Some will have dates, some not. I encourage you to give yourself a deadline. You are much more likely to accomplish the task.
Category B- A Little Help
Category B is all the items you can ask for help with or delegate to someone else. Have a housefull of kids? Try enlisting their help with small chores around the house. Does your Aunt come over every 3rd Sunday for tea and offer to help you with wedding planning? Let her. If getting to the grocery store seems like a pipe dream remember services like InstaCart deliver food and booze today. There are a variety of meal prep services that will help you with preparing bulk meals to store in the freezer that reheat for delicious, balanced meals.
Category C - See You Tomorrow
Category C is where the bulk of your list will likely lie. These are items that you can put off to another day. Tasks like cleaning the gutters, 2019's meal planning, and calling Grandma. Many of us are so busy trying to create the life we want that we don't stop to say is this really important? Is there something else I should be doing? If you are having trouble deciding if the item is a category A or C ask yourself if I don't do it today will there be a consequence? If yes, it's an A and if not then it fits here in Category C.
Category D - Fun, Fun, Fun
Yes, we did it. A whole category dedicated to only items that are fun. These are all of the things that you want to do. Let's face it, life is short. We have a very limited amount of time on this planet and while we're doing all the things we need to it is vital that we take care of ourselves with activities that we want to do. This can be water skiing, coloring, rock climbing, learning a language, going to the Target Dollar Spot, the possibilities are endless. Don't have any? You better start now because this model depends on it.
Only One Rule
This is it kids! I promised you a treat. Your list MUST have an equal number of A's to D's. Yes, you heard that right. Every time you accomplish something you had to do from Category A congratulate yourself with a treat from Category D. It is so vital to stay motivated and to celebrate every moment in life. Make sure your list has items of varied time, so you're not waiting until your trip to Colorado before you get anything else done. It's all about balance. Our most popular daily journal format is available in our downloads section if you're ready to take it to the next level.