WITNESS LIST Appropriations-S/C on Health & Human Services Committee March 7, 2000-9:00A Federal Funds On: Archer, William (Dept. of Health) Bishop, Joani (Office of the Comptroller) Campbell, Bill (Dept. of Mental Health and Mental Retardation) Delgado, Ben (Dept. of Health) Edge, Darrell (Office of the Comptroller) Elder, Mary (Council on Early Childhood Intervention) Gilbert, Don (Health & Human Services Commission) Green, Don (Health and Human Services Commission) Halfmann, Bobby (Dept. of Human Services) Thigpen, Drew (Dept. of Protective and Regulatory Services) Welch, Ken (Office of the Comptroller)


Learning objectives
- Use
group_by()
withsummarise()
to compute summary values for groups of observations- Use
count()
to count the numbers of observations within categories- Combine data from two tables based on a common identifier (
join
operations)- Customize plots created using ggplot2 by changing labels, scales and colours
In this session, we’ll look at some more useful functions provided by the dplyr package, the ‘workhorse’ in the tidyverse family for manipulating tabular data. Continuing from last week, we’ll see how we can summarise data for groups of observations within different categories. We’ll also show how dplyr allows us to combine data for the same observational unit, e.g. person or date, that comes from different sources and is read into R in different tables.
We’ll also look at how to customize the plots we create using ggplot2, in particular how we can add or change titles and labels, how we can adjust the way the axes are displayed and how we can use a colour scheme of our choosing.
dplyr and ggplot2 are core component packages within the tidyverse and both get loaded as part of the tidyverse.
library(tidyverse)
To demonstrate how these grouping and combining functions work and to illustrate customization of plots, we’ll again use the METABRIC data set.
metabric <- read_csv("data/metabric_clinical_and_expression_data.csv")
metabric
## # A tibble: 1,904 x 32
## Patient_ID Cohort Age_at_diagnosis Survival_time Survival_status
## <chr> <dbl> <dbl> <dbl> <chr>
## 1 MB-0000 1 75.6 140. LIVING
## 2 MB-0002 1 43.2 84.6 LIVING
## 3 MB-0005 1 48.9 164. DECEASED
## 4 MB-0006 1 47.7 165. LIVING
## 5 MB-0008 1 77.0 41.4 DECEASED
## 6 MB-0010 1 78.8 7.8 DECEASED
## 7 MB-0014 1 56.4 164. LIVING
## 8 MB-0022 1 89.1 99.5 DECEASED
## 9 MB-0028 1 86.4 36.6 DECEASED
## 10 MB-0035 1 84.2 36.3 DECEASED
## # … with 1,894 more rows, and 27 more variables: Vital_status <chr>,
## # Chemotherapy <chr>, Radiotherapy <chr>, Tumour_size <dbl>,
## # Tumour_stage <dbl>, Neoplasm_histologic_grade <dbl>,
## # Lymph_nodes_examined_positive <dbl>, Lymph_node_status <dbl>,
## # Cancer_type <chr>, ER_status <chr>, PR_status <chr>,
## # HER2_status <chr>, HER2_status_measured_by_SNP6 <chr>, PAM50 <chr>,
## # `3-gene_classifier` <chr>, Nottingham_prognostic_index <dbl>,
## # Cellularity <chr>, Integrative_cluster <chr>, Mutation_count <dbl>,
## # ESR1 <dbl>, ERBB2 <dbl>, PGR <dbl>, TP53 <dbl>, PIK3CA <dbl>,
## # GATA3 <dbl>, FOXA1 <dbl>, MLPH <dbl>
In the previous session we introduced the summarise()
function for computing a summary value for one or more variables from all rows in a table (data frame or tibble). For example, we computed the mean expression of ESR1, the estrogen receptor alpha gene, as follows.
summarise(metabric, mean(ESR1))
## # A tibble: 1 x 1
## `mean(ESR1)`
## <dbl>
## 1 9.61
While the summarise()
function is useful on its own, it becomes really powerful when applied to groups of observations within a dataset. For example, we might be more interested in the mean ESR1 expression calculated separately for ER positive and ER negative tumours. We could take each group in turn, filter the data frame to contain only the rows for a given ER status, then apply the summarise()
function to compute the mean expression, but that would be somewhat cumbersome. Even more so if we chose to do this for a categorical variable with more than two states, e.g. for each of the integrative clusters. Fortunately, the group_by()
function allows this to be done in one simple step.
metabric %>%
group_by(ER_status) %>%
summarise(mean(ESR1))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 2 x 2
## ER_status `mean(ESR1)`
## <chr> <dbl>
## 1 Negative 6.21
## 2 Positive 10.6
We get an additional column in our output for the categorical variable, ER_status
, and a row for each category.
Incidentally, we should expect this result of ER-positive tumours having a higher expression of ESR1 on average than ER-negative tumours. Simple summaries like this are a good way of checking that what we think we know actually holds true in the data we’re looking at. Note that the expression values are on a log2scale so ER-positive breast cancers express ESR1 at a level that is approximately 20 times greater, on average, than that of ER-negative tumours.
2 ** (10.6 - 6.21) # equivalent to (2 ** 10.6) / (2 ** 6.21)
## [1] 20.96629
Let’s have a look at how ESR1 expression varies between the integrative cluster subtypes defined by the METABRIC study.
metabric %>%
group_by(Integrative_cluster) %>%
summarise(ESR1 = mean(ESR1))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 11 x 2
## Integrative_cluster ESR1
## <chr> <dbl>
## 1 1 10.3
## 2 10 6.39
## 3 2 10.9
## 4 3 10.5
## 5 4ER- 6.55
## 6 4ER+ 9.78
## 7 5 7.78
## 8 6 10.9
## 9 7 10.9
## 10 8 11.1
## 11 9 9.96
The following schematic contains another example using a simplified subset of the METABRIC tumour samples to show what’s going on.
As we saw last week, we can summarize multiple observations, e.g. the mean expression for other genes of interest, with summarise_at()
, this time using the PAM50 classification to define the groups.
metabric %>%
group_by(PAM50) %>%
summarise_at(vars(ESR1, PGR, ERBB2), mean)
## # A tibble: 7 x 4
## PAM50 ESR1 PGR ERBB2
## <chr> <dbl> <dbl> <dbl>
## 1 Basal 6.42 5.46 10.2
## 2 claudin-low 7.47 5.60 9.85
## 3 Her2 7.81 5.62 12.6
## 4 LumA 10.8 6.75 10.7
## 5 LumB 11.0 6.39 10.6
## 6 NC 10.9 6.47 10.3
## 7 Normal 9.50 6.21 10.8
We can also refine our groups by using more than one categorical variable. Let’s subdivide the PAM50 groups by HER2 status to illustrate this.
metabric %>%
group_by(PAM50, HER2_status) %>%
summarise(ESR1_mean = mean(ESR1))
## `summarise()` regrouping output by 'PAM50' (override with `.groups` argument)
## # A tibble: 13 x 3
## # Groups: PAM50 [7]
## PAM50 HER2_status ESR1_mean
## <chr> <chr> <dbl>
## 1 Basal Negative 6.39
## 2 Basal Positive 6.71
## 3 claudin-low Negative 7.52
## 4 claudin-low Positive 6.80
## 5 Her2 Negative 8.82
## 6 Her2 Positive 7.04
## 7 LumA Negative 10.8
## 8 LumA Positive 10.1
## 9 LumB Negative 11.1
## 10 LumB Positive 10.2
## 11 NC Negative 10.9
## 12 Normal Negative 9.68
## 13 Normal Positive 7.77
It can be quite useful to know how many observations are within each group. We can use a special function, n()
, that counts the number of rows rather than computing a summary value from one of the columns.
metabric %>%
group_by(PAM50, HER2_status) %>%
summarise(N = n(), ESR1_mean = mean(ESR1))
## `summarise()` regrouping output by 'PAM50' (override with `.groups` argument)
## # A tibble: 13 x 4
## # Groups: PAM50 [7]
## PAM50 HER2_status N ESR1_mean
## <chr> <chr> <int> <dbl>
## 1 Basal Negative 179 6.39
## 2 Basal Positive 20 6.71
## 3 claudin-low Negative 184 7.52
## 4 claudin-low Positive 15 6.80
## 5 Her2 Negative 95 8.82
## 6 Her2 Positive 125 7.04
## 7 LumA Negative 658 10.8
## 8 LumA Positive 21 10.1
## 9 LumB Negative 419 11.1
## 10 LumB Positive 42 10.2
## 11 NC Negative 6 10.9
## 12 Normal Negative 127 9.68
## 13 Normal Positive 13 7.77
Counting observations within groups is such a common operation that dplyr provides a count()
function to do just that. So we could count the number of patient samples in each of the PAM50 classes as follows.
count(metabric, PAM50)
## # A tibble: 7 x 2
## PAM50 n
## <chr> <int>
## 1 Basal 199
## 2 claudin-low 199
## 3 Her2 220
## 4 LumA 679
## 5 LumB 461
## 6 NC 6
## 7 Normal 140
This is much like the table()
function we’ve used several times already to take a quick look at what values are contained in one of the columns in a data frame. They return different data structures however, with count()
always returning a data frame (or tibble) that can then be passed to subsequent steps in a ‘piped’ workflow.
If we wanted to subdivide our categories by HER2 status, we can add this as an additional categorical variable just as we did with the previous group_by()
examples.
count(metabric, PAM50, HER2_status)
## # A tibble: 13 x 3
## PAM50 HER2_status n
## <chr> <chr> <int>
## 1 Basal Negative 179
## 2 Basal Positive 20
## 3 claudin-low Negative 184
## 4 claudin-low Positive 15
## 5 Her2 Negative 95
## 6 Her2 Positive 125
## 7 LumA Negative 658
## 8 LumA Positive 21
## 9 LumB Negative 419
## 10 LumB Positive 42
## 11 NC Negative 6
## 12 Normal Negative 127
## 13 Normal Positive 13
The count column is named ‘n’ by default but you can change this.
count(metabric, PAM50, HER2_status, name = "Samples")
## # A tibble: 13 x 3
## PAM50 HER2_status Samples
## <chr> <chr> <int>
## 1 Basal Negative 179
## 2 Basal Positive 20
## 3 claudin-low Negative 184
## 4 claudin-low Positive 15
## 5 Her2 Negative 95
## 6 Her2 Positive 125
## 7 LumA Negative 658
## 8 LumA Positive 21
## 9 LumB Negative 419
## 10 LumB Positive 42
## 11 NC Negative 6
## 12 Normal Negative 127
## 13 Normal Positive 13
count()
is equivalent to grouping observations with group_by()
and calling summarize()
using the special n()
function to count the number of rows. So the above statement could have been written in a more long-winded way as follows.
metabric %>%
group_by(PAM50, HER2_status) %>%
summarize(Samples = n())
Summarizing with n()
is useful when showing the number of observations in a group alongside a summary value, as we did earlier looking at the mean ESR1 expression within specified groups; it allows you to see if you’re drawing conclusions from only a few data points.
Many summarization functions return NA
if any of the values are missing, i.e. the column contains NA
values. As an example, we’ll compute the average size of ER-negative and ER-positive tumours.
metabric %>%
group_by(ER_status) %>%
summarize(N = n(), `Average tumour size` = mean(Tumour_size))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 2 x 3
## ER_status N `Average tumour size`
## <chr> <int> <dbl>
## 1 Negative 445 NA
## 2 Positive 1459 NA
The mean()
function, along with many similar summarization functions, has an na.rm
argument that can be set to TRUE
to exclude those missing values from the calculation.
metabric %>%
group_by(ER_status) %>%
summarize(N = n(), `Average tumour size` = mean(Tumour_size, na.rm = TRUE))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 2 x 3
## ER_status N `Average tumour size`
## <chr> <int> <dbl>
## 1 Negative 445 28.5
## 2 Positive 1459 25.6
An alternative would be to filter out the observations with missing values but then the number of samples in each ER status group would take on a different meaning, which may or may not be what we actually want.
metabric %>%
filter(!is.na(Tumour_size)) %>%
group_by(ER_status) %>%
summarize(N = n(), `Average tumour size` = mean(Tumour_size))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 2 x 3
## ER_status N `Average tumour size`
## <chr> <int> <dbl>
## 1 Negative 438 28.5
## 2 Positive 1446 25.6
The sum()
and mean()
summarization functions are often used with logical values. It might seem surprising to compute a summary for a logical variable but but this turns out to be quite a useful thing to do, for counting the number of TRUE
values or obtaining the proportion of values that are TRUE
.
Following on from the previous example we could add a column to our summary of average tumour size for ER-positive and ER-negative patients that contains the number of missing values.
metabric %>%
group_by(ER_status) %>%
summarize(N = n(), Missing = sum(is.na(Tumour_size)), `Average tumour size` = mean(Tumour_size, na.rm = TRUE))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 2 x 4
## ER_status N Missing `Average tumour size`
## <chr> <int> <int> <dbl>
## 1 Negative 445 7 28.5
## 2 Positive 1459 13 25.6
Why does this work? Well, the is.na()
function takes a vector and sees which values are NA
, returning a logical vector of TRUE
where the value was NA
and FALSE
if not.
test_vector <- c(1, 3, 2, NA, 6, 5, NA, 10)
is.na(test_vector)
## [1] FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE
The sum()
function treats the logical vector as a set of 0
s and 1
s where FALSE
is 0
and TRUE
is 1
. In effect sum()
counts the number of TRUE
values.
sum(is.na(test_vector))
## [1] 2
Similarly, mean()
will compute the proportion of the values that are TRUE
.
mean(is.na(test_vector))
## [1] 0.25
So let’s calculate the number and proportion of samples that do not have a recorded tumour size in each of the ER-negative and ER-positive groups.
metabric %>%
group_by(ER_status) %>%
summarize(N = n(), `Missing tumour size` = sum(is.na(Tumour_size)), `Proportion missing` = mean(is.na(Tumour_size)))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 2 x 4
## ER_status N `Missing tumour size` `Proportion missing`
## <chr> <int> <int> <dbl>
## 1 Negative 445 7 0.0157
## 2 Positive 1459 13 0.00891
We can use sum()
and mean()
for any condition that returns a logical vector. We could, for example, find the number and proportion of patients that survived longer than 10 years (120 months) in each of the ER-negative and ER-positive groups.
metabric %>%
filter(Survival_status == "DECEASED") %>%
group_by(ER_status) %>%
summarise(N = n(), N_long_survival = sum(Survival_time > 120), Proportion_long_survival = mean(Survival_time > 120))
## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 2 x 4
## ER_status N N_long_survival Proportion_long_survival
## <chr> <int> <int> <dbl>
## 1 Negative 250 40 0.16
## 2 Positive 853 325 0.381
There are occassions when we want to count the number of distinct values in a variable or a combination of variables. In this week’s assignment, we introduce another set of data from the METABRIC study which contains details of the mutations detected by targeted sequencing of a panel of 173 genes. We’ll read this data into R now as this provides a good example of having multiple observations in different rows for a single observational unit, in this case several mutations detected in each tumour sample.
mutations <- read_csv("data/metabric_mutations.csv")
select(mutations, Patient_ID, Chromosome, Position = Start_Position, Ref = Reference_Allele, Alt = Tumor_Seq_Allele1, Type = Variant_Type, Gene)
## # A tibble: 17,272 x 7
## Patient_ID Chromosome Position Ref Alt Type Gene
## <chr> <chr> <dbl> <chr> <chr> <chr> <chr>
## 1 MTS-T0058 17 7579344 <NA> <NA> INS TP53
## 2 MTS-T0058 17 7579346 <NA> <NA> INS TP53
## 3 MTS-T0058 6 168299111 G G SNP MLLT4
## 4 MTS-T0058 22 29999995 G G SNP NF2
## 5 MTS-T0059 2 198288682 A A SNP SF3B1
## 6 MTS-T0059 6 86195125 T T SNP NT5E
## 7 MTS-T0059 7 55241717 C C SNP EGFR
## 8 MTS-T0059 10 6556986 C C SNP PRKCQ
## 9 MTS-T0059 11 62300529 T T SNP AHNAK
## 10 MTS-T0059 15 74912475 G G SNP CLK3
## # … with 17,262 more rows
We can see from just these few rows that each patient sample has multiple mutations and sometimes there are more than one mutation in the same gene within a sample, as can be seen in the first two rows at the top of the table above.
If we want to count the number of patients in which mutations were detected we could select the distinct set of patient identifiers using the distinct()
function.
mutations %>%
distinct(Patient_ID) %>%
nrow()
## [1] 2369
Similarly, we could select the distinct set of mutated genes for each patient as follows.
mutations %>%
distinct(Patient_ID, Gene)
## # A tibble: 15,656 x 2
## Patient_ID Gene
## <chr> <chr>
## 1 MTS-T0058 TP53
## 2 MTS-T0058 MLLT4
## 3 MTS-T0058 NF2
## 4 MTS-T0059 SF3B1
## 5 MTS-T0059 NT5E
## 6 MTS-T0059 EGFR
## 7 MTS-T0059 PRKCQ
## 8 MTS-T0059 AHNAK
## 9 MTS-T0059 CLK3
## 10 MTS-T0059 TP53
## # … with 15,646 more rows
This has reduced the number of rows as only distinct combinations of patient and gene are retained. This would be necessary if we wanted to count the number of patients that have mutations in each gene rather than the number of mutations for that gene regardless of the patient.
# number of mutations for each gene
count(mutations, Gene)
## # A tibble: 173 x 2
## Gene n
## <chr> <int>
## 1 ACVRL1 20
## 2 AFF2 70
## 3 AGMO 43
## 4 AGTR2 14
## 5 AHNAK 327
## 6 AHNAK2 859
## 7 AKAP9 182
## 8 AKT1 115
## 9 AKT2 23
## 10 ALK 98
## # … with 163 more rows
# number of tumour samples in which each gene is mutated
mutations %>%
distinct(Patient_ID, Gene) %>%
count(Gene)
## # A tibble: 173 x 2
## Gene n
## <chr> <int>
## 1 ACVRL1 20
## 2 AFF2 69
## 3 AGMO 43
## 4 AGTR2 14
## 5 AHNAK 272
## 6 AHNAK2 530
## 7 AKAP9 173
## 8 AKT1 115
## 9 AKT2 23
## 10 ALK 95
## # … with 163 more rows
The genes that differ in these two tables are those that have more than one mutation within a patient tumour sample.
In many real life situations, data are spread across multiple tables or spreadsheets. Usually this occurs because different types of information about a subject, e.g. a patient, are collected from different sources. It may be desirable for some analyses to combine data from two or more tables into a single data frame based on a common column, for example, an attribute that uniquely identifies the subject such as a patient identifier.
dplyr provides a set of join functions for combining two data frames based on matches within specified columns. These operations are very similar to carrying out join operations between tables in a relational database using SQL.
left_join
To illustrate join operations we’ll first consider the most common type, a “left join”. In the schematic below the two data frames share a common column, V1. We can combine the two data frames into a single data frame by matching rows in the first data frame with those in the second data frame that share the same value of variable V1.
left_join()
returns all rows from the first data frame regardless of whether there is a match in the second data frame. Rows with no match are included in the resulting data frame but have NA
values in the additional columns coming from the second data frame.
Here’s an example in which details about members of the Beatles and Rolling Stones are contained in two tables, using data frames conveniently provided by dplyr (we’ll look at a real example shortly).
The name column identifies each of the band members and is used for matching rows from the two tables.
band_members
## # A tibble: 3 x 2
## name band
## <chr> <chr>
## 1 Mick Stones
## 2 John Beatles
## 3 Paul Beatles
band_instruments
## # A tibble: 3 x 2
## name plays
## <chr> <chr>
## 1 John guitar
## 2 Paul bass
## 3 Keith guitar
left_join(band_members, band_instruments, by = "name")
## # A tibble: 3 x 3
## name band plays
## <chr> <chr> <chr>
## 1 Mick Stones <NA>
## 2 John Beatles guitar
## 3 Paul Beatles bass
We have joined the band members and instruments tables based on the common name
column. Because this is a left join, only observations for band members in the ‘left’ table (band_members
) are included with information brought in from the ‘right’ table (band_instruments
) where such exists. There is no entry in band_instruments
for Mick so an NA
value is inserted into the plays
column that gets added in the combined data frame. Keith is only included in the band_instruments
data frame so doesn’t make it into the final output as this is based on those band members in the ‘left’ table.
right_join()
is similar but returns all rows from the second data frame, i.e. the ‘right’ data frame, that have a match with rows in the first data frame.
right_join(band_members, band_instruments, by = "name")
## # A tibble: 3 x 3
## name band plays
## <chr> <chr> <chr>
## 1 John Beatles guitar
## 2 Paul Beatles bass
## 3 Keith <NA> guitar
right_join()
is used very infrequently compared with left_join()
.
inner_join
Another joining operation is the “inner join” in which only observations that are common to both data frames are included.
inner_join(band_members, band_instruments, by = "name")
## # A tibble: 2 x 3
## name band plays
## <chr> <chr> <chr>
## 1 John Beatles guitar
## 2 Paul Beatles bass
In this case when considering observations identified by name
, only John and Paul are contained in both the band_members
and band_instruments
tables, so only these make it into the combined table.
full_join
We’ve seen how missing rows from one table can be retained in the joined data frame using left_join
or right_join
but sometimes data for a given subject may be missing from either of the tables and we still want that subject to appear in the combined table. A full_join
will return all rows and all columns from the two tables and where there are no matching values, NA
values are used to fill in the missing values.
full_join(band_members, band_instruments, by = "name")
## # A tibble: 4 x 3
## name band plays
## <chr> <chr> <chr>
## 1 Mick Stones <NA>
## 2 John Beatles guitar
## 3 Paul Beatles bass
## 4 Keith <NA> guitar
Now, with full_join()
, we have rows for both Mick and Keith even though they are only in one or other of the tables being joined.
It isn’t uncommon for the columns used for joining two tables to have different names in each table. Of course we could rename one of the two columns, e.g. using the dplyr rename()
function, but the dplyr join functions allow you to match using differently-named columns as illustrated using another version of the band_instruments
data frame.
band_instruments2
## # A tibble: 3 x 2
## artist plays
## <chr> <chr>
## 1 John guitar
## 2 Paul bass
## 3 Keith guitar
left_join(band_members, band_instruments2, by = c("name" = "artist"))
## # A tibble: 3 x 3
## name band plays
## <chr> <chr> <chr>
## 1 Mick Stones <NA>
## 2 John Beatles guitar
## 3 Paul Beatles bass
The name for the column used for joining is the one given in the first table, i.e. the ‘left’ table, so name
rather than artist
in this case.
You may be wondering what happens if there are multiple rows in one of both of the two tables for the thing that is being joined, for example what would happen if our second table had two entries for instruments that Paul plays.
band_instruments3 <- tibble(
name = c("John", "Paul", "Paul", "Keith"),
plays = c("guitar", "bass", "guitar", "guitar")
)
band_instruments3
## # A tibble: 4 x 2
## name plays
## <chr> <chr>
## 1 John guitar
## 2 Paul bass
## 3 Paul guitar
## 4 Keith guitar
left_join(band_members, band_instruments3, by = "name")
## # A tibble: 4 x 3
## name band plays
## <chr> <chr> <chr>
## 1 Mick Stones <NA>
## 2 John Beatles guitar
## 3 Paul Beatles bass
## 4 Paul Beatles guitar
We get both entries from the second table added to the first table.
Let’s add an entry for Paul being in a second band and see what happens then when we combine the two tables, each with two entries for Paul.
band_members3 <- tibble(
name = c("Mick", "John", "Paul", "Paul"),
band = c("Stones", "Beatles", "Beatles", "Wings")
)
band_members3
## # A tibble: 4 x 2
## name band
## <chr> <chr>
## 1 Mick Stones
## 2 John Beatles
## 3 Paul Beatles
## 4 Paul Wings
left_join(band_members3, band_instruments3, by = "name")
## # A tibble: 6 x 3
## name band plays
## <chr> <chr> <chr>
## 1 Mick Stones <NA>
## 2 John Beatles guitar
## 3 Paul Beatles bass
## 4 Paul Beatles guitar
## 5 Paul Wings bass
## 6 Paul Wings guitar
The resulting table includes all combinations of band and instrument for Paul.
Sometimes the observations being combined are identified by multiple columns, for example, a forename and a surname. We can specify a vector of column names to be used in the join operation.
band_members4 <- tibble(
forename = c("Mick", "John", "Paul", "Mick", "John"),
surname = c("Jagger", "Lennon", "McCartney", "Avory", "Squire"),
band = c("Stones", "Beatles", "Beatles", "Kinks", "Roses")
)
band_instruments4 <- tibble(
forename = c("John", "Paul", "Keith", "Mick", "John"),
surname = c("Lennon", "McCartney", "Richards", "Avory", "Squire"),
plays = c("guitar", "bass", "guitar", "drums", "guitar")
)
full_join(band_members4, band_instruments4, by = c("forename", "surname"))
## # A tibble: 6 x 4
## forename surname band plays
## <chr> <chr> <chr> <chr>
## 1 Mick Jagger Stones <NA>
## 2 John Lennon Beatles guitar
## 3 Paul McCartney Beatles bass
## 4 Mick Avory Kinks drums
## 5 John Squire Roses guitar
## 6 Keith Richards <NA> guitar
Occasionally we may find that there are duplicated columns in the two tables we want to join, columns that aren’t those used for joining. These variables may even contain different data but happen to have the same name. In such cases dplyr joins add a suffix to each column in the combined table.
band_members5 <- tibble(
name = c("John", "Paul", "Ringo", "George", "Mick"),
birth_year = c(1940, 1942, 1940, 1943, 1943),
band = c("Beatles", "Beatles", "Beatles", "Beatles", "Stones")
)
band_instruments5 <- tibble(
name = c("John", "Paul", "Ringo", "George"),
birth_year = c(1940, 1942, 1940, 1943),
instrument = c("guitar", "bass", "drums", "guitar")
)
left_join(band_members5, band_instruments5, by = "name")
## # A tibble: 5 x 5
## name birth_year.x band birth_year.y instrument
## <chr> <dbl> <chr> <dbl> <chr>
## 1 John 1940 Beatles 1940 guitar
## 2 Paul 1942 Beatles 1942 bass
## 3 Ringo 1940 Beatles 1940 drums
## 4 George 1943 Beatles 1943 guitar
## 5 Mick 1943 Stones NA <NA>
It is advisable to rename or remove the duplicated columns that aren’t used for joining.
A variation on the join operations we’ve considered are semi_join()
and anti_join()
that filter the rows in one table based on matches or lack of matches to rows in another table.
semi_join()
returns all rows from the first table where there are matches in the other table.
semi_join(band_members, band_instruments, by = "name")
## # A tibble: 2 x 2
## name band
## <chr> <chr>
## 1 John Beatles
## 2 Paul Beatles
anti_join()
returns all rows where there is no match in the other table, i.e. those that are unique to the first table.
anti_join(band_members, band_instruments, by = "name")
## # A tibble: 1 x 2
## name band
## <chr> <chr>
## 1 Mick Stones
Let’s move on to a real example of joining data from two different tables that we used in putting together the combined METABRIC clinical and expression data set.
We first read the clinical data into R and then just select a small number of columns to make it easier to see what is going on when combining the data.
clinical_data <- read_csv("data/metabric_clinical_data.csv")
clinical_data <- select(clinical_data, Patient_ID, ER_status, PAM50)
clinical_data
## # A tibble: 2,509 x 3
## Patient_ID ER_status PAM50
## <chr> <chr> <chr>
## 1 MB-0000 Positive claudin-low
## 2 MB-0002 Positive LumA
## 3 MB-0005 Positive LumB
## 4 MB-0006 Positive LumB
## 5 MB-0008 Positive LumB
## 6 MB-0010 Positive LumB
## 7 MB-0014 Positive LumB
## 8 MB-0020 Negative Normal
## 9 MB-0022 Positive claudin-low
## 10 MB-0025 Positive <NA>
## # … with 2,499 more rows
We then read in the mRNA expression data that was downloaded separately from cBioPortal.
mrna_expression_data <- read_tsv("data/metabric_mrna_expression.txt")
mrna_expression_data
## # A tibble: 2,509 x 10
## STUDY_ID SAMPLE_ID ESR1 ERBB2 PGR TP53 PIK3CA GATA3 FOXA1 MLPH
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 brca_metabric MB-0000 8.93 9.33 5.68 6.34 5.70 6.93 7.95 9.73
## 2 brca_metabric MB-0002 10.0 9.73 7.51 6.19 5.76 11.3 11.8 12.5
## 3 brca_metabric MB-0005 10.0 9.73 7.38 6.40 6.75 9.29 11.7 10.3
## 4 brca_metabric MB-0006 10.4 10.3 6.82 6.87 7.22 8.67 11.9 10.5
## 5 brca_metabric MB-0008 11.3 9.96 7.33 6.34 5.82 9.72 11.6 12.2
## 6 brca_metabric MB-0010 11.2 9.74 5.95 5.42 6.12 9.79 12.1 11.4
## 7 brca_metabric MB-0014 10.8 9.28 7.72 5.99 7.48 8.37 11.5 10.8
## 8 brca_metabric MB-0020 NA NA NA NA NA NA NA NA
## 9 brca_metabric MB-0022 10.4 8.61 5.59 6.17 7.59 7.87 10.7 9.95
## 10 brca_metabric MB-0025 NA NA NA NA NA NA NA NA
## # … with 2,499 more rows
Now we have both sets of data loaded into R as data frames, we can combine them into a single data frame using an inner_join()
. Our resulting table will only contain entries for the patients for which expression data are available.
combined_data <- inner_join(clinical_data, mrna_expression_data, by = c("Patient_ID" = "SAMPLE_ID"))
combined_data
## # A tibble: 2,509 x 12
## Patient_ID ER_status PAM50 STUDY_ID ESR1 ERBB2 PGR TP53 PIK3CA GATA3
## <chr> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 MB-0000 Positive clau… brca_me… 8.93 9.33 5.68 6.34 5.70 6.93
## 2 MB-0002 Positive LumA brca_me… 10.0 9.73 7.51 6.19 5.76 11.3
## 3 MB-0005 Positive LumB brca_me… 10.0 9.73 7.38 6.40 6.75 9.29
## 4 MB-0006 Positive LumB brca_me… 10.4 10.3 6.82 6.87 7.22 8.67
## 5 MB-0008 Positive LumB brca_me… 11.3 9.96 7.33 6.34 5.82 9.72
## 6 MB-0010 Positive LumB brca_me… 11.2 9.74 5.95 5.42 6.12 9.79
## 7 MB-0014 Positive LumB brca_me… 10.8 9.28 7.72 5.99 7.48 8.37
## 8 MB-0020 Negative Norm… brca_me… NA NA NA NA NA NA
## 9 MB-0022 Positive clau… brca_me… 10.4 8.61 5.59 6.17 7.59 7.87
## 10 MB-0025 Positive <NA> brca_me… NA NA NA NA NA NA
## # … with 2,499 more rows, and 2 more variables: FOXA1 <dbl>, MLPH <dbl>
Having combined the data, we can carry out exploratory data analysis using elements from both data sets.
combined_data %>%
filter(!is.na(PAM50), !is.na(ESR1)) %>%
ggplot(mapping = aes(x = PAM50, y = ESR1, colour = PAM50)) +
geom_boxplot(show.legend = FALSE)
Finally, we’ll turn our attention back to visualization using ggplot2 and how we can customize our plots by adding or changing titles and labels, changing the scales used on the x and y axes, and choosing colours.
Adding titles and subtitles to a plot and changing the x- and y-axis labels is very straightforward using the labs()
function.
plot <- ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
labs(
title = "mRNA expression in the METABRIC breast cancer data set",
subtitle = "GATA3 and ESR1 mRNA expression measured using Illumina HT-12 bead arrays",
x = "log2 GATA3 expression",
y = "log2 ESR1 expression",
colour = "ER status"
)
plot
The labels are another component of the plot object that we’ve constructed, along with aesthetic mappings and layers (geoms). The plot object is a list and contains various elements including those mappings and layers and one element named labels
.
labs()
is a simple function for creating a list of labels you want to specify as name-value pairs as in the above example. You can name any aesthetic (in this case x and y) to override the default values (the column names) and you can add a title, subtitle and caption if you wish. In addition to changing the x- and y-axis labels, we also removed the underscore from the legend title by setting the label for the colour aesthetic.
Take a look at the x and y scales in the above plot. ggplot2 has chosen the x and y scales and where to put breaks and ticks.
Let’s have a look at the elements of the list object we created that specifies how the plot should be displayed.
names(plot)
## [1] "data" "layers" "scales" "mapping" "theme"
## [6] "coordinates" "facet" "plot_env" "labels"
One of the components of the plot is called scales
. ggplot2 automatically adds default scales behind the scenes equivalent to the following:
plot <- ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_x_continuous() +
scale_y_continuous() +
scale_colour_discrete()
Note that we have three aesthetics and ggplot2 adds a scale for each.
plot$mapping
## Aesthetic mapping:
## * `x` -> `GATA3`
## * `y` -> `ESR1`
## * `colour` -> `ER_status`
The x and y variables (GATA3
and ESR1
) are continuous so ggplot2 adds a continuous scale for each. ER_status
is a discrete variable in this case so ggplot2 adds a discrete scale for colour.
Generalizing, the scales that are required follow the naming scheme:
scale_<NAME_OF_AESTHETIC>_<NAME_OF_SCALE>
Look at the help page for scale_y_continuous
to see what we can change about the y-axis scale.
First we’ll change the breaks, i.e. where ggplot2 puts ticks and numeric labels, on the y axis.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_y_continuous(breaks = seq(5, 15, by = 2.5))
seq()
is a useful function for generating regular sequences of numbers. In this case we wanted numbers from 5 to 15 going up in steps of 2.5.
seq(5, 15, by = 2.5)
## [1] 5.0 7.5 10.0 12.5 15.0
We could do the same thing for the x axis using scale_x_continuous()
.
We can also adjust the extents of the x or y axis.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_y_continuous(breaks = seq(5, 15, by = 2.5), limits = c(4, 12))
## Warning: Removed 160 rows containing non-finite values (stat_smooth).
## Warning: Removed 160 rows containing missing values (geom_point).
Here, just for demonstration purposes, we set the upper limit to be less than the largest values of ESR1 expression and ggplot2 warned us that some rows have been removed from the plot.
We can change the minor breaks, e.g. to add more lines that act as guides. These are shown as thin white lines when using the default theme (we’ll take a look at alternative themes next week).
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_y_continuous(breaks = seq(5, 12.5, by = 2.5), minor_breaks = seq(5, 13.5, 0.5), limits = c(5, 13.5))
Or we can remove the minor breaks entirely.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_y_continuous(breaks = seq(6, 14, by = 2), minor_breaks = NULL, limits = c(5, 13.5))
Similarly we could remove all breaks entirely.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_y_continuous(breaks = NULL)
A more typical scenario would be to keep the breaks, because we want to display the ticks and their lables, but remove the grid lines. Somewhat confusingly the position of grid lines are controlled by a scale but preventing these from being displayed requires changing the theme. The theme controls the way in which non-data components are displayed – we’ll look at how these can be customized next week. For now, though, here’s an example of turning off the display of all grid lines for major and minor breaks for both axes.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_y_continuous(breaks = seq(4, 14, by = 2), limits = c(4, 14)) +
theme(panel.grid = element_blank())
By default, the scales are expanded by 5% of the range on either side. We can add or reduce the space as follows.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_x_continuous(expand = expand_scale(mult = 0.01)) +
scale_y_continuous(expand = expand_scale(mult = 0.25))
Here we only added 1% (0.01) of the range of GATA3 expression values on either side along the x axis but we added 25% (0.25) of the range of ESR1 expression on either side along the y axis.
We can move the axis to the other side of the plot –- not sure why you’d want to do this but with ggplot2 just about anything is possible.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_x_continuous(position = "top")
The colour asthetic is used with a categorical variable, ER_status
, in the scatter plots we’ve been customizing. The default colour scale used by ggplot2 for categorical variables is scale_colour_discrete
. We can manually set the colours we wish to use using scale_colour_manual
instead.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_colour_manual(values = c("dodgerblue2", "firebrick2"))
Setting colours manually is ok when we only have two or three categories but when we have a larger number it would be handy to be able to choose from a selection of carefully-constructed colour palettes. Helpfully, ggplot2 provides access to the ColorBrewer palettes through the functions scale_colour_brewer()
and scale_fill_brewer()
.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = `3-gene_classifier`)) +
geom_point(size = 0.6, alpha = 0.5, na.rm = TRUE) +
scale_colour_brewer(palette = "Set1")
Look at the help page for scale_colour_brewer
to see what other colour palettes are available and visit the ColorBrewer website to see what these look like.
Interestingly, you can set other attributes other than just the colours at the same time.
ggplot(data = metabric, mapping = aes(x = GATA3, y = ESR1, colour = ER_status)) +
geom_point(size = 0.6, alpha = 0.5) +
geom_smooth(method = "lm") +
scale_colour_manual(values = c("dodgerblue2", "firebrick2"), labels = c("ER-negative", "ER-positive")) +
labs(colour = NULL) # remove legend title for colour now that the labels are self-explanatory
We have applied our own set of mappings from levels in the data to aesthetic values.
For continuous variables we may wish to be able to change the colours used in the colour gradient. To demonstrate this we’ll correct the Nottingham prognostic index (NPI) values and use this to colour points in the scatter plot of ESR1 vs GATA3 expression on a continuous scale.
# Nottingham_prognostic_index is incorrectly calculated in the data downloaded from cBioPortal
metabric <- mutate(metabric, Nottingham_prognostic_index = 0.02 * Tumour_size + Lymph_node_status + Neoplasm_histologic_grade)
#
metabric %>%
filter(!is.na(Nottingham_prognostic_index)) %>%
ggplot(mapping = aes(x = GATA3, y = ESR1, colour = Nottingham_prognostic_index)) +
geom_point(size = 0.5)
Higher NPI scores correspond to worse prognosis and lower chance of 5 year survival. We’ll emphasize those points on the scatter plot by adjusting our colour scale.
metabric %>%
filter(!is.na(Nottingham_prognostic_index)) %>%
ggplot(mapping = aes(x = GATA3, y = ESR1, colour = Nottingham_prognostic_index)) +
geom_point(size = 0.75) +
scale_colour_gradient(low = "white", high = "firebrick2")
In some cases it might make sense to specify two colour gradients either side of a mid-point.
metabric %>%
filter(!is.na(Nottingham_prognostic_index)) %>%
ggplot(mapping = aes(x = GATA3, y = ESR1, colour = Nottingham_prognostic_index)) +
geom_point(size = 0.75) +
scale_colour_gradient2(low = "dodgerblue1", mid = "grey90", high = "firebrick1", midpoint = 4.5)
As before we can override the default labels and other aspects of the colour scale within the scale function.
metabric %>%
filter(!is.na(Nottingham_prognostic_index)) %>%
ggplot(mapping = aes(x = GATA3, y = ESR1, colour = Nottingham_prognostic_index)) +
geom_point(size = 0.5) +
scale_colour_gradient(
low = "lightblue", high = "darkblue",
name = "NPI",
breaks = 2:6,
limits = c(1.5, 6.5)
)
In this session we have covered the following:
Assignment: assignment5.Rmd
Solutions: assignment5_solutions.Rmd and assignment5_solutions.html


Converting fisheye image to equirectangular image¶
The python version of the code is inspired by this repository.[2] Significant changes have been made to improve the real-time performance of the code.
Contributions
Elimination of nested for a loop by implementing NumPy based vectorized calculations.
Nested for loop calculations in the reference repository results in high computation time.
Conversion map is saved as an object attribute so it can be reused for consecutive frames without any need to recompute the mapping in cases where the mapping parameters are not changing.
Average computation time for conversion for both methods has been tabulated for various output dimensions to explain the significant improvement in real-time performance of the code. Computation time for conversion by reusing the map stored as an object attribute is also given in the table.
Output image resolution |
Nested for loop based calculations [2] |
Vectorized calculations (ours) |
Reusing the map (ours) |
|---|---|---|---|
1600x3200 |
52.46 s |
3.26 s |
0.19281 s |
800x1600 |
12.73 s |
0.76 s |
0.03779 s |
400x800 |
3.48 s |
0.197 s |
0.00457 s |
200x400 |
0.83 s |
0.069 s |
0.00249 s |
Two different modes are provided for converting fisheye image to an equirectangular image. The second mode is especially useful in cases where the camera is placed parallel to the ground, facing the sky.
Mode 1 |
Mode 2 |
|---|---|
Conversion flowchart
The following figure explains the inter-conversion between a fisheye image and an equirectangular image (source.)[1]
Python Example¶
-
fisheyeImgConv.fisheye2equirect(srcFrame,outShape,aperture=0,delx=0,dely=0,radius=0,edit_mode=False)`
- Parameters
srcFrame – Input fisheye image
outShape – Desired shape of the output image in form of a tuple [Image_Height,Image_Width]
aperture – Aperture of the fisheye camera. Can be determined using the GUI provided.
delx – Shift in center x coordinate
dely – Shift in center y coordinate
radius – Radius of the circle containing the fisheye image
edit_mode – Boolean to allow editing the parameters. Useful in the GUI provided.
#!/usr/bin/env/python
import cv2
import numpy as np
import math
import sys
from omnicv import fisheyeImgConv
Img_path = sys.argv[1]
# Path to the file containing fisheye camera parameters (obtained using GUI)
param_file_path = "../fisheyeParams.txt"
# Reading the fisheye image
frame = cv2.imread(Img_path)
outShape = [400,800]
inShape = frame.shape[:2]
# In case of fisheye lens placed vertically
mapper = fisheyeImgConv(param_file_path)
frame2 = mapper.fisheye2equirect(frame,outShape)
# Use the below line if there are multiple images and the mapping is not changing in case of a video
frame2 = mapper.applyMap(0,frame)
cv2.imshow("image",frame2)
cv2.waitKey(0)
C++ Example¶
-
fisheyeImgConv::fisheye2equirect(const cv::Mat &srcFrame, cv::Mat &dstFrame, cv::Size outShape,
-
int aperture, int delx, int dely, int radius, bool edit_mode)`
- Parameters
srcFrame – Input fisheye image
outShape – Desired shape of the output image in form of a tuple [Image_Height,Image_Width]
aperture – Aperture of the fisheye camera. Can be determined using the GUI provided.
delx – Shift in center x coordinate
dely – Shift in center y coordinate
radius – Radius of the circle containing the fisheye image
edit_mode – Boolean to allow editing the parameters. Useful in the GUI provided.
#include<iostream>
#include<opencv2/opencv.hpp>
#include"../omnicv/utils.hpp"
#include <opencv2/core/core.hpp>
// Creating the display window
int H = 500;
int W = 500;
std::string WINDOW_NAME{"viewer"};
int main()
{
cv::namedWindow(WINDOW_NAME,CV_WINDOW_NORMAL);
cv::Mat frame;
cv::Mat outFrame;
frame = cv::imread("../data/fisheye_horizontal2.jpg");
cv::imshow(WINDOW_NAME,frame);
cv::waitKey(0);
fisheyeImgConv mapper1("../fisheyeParams.txt");
mapper1.fisheye2equirect(frame, outFrame, cv::Size (800,400));
cv::imshow("output",outFrame);
cv::waitKey(0);
mapper1.applyMap(0,frame,outFrame);
cv::imshow("output 2",outFrame);
cv::waitKey(0);
return 0;
}
For more examples of this method read this code.
Click here to refer the documentation for the GUI to determine the fisheye parameters
References¶
[1] Converting dual fisheye images into a spherical (equirectangular) projection - http://paulbourke.net/dome/dualfish2sphere/
[2] Github repository py-fisheye-dewarp - https://github.com/BlueHorn07/py-fisheye-dewarp


| DOSE: |
repeated |
smoked |
Cannabis |
(plant material) |
| |
4 shots |
oral |
Alcohol - Hard |
(liquid) |
| |
repeated |
insufflated |
Cocaine |
(powder / crystals) |
| |
18 mg |
oral |
2C-E |
(powder / crystals) |
| |
2 hits |
smoked |
Crack |
(freebase) |
I've never heard of anyone being crazy, or stupid enough to attempt to combine tripping with a nice coke experience. The night I took all these chemicals began simply enough. Me and some friends were just drinking little and smoking a little bud, waiting for another one of my friends to get off work, so we could get some coke. I already had gotten together a bunch of 2C-E doses, and some of us were thinking about tripping.
After we finally got the coke and all took a sample line, we returned back to my friends house. It is then I decided to take the 2C-E. By this time I had taken about 4 shots of some pretty garnly liquor. After taking a couple more lines and chilling for around an hour, I began to feel sick. This was the 2C-E coming up. I finally ended up feeling sick for about 25 minutes, and throwing up once. 30 minutes after that I was into my trip.
I was allready blown by the time the 2C-E began to take full effect. The cocaine at times almost seemed to intensify all my thoughts and feelings. Yet I felt incredibly mellow during most of the trip. I felt like I could just sit back and melt into the reality of existence. When I was probably halfway into my trip I decided it was time to start sessioning the coke again. Fortunealtey my one friend, who was tripping was just about to cook up some sheets.
I can safely say smoking crack on 2C-E is a very strage experience. As soon as I exhaled my first hit, the pattrens I was seeing in the bathroom started to slow down a bit and mellow out. The colors I was seeing however allready intensifed by the 2C-E jumped out even more. I felt amazing. The coke high combined with the trip gave me a feeling of utter peace and control. It was almost as if I was the master of the universe. I took another hit and then sat on my friends couch for awhile.
Again, my thoughts raced in my head while the world seemed almost slow around my. I don't remember thinking about much, all I can remember is that my thoughts were intense. The weirdest part is, about 30 minutes later I began to come down from my crack hits the patterns I was seeing began to become more intense again. After the crack, it was a much more intense clear reality, at least that is the best way I can describe the experience.
This was my 2nd time tripping on 2C-E, so I am still very unexperienced with how it feels. Overall, the coke mostly seemed to mask the trip and the effects of the 2C-E, although in some ways it seemed to intensify it, mostly the rate at which I thought and my euphoria/body high. I eventually ended this night smoking bowls, and was surprisingly able to fall asleep about 11 hours after dosing on 2C-E. I felt brain dead for a few days after this experience, and this is definitely a combo that I will not be repeating.
Experience Reports are the writings and opinions of the individual authors who submit them.
Some of the activities described are dangerous and/or illegal and none are recommended by Erowid.


|
Contiki 2.5
|
This tutorial explains how to run Contiki with IPv6 and 6lowpan support on Atmel RAVEN hardware. More...
This tutorial explains how to run Contiki with IPv6 and 6lowpan support on Atmel RAVEN hardware.
Introduction
Hardware requirements
Software requirements
Demo Overview
Compiling, installing, configuring
Running the basic demo
Advanced use
Known issues
Annex
This tutorial explains how to run Contiki with IPv6 and 6lowpan support on Atmel RAVEN evaluation kit (ATAVRRZRAVEN) hardware. We present basic example system architecture and application scenarios, as well as instructions to run more advanced demos.
To run the demo, you will need at least
To install the demo you need:
Instructions to install these tools are in the section Software setup details.
To run the demo, you need:
ipv6 install
The network comprises:
In the basic demo, you can:
The binaries needed are:
To compile each of them, type in Cygwin:
cd c:/contiki/examples/webserver-ipv6-raven make
cd c:/contiki/platform/avr-ravenlcd make
cd c:/contiki/examples/ravenusbstick make
To power the RAVEN, put the EXT/BAT jumper in BAT position. This will enable power on batteries. If you want to power the RAVEN externally, check instructions in Using an external board for power and Debug.
The RZ USB Stick needs to be plugged in the PC you will run the demo on. If you plan to run the demo on a Windows PC, you will need to install drivers once contiki is loaded on the stick. Until then, you can exit any driver installation popup.
What you need to do
Hardware connections
To load the binary on each AVR in Windows
For webserver6.elf, set the processor to ATmega1284P For ravenlcd_3290.elf, set the processor to ATmega3290P For ravenusbstick.elf, set the processor to AT90USB1287
Once the RZ USB Stick is programmed, unplug it from the PC. Note this programmed the fuses, EEPROM, and FLASH all at once.
On Linux
Plug the RZ USB Stick in the PC. It should appear as a USB network interface (e.g. usb0).
usb0 should automatically get an IPv6 link local address, i.e. fe80::0012:13ff:fe14:1516/64. Check this is the case by typing
ifconfig
and checking the addresses of interface usb0
If it does not, add it manually:
ip -6 address add fe80::0012:13ff:fe14:1516/64 scope link dev usb0
Configure the IP addresses on usb0
ip -6 address add aaaa::1/64 dev usb0
Install the radvd deamon and configure it so the usb0 interface advertises the aaaa::/64 prefix as on link and usable for address autoconfiguration.
Radvd configuration (usually in /etc/radvd.conf)
interface usb0 { AdvSendAdvert on; AdvLinkMTU 1280; AdvCurHopLimit 128; AdvReachableTime 360000; MinRtrAdvInterval 100; MaxRtrAdvInterval 150; AdvDefaultLifetime 200; prefix AAAA::/64 { AdvOnLink on; AdvAutonomous on; AdvPreferredLifetime<PHONE_NUMBER>; AdvValidLifetime<PHONE_NUMBER>; }; };
Restart the radvd daemon. Example command:
/etc/init.d/radvd restart
If you get a message that radvd won't start as forwarding isn't enabled, you can run this as root:
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
On Windows
Plug the RZ USB Stick in the PC. A "new hardware installation" window should pop up. If it does not, go to "Control Panel"-> "Add Hardware". Choose "Install the driver manually", then select the search path C:\contiki\cpu\avr\dev\usb\INF. Finish the installation.
You now need to get the "interface index" of the USB Stick interface (noted [interface index] in the following) and the Ethernet interface (noted [ethernet interface index] in the following).
In a DOS or Cygwin shell, type
ipv6 if
As an example, the output might look something like this:
... Interface 7: Ethernet ... link-layer address: 02-12-13-14-15-16 preferred link-local fe80::12:13ff:fe14:1516, life infinite ... Interface 4: Ethernet: Local Area Connection ... link-layer address: 00-1e-37-16-5d-83 preferred link-local fe80::21e:37ff:fe16:5d83, life infinite ... ...
Note the link-layer address associated with interface 7 is the USB Stick. Hence [interface index] is 7, [ethernet interface index] is 4 and [ethernet link-local address] is fe80::21e:37ff:fe16:5d83.
Then you need to
To do so, type:
ipv6 ifc [interface index] advertises forwards ipv6 adu [interface index]/aaaa::1 ipv6 rtu ::/0 [ethernet interface index]/[ethernet link-local address] publish ipv6 rtu aaaa::/64 [interface index] publish
Reboot the RAVEN board. The PC sends router advertisements and the RAVEN Board configures an IPv6 global address based on them. The PC global addresses were set above. Communication is ready.
On Windows (Cygwin shell) or Linux, type
ping6 -n 5 aaaa::11:22ff:fe33:4455
or
ping6 -s aaaa::1 aaaa::11:22ff:fe33:4455
The router is sending 5 echo requests to the RAVEN board. The RAVEN board answers with 5 echo replies.
To send a ping from the RAVEN to the router you need to use the RAVEN's joystick and LCD screen. Initially, the LCD screen should print CONTIKI - 6LOWPAN in a loop. You can navigate the LCD menu by using the small joystick just below its lower right corner. To 'ping' push the joystick twice to the right. The RAVEN board sends 4 echo requests to the router, which answers by 4 echo replies.
For more information about the LCD menu, please see RZRAVEN LCD 3290p.
In a Web browser, point to http://[aaaa::0011:22ff:fe33:4455]. Then click on 'Sensor Readings'. If no temperature is displayed it means that you need to start the temperature update process on the RAVEN. To do so you must use the RAVEN's LCD menu and joystick. Starting from the CONTIKI - 6LOWPAN display navigate to TEMP and then to SEND. You can pick either ONCE or AUTO, but in any case you always need to reload the webpage to see the latest temperature reading.
For more information about the LCD menu, please see RZRAVEN LCD 3290p.
To power the RAVEN boards externally and enable debug output on RS232, you can use the stk500 board together with the raven.
Power:
Debug Connection
The following image shows this connection, with the red and black being VCC and GND. The green wire is debug out:
There is no widely available 802.15.4 and 6lowpan stack for PCs. As a temporary solution and to be able to connect IPv6 hosts such as RAVEN boards to IP networks, we implemented a "bridge" function on the RZ USB Stick. The RZ USB stick bridges 802.15.4 packets to Ethernet (The Ethernet interface is emulated on the USB port).
As Ethernet frames and addresses are very different from 802.15.4 ones, a few adjustements are needed on addresses and some neighbor discovery packets. As a consequence, 802.15.4 MAC addresses configured on both the RAVEN boards and the RZ USB stick must have the format:
x2:xx:xx:ff:fe:xx:xx:xx
where x can take any hexadecimal value. Read the section below to change the MAC address on one device.
You can change the MAC address of a RAVEN board or the RZ USB Stick by setting the 8 first bytes of the EEPROM, following the convention above. You can do this three ways.
The first is to set EEPROM bytes directly in an AVR Studio project, in Debug mode
The second is to reprogram the whole EEPROM individually from the Flash and Fuses.
The third is to modify the default value in the code:
/* Put default MAC address in EEPROM */
uint8_t mac_address[8] EEMEM = {0x02, 0x11, 0x22, 0xff, 0xfe, 0x33, 0x44, 0x55};
In case you need to reset the fuses on one AVR, do the following:
0xFF, 0x99, 0xE2 for the ATmega1284P on the RAVEN board 0xFF, 0x99, 0xE2 for the ATmega3290P on the RAVEN board 0xFB, 0x99, 0xDE for the AT90USB1287 on the USB Stick
To view packets being sent over the air, you can use Atmel AVR Wireless Services in Sniffer Mode, with the RZ USB Stick. You need the software preinstalled on the RZ USB Stick to do this. Packets are sent on channel 24. Links to detailed information about AVR Wireless Services is provided with the RZ USB Stick.
See the RZRAVEN USB Stick (Jackdaw) documentation for more details about using Wireshark.
One can use avrdude to load the binaries in Linux.
IETF Internet Draft draft-hui-6lowpan-hc-01 defines a stateful header compression mechanism (called HC01) which will soon deprecate the stateless header compression mechanism (called HC1) defined in RFC4944. HC01 is much more powerfull and flexible, in particular it allows compression of some multicast addresses and of all global unicast addresses.
Contiki is compiled by default with HC1 support. To use HC01 instead, edit platform/xxx/contiki-conf.h (replace xxx with avr-raven, then avr-ravenusb.) and replace the line
#define SICSLOWPAN_CONF_COMPRESSION SICSLOWPAN_CONF_COMPRESSION_HC1
with
#define SICSLOWPAN_CONF_COMPRESSION SICSLOWPAN_CONF_COMPRESSION_HC01
Recompile and load Contiki for both the RAVEN ATmega1284P and RZ USB Stick.
If you capture packets being sent over the air (on the 802.15.4 network), you will see that much more packets are compressed than when HC1 is used. Overall, packets sent are much smaller.
You can integrate the RAVEN boards and RZ USB stick to a more complete IPv6 network by connecting the PC which you plug the RZ USB Stick in to any IPv6 network with correct routing configured.
This way, you will be able to reach the RAVEN boards (to read sensor data for example) from anywhere within this IPv6 network, or even any IPv4 network if v4 to v6 translation mechanisms are used between both networks.
You can also have several RAVEN boards in your setup. If you do so, be sure to configure different MAC addresses on each board.
RZ USB Stick Link local address not created on Linux
When plugging the RZ USB Stick in a Linux PC, it should automatically configure a link local address (fe80::0012:13ff:fe14:1516/64 with default MAC address). On some Linux distributions, it seems to fail. To check this, in a terminal, type
ifconfig
If the interface usb0 does not have an IPv6 address starting with fe80::, add it manually by typing:
ip -6 address add fe80::0012:13ff:fe14:1516/64 scope link dev usb0
make version issues
You need to use the "make" executable from WinAVR. There are compilation issues with GNU make coming with Cygwin.
RAVEN evaluation and starter kits
RAVEN AVRs and Wireless transceivers
Additional hardware
Buying the hardware (part number ATAVRRZRAVEN and ATJTAGICE2)
Contiki
Download Contiki code from http://www.sics.se/contiki and extract the source code. We assume the directory you extract to is c:/contiki.
Cygwin
AVR Studio
Download and install AVR Studio from http://www.atmel.com/dyn/products/tools_card.asp?tool_id=2725
WinAVR
WinAVR which contains a number of AVR tools such as the avr-gcc compiler.
Download and install WinAVR latest version from http://winavr.sourceforge.net/
JTAGICE mkII Drivers
Plug the JTAGICE mkII in a USB port of a windows PC. Follow the indications to install the Windows drivers automatically.


Levy Stable models of Stochastic Volatility¶
This tutorial demonstrates inference using the Levy Stable distribution through a motivating example of a non-Gaussian stochastic volatilty model.
Inference with stable distribution is tricky because the density Stable.log_prob()
is not defined. In this tutorial we demonstrate two approaches to inference: (i) using the poutine.reparam effect to transform models in to a tractable form, and (ii) using the likelihood-free loss EnergyDistance with SVI.
Summary¶
Stable.log_prob() is undefined.
Stable inference requires either reparameterization or a likelihood-free loss.
Reparameterization:
The poutine.reparam() handler can transform models using various strategies.
The StableReparam strategy can be used for Stable distributions in SVI or HMC.
The LatentStableReparam strategy is a little cheaper, but cannot be used for likelihoods.
The DiscreteCosineReparam strategy improves geometry in batched latent time series models.
Likelihood-free loss with SVI:
The EnergyDistance loss allows stable distributions in the guide and in model likelihoods.
Table of contents¶
Fitting a single distribution to log returns using
EnergyDistance
Modeling stochastic volatility using
poutine.reparam
Daily S&P 500 data¶
The following daily closing prices for the S&P 500 were loaded from Yahoo finance.
[1]:
import math
import os
import torch
import pyro
import pyro.distributions as dist
from matplotlib import pyplot
from torch.distributions import constraints
from pyro import poutine
from pyro.contrib.examples.finance import load_snp500
from pyro.infer import EnergyDistance, Predictive, SVI, Trace_ELBO
from pyro.infer.autoguide import AutoDiagonalNormal
from pyro.infer.reparam import DiscreteCosineReparam, StableReparam
from pyro.optim import ClippedAdam
from pyro.ops.tensor_utils import convolve
%matplotlib inline
assert pyro.__version__.startswith('1.8.6')
smoke_test = ('CI' in os.environ)
[2]:
df = load_snp500()
dates = df.Date.to_numpy()
x = torch.tensor(df["Close"]).float()
x.shape
[2]:
torch.Size([23116])
[3]:
pyplot.figure(figsize=(9, 3))
pyplot.plot(x)
pyplot.yscale('log')
pyplot.ylabel("index")
pyplot.xlabel("trading day")
pyplot.title("S&P 500 from {} to {}".format(dates[0], dates[-1]));
Of interest are the log returns, i.e. the log ratio of price on two subsequent days.
[4]:
pyplot.figure(figsize=(9, 3))
r = (x[1:] / x[:-1]).log()
pyplot.plot(r, "k", lw=0.1)
pyplot.title("daily log returns")
pyplot.xlabel("trading day");
[5]:
pyplot.figure(figsize=(9, 3))
pyplot.hist(r.numpy(), bins=200)
pyplot.yscale('log')
pyplot.ylabel("count")
pyplot.xlabel("daily log returns")
pyplot.title("Empirical distribution. mean={:0.3g}, std={:0.3g}".format(r.mean(), r.std()));
Fitting a single distribution to log returns¶
Log returns appear to be heavy-tailed. First let’s fit a single distribution to the returns. To fit the distribution, we’ll use a likelihood free statistical inference algorithm EnergyDistance, which matches fractional moments of observed data and can handle data with heavy tails.
[6]:
def model():
stability = pyro.param("stability", torch.tensor(1.9),
constraint=constraints.interval(0, 2))
skew = 0.
scale = pyro.param("scale", torch.tensor(0.1), constraint=constraints.positive)
loc = pyro.param("loc", torch.tensor(0.))
with pyro.plate("data", len(r)):
return pyro.sample("r", dist.Stable(stability, skew, scale, loc), obs=r)
[7]:
%%time
pyro.clear_param_store()
pyro.set_rng_seed(1234567890)
num_steps = 1 if smoke_test else 201
optim = ClippedAdam({"lr": 0.1, "lrd": 0.1 ** (1 / num_steps)})
svi = SVI(model, lambda: None, optim, EnergyDistance())
losses = []
for step in range(num_steps):
loss = svi.step()
losses.append(loss)
if step % 20 == 0:
print("step {} loss = {}".format(step, loss))
print("-" * 20)
pyplot.figure(figsize=(9, 3))
pyplot.plot(losses)
pyplot.yscale("log")
pyplot.ylabel("loss")
pyplot.xlabel("SVI step")
for name, value in sorted(pyro.get_param_store().items()):
if value.numel() == 1:
print("{} = {:0.4g}".format(name, value.squeeze().item()))
step 0 loss = 8.961664199829102
step 20 loss = 4.8506011962890625
step 40 loss = 1.5543489456176758
step 60 loss = 1.7787070274353027
step 80 loss = 1.4140945672988892
step 100 loss = 1.3671720027923584
step 120 loss = 1.287503719329834
step 140 loss = 1.2791334390640259
step 160 loss = 1.2810490131378174
step 180 loss = 1.2784368991851807
step 200 loss = 1.2823134660720825
--------------------
loc = 0.0003696
scale = 0.00872
stability = 1.977
CPU times: user 15.6 s, sys: 521 ms, total: 16.1 s
Wall time: 2.38 s
[8]:
samples = poutine.uncondition(model)().detach()
pyplot.figure(figsize=(9, 3))
pyplot.hist(samples.numpy(), bins=200)
pyplot.yscale("log")
pyplot.xlabel("daily log returns")
pyplot.ylabel("count")
pyplot.title("Posterior predictive distribution");
This is a poor fit, but that was to be expected since we are mixing all time steps together: we would expect this to be a scale-mixture of distributions (Normal, or Stable), but are modeling it as a single distribution (Stable in this case).
Modeling stochastic volatility¶
We’ll next fit a stochastic volatility model. Let’s begin with a constant volatility model where log price \(p\) follows Brownian motion
where \(w_t\) is a sequence of standard white noise. We can rewrite this model in terms of the log returns \(r_t=\log(p_t\,/\,p_{t-1})\):
Now to account for volatility clustering we can generalize to a stochastic volatility model where volatility \(h\) depends on time \(t\). Among the simplest such models is one where \(h_t\) follows geometric Brownian motion
where again \(v_t\) is a sequence of standard white noise. The entire model thus consists of a geometric Brownian motion \(h_t\) that determines the diffusion rate of another geometric Brownian motion \(p_t\):
Usually \(v_1\) and \(w_t\) are both Gaussian. We will generalize to a Stable distribution for \(w_t\), learning three parameters (stability, skew, and location), but still scaling by \(\sqrt h_t\).
Our Pyro model will sample the increments \(v_t\) and record the computation of \(\log h_t\) via pyro.deterministic. Note that there are many ways of implementing this model in Pyro, and geometry can vary depending on implementation. The following version seems to have good geometry, when combined with reparameterizers.
[9]:
def model(data):
# Note we avoid plates because we'll later reparameterize along the time axis using
# DiscreteCosineReparam, breaking independence. This requires .unsqueeze()ing scalars.
h_0 = pyro.sample("h_0", dist.Normal(0, 1)).unsqueeze(-1)
sigma = pyro.sample("sigma", dist.LogNormal(0, 1)).unsqueeze(-1)
v = pyro.sample("v", dist.Normal(0, 1).expand(data.shape).to_event(1))
log_h = pyro.deterministic("log_h", h_0 + sigma * v.cumsum(dim=-1))
sqrt_h = log_h.mul(0.5).exp().clamp(min=1e-8, max=1e8)
# Observed log returns, assumed to be a Stable distribution scaled by sqrt(h).
r_loc = pyro.sample("r_loc", dist.Normal(0, 1e-2)).unsqueeze(-1)
r_skew = pyro.sample("r_skew", dist.Uniform(-1, 1)).unsqueeze(-1)
r_stability = pyro.sample("r_stability", dist.Uniform(0, 2)).unsqueeze(-1)
pyro.sample("r", dist.Stable(r_stability, r_skew, sqrt_h, r_loc * sqrt_h).to_event(1),
obs=data)
We use two reparameterizers: StableReparam to handle the Stable
likelihood (since Stable.log_prob()
is undefined), and DiscreteCosineReparam to improve geometry of the latent Gaussian process for v
. We’ll then use reparam_model
for both inference and prediction.
[10]:
reparam_model = poutine.reparam(model, {"v": DiscreteCosineReparam(),
"r": StableReparam()})
[11]:
%%time
pyro.clear_param_store()
pyro.set_rng_seed(1234567890)
num_steps = 1 if smoke_test else 1001
optim = ClippedAdam({"lr": 0.05, "betas": (0.9, 0.99), "lrd": 0.1 ** (1 / num_steps)})
guide = AutoDiagonalNormal(reparam_model)
svi = SVI(reparam_model, guide, optim, Trace_ELBO())
losses = []
for step in range(num_steps):
loss = svi.step(r) / len(r)
losses.append(loss)
if step % 50 == 0:
median = guide.median()
print("step {} loss = {:0.6g}".format(step, loss))
print("-" * 20)
for name, (lb, ub) in sorted(guide.quantiles([0.325, 0.675]).items()):
if lb.numel() == 1:
lb = lb.squeeze().item()
ub = ub.squeeze().item()
print("{} = {:0.4g} ± {:0.4g}".format(name, (lb + ub) / 2, (ub - lb) / 2))
pyplot.figure(figsize=(9, 3))
pyplot.plot(losses)
pyplot.ylabel("loss")
pyplot.xlabel("SVI step")
pyplot.xlim(0, len(losses))
pyplot.ylim(min(losses), 20)
step 0 loss = 80.7915
step 50 loss = 2.49764
step 100 loss = 6.18623
step 150 loss = -1.42891
step 200 loss = -2.48601
step 250 loss = -2.75234
step 300 loss = -2.80716
step 350 loss = -2.64854
step 400 loss = -2.93349
step 450 loss = -2.90964
step 500 loss = -2.93564
step 550 loss = -2.98376
step 600 loss = -3.01648
step 650 loss = -3.01208
step 700 loss = -3.04329
step 750 loss = -3.03045
step 800 loss = -3.04258
step 850 loss = -3.06856
step 900 loss = -3.05272
step 950 loss = -3.06414
step 1000 loss = -3.06487
--------------------
h_0 = 0.3713 ± 0.01079
r_loc = 0.05134 ± 0.002976
r_skew = 0.0001597 ± 0.0002002
r_stability = 1.92 ± 0.001772
sigma = 0.2373 ± 0.000313
CPU times: user 38.1 s, sys: 6.95 s, total: 45.1 s
Wall time: 45.1 s
It appears the log returns exhibit very little skew, but exhibit a stability parameter slightly but significantly less than 2. This contrasts the usual Normal model corresponding to a Stable distribution with skew=0 and stability=2. We can now visualize the estimated volatility:
[12]:
fig, axes = pyplot.subplots(2, figsize=(9, 5), sharex=True)
pyplot.subplots_adjust(hspace=0)
axes[1].plot(r, "k", lw=0.2)
axes[1].set_ylabel("log returns")
axes[1].set_xlim(0, len(r))
# We will pull out median log returns using the autoguide's .median() and poutines.
with torch.no_grad():
pred = Predictive(reparam_model, guide=guide, num_samples=20, parallel=True)(r)
log_h = pred["log_h"]
axes[0].plot(log_h.median(0).values, lw=1)
axes[0].fill_between(torch.arange(len(log_h[0])),
log_h.kthvalue(2, dim=0).values,
log_h.kthvalue(18, dim=0).values,
color='red', alpha=0.5)
axes[0].set_ylabel("log volatility")
stability = pred["r_stability"].median(0).values.item()
axes[0].set_title("Estimated index of stability = {:0.4g}".format(stability))
axes[1].set_xlabel("trading day");
Observe that volatility roughly follows areas of large absolute log returns. Note that the uncertainty is underestimated, since we have used an approximate AutoDiagonalNormal
guide. For more precise uncertainty estimates, one could use HMC or NUTS inference.


This section describes region compression, its benefits and usage.
One way to reduce memory consumption by Geode is to enable compression in your regions. Geode allows you to compress in-memory region values using pluggable compressors (compression codecs). Geode includes the Snappy compressor as the built-in compression codec; however, you can implement and specify a different compressor for each compressed region.
What Gets Compressed
When you enable compression in a region, all values stored in the region are compressed while in memory. Keys and indexes are not compressed. New values are compressed when put into the in-memory cache and all values are decompressed when being read from the cache. Values are not compressed when persisted to disk. Values are decompressed before being sent over the wire to other peer members or clients.
When compression is enabled, each value in the region is compressed, and each region entry is compressed as a single unit. It is not possible to compress individual fields of an entry.
You can have a mix of compressed and non-compressed regions in the same cache.
Guidelines on Using Compression
This topic describes factors to consider when deciding on whether to use compression.
How to Enable Compression in a Region
This topic describes how to enable compression on your region.
-
When using region compression, you can use the default Snappy compressor included with Geode or you can specify your own compressor.
Comparing Performance of Compressed and Non-Compressed Regions
The comparative performance of compressed regions versus non-compressed regions can vary depending on how the region is being used and whether the region is hosted in a memory-bound JVM.
Guidelines on Using Compression
This topic describes factors to consider when deciding on whether to use compression.
Review the following guidelines when deciding on whether or not to enable compression in your region:
Use compression when JVM memory usage is too high. Compression allows you to store more region data in-memory and to reduce the number of expensive garbage collection cycles that prevent JVMs from running out of memory when memory usage is high.
To determine if JVM memory usage is high, examine the the following statistics:
- vmStats>freeMemory
- vmStats->maxMemory
- ConcurrentMarkSweep->collectionTime
If the amount of free memory regularly drops below 20% - 25% or the duration of the garbage collection cycles is generally on the high side, then the regions hosted on that JVM are good candidates for having compression enabled.
Consider the types and lengths of the fields in the region’s entries. Since compression is performed on each entry separately (and not on the region as a whole), consider the potential for duplicate data across a single entry. Duplicate bytes are compressed more easily. Also, since region entries are first serialized into a byte area before being compressed, how well the data might compress is determined by the number and length of duplicate bytes across the entire entry and not just a single field. Finally, the larger the entry the more likely compression will achieve good results as the potential for duplicate bytes, and a series of duplicate bytes, increases.
Consider the type of data you wish to compress. The type of data stored has a significant impact on how well the data may compress. String data will generally compress better than numeric data simply because string bytes are far more likely to repeat; however, that may not always be the case. For example, a region entry that holds a couple of short, unique strings may not provide as much memory savings when compressed as another region entry that holds a large number of integer values. In short, when evaluating the potential gains of compressing a region, consider the likelihood of having duplicate bytes, and more importantly the length of a series of duplicate bytes, for a single, serialized region entry. In addition, data that has already been compressed, such as JPEG format files, can actually cause more memory to be used.
Compress if you are storing large text values. Compression is beneficial if you are storing large text values (such as JSON or XML) or blobs in Geode that would benefit from compression.
Consider whether fields being queried against are indexed. You can query against compressed regions; however, if the fields you are querying against have not been indexed, then the fields must be decompressed before they can be used for comparison. In short, you may incur some query performance costs when querying against non-indexed fields.
Objects stored in the compression region must be serializable. Compression only operates on byte arrays, therefore objects being stored in a compressed region must be serializable and deserializable. The objects can either implement the Serializable interface or use one of the other Geode serialization mechanisms (such as PdxSerializable). Implementers should always be aware that when compression is enabled the instance of an object put into a region will not be the same instance when taken out. Therefore, transient attributes will lose their value when the containing object is put into and then taken out of a region.
Compressed regions will enable cloning by default. Setting a compressor and then disabling cloning results in an exception. The options are incompatible because the process of compressing/serializing and then decompressing/deserializing will result in a different instance of the object being created and that may be interpreted as cloning the object.
How to Enable Compression in a Region
This topic describes how to enable compression on your region.
To enable compression on your region, set the following region attribute in your cache.xml:
<?xml version="1.0" encoding= "UTF-8"?>
<cache xmlns="http://geode.apache.org/schema/cache"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://geode.apache.org/schema/cache http://geode.apache.org/schema/cache/cache-1.0.xsd"
version="1.0” lock-lease="120" lock-timeout= "60" search-timeout= "300" is-server= "true" copy-on-read= "false" >
<region name="compressedRegion" >
<region-attributes data-policy="replicate" ... />
<compressor>
<class-name>org.apache.geode.compression.SnappyCompressor</class-name>
</compressor>
...
</region-attributes>
</region>
</cache>
In the Compressor element, specify the class-name for your compressor implementation. This example specifies the Snappy compressor, which is bundled with Geode . You can also specify a custom compressor. See Working with Compressors for an example.
Compression can be enabled during region creation using gfsh or programmatically as well.
Using gfsh:
gfsh>create-region --name=”CompressedRegion” --compressor=”org.apache.geode.compression.SnappyCompressor”;
API:
regionFactory.setCompressor(new SnappyCompressor());
or
regionFactory.setCompressor(SnappyCompressor.getDefaultInstance());
How to Check Whether Compression is Enabled
You can also check whether a region has compression enabled by querying which codec is being used. A null codec indicates that no compression is enabled for the region.
Region myRegion = cache.getRegion("myRegion");
Compressor compressor = myRegion.getAttributes().getCompressor();
Working with Compressors
When using region compression, you can use the default Snappy compressor included with Geode or you can specify your own compressor.
The compression API consists of a single interface that compression providers must implement. The default compressor (SnappyCompressor) is the single compression implementation that comes bundled with the product. Note that since the Compressor is stateless, there only needs to be a single instance in any JVM; however, multiple instances may be used without issue. The single, default instance of the SnappyCompressor may be retrieved with the SnappyCompressor.getDefaultInstance()
static method.
Note: The Snappy codec included with Geode cannot be used with Solaris deployments. Snappy is only supported on Linux, Windows, and macOS deployments of Geode.
This example provides a custom Compressor implementation:
package com.mybiz.myproduct.compression;
import org.apache.geode.compression.Compressor;
public class LZWCompressor implements Compressor {
private final LZWCodec lzwCodec = new LZWCodec();
@Override
public byte[] compress(byte[] input) {
return lzwCodec.compress(input);
}
@Override
public byte[] decompress(byte[] input) {
return lzwCodec.decompress(input);
}
}
To use the new custom compressor on a region:
- Make sure that the new compressor package is available in the classpath of all JVMs that will host the region.
Configure the custom compressor for the region using any of the following mechanisms:
Using gfsh:
gfsh>create-region --name=”CompressedRegion” \ --compressor=”com.mybiz.myproduct.compression.LZWCompressor”
Using API:
For example:
regionFactory.setCompressor(new LZWCompressor());
cache.xml:
<region-attributes> <Compressor> <class-name>com.mybiz.myproduct.compression.LZWCompressor</class-name> </Compressor> </region-attributes>
Changing the Compressor for an Already Compressed Region
You typically enable compression on a region at the time of region creation. You cannot modify the Compressor or disable compression for the region while the region is online.
However, if you need to change the compressor or disable compression, you can do so by performing the following steps:
- Shut down the members hosting the region you wish to modify.
- Modify the cache.xml file for the member either specifying a new compressor or removing the compressor attribute from the region.
- Restart the member.
Comparing Performance of Compressed and Non-Compressed Regions
The comparative performance of compressed regions versus non-compressed regions can vary depending on how the region is being used and whether the region is hosted in a memory-bound JVM.
When considering the cost of enabling compression, you should consider the relative cost of reading and writing compressed data as well as the cost of compression as a percentage of the total time spent managing entries in a region. As a general rule, enabling compression on a region will add 30% - 60% more overhead for region create and update operations than for region get operations. Because of this, enabling compression will create more overhead on regions that are write heavy than on regions that are read heavy.
However, when attempting to evaluate the performance cost of enabling compression you should also consider the cost of compression relative to the overall cost of managing entries in a region. A region may be tuned in such a way that it is highly optimized for read and/or write performance. For example, a replicated region that does not save to disk will have much better read and write performance than a partitioned region that does save to disk. Enabling compression on a region that has been optimized for read and write performance will provide more noticeable results than using compression on regions that have not been optimized this way. More concretely, performance may degrade by several hundred percent on a read/write optimized region whereas it may only degrade by 5 to 10 percent on a non-optimized region.
A final note on performance relates to the cost when enabling compression on regions in a memory bound JVM. Enabling compression generally assumes that the enclosing JVM is memory bound and therefore spends a lot of time for garbage collection. In that case performance may improve by as much as several hundred percent as the JVM will be running far fewer garbage collection cycles and spending less time when running a cycle.
Monitoring Compression Performance
The following statistics provide monitoring for cache compression:
-
compressTime
-
decompressTime
-
compressions
-
decompressions
-
preCompressedBytes
-
postCompressedBytes
See Cache Performance (CachePerfStats) for statistic descriptions.


Samples
All i want for Christmas is you
Why so serious!
I'm so sad so very very sad
It was boring for me.

Title: Totally Bored Boris!
contents: Boris, the bear, is very bored, until he joins the party his sister and her friends are having.
Title: Never Lonely Again


Before Change
@property
def binary(self):
// TODO(pl): Assert there is only one dep and it is a JvmBinary
return self.dependencies[0]
@property
def jar_dependencies(self):
return self.binary.jar_dependencies
After Change
@property
def binary(self):
dependencies = self.dependencies
if len(dependencies) != 1:
raise TargetDefinitionException(self, "A JvmApp must define exactly one JvmBinary "
"dependency, have: %s" % dependencies)
binary = dependencies[0]
if not isinstance(binary, JvmBinary):
raise TargetDefinitionException(self, "Expected JvmApp binary dependency to be a JvmBinary "
"target, found %s" % binary)


Adds CSS to the style tag in the head of a web page. You can use <o2 addCss> as a normal style tag, or you can specify a class-name
and the class' corresponding style in one of two ways (see the examples). When you don't use the addCss tag as a style tag, you can only
specify the style for a given class-name - not for IDs or HTML tags.
#item1 {
color: green;
}
or
color: red;
or
color: red;
class - the class name
param - the class name
content - the style
<o2 header> or <o2 incStylesheet> must be called on the same page for addCss to work.
See also or .


M3 Nylon Nut - NN3
Summary
Name: M3 Nylon Nut
ID: NNUT-M3-X-STAN-01
Hex ID: NN3
WebPage: https://github.com/oomlout/oomlout-OOMP/wiki/NNUT-M3-X-STAN-01
Short URL: http://oom.lt/NN3
Revision History: https://github.com/oomlout/oomlout-OOMP/blob/master/parts/NNUT-M3-X-STAN-01/
| Type |
Size |
Color |
Description |
Index |
NNUT Nylon Nut |
M3 M3 |
X
|
STAN
|
01
|
Images
About
This part is awaiting a description.
Specifications
| Info |
Value |
| Type |
Nylon Nut |
| Size |
M3 |
Extra Details
Spotted a mistake, want to add more? Let us know<EMAIL_ADDRESS>All images and resources are licensed [CC BY-SA] unless otherwise stated (ie. the datasheets)


PhongMaterial Class
A PhongMaterial is a Material that defines the surface appearance of attached GameObjects using the Phong lighting model.
Overview
- These PhongMaterial properties, along with emissive, opacity and reflectivity, specify attributes that are to be applied uniformly across the surface of attached Geometries.
- Most of those attributes can be textured, effectively replacing the values set for those properties, by assigning Textures to the PhongMaterial's diffuseMap, specularMap, emissiveMap, opacityMap and reflectivityMap properties.
- For example, the value of diffuse will be ignored if your PhongMaterial also has a diffuseMap set to a Texture. The Texture's pixel colors directly provide the diffuse color of each fragment across the Geometry surface, ie. they are not multiplied by the diffuse for each pixel, as is done in many shading systems.
- See Shader Inputs for the variables that PhongMaterials create within xeoEngine's shaders.
Example
In this example we have
- a Texture,
- a Fresnel,
- a PhongMaterial which applies the Texture as a diffuse map and the Fresnel as a specular Fresnel effect,
- a Lights containing an AmbientLight and a DirLight,
- a Geometry that is the default box shape, and
- a GameObject attached to all of the above.
Note that the value for the PhongMaterial's diffuse property is ignored and redundant, since we assign a Texture to the PhongMaterial's diffuseMap property. The Texture's pixel colors directly provide the diffuse color of each fragment across the Geometry surface.
var scene = new XEO.Scene();
var diffuseMap = new XEO.Texture(scene, {
src: "diffuseMap.jpg"
});
var fresnel = new XEO.Fresnel(scene, {
leftColor: [1.0, 1.0, 1.0],
rightColor: [0.0, 0.0, 0.0],
power: 4
});
var material = new XEO.PhongMaterial(scene, {
ambient: [0.3, 0.3, 0.3],
diffuse: [0.5, 0.5, 0.0], // Ignored, since we have assigned a Texture to diffuseMap, below
diffuseMap: diffuseMap,
specular: [1, 1, 1],
shininess: 30,
specularFresnel: fresnel
});
var ambientLight = new XEO.AmbientLight(scene, {
color: [0.7, 0.7, 0.7]
});
var dirLight = new XEO.DirLight(scene, {
dir: [-1, -1, -1],
color: [0.5, 0.7, 0.5],
intensity: [1.0, 1.0, 1.0],
space: "view"
});
var lights = new XEO.Lights(scene, {
lights: [
ambientLight,
dirLight
]
});
var geometry = new XEO.Geometry(scene); // Geometry without parameters will default to a 2x2x2 box.
var object = new XEO.GameObject(scene, {
lights: lights,
material: material,
geometry: geometry
});
Constructor
PhongMaterial
-
[scene]
-
[cfg]
Parameters:
-
[scene]
Scene optional -
[cfg]
optionalThe PhongMaterial configuration
-
[id]
String optionalOptional ID, unique among all components in the parent Scene, generated automatically when omitted.
-
[meta=null]
String:Object optionalMetadata to attach to this PhongMaterial.
-
[ambient=[0.7, 0.7, 0.8 ]]
Array of Number optionalPhongMaterial ambient color.
-
[diffuse=[ 1.0, 1.0, 1.0 ]]
Array of Number optionalPhongMaterial diffuse color.
-
[specular=[ 1.0, 1.0, 1.0 ]]
Array of Number optionalPhongMaterial specular color.
-
[emissive=[ 0.0, 0.0, 0.0 ]]
Array of Number optionalPhongMaterial emissive color.
-
[opacity=1]
Number optionalScalar in range 0-1 that controls opacity, where 0 is completely transparent and 1 is completely opaque. Only applies while Modes transparent equals
true
. -
[shininess=30]
Number optionalScalar in range 0-70 that determines the size and sharpness of specular highlights.
-
[reflectivity=1]
Number optionalScalar in range 0-1 that controls how much CubeMap is reflected.
-
[diffuseMap=null]
Texture optional -
[specularMap=null]
Texture optional -
[emissiveMap=null]
Texture optional -
[normalMap=null]
Texture optional -
[opacityMap=null]
Texture optional -
[reflectivityMap=null]
Texture optional -
[diffuseFresnel=null]
Fresnel optionalA diffuse Fresnel.
-
[specularFresnel=null]
Fresnel optionalA specular Fresnel.
-
[emissiveFresnel=null]
Fresnel optionalAn emissive Fresnel.
-
[opacityFresnel=null]
Fresnel optionalAn opacity Fresnel.
-
[reflectivityFresnel=null]
Fresnel optionalA reflectivity Fresnel.
-
Item Index
Properties
Methods
destroy
()
Destroys this component.
Fires a destroyed event on this Component.
Automatically disassociates this component from other components, causing them to fall back on any defaults that this component overrode on them.
error
-
message
Logs an error for this component to the JavaScript console.
The console message will have this format: [ERROR] [<component type> =<component id>: <message>
Also fires the message as an error event on the parent Scene.
Parameters:
-
message
StringThe message to log
fire
-
event
-
value
-
[forget=false]
Fires an event on this component.
Notifies existing subscribers to the event, retains the event to give to any subsequent notifications on that location as they are made.
Parameters:
-
event
StringThe event type name
-
value
ObjectThe event parameters
-
[forget=false]
Boolean optionalWhen true, does not retain for subsequent subscribers
log
-
message
Logs a console debugging message for this component.
The console message will have this format: [LOG] [<component type> <component id>: <message>
Parameters:
-
message
StringThe message to log
off
-
handle
Parameters:
-
handle
StringPublication handle
on
-
event
-
callback
-
[scope=this]
Subscribes to an event on this component.
The callback is be called with this component as scope.
Parameters:
-
event
StringThe event
-
callback
FunctionCalled fired on the event
-
[scope=this]
Object optionalScope for the callback
Returns:
Handle to the subscription, which may be used to unsubscribe with {@link #off}.
once
-
event
-
callback
-
[scope=this]
Subscribes to the next occurrence of the given event, then un-subscribes as soon as the event is handled.
This is equivalent to calling on, and then calling off inside the callback function.
Parameters:
-
event
StringData event to listen to
-
callback
Function(data)Called when fresh data is available at the event
-
[scope=this]
Object optionalScope for the callback
Properties
ambient
Array(Number)
The PhongMaterial's ambient color.
Fires a ambient event on change.
Default: [1.0, 1.0, 1.0]
destroyed
Boolean
True as soon as this Component has been destroyed
diffuse
Array(Number)
The PhongMaterial's diffuse color.
This property may be overridden by diffuseMap.
Fires a diffuse event on change.
Default: [1.0, 1.0, 1.0]
diffuseFresnel
Fresnel
A diffuse Fresnel attached to this PhongMaterial.
This property overrides diffuseFresnel when not null or undefined.
Fires a diffuseFresnel event on change.
Default: null
diffuseMap
Texture
A diffuse Texture attached to this PhongMaterial.
This property overrides diffuseMap when not null or undefined.
Fires a diffuseMap event on change.
Default: null
emissive
Array(Number)
The PhongMaterial's emissive color.
This property may be overridden by emissiveMap.
Fires a emissive event on change.
Default: [0.0, 0.0, 0.0]
emissiveFresnel
Fresnel
An emissive Fresnel attached to this PhongMaterial.
This property overrides emissive when not null or undefined.
Fires an emissiveFresnel event on change.
Default: null
emissiveMap
Texture
An emissive Texture attached to this PhongMaterial.
This property overrides emissive when not null or undefined.
Fires an emissiveMap event on change.
Default: null
Items in this map
Unknown
metadata
Object
Arbitrary, user-defined metadata on this component.
normalMap
Texture
A normal Texture attached to this PhongMaterial.
This property overrides normalMap when not null or undefined.
Fires a normalMap event on change.
Default: null
opacity
Number
Factor in the range [0..1] indicating how transparent the PhongMaterial is.
A value of 0.0 indicates fully transparent, 1.0 is fully opaque.
Attached GameObjects will appear transparent only if they are also attached to Modes that have transparent set to true.
This property may be overridden by opacityMap.
Fires an opacity event on change.
Default: 1.0
opacityFresnel
Fresnel
An opacity Fresnel attached to this PhongMaterial.
This property overrides opacity when not null or undefined.
Fires an opacityFresnel event on change.
Default: null
opacityMap
Texture
An opacity Texture attached to this PhongMaterial.
This property overrides opacity when not null or undefined.
Fires an opacityMap event on change.
Default: null
reflectivity
Number
Scalar in range 0-1 that controls how much CubeMap is reflected by this PhongMaterial.
The surface will be non-reflective when this is 0, and completely mirror-like when it is 1.0.
This property may be overridden by reflectivityMap.
Fires a reflectivity event on change.
Default: 1.0
reflectivityFresnel
Fresnel
A reflectivity Fresnel attached to this PhongMaterial.
This property overrides reflectivity when not null or undefined.
Fires a reflectivityFresnel event on change.
Default: null
reflectivityMap
Texture
A reflectivity Texture attached to this PhongMaterial.
This property overrides reflectivity when not null or undefined.
Fires a reflectivityMap event on change.
Default: null
shininess
Number
A factor in range [0..128] that determines the size and sharpness of the specular highlights create by this PhongMaterial.
Larger values produce smaller, sharper highlights. A value of 0.0 gives very large highlights that are almost never desirable. Try values close to 10 for a larger, fuzzier highlight and values of 100 or more for a small, sharp highlight.
Fires a shininess event on change.
Default: 30.0
specular
Array(Number)
The material's specular color.
This property may be overridden by specularMap.
Fires a specular event on change.
Default: [0.3, 0.3, 0.3]
specularFresnel
Fresnel
A specular Fresnel attached to this PhongMaterial.
This property overrides specular when not null or undefined.
Fires a specularFresnel event on change.
Default: null
specularMap
Texture
A specular Texture attached to this PhongMaterial.
This property overrides specular when not null or undefined.
Fires a specularMap event on change.
Default: null
type
String
final
JavaScript class name for this Component.
This is used when loading Scenes from JSON, and is included in the JSON representation of this Component, so that this class may be instantiated when loading it from the JSON representation.
For example: "XEO.AmbientLight", "XEO.ColorTarget", "XEO.Lights" etc.
Events
ambient
Fired whenever this PhongMaterial's ambient property changes.
Event Payload:
-
value
Array(Number)The property's new value
destroyed
Fired when this Component is destroyed.
diffuse
Fired whenever this PhongMaterial's diffuse property changes.
Event Payload:
-
value
Array(Number)The property's new value
diffuseFresnel
Fired whenever this PhongMaterial's diffuse property changes.
Event Payload:
-
value
ObjectNumber The property's new value
diffuseMap
Fired whenever this PhongMaterial's diffuse property changes.
Event Payload:
-
value
ObjectNumber The property's new value
emissive
Fired whenever this PhongMaterial's emissive property changes.
Event Payload:
-
value
Array(Number)The property's new value
emissiveFresnel
Fired whenever this PhongMaterial's emissiveFresnel property changes.
Event Payload:
-
value
ObjectNumber The property's new value
emissiveMap
Fired whenever this PhongMaterial's emissiveMap property changes.
Event Payload:
-
value
ObjectNumber The property's new value
normalMap
Fired whenever this PhongMaterial's PhongMaterial/normal:property property changes.
Event Payload:
-
value
ObjectNumber The property's new value
opacity
Fired whenever this PhongMaterial's opacity property changes.
Event Payload:
-
value
NumberThe property's new value
opacityFresnel
Fired whenever this PhongMaterial's opacityFresnel property changes.
Event Payload:
-
value
ObjectNumber The property's new value
opacityMap
Fired whenever this PhongMaterial's opacityMap property changes.
Event Payload:
-
value
ObjectNumber The property's new value
reflectivity
Fired whenever this PhongMaterial's reflectivity property changes.
Event Payload:
-
value
ObjectNumber The property's new value
reflectivityFresnel
Fired whenever this PhongMaterial's reflectivityFresnel property changes.
Event Payload:
-
value
ObjectNumber The property's new value
reflectivityMap
Fired whenever this PhongMaterial's reflectivityMap property changes.
Event Payload:
-
value
ObjectNumber The property's new value
shininess
Fired whenever this PhongMaterial's shininess property changes.
Event Payload:
-
value
ObjectNumber The property's new value
specular
Fired whenever this PhongMaterial's specular property changes.
Event Payload:
-
value
Array(Number)The property's new value
specularFresnel
Fired whenever this PhongMaterial's specularFresnel property changes.
Event Payload:
-
value
ObjectNumber The property's new value
specularMap
Fired whenever this PhongMaterial's specularMap property changes.
Event Payload:
-
value
ObjectNumber The property's new value


| competition_tracking/__init__.py |
0 |
0 |
0 |
100% |
| competition_tracking/admin.py |
1 |
0 |
0 |
100% |
| competition_tracking/api/competition_track.py |
232 |
202 |
0 |
13% |
| competition_tracking/api/urls.py |
4 |
0 |
0 |
100% |
| competition_tracking/api/utils.py |
82 |
70 |
0 |
15% |
| competition_tracking/api/views.py |
40 |
27 |
0 |
32% |
| competition_tracking/apps.py |
3 |
3 |
0 |
0% |
| competition_tracking/migrations/__init__.py |
0 |
0 |
0 |
100% |
| competition_tracking/models.py |
1 |
0 |
0 |
100% |
| competition_tracking/tasks.py |
27 |
27 |
0 |
0% |
| competition_tracking/tests.py |
1 |
0 |
0 |
100% |
| competition_tracking/urls.py |
3 |
0 |
0 |
100% |
| competition_tracking/views.py |
1 |
0 |
0 |
100% |
| everpro/__init__.py |
3 |
0 |
0 |
100% |
| everpro/asgi.py |
4 |
4 |
0 |
0% |
| everpro/celery.py |
10 |
1 |
0 |
90% |
| everpro/settings.py |
26 |
0 |
0 |
100% |
| everpro/urls.py |
4 |
0 |
0 |
100% |
| everpro/wsgi.py |
4 |
4 |
0 |
0% |
| inventory_scrapping/__init__.py |
0 |
0 |
0 |
100% |
| inventory_scrapping/admin.py |
3 |
0 |
0 |
100% |
| inventory_scrapping/api/serializers.py |
19 |
6 |
0 |
68% |
| inventory_scrapping/api/urls.py |
4 |
0 |
0 |
100% |
| inventory_scrapping/api/views.py |
57 |
41 |
0 |
28% |
| inventory_scrapping/apps.py |
3 |
3 |
0 |
0% |
| inventory_scrapping/migrations/0001_initial.py |
7 |
0 |
0 |
100% |
| inventory_scrapping/migrations/__init__.py |
0 |
0 |
0 |
100% |
| inventory_scrapping/models.py |
15 |
1 |
0 |
93% |
| inventory_scrapping/scrapper.py |
46 |
46 |
0 |
0% |
| inventory_scrapping/tasks.py |
8 |
8 |
0 |
0% |
| inventory_scrapping/tests.py |
1 |
0 |
0 |
100% |
| inventory_scrapping/urls.py |
3 |
0 |
0 |
100% |
| inventory_scrapping/utils.py |
18 |
18 |
0 |
0% |
| inventory_scrapping/views.py |
3 |
0 |
0 |
100% |
| manage.py |
12 |
2 |
0 |
83% |
| search/__init__.py |
0 |
0 |
0 |
100% |
| search/admin.py |
1 |
0 |
0 |
100% |
| search/apps.py |
3 |
3 |
0 |
0% |
| search/engine.py |
15 |
3 |
0 |
80% |
| search/engines/__init__.py |
0 |
0 |
0 |
100% |
| search/engines/amazon.py |
157 |
135 |
0 |
14% |
| search/engines/flipkart.py |
303 |
275 |
0 |
9% |
| search/migrations/__init__.py |
0 |
0 |
0 |
100% |
| search/models.py |
1 |
0 |
0 |
100% |
| search/tests.py |
8 |
0 |
0 |
100% |
| search/urls.py |
3 |
0 |
0 |
100% |
| search/views.py |
20 |
4 |
0 |
80% |


Mar-a-Lago trespasser sentenced to 8 months in prison
Chinese national Yujing Zhang was sentenced to eight months in prison and two years of supervised release over her arrest for trespassing at President Donald Trump's Mar-a-Lago club in Florida last March, a Justice Department official told ABC News Monday.
Zhang was convicted in September for one count of lying to a federal officer and one count of entering restricted grounds.
On March 30, staff mistakenly allowed Zhang in to Mar-a-Lago and said that she misrepresented herself to several Secret Service officers as she made her way through security checkpoints. She was flagged to security after a receptionist said her name was not on the access list for the club.
Following her arrest, investigators suggested they had suspicions about potential ties to Chinese intelligence after finding Zhang in possession of several electronic devices, including one device that could be used to detect hidden cameras.
Zhang was never formally charged with espionage, however, and in a statement following her verdict she re-asserted her belief that she had done nothing wrong.
During her trial, Zhang represented herself after rejecting the court's offer for a public defender.
She was admonished several times by Judge Roy Altman for instances of bizarre behavior, including complaints that she was feeling "sick and dizzy," and claims that she was unable to hear several of Altman's statements to her.