In this project, you'll use generative adversarial networks to generate new images of faces.
You'll be using two datasets in this project:
Since the celebA dataset is complex and you're doing GANs in a project for the first time, we want you to test your neural network on MNIST before CelebA. Running the GANs on MNIST will allow you to see how well your model trains sooner.
If you're using FloydHub, set data_dir
to "/input" and use the FloydHub data ID "R5KrjnANiKVhLWAkpXhNBe".
data_dir = './data'
# FloydHub - Use with data ID "R5KrjnANiKVhLWAkpXhNBe"
#data_dir = '/input'
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
helper.download_extract('mnist', data_dir)
helper.download_extract('celeba', data_dir)
show_n_images = 25
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
import os
from glob import glob
from matplotlib import pyplot
mnist_images = helper.get_batch(glob(os.path.join(data_dir, 'mnist/*.jpg'))[:show_n_images], 28, 28, 'L')
pyplot.imshow(helper.images_square_grid(mnist_images, 'L'), cmap='gray')
The CelebFaces Attributes Dataset (CelebA) dataset contains over 200,000 celebrity images with annotations. Since you're going to be generating faces, you won't need the annotations. You can view the first number of examples by changing show_n_images
.
show_n_images = 25
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
mnist_images = helper.get_batch(glob(os.path.join(data_dir, 'img_align_celeba/*.jpg'))[:show_n_images], 28, 28, 'RGB')
pyplot.imshow(helper.images_square_grid(mnist_images, 'RGB'))
Since the project's main focus is on building the GANs, we'll preprocess the data for you. The values of the MNIST and CelebA dataset will be in the range of -0.5 to 0.5 of 28x28 dimensional images. The CelebA images will be cropped to remove parts of the image that don't include a face, then resized down to 28x28.

You'll build the components necessary to build a GANs by implementing the following functions below:
model_inputs
discriminator
generator
model_loss
model_opt
train
This will check to make sure you have the correct version of TensorFlow and access to a GPU
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf
# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer. You are using {}'.format(tf.__version__)
print('TensorFlow Version: {}'.format(tf.__version__))
# Check for a GPU
if not tf.test.gpu_device_name():
warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))
Implement the model_inputs
function to create TF Placeholders for the Neural Network. It should create the following placeholders:
image_width
, image_height
, and image_channels
.z_dim
.Return the placeholders in the following the tuple (tensor of real input images, tensor of z data)
import problem_unittests as tests
def model_inputs(image_width, image_height, image_channels, z_dim):
"""
Create the model inputs
:param image_width: The input image width
:param image_height: The input image height
:param image_channels: The number of image channels
:param z_dim: The dimension of Z
:return: Tuple of (tensor of real input images, tensor of z data, learning rate)
"""
# Real images dimention
real_dim = (image_width, image_height, image_channels)
# Real images placeholder
inputs_real = tf.placeholder(tf.float32, (None, *real_dim))
# Generator input placeholder
z = tf.placeholder(tf.float32, (None, z_dim))
# Learning rate
learning_rate = tf.placeholder(tf.float32, shape=())
return inputs_real, z, learning_rate
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_inputs(model_inputs)
Implement discriminator
to create a discriminator neural network that discriminates on images
. This function should be able to reuse the variabes in the neural network. Use tf.variable_scope
with a scope name of "discriminator" to allow the variables to be reused. The function should return a tuple of (tensor output of the discriminator, tensor logits of the discriminator).
def discriminator(images, reuse=False, alpha=0.1):
"""
Create the discriminator network
:param image: Tensor of input image(s)
:param reuse: Boolean if the weights should be reused
:return: Tuple of (tensor output of the discriminator, tensor logits of the discriminator)
"""
with tf.variable_scope('discriminator', reuse=reuse):
# First convolutional layer - 14 x 14 x 64
conv1 = tf.layers.conv2d(images, 64, 5, strides=2, padding='same')
conv1r = tf.maximum(alpha * conv1, conv1)
# Second convolutional layer - 7 x 7 x 128
conv2 = tf.layers.conv2d(conv1r, 128, 5, strides=2, padding='same')
conv2n = tf.layers.batch_normalization(conv2, training=True)
conv2r = tf.maximum(alpha * conv2n, conv2n)
# Third convolutional layer - 4 x 4 x 256
conv3 = tf.layers.conv2d(conv2r, 256, 5, strides=2, padding='same')
conv3n = tf.layers.batch_normalization(conv3, training=True)
conv3r = tf.maximum(alpha * conv3n, conv3n)
# Fourth convolutional layer - 2 x 2 x 512
conv4 = tf.layers.conv2d(conv3r, 512, 5, strides=2, padding='same')
conv4n = tf.layers.batch_normalization(conv4, training=True)
conv4r = tf.maximum(alpha * conv4n, conv4n)
# Reshape output for the final layer
reshape = tf.reshape(conv4r,(-1, 8 * 64 * 2 * 2))
# Logits
logits = tf.layers.dense(reshape, 1)
# Output
out = tf.sigmoid(logits)
return out, logits
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_discriminator(discriminator, tf)
Implement generator
to generate an image using z
. This function should be able to reuse the variabes in the neural network. Use tf.variable_scope
with a scope name of "generator" to allow the variables to be reused. The function should return the generated 28 x 28 x out_channel_dim
images.
def generator(z, out_channel_dim, is_train=True, alpha=0.1):
"""
Create the generator network
:param z: Input z
:param out_channel_dim: The number of channels in the output image
:param is_train: Boolean if generator is being used for training
:return: The tensor output of the generator
"""
with tf.variable_scope('generator', reuse= not is_train):
# Dense layer
d = tf.layers.dense(z, 16 * 32 * 3 * 3)
dr = tf.reshape(d, (-1, 3, 3, 16 * 32))
drn = tf.layers.batch_normalization(dr, training=is_train)
drnr = tf.maximum(alpha * drn, drn)
# First transpose convolution - 7 x 7 x 128
c1 = tf.layers.conv2d_transpose(drnr, 128, 3, strides=2, padding='valid')
c1n = tf.layers.batch_normalization(c1, training=is_train)
c1nr = tf.maximum(alpha * c1n, c1n)
# Second transpose convolution - 14 x 14 x 64
c2 = tf.layers.conv2d_transpose(c1nr, 64, 5, strides=2, padding='same')
c2n = tf.layers.batch_normalization(c2, training=is_train)
c2nr = tf.maximum(alpha * c2n, c2n)
# Third transpose convolution - 28 x 28 x 32
c3 = tf.layers.conv2d_transpose(c2nr, 32, 5, strides=2, padding='same')
c3n = tf.layers.batch_normalization(c3, training=is_train)
c3nr = tf.maximum(alpha * c3n, c3n)
# Fourth transpose convolution - 28 x 28 x out_channel_dim
c4 = tf.layers.conv2d_transpose(c3nr, out_channel_dim, 5, strides=1, padding='same')
# Output
out = tf.tanh(c4)
return out
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_generator(generator, tf)
Implement model_loss
to build the GANs for training and calculate the loss. The function should return a tuple of (discriminator loss, generator loss).  Use tf.trainable_variables
to get all the trainable variables. Filter the variables with names that are in the discriminator and generator scope names. The function should return a tuple of (discriminator training operation, generator training operation).
def model_opt(d_loss, g_loss, learning_rate, beta1):
"""
Get optimization operations
:param d_loss: Discriminator loss Tensor
:param g_loss: Generator loss Tensor
:param learning_rate: Learning Rate Placeholder
:param beta1: The exponential decay rate for the 1st moment in the optimizer
:return: A tuple of (discriminator training operation, generator training operation)
"""
# Trainable variables
t_vars = tf.trainable_variables()
# Trainable discriminator variables
d_vars = [var for var in t_vars if var.name.startswith('discriminator')]
# Trainable generator variables
g_vars = [var for var in t_vars if var.name.startswith('generator')]
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
# Generator update
gen_updates = [op for op in update_ops if op.name.startswith('generator')]
# Optimizers
with tf.control_dependencies(gen_updates):
# Train optimizer for Discriminator
d_train_opt = tf.train.AdamOptimizer(learning_rate, beta1=beta1).minimize(d_loss, var_list=d_vars)
# Train optimizer for Generator
g_train_opt = tf.train.AdamOptimizer(learning_rate, beta1=beta1).minimize(g_loss, var_list=g_vars)
return d_train_opt, g_train_opt
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_opt(model_opt, tf)
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np
def show_generator_output(sess, n_images, input_z, out_channel_dim, image_mode):
"""
Show example output for the generator
:param sess: TensorFlow session
:param n_images: Number of Images to display
:param input_z: Input Z Tensor
:param out_channel_dim: The number of channels in the output image
:param image_mode: The mode to use for images ("RGB" or "L")
"""
cmap = None if image_mode == 'RGB' else 'gray'
z_dim = input_z.get_shape().as_list()[-1]
example_z = np.random.uniform(-1, 1, size=[n_images, z_dim])
samples = sess.run(
generator(input_z, out_channel_dim, False),
feed_dict={input_z: example_z})
images_grid = helper.images_square_grid(samples, image_mode)
pyplot.imshow(images_grid, cmap=cmap)
pyplot.show()
Implement train
to build and train the GANs. Use the following functions you implemented:
model_inputs(image_width, image_height, image_channels, z_dim)
model_loss(input_real, input_z, out_channel_dim)
model_opt(d_loss, g_loss, learning_rate, beta1)
Use the show_generator_output
to show generator
output while you train. Running show_generator_output
for every batch will drastically increase training time and increase the size of the notebook. It's recommended to print the generator
output every 100 batches.
def train(epoch_count, batch_size, z_dim, learning_rate, beta1, get_batches, data_shape, data_image_mode):
"""
Train the GAN
:param epoch_count: Number of epochs
:param batch_size: Batch Size
:param z_dim: Z dimension
:param learning_rate: Learning Rate
:param beta1: The exponential decay rate for the 1st moment in the optimizer
:param get_batches: Function to get batches
:param data_shape: Shape of the data
:param data_image_mode: The image mode to use for images ("RGB" or "L")
"""
# Number of color channels
_, image_w, image_h, n_channels = data_shape
# Model input
img, z, lr = model_inputs(image_w, image_h, n_channels, z_dim)
# Losses
d_loss, g_loss = model_loss(img, z, n_channels)
# Optimizers
d_opt, g_opt = model_opt(d_loss, g_loss, lr, beta1)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for epoch_i in range(epoch_count):
# Set initial steps and sums
steps = 0
d_loss_sum = 0
g_loss_sum = 0
batch_count = 0
for batch_images in get_batches(batch_size):
steps += 1
batch_count += 1
batch_images * 2
# Sample random noise for generator
batch_z = np.random.uniform(-1, 1, size=(batch_size, z_dim))
# Run optimizers
_ = sess.run(d_opt, feed_dict={img: batch_images, z: batch_z, lr: learning_rate})
_ = sess.run(g_opt, feed_dict={z: batch_z, lr: learning_rate})
# Update loss sums
d_loss_sum += d_loss.eval({z: batch_z, img: batch_images})
g_loss_sum += g_loss.eval({z: batch_z})
# Print the losses
if steps%20 == 0:
# Generator output
show_generator_output(sess, 16, z, n_channels, data_image_mode)
print("Epoch {}/{}...".format(epoch_i+1, epoch_count),
"Avg. Discriminator Loss: {:.4f}...".format(d_loss_sum / batch_count),
"Avg. Generator Loss: {:.4f}".format(g_loss_sum / batch_count))
# Set loss sums back to zero
d_loss_sum = 0
g_loss_sum = 0
# Set batch count back to zero
batch_count = 0
Test your GANs architecture on MNIST. After 2 epochs, the GANs should be able to generate images that look like handwritten digits. Make sure the loss of the generator is lower than the loss of the discriminator or close to 0.
batch_size = 64
z_dim = 100
learning_rate = 0.001
beta1 = 0.5
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
epochs = 2
mnist_dataset = helper.Dataset('mnist', glob(os.path.join(data_dir, 'mnist/*.jpg')))
with tf.Graph().as_default():
train(epochs, batch_size, z_dim, learning_rate, beta1, mnist_dataset.get_batches,
mnist_dataset.shape, mnist_dataset.image_mode)
Run your GANs on CelebA. It will take around 20 minutes on the average GPU to run one epoch. You can run the whole epoch or stop when it starts to generate realistic faces.
batch_size = 128
z_dim = 100
learning_rate = 0.001
beta1 = 0.5
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
epochs = 1
celeba_dataset = helper.Dataset('celeba', glob(os.path.join(data_dir, 'img_align_celeba/*.jpg')))
with tf.Graph().as_default():
train(epochs, batch_size, z_dim, learning_rate, beta1, celeba_dataset.get_batches,
celeba_dataset.shape, celeba_dataset.image_mode)
When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_face_generation.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.


{% extends 'templates/nhs_template.html' %} {% block banner %} {% include "includes/nhs.uk/banner_this_is_new_page.html" %} {% endblock %} {% block pageTitle %}Test pages for NHS.UK{% endblock %} {% block content %}
This is content.
Here is a piece of content with some italics, some strong type, and a link that goes nowhere.
You want a bullet list?


Quizzes at your hand
Here you have the chance to be creative and create your OWN quiz
Keep scrolling down to discover more quizzes that you can solve.
Math |
Languages |
Science |
Programming |
Fun |
Geography |


This player is in a very early stage of development. It supports only predefined, simple instruments and some percussion. For better quality, export to Mp3 or Ogg and play the song from the Export window instead.
Do you want to give some feedback about this application? Great!
You can just write it down in the text area below and then press the "Submit" button.
If you want some kind of answer you can just provide an email address or simply provide such info in the "Account" dialog.
Loading page, please wait...


|
BubbleProfiler
0.3.0
by Peter Athron, Csaba Balazs, Michael Bardsley, Andrew Fowlie, Dylan Harries & Graham White
|
Provides guess for field profiles read from an input stream. More...
#include <instream_profile_guesser.hpp>
Public Member Functions | |
| Instream_profile_guesser (std::istream &) | |
| reads a profile guess from the given input stream More... | |
| Instream_profile_guesser (std::istream &, const std::string &) | |
| reads a profile guess from the given input stream More... | |
| Instream_profile_guesser (std::istream &, const std::string &, const std::string &) | |
| reads a profile guess from the given input stream More... | |
| virtual | ~Instream_profile_guesser ()=default |
| Instream_profile_guesser (const Instream_profile_guesser &)=default | |
| Instream_profile_guesser & | operator= (const Instream_profile_guesser &)=default |
| Instream_profile_guesser (Instream_profile_guesser &&)=default | |
| Instream_profile_guesser & | operator= (Instream_profile_guesser &&)=default |
| virtual Field_profiles | get_profile_guess (const Potential &, const Eigen::VectorXd &, int, double, double, double, double) override |
| Calculate an initial guess for the bubble profile. More... | |
| std::size_t | get_number_of_grid_points () const |
| returns the number of grid points read from the stream More... | |
| std::size_t | get_number_of_fields () const |
| Eigen::VectorXd | get_spatial_grid () const |
| returns the values of the spacetime coordinate read from the stream More... | |
| Eigen::MatrixXd | get_field_values () const |
| returns the values of the fields read from the stream More... | |
| void | set_spatial_tolerance (double t) |
| sets the spatial tolerance used when comparing grid points More... | |
| void | set_delimiters (const std::string &d) |
| set the characters to use as delimiters between fields More... | |
| void | set_line_comment (const std::string &c) |
| sets the character sequence used to indicate a comment More... | |
| void | read_profiles (std::istream &) |
| read a profile guess from the given stream More... | |
| Public Member Functions inherited from BubbleProfiler::Profile_guesser | |
| virtual | ~Profile_guesser ()=default |
Private Member Functions | |
| void | reset () |
| void | strip_comments (std::string &) const |
| void | process_line (const std::string &) |
Private Attributes | |
| std::string | delimiters {" \t"} |
| std::string | line_comment_start {"#"} |
| bool | has_n_fields {false} |
| std::size_t | n_fields {0} |
| double | tolerance {1.e-6} |
| std::vector< double > | spatial_grid {} |
| std::vector< std::vector< double > > | field_values {} |
| logging::Basic_logger | logger {} |
Provides guess for field profiles read from an input stream.
Definition at line 41 of file instream_profile_guesser.hpp.
|
explicit |
reads a profile guess from the given input stream
| input_stream_ | the stream to read from |
Definition at line 36 of file instream_profile_guesser.cpp.
References read_profiles().
| BubbleProfiler::Instream_profile_guesser::Instream_profile_guesser | ( | std::istream & | input_stream_, |
| const std::string & | delimiters_ | ||
| ) |
reads a profile guess from the given input stream
| input_stream_ | the stream to read from |
| delimiters_ | a set of characters to use as field delimiters |
Definition at line 42 of file instream_profile_guesser.cpp.
References read_profiles().
| BubbleProfiler::Instream_profile_guesser::Instream_profile_guesser | ( | std::istream & | input_stream_, |
| const std::string & | delimiters_, | ||
| const std::string & | line_comment_start_ | ||
| ) |
reads a profile guess from the given input stream
| input_stream_ | the stream to read from |
| delimiters_ | a set of characters to use as field delimiters |
| line_comment_start_ | characters indicating the start of a comment |
Definition at line 49 of file instream_profile_guesser.cpp.
References read_profiles().
|
virtualdefault |
|
default |
|
default |
| Eigen::MatrixXd BubbleProfiler::Instream_profile_guesser::get_field_values | ( | ) | const |
returns the values of the fields read from the stream
Definition at line 213 of file instream_profile_guesser.cpp.
References field_values, n_fields, and spatial_grid.
Referenced by get_number_of_fields(), and get_profile_guess().
|
inline |
returns the number of fields for which values were read
Definition at line 84 of file instream_profile_guesser.hpp.
References field_values, get_field_values(), and get_spatial_grid().
|
inline |
returns the number of grid points read from the stream
Definition at line 78 of file instream_profile_guesser.hpp.
References spatial_grid.
|
overridevirtual |
Calculate an initial guess for the bubble profile.
The calculated profiles should be evaluated at the requested number of points, n_grid_points
, between the minimum and maximum radial coordinate values, domain_start
and domain_end
, respectively. If a negative value is given for either of these, appropriate values are to be guessed based on the given potential and vacuum location.
| potential | the potential for which the profiles are to be computed |
| true_vacuum | the location of the true vacuum in field space |
| n_dimensions | the number of space-time dimensions |
| domain_start | the minimum value of the radial coordinate at which the profiles are evaluated. If negative, the function should guess an appropriate value. |
| domain_end | the maximum value of the radial coordinate at which the profiles are evaluated. If negative, the function should guess an appropriate value. |
| initial_step_size | the initial step size to be used in constructing the discretized solution |
| interpolation_points_fraction | fraction of grid points to use for interpolation |
Implements BubbleProfiler::Profile_guesser.
Definition at line 144 of file instream_profile_guesser.cpp.
References BubbleProfiler::Abs(), get_field_values(), BubbleProfiler::Potential::get_number_of_fields(), get_spatial_grid(), has_n_fields, n_fields, BubbleProfiler::Field_profiles::set_number_of_dimensions(), spatial_grid, and tolerance.
| Eigen::VectorXd BubbleProfiler::Instream_profile_guesser::get_spatial_grid | ( | ) | const |
returns the values of the spacetime coordinate read from the stream
Definition at line 206 of file instream_profile_guesser.cpp.
References spatial_grid.
Referenced by get_number_of_fields(), and get_profile_guess().
|
default |
|
default |
|
private |
Definition at line 106 of file instream_profile_guesser.cpp.
References delimiters, field_values, has_n_fields, n_fields, and spatial_grid.
Referenced by read_profiles().
| void BubbleProfiler::Instream_profile_guesser::read_profiles | ( | std::istream & | input_stream | ) |
read a profile guess from the given stream
| input_stream | the stream to read from |
Definition at line 65 of file instream_profile_guesser.cpp.
References has_n_fields, BubbleProfiler::logging::Basic_logger::log_message(), logger, n_fields, process_line(), reset(), spatial_grid, strip_comments(), and BubbleProfiler::logging::Trace.
Referenced by Instream_profile_guesser(), and set_line_comment().
|
private |
Definition at line 58 of file instream_profile_guesser.cpp.
References field_values, has_n_fields, and spatial_grid.
Referenced by read_profiles().
|
inline |
set the characters to use as delimiters between fields
| d | a string containing the characters to be used as delimiters |
Definition at line 115 of file instream_profile_guesser.hpp.
References delimiters.
|
inline |
sets the character sequence used to indicate a comment
Lines in the provided input stream may contain a sequence of one or more characters indicating a comment. All content in the line following a comment sequence is ignored by the profile guesser.
| c | the string used to indicate the start of a comment |
Definition at line 127 of file instream_profile_guesser.hpp.
References line_comment_start, and read_profiles().
|
inline |
sets the spatial tolerance used when comparing grid points
When using the read-in data to construct a set of guessed profiles, the boundaries of the stored domain must agree to within the specified tolerance with the values of domain_start
and domain_end
provided to get_profile_guess() .
| t | the tolerance to be used |
Definition at line 109 of file instream_profile_guesser.hpp.
References tolerance.
|
private |
Definition at line 98 of file instream_profile_guesser.cpp.
References line_comment_start.
Referenced by read_profiles().
|
private |
Definition at line 136 of file instream_profile_guesser.hpp.
Referenced by process_line(), and set_delimiters().
|
private |
Definition at line 142 of file instream_profile_guesser.hpp.
Referenced by get_field_values(), get_number_of_fields(), process_line(), and reset().
|
private |
Definition at line 138 of file instream_profile_guesser.hpp.
Referenced by get_profile_guess(), process_line(), read_profiles(), and reset().
|
private |
Definition at line 137 of file instream_profile_guesser.hpp.
Referenced by set_line_comment(), and strip_comments().
|
private |
Definition at line 143 of file instream_profile_guesser.hpp.
Referenced by read_profiles().
|
private |
Definition at line 139 of file instream_profile_guesser.hpp.
Referenced by get_field_values(), get_profile_guess(), process_line(), and read_profiles().
|
private |
Definition at line 141 of file instream_profile_guesser.hpp.
Referenced by get_field_values(), get_number_of_grid_points(), get_profile_guess(), get_spatial_grid(), process_line(), read_profiles(), and reset().
|
private |
Definition at line 140 of file instream_profile_guesser.hpp.
Referenced by get_profile_guess(), and set_spatial_tolerance().


2022-01-28
Most interviews have a step called "The Sell". As the name suggests, the purpose is to sell the candidate on the opportunity — to convince them to join. Sometimes there are multiple sell rounds depending on the difficulty or seniority of the role. I do sells. But I also have a less common tactic and it's extremely effective.
It's called The Unsell Email.
Dun, dun, dun. Pan to the audience gasping.
Once I decide to extend an offer to someone, I send an email with ~10 bullet points that outline the gnarliest things they will discover about the role, company, and culture in their first 90 days on the job. Things that are difficult to get from the interview process (and frankly, most don't want to share). I made this example to help illustrate what I mean.
Why on Earth would you do that?!
I want them to really understand what they're getting into — the good, the bad, and the ugly. It seems counterintuitive and all my recruiting partners hate it.
Kevin! We just went through all that effort to get them to the finish line, why would you ruin it like that??
But it is one of the most effective tactics I have developed for hiring the right person for the right role.
How did this come about?
There was one role on my team that was a revolving door. Every person in that role was excellent in the interview process but red flags would appear within the first six months on the job. There was nothing wrong with the person — they were good product managers. But they realized they didn't want that specific role or that they were a different type of product manager. It took several mishires and a lot of reflection to understand what happened [1].
So I devised The Unsell Email to ensure the next person would understand the legacy baggage behind the role, the reasons others left the role, the challenges they will need to overcome, and the type of person who will enjoy it the most.
Holy cow! Doesn't that scare everyone off?
I lose about 30% of the candidates I send this to, and that's at the offer stage. It's painful. But way less painful than the year you would otherwise spend onboarding someone, only to have them leave [2, 3]. The people who make it past The Unsell Email are superb. They thrive in the role. They come in eyes wide open, hungry to tackle the challenges we discussed, confident in their fit for the role, and unafraid to ask difficult questions because they know I'll be honest with them. It leads to an incredibly powerful starting point for a relationship.
Should everyone send an Unsell Email?
No. This is not a silver bullet (nothing is). You need to figure out whether there are notable issues that repeatably surprise candidates after they join that impact tenure. If you happen to hire people who stay on your team for multiple years, happy as a duck, don't change a thing [4]. But if the idea of high attrition roles resonates, give it a shot.
[1] I spoke with everyone who left a role on one of my teams (regardless of whether I managed them directly or not). There were a few common themes that have informed the contents of my Unsell Emails.
[2] I realize that some people will feel like they don't have this luxury. They already feel like they're fighting to get every single new hire. That's fair. But the point you are missing is that it is still way more expensive to hire someone who leaves within a year than to wait for the right person. The time and effort that goes into onboarding (not to mention the impact on team morale) is incredibly large.
[3] You may be thinking that this sort of revolving door means that there is something wrong with the team or role. That the person isn't the problem, the environment is. And you'd be right! It's a system. So you likely need to change multiple parts.
[4] An interesting point to note is that people staying on a team for a long time isn't always a good sign. In the worst case, it could mean a devious manager that makes internal mobility very difficult. Hopefully your company has checks and balances in place for those types of issues.


Odo Draggable
A class for versatile, performant draggability.
Support
IE9+.
Dependencies
Odo Pointer, Odo Device, Odo Helpers, Tiny Emitter.
Odo Pointer also expects Object.assign
from ES6 to be available.
X Axis
Setup
var x = new OdoDraggable(document.getElementById('draggable-x'), {
axis: 'x'
});
Position
Set the position of the draggable element. The parameters are percentages, so 50
equates to 50%
.
// Set to 50%
x.setPosition(50, 50);
// Set to 0
x.setPosition(0, 0);
The position of the draggable element can be retrieved with getPosition
. Optionally specify the first parameter as true
to get the percentage coordinates instead of pixels.
x.setPosition(50, 50);
// Get pixels
x.getPosition(); // Coordinate {x: 480, y: 0}
// Get percentages
x.getPosition(true);// Coordinate {x: 50, y: 0}
Limits
You can also sets limits
on the draggable instance, which restricts it to given boundaries. The only parameter is a Rect
. Rect
s have a left
, top
, width
, and height
and can be accessed from the math
property of OdoHelpers
.
In this example, the draggable stops at the edges of the .container
element. Since the draggable is centered within the container and the draggable starts at 0, 0
, the left
limit needs to be negative half the leftover space. The minimum x position is the left
value. The maximum x is the left + width
. The width in this case, is the leftover space (20% because draggable is 80%). The y axis works the same way, but in this example, the top
and height
parameters don't matter because this draggable is restricted to the x
axis.
var math = OdoHelpers.math;
var draggableWidth = x.element.offsetWidth;
var containerWidth = container.offsetWidth;
var left = (containerWidth - draggableWidth) / -2;
var top = 0;
var width = containerWidth - draggableWidth;
var height = 0;
var rect = new OdoHelpers.Rect(left, top, width, height);
x.setLimits(rect);
Y Axis
Heads up! Any draggable element on the Y axis requires its parent to have a height to work with the fallback left
and top
positioning.
Setup
var y = new OdoDraggable(document.getElementById('draggable-y'), {
axis: 'y'
});
Disabling
The draggable instance can be disabled (and re-enabled). The carousel, for example, disables dragging while its animating.
// Disable draggable
y.isEnabled = false;
// Enable draggable.
y.isEnabled = true;
Both Axis
Just like the Y axis note above, the element which contains the draggable element must have a height, otherwise the top
percentage value is not relative to anything and is calculated as zero.
Setup
var xy = new OdoDraggable(document.getElementById('draggable-xy'), {
axis: 'xy'
});
Friction
You can make a draggable instance act like it is resisting the user's touch by setting a friction value. A value of 1
represents no friction; 100% of the user's movements are translated into movement of the draggable instance. Use a value less than 1, like 0.4
, and the draggable element does not move completely in sync with your pointing device.
// Give it friction, like the edge slide of a carousel.
xy.friction = 0.4;
// Remove friction.
xy.friction = 1;
Throwable
Give it a toss.
By default, draggable instances are not throwable. Use the isThrowable
option to enable it. You must also set a boundary for the throwable element, otherwise you can throw it off the screen!
// Create new draggable
var throwable = new OdoDraggable(document.getElementById('draggable-throw-me'), {
axis: 'xy',
isThrowable: true
});
var element = throwable.element;
var parent = element.parentNode;
var draggableWidth = element.offsetWidth;
var draggableHeight = element.offsetHeight;
var containerWidth = parent.offsetWidth;
var containerHeight = parent.offsetHeight;
var left = 0;
var top = 0;
var width = containerWidth - draggableWidth;
var height = containerHeight - draggableHeight;
var rect = new OdoHelpers.Rect(left, top, width, height);
// Set a boundary for the draggable so that it won't be thrown outside of its parent.
throwable.setLimits(rect);
More like a carousel
This example uses the same markup and css as the Odo Carousel, except the .carousel-element
's width is set to fit the exact amount of slides (width is 500%
instead of 2000%
).
Here we follow the same steps. Set the isThrowable
option and then set limits.
var freescroll = new OdoDraggable(document.getElementById('draggable-carousel'), {
axis: 'x',
isThrowable: true
});
var element = freescroll.element;
var parent = element.parentNode;
var draggableWidth = element.offsetWidth;
var containerWidth = parent.offsetWidth;
var left = -draggableWidth + containerWidth;
var top = 0;
var width = -left;
var height = 0;
var rect = new OdoHelpers.Rect(left, top, width, height);
freescroll.setLimits(rect);
Events
Draggable emits three events: START
, MOVE
, and END
.
instance.on(OdoDraggable.EventType.START, function(pointerEvent) {
console.log('drag started', pointerEvent);
});
instance.on(OdoDraggable.EventType.MOVE, function (pointerEvent) {
console.log('drag move:', pointerEvent);
});
instance.on(OdoDraggable.EventType.END, function (pointerEvent) {
console.log('drag ended:', pointerEvent);
});
PointerEvent
Here is the data contained within a drag end event from the first draggable on this page. There are a few simple properties here that have already determined the drag movement compared to its axis.
{
axisDirection: "left",
currentVelocity: {
x: 1.88,
y: 0.14
}
delta: {
x: -32,
y: 0
},
deltaTime: 203,
didMoveOnAxis: true,
direction: "left", // OdoPointer.Direction.LEFT
distance: 32,
end: {
x: -32,
y: 0
},
isDirectionOnAxis: true,
position: {
percent: {
x: -3.2160804020100504,
y: 0
},
pixel: {
x: -32,
y: 0
},
},
start: {
x: 0,
y: 0
},
target: div,
type: "ododraggable:end", // OdoDraggable.EventType.END
velocity: {
x: 0.15763546798029557,
y: 0
}
}
Disposal
To properly destroy a draggable instance, you must call the dispose
method. This removes event listeners, nullifies DOM references, and performs other clean up.
xy.dispose();
Options
OdoDraggable.Defaults = {
// Draggable axis.
axis: OdoPointer.Axis.X,
// Amplifies throw velocity by this value. Higher values make the throwable
// travel farther and faster.
amplifier: 24,
// Once the velocity has gone below this threshold, throwing stops.
velocityStop: 0.08,
// On each throw frame, the velocity is multiplied by this friction value.
// It must be less than 1. Higher values let the throwable slide farther and longer.
throwFriction: 0.94,
// Whether the draggable will keep its movement momentum after the user releases.
isThrowable: false
}


Eric.Stermer( );
Home
About
Projects
My Projects
Ruby on Rails
Mongo, Express, Angular, and Node
JavaScript, jQuery, AJAX, DOM Manipulation
Algorithms, Practice, and Fiddles


There is a problem
What happens next?
Your submission will now be evaluated by our team.
What should I include?
Accepted file types
It must be less than 32MB in size.
Feedback
You can give feedback to help us improve this service. This should take no more than 2 minutes.
Give us feedback


The latest advice from the British F.O. is to avoid all but essential travel to most parts of the country.
There are violent demonstrations taking place in many towns and cities and there have been attacks on foreign nationals including in Hurghada.
See the FCO website for more details.
April 2016.
Very hot summers; mild winters with little rain. The prevailing winds in the Red Sea are northerly, while along the Mediterranean coast there are daily alternating land and sea breezes. Occasionally the khamsin, a hot dry wind, blows off the land laden with dust and sand reducing visibility.
For links to free global weather information, forecast services and extreme weather information see the Noonsite Weather Page
Canal: Ismailia
Mediterranean: Alexandria * , Port Said *
Red Sea: El Gouna * , Hurghada * , Port Ghalib * , Safaga * , Sharm el Sheik * , Suez * , Taba Heights * , Wadi-El-Dome Marina
* indicates port of entry


font-size:100px and color:#f00 are both displayed in the rule view but in reality they are overwritten by user agent styles.
font-size:100px
color:#f00


An application of modern Machine Learning methods to the problem of classifying human movement. Data is collected from a hip-mounted Galaxy S2 smartphone, utilizing the gyroscope and GPS sensors of the device. Data is pre-processed and explored with a variety of linear and non-linear models. Our final model (a Linear Support Vector Classifier) achieves a roughly 96% classification accuracy. Presented at SIAM conference 2015.
A statistical paper that makes updates to the data set of global CO2 emissions presented by Bjorn Lomborg (environmental economist) in his popular book “The Skeptical Environmentalist”and reflects on the validity of his predictions in current times. Correlations are presented with CO2 emissions and over a half-dozen socio-economic indicators, as well as a model to predict future emissions using a Holt Winters smoothing
An exploration of a novel method to increase robustness (resistance to outliers) in Linear Regression using adjustment of residual points as classified by Cook’s Distance. Adjustment level is a parameter (a function of outlier magnitude). Various functions are discussed, including Bayesian methods and domain knowledge approaches. Presented at UIUC Economics Research Symposium.
An examination of the correlation between ratings of college professors from the website RateMyProfessor.com and GPA data across three major U.S public universities. We attempt to examine whether student perceptions (and consequently, class registration) are affected mainly by the average grade they received in the class. Submitted to 2015 SAS data convention
Research extending the research done regarding the “gender wage gap” (statistically significant different between male and female employee salaries in industry) into academia using Bayesian Hierarchical Modeling. All academic employees from the state of Illinois were examined and data was separated by position, school and gender
A lecture/talk series given to a crowd of ~40 students covering the statistical methods utilized in Financial Engineering and Econometrics. Topics covered include scraping of real-time financial data, stocks and bonds markets, statistical arbitrage, options pricing models, hedging, ARMA models and forecasting
A paper outlining modern Distributed Processing Systems and Methods with Their Effect on Data Processing Times (in-development)
A collection of my literary works, currentely only contains poetry (in-development)
Advanced proficiency with designing and converting statistical algorithms to run in parallel using software such as PySpark, PL/R and PL/Python
Advanced proficiency with modern machine-learning methods such as regression, tree-based models, clustering, and neural networks. Specialization in time-ordered, text and vision data.
End-to-end handling of large-scale data, including data warehousing and analysis. Experiance with data ranging in Petabytes of size. Work experiance using full big data tech stack (Hive, Hadoop, SQL, Spark) with a specialization in time-series modeling using PySpark.
Design and implementation of forecasting models, with a focus in time-series and machine learning models.
Construction of user-friendly web applications to display live analytic results and data visualizations using Tableu, R Shiny and HTML/CSS/JS.


You are the prototype!
Click me if you want to live your life through photos.
Can sign up for just about anything.
Ooooh Astrology!


Security Advisory: Multiple Critical Vulnerabilities in Hola Overlay Network Client
By slipstream/RoL, Donncha O'Cearbhaill, joepie91 (Sven Slootweg), IceMans/RoL, infodox, pathfinder/braenaru, APT1337 and spoonzy, LeShadow.
A public information website is available at adios-hola.org.
Overview
Hola 'Unblocker'/Hola 'Better Internet'/Hola 'VPN' is a commercial service offered on a 'freemium' basis that advertises itself variously as a censorship-evasion and privacy/anonymity-enhancement tool. The number of users is difficult to quantify objectively, but estimates range from 8 million users at the low end, to 42 million users as claimed on the Hola website.
The Hola Unblocker Windows client, Firefox addon, Chrome extension and Android application contain multiple vulnerabilities which allow a remote or local attacker to gain code execution and potentially escalate privileges on a user's system. Additional design flaws allow a Hola user to be tracked across the internet via a persistent ID. Furthermore, as Hola users - wittingly, or otherwise - act as exit-nodes for the overlay network, each is capable of acting as a Man-in-the-Middle for other users of the free or premium Hola network, or its commercial 'bandwidth' service, Luminati, and thereby compromising the privacy and anonymity of their browsing and exposing them to further attacks.
Issues
The Hola software creates an API server listening on port 6853 of the loopback interface (<IP_ADDRESS>). The server exposes JSON and other endpoints over HTTP, setting an Access-Control-Allow-Origin: *
CORS header, which allows a remote website to read the responses for requests to these endpoints, and thereby enumerate sensitive information, as well as affecting the security and integrity of the system.
1. Local File Read
The /file_read.json
endpoint allows a remote website to read arbitrary files on the local system via path traversal. The files can be read up to the first null byte.
2. Information Disclosure
The
/callback.json
endpoint provides uniquely identifying information about the users local Hola install. An attacker could exploit this issue to persistently track a Hola user across the Internet. When nodes are acting as exit nodes, they also obtain implicit unique identifiers for the traffic they relay, compounding the privacy concerns.The
/procinfo/ps
endpoint provides internal debugging information of the Hola software. With certain query arguments, the information provided contains pointers to functions that can be used to defeat Address Space Layout Randomization (ASLR). This would make it easier for an attacker to exploit memory corruption, such as a buffer overflow and other vulnerabilities, potentially allowing the attacker to achieve remote code execution in the context of the service in conjunction with other bugs.
As it happens, remote code execution is achieved without the use of the information leak.
3. Remote Code Execution
The
/vlc_mv.json
endpoint allows files on the local system to be moved. The/vlc_start.json
endpoint allows for the bundled VLC executable to be run with full controlled arguments. Together these vulnerabilities can be utilized to gain arbitrary remote code execution. Details will be fully disclosed at the discretion of the researchers when some time has elapsed for users to remove the software.If an attacker can perform a Man-in-the-Middle attack against a target running the Hola client on Windows - either as a network adversary, ISP, intelligence agency or another Hola client acting as an exit node -- they can create a connection seeming to originate from the hola.org or client.hola.org hosts to the local websocket port.
This web socket connection can be used to download a remote file and execute it with either the privileges of the logged in user, or as SYSTEM, depending on what version of the client is installed (browser extension or Windows client). The Android app is also affected by this vulnerability, which can be used to remotely execute arbitrary code as the Hola app user. An XSS vulnerability on the hola.org or client.hola.org domains could also allow an attacker to exploit this issue to remotely execute arbitrary code.
4. Privilege Escalation
The Hola software creates a websocket server listening on port 6863 of the loopback interface. The software checks the Origin header against some whitelisted values, but this can be spoofed if RCE has been achieved. The Windows client runs its service (hola_svc.exe
) as SYSTEM. The websocket server allows access to commands implemented within the service such as wget
, which downloads a file from a remote server, and exec
which executes a file with the same privileges as the service. Therefore, the websocket can be abused to escalate privileges if the Windows client is installed.
Affected Versions
Hola Engine for Windows and the Hola Firefox addon on Windows are vulnerable to the first described remote code execution vulnerability. Hola Engine for Windows is also vulnerable to the privilege escalation issue. These versions along with the "Chrome app on Windows" .crx extension (hola_chrome_ext_dll_1.7.333.crx
) distributed on the Hola website and the Hola Android app are vulnerable to the second described remote code execution issue, the information disclosure and local file read issues.
Solution
No solution yet exists. Uninstallation of the Hola service, Firefox addon and Android application will prevent exploitation of these issues. To fully uninstall the Firefox addon, `%localappdata%\Hola
must also be deleted. The researchers cannot sanction any mitigations except to remove this software definitively from any affected devices. To fully uninstall Hola Engine for Windows, in some cases, deletion of C:\Program Files\Hola
is also needed.


This program is written for 32-Bit Microsoft Windows. The
source code
is provided in Delphi 5.0.
OPERATION OF 'TIME'
The 'TIME' example program contains only one window. Its reads
and displays
the date/time from the DS1994's internal real time clock/calendar.
README.TXT - This file.
TIMEDL32.DPR - Delphi project file.
TIMEDL32.DOF - Delphi options.
TIMEDL32.RES - Resource module.
TIMEUN32.PAS - Main pascal source file for the window.
iBTMEXPW.PAS - Pascal source file for the TMEX calls.
TIMEUN32.DFM - Delphi form file.
TIMEDL32.EXE - Program executable.


--- layout: post status: publish published: true title: '"Lil'' Smargy" - Podcast Episode 6.14' author: display_name: Corey Fineran login: Corey email<EMAIL_ADDRESS>url: '' author_login: Corey author_email<EMAIL_ADDRESS>wordpress_id: 9940 wordpress_url: http://ivyenvy.com/?p=9940 date: '2014-05-18 22:42:56 +0000' date_gmt: '2014-05-19 03:42:56 +0000' categories: - Podcast tags: - Jeff Samardzija - Javier Baez - Edwin Jackson - Kris Bryant - Tyler Skulina comments: [] ---
Andy, Corey and Kurt convene for the 252nd episode of the Ivy Envy Podcast.
Our Iowa Cubs Game Meet-up is this upcoming Saturday. If you'd like to meet up with us for beers before the game and join us in the suite for the game, check out the details at IvyEnvy.com/Iowa. Also, the following Saturday (May 31st) is the Wasting Away in Wrigleyville Pub Crawl. If you'd like to join us for a tour of Wrigley Field and hit 9 bars in 9 innings, you can find the details at IvyEnvy.com/Wrigleyville.
The Cubs minor league system has another no-hitter, this time from Kane County. Jorge Soler suffers another injury and we talk about his injury struggles. Edwin Jackson has put together a nice little string of starts. Kris Bryant continues to tear it up at AA Tennessee and we discuss the timing of his promotion to AAA Iowa.
We also look at the Cubs run differential in 2014. It indicates the Cubs should be very close to a .500 team, yet they are far from it. So what's the reason for this and can we expect things to level out over the course of the season.
We answer listener questions, one of which leads into Kurt's Scary Stat of the Week. After the break, we read a few Internet Muscle Flexes and pick a winner for this week's photo caption contest.
We would really appreciate it if you would take 5 minutes to fill out this listener survey.
If you'd like to support the Ivy Envy podcast for as little as 25 cents per episode, check out our Patreon campaign.


Site – St Johns, Study – 021-22045 - Central Monitoring
Site – All, Study – 025-250901 - Deviation Review
Site – Appolo, Study – 025-25081 - Physical Visit
Study – 021-22045 – Status changed to "RED"
Study – 025-22091 – Status changed to "YELLOW"
Site – St Johns, Status Changed to "RED" – Document mismatch
Site – Appolo, Status Changed to "Yellow" – CRF not updated for patient x visit
New Update to Protocol
Site: Appollo - Training document not found
Site – St Johns, EC letter – overdue – 2 days
Site – St Johns, Monitoring due 8th May 2012
Site – Appollo, Monitoring due 9th May 2012
Site – St Johns, Investigator Payment overdue since March 2012
Site – St Johns, Material shortage
Site – St Johns, Subject – 001, ICF not found


We're excited to announce the release of MXNet 1.3.0! This release includes Clojure bindings, Gluon package enhancements, ONNX export, TensorRT integration, and more!
Learn MoreCheck out our quick overview of how to use Gluon, the imperative interface of MXNet.
Learn MoreContributors and customers of MXNet post about how-tos, guides, examples, and stories about their MXNet experiences.
Learn MoreOne-click pre-trained models, included in Gluon. Fast implementations of many state-of-the-art models, for plug-and-play effortless use.
Explore projects from simple demos, examples, tutorials to state-of-the-art research.
These tutorials introduce a few fundamental concepts in deep learning and how to implement them in MXNet.


Basic component that demo couple things:
- Define component using custom markup
- Render template (static in this case)
- CSS is writen using seperate scss file
- Styles are scoped which prevents leaking style out
- Hard to use scoped styles (random class names) to style internals of this component
- Exposes [part="..."] attribute to ease stylling from outside
Dynamicly added nodes
Clock Component
Counter component
- Property exposed by attribute
- Reacts to attribute changes
- Reacts to property changes
- Reflec property value to attribute
- Expose method as public api
State component
- Reacts to changes in simple state store


{include file="../application/index/view/public/_header.html"}
KOOA - Forgotten Password
{$Think.lang.phone}:
{$Think.lang.verify_code}:
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.


This is turning into a bit of a FAQ, but the answer is too big to fit in the margin we use for those.
There are a number of reasons why there are no plans in sight that will grow SSL support in Varnish.
First, I have yet to see a SSL library where the source code is not a nightmare.
As I am writing this, the varnish source-code tree contains 82.595 lines of .c and .h files, including JEmalloc (12.236 lines) and Zlib (12.344 lines).
OpenSSL, as imported into FreeBSD, is 340.722 lines of code, nine times larger than the Varnish source code, 27 times larger than each of Zlib or JEmalloc.
This should give you some indication of how insanely complex the canonical implementation of SSL is.
Second, it is not exactly the best source-code in the world. Even if I have no idea what it does, there are many aspect of it that scares me.
Take this example in a comment, randomly found in s3-srvr.c:
/* Throw away what we have done so far in the current handshake,
* which will now be aborted. (A full SSL_clear would be too much.)
* I hope that tmp.dh is the only thing that may need to be cleared
* when a handshake is not completed ... */
I hope they know what they are doing, but this comment doesn't exactly carry that point home, does it ?
But let us assume that a good SSL library can be found, what would Varnish do with it ?
We would terminate SSL sessions, and we would burn CPU cycles doing that. You can kiss the highly optimized delivery path in Varnish goodby for SSL, we cannot simply tell the kernel to put the bytes on the socket, rather, we have to corkscrew the data through the SSL library and then write it to the socket.
Will that be significantly different, performance wise, from running a SSL proxy in separate process ?
No, it will not, because the way varnish would have to do it would be to ... start a separate process to do the SSL handling.
There is no other way we can guarantee that secret krypto-bits do not leak anywhere they should not, than by fencing in the code that deals with them in a child process, so the bulk of varnish never gets anywhere near the certificates, not even during a core-dump.
Would I be able to write a better stand-alone SSL proxy process than the many which already exists ?
Probably not, unless I also write my own SSL implementation library, including support for hardware crypto engines and the works.
That is not one of the things I dreamt about doing as a kid and if I dream about it now I call it a nightmare.
So the balance sheet, as far as I can see it, lists "It would be a bit easier to configure" on the plus side, and everything else piles up on the minus side, making it a huge waste of time and effort to even think about it..
Poul-Henning, 2011-02-15


Quirkey
Code
Quirkey
Portfolio
Selected work.
Quirkey
Log
Up-to-the-minute updates from the interweb.
Quirkey
Blog
Articles, Tutorials, etc.
Quirkey
.com
The Lobby; General Information.
^
Here's a selection of the open-source projects I've worked on. Hosted at github.
Most popular (
watchers
)
Loading
See also
Loading


Links Below
Home
sprint-1 cultural
sprint-1 technical
sprint-2 cultural
sprint-2 technical
sprint-3 cultural
sprint-3 technical
sprint-4 cultural
sprint-4 technical
sprint-5 cultural
sprint-5 games
Chaz Ratima
That's me!
This was taken on android LOL
This page is my home page?
There's nothing here... or is there...


.NET Foundation April 2019 Update
Here's the April edition of the .NET Foundation newsletter. Every month, we'll give you a quick overview of the .NET Open Source landscape, including top project news, events, community links and more.
This month's newsletter includes:
- Get ready for Build 2019!
- Visual Studio 2019 Launch Recap
- Visual Studio Release: Announcement Post Rollup
- .NET Product Team Announcement Posts
- .NET Foundation Project Updates
- New .NET Foundation Projects
- Meetups
As always, these are available both on our blog and via e-mail: Sign up to get the .NET Foundation Update via e-mail
Get ready for Build 2019!
Build 2019 is coming up May 6-8. Just about everything will be live streamed, so if you're not going to be there in person you can watch online. Build 2019: Save The Data And if you're there in person, stop by the .NET Open Source booth to say hi! We'll have .NET Foundation project leaders and board members there to say hi and talk about .NET open source, as well as some pretty great stickers and t-shirts... Come say hi!
Visual Studio 2019 Launch Recap
Visual Studio 2019 was released on April 2! Lots of new announcement posts, videos you may have missed and some ongoing local launch events to get in on.
Launch Event Videos
First, all the goodies from the Visual Studio 2019 Launch Event:
Visual Studio 2019 and Visual Studio 2019 for Mac are now available for download on https://visualstudio.com/downloads
See what makes .NET Core 3.0 such a big deal for .NET developers no matter what kind of apps you're building. See the productivity and performance improvements and innovative new ways to build and enhance your .NET applications.
See Visual Studio 2019 in action building modern, cross-platform web apps using ASP.NET Core 3.0 Preview.Visual Studio 2019 in action building modern, cross-platform web apps using ASP.NET Core 3.0 Preview.
Learn how to program the next internet connected toothbrush or industrial IoT device with IoT and Visual Studio 2019.
Who needs fancy data scientists? See how you can infuse custom AI/ML into any .NET application with ease and how Visual Studio 2019 improves your debugging & development experience.
What’s new in Visual Studio for Mac 2019? Find out all the new features and capabilities in the IDE and even get a glance of our product roadmap.
Visual Studio 2019 Launch Events Near You
There are still dozens of worldwide local events running through the end of June. Find one near you here.
Visual Studio Release: Announcement Post Rollup
Here are some of the top posts during the Visual Studio 2019 Launch:
- Visual Studio 2019: Code faster. Work smarter. Create the future.
- Visual Studio 2019 .NET productivity
- ASP.NET Core updates in .NET Core 3.0 Preview 4
- Web and Azure Tool Updates in Visual Studio 2019
- Announcing Entity Framework Core 3.0 Preview 4
- Announcing ML.NET 1.0 RC – Machine Learning for .NET
- Announcing the .NET Framework 4.8
- Announcing .NET Core 3 Preview 4
- Visual Studio 2019 for Mac is now available
.NET Product Team Announcement Posts
Lots of exciting new announcements during the rest of the month, too:
- Using .NET and Docker Together – DockerCon 2019 Update
- Blazor now in official preview!
- Updated Razor support in Visual Studio Code, now with Blazor support
- .NET Core Workers in Azure Container Instances
- .NET Core Workers as Windows Services
.NET Foundation Project Updates
BenchmarkDotNet v0.11.5 has been released!
Power plan management, better Environment Variables API and outlier messages, support modern CPU architecture names, tons of improvements and bug fixes!
DNN RC
Version 9.3.2 will release soon... the RC posted 12 days ago.
Orleans
Orleans 2.3.1 includes a few improvements, one bug fix, and batch stream API added back.
PeachPie: Progress toward full PHP compatibility
Great post from the PeachPie project on the impact the open source community has had on the project, the impact on support for popular PHP applications and frameworks (like WooCommerce, Laravel, Symfony, Drupal and more), and a look at what's next.
AutoMapper 8.1.0 Released
AutoMapper 8.1 adds a major new feature - attribute-based maps. Attribute maps let you easily declare maps on destination types when you have straightforward scenarios.
IdentityServer4 Roadmap
The IdentityServer team released a roadmap update, recapping recent releases and looking ahead to the 3.0 release (built against ASP.NET Core 3) and a 4.0 release to support ASP.NET Core 3.1 (LTS).
Using Microsoft Configuration Extensions with Steeltoe
David Dieruf explains how to choose between environment variables and external configuration servers, then explains why Steeltoe and Microsoft.Extensions are a great fit for external configuration.
Orchard Core Beta 3
Congrats to the Orchard team for releasing Orchard Core Beta 3. This release includes support for ASP.NET Core 2.2, GraphQL, support for all social authentication, decoupled and headless CMS support, and lots more.
Cake v0.33.0 released
This release sees a number of long waited features being added, including the ability to use a globbing pattern within the #load directive. As well as this, a number of improvements were included, and a number of bugs squashed. All of these are detailed below in the release notes.
Fifteen years of WiX
Wow! The WiX team is celebrating fifteen years! Congrats! Read these posts from Bob Arnson and Rob Mensching to hear about the history of this iconic .NET open source project.
Json.NET 12.0.2
A new release of Json.NET is out, adding several new features and a few bug fixes.
New .NET Foundation Projects
We're excited to welcome the following new .NET Foundation projects, with several more on the way this next month:
Introducing .NET for Apache® Spark™ Preview
At Spark + AI summit, the team announced .NET for Apache Spark. Spark is a popular open source distributed processing engine for analytics over large data sets. Spark can be used for processing batches of data, real-time streams, machine learning, and ad-hoc query.
.NET for Apache Spark is aimed at making Apache® Spark™ accessible to .NET developers across all Spark APIs. So far Spark has been accessible through Scala, Java, Python and R but not .NET.
Welcoming DotVVM to the .NET Foundation!
DotVVM is an MVVM framework for ASP.NET Core and OWIN. The DotVVM project has been under active development for almost 5 years.
In addition to providing some nice productivity features through MVVM development and controls, it also offers an incremental upgrade path for ASP.NET Web Forms applications through the DotVVM Adapter for ASP.NET WebForms.
Meetups
Our .NET Foundation sponsored .NET Meetup Pro groups are continuing to grow worldwide. Here are some quick stats:
- 291 Groups (+10 from the past month)
- 59 Countries
- 208K Members (+20K in the past 30 days!)
We've also started sending newsletters to Meetup organizers, including some links for some free swag for their groups. If your meetup hasn't joined yet, you can right here.
.
Connect with the .NET Foundation online
The .NET Foundation is on Facebook now. Please like our page! We’ll post regular updates and interesting things happening with .NET to share.
The .NET Foundation is also on YouTube. Watch community standups and design reviews as well as code-focused shows and interviews across our multiple playlists.
Remember to Subscribe!
Please sign up to get the .NET Foundation Update via e-mail. Don’t worry, we want to keep these short, interesting, and low-noise, so we won’t overload your e-mail.


--- layout: post title: Building a Technical Blog Site With Jekyll And Emacs date: 2014-12-13 summary: This article shows how I set up my own blog site on GitHub, using the org-mode of Emacs for editing articles and HPSTR Jekyll Theme for displaying them. tags: [Jekyll, Emacs, org-mode, Blogging] ---
Jekyll is a static web site generator written in Ruby. It can transform various text markups, using a templating language, into static html. The resulting site can be served by almost any web server without requiring additional components such as php. Jekyll is the tool used to produce Github's pages.
Org-mode of Emacs is very convenient for all-around editing, from simple note taking to managing one's tasks and to writing fully-fledged articles (which can later be exported to LaTeX, for instance). Hence it feels just natural to write blogs and articles in org-mode (rather than in markdown or textile, which are the formats supported by Jekyll).
This article shows how I set up my own blog site on GitHub, using the org-mode of Emacs for editing articles and HPSTR Jekyll Theme for displaying them.
username.github.io
(where username
is your GitHub
username). Alternatively, you can fork the repo of my website.
sudo apt-get install git git clone https://github.com/username/username.github.io
sudo apt-get install build-essential nodejs sudo apt-get install ruby ruby-dev sudo gem install bundler
cd username.github.io/ bundle install
bundle exec jekyll serve
Now open in browser: http://localhost:4000
Edit _config.yml
to personalize your site. Most of the variables
found here are used in the .html files found in _includes/
(if you
need to add or remove anything).
Check out the sample posts in _posts/
to see examples for pulling
in large feature images, assigning categories and tags, and other
YAML data.
For more details about customizing the theme see: http://mmistakes.github.io/hpstr-jekyll-theme/theme-setup/
Since it is a git repository, we can commit our changes and push
them to GitHub with git push origin master
. In a few moments our
changes will be available online and we can access it at
http://username.github.io. However we should make sure first that
the URL of the site on _config.yml
is correct. It should be like
this: url: http://username.github.io
, not empty or url:
http://localhost:4000
, which are used for local development.
Here are a few tips that can be useful during development:
git push origin master
, you
have to give the github username and password. To avoid this, you
can modify the file .git/config
like this:
[remote "origin"] url =<EMAIL_ADDRESS>(adding username:password@ before github.com).
url:
in _config.yml
has to be different
for the online and local copy of the website, I have created a git
branch named local where I do the development and testing, and
then cherry-pick commits to the branch master and push them to
github.
push.sh
with a content like this:
#!/bin/bash ### get the parameter commit if [ "$1" = '' ] then echo "Usage: $0 <commit_id>" exit 1 fi commit=$1 ### transfer the commit to branch gh-pages git stash git checkout master git cherry-pick $commit ### push to github gh-pages git push origin master ### switch back to the local branch git checkout local git stash pop
Jekyll expects the posted articles to be on the directory _posts/
,
with a file name like yyyy-mm-dd-title-of-the-article.extension
.
They can be in markdown or textile format and jekyll will convert
them to HTML. But they can also be just plain HTML and jekyll will
not touch their content. The extension can be .markdown
(or
.md
), .textile
, or .html
.
In each case jekyll expects a front-matter in YAML format at the very top of each file, which looks like this:
--- layout: post title: Building a Technical Blog Site With Jekyll And Emacs date: 2014-12-13 summary: This article shows how I set up my own blog site on GitHub, using the org-mode of Emacs for editing articles and HPSTR Jekyll Theme for displaying them. tags: [Jekyll, Emacs, org-mode, Blogging] ---
It defines the layout (from directory _layouts/
) that will be used
to render the article and other variables that are used in the
templates.
To customize CSS, I added this line on assets/css/main.scss
:
@import "custom";
Then created the file _sass/_custom.scss
with a content like this:
Of course you will need to make your own customizations.
I have created the directory _org/_posts/
for the articles in
org-mode format. The filename of an article looks like this:
2014-12-13-building-technical-blog-site-with-jekyll-and-emacs.org
.
The content of the article looks like this:
#+TITLE: Building a Technical Blog Site With Jekyll And Emacs #+AUTHOR: Dashamir Hoxha #+EMAIL<EMAIL_ADDRESS>#+DATE: 2014-12-13 #+OPTIONS: H:3 num:t toc:t \n:nil @:t ::t |:t ^:nil -:t f:t *:t <:t #+OPTIONS: TeX:nil LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc # #+INFOJS_OPT: view:overview toc:t ltoc:t mouse:#aadddd buttons:0 path:js/org-info.js # #+STYLE: <link rel="stylesheet" type="text/css" href="css/org-info.css" /> #+BEGIN_HTML --- layout: post title: Building a Technical Blog Site With Jekyll And Emacs date: 2014-12-13 summary: This article shows how I set up my own blog site on GitHub, using the org-mode of Emacs for editing articles and HPSTR Jekyll Theme for displaying them. tags: [Jekyll, Emacs, org-mode, Blogging] --- # +END_HTML Jekyll is a static web site generator written in Ruby. It can transform various text markups, using a templating language, into static html. The resulting site can be served by almost any web server without requiring additional components such as php.
The top lines that start with #+TITLE:
, #+AUTHOR:
, etc. are not
displayed on the converted HTML document. So, the first thing that
is outputed is the YAML front matter, which is included between the
lines #+BEGIN_HTML
and #+END_HTML
:
--- layout: post title: Building a Technical Blog Site With Jekyll And Emacs date: 2014-12-13 summary: This article shows how I set up my own blog site on GitHub, using the org-mode of Emacs for editing articles and HPSTR Jekyll Theme for displaying them. tags: [Jekyll, Emacs, org-mode, Blogging] ---
We need to export (convert) org-mode articles to HTML format and to
place the HTML article in the directory _posts/
, so that it can be
found and processed by Jekyll. We do this by defining an org-mode
publish project in the file ~/.emacs, with a content like this:
The first project ("org-blog") defines how the org files are
published. :base-directory ~/username.github.io/_org/
is the
directory that will be searched recursively for .org
files. They
will be converted to html with :publishing-function
org-html-publish-to-html
and will be saved to
:publishing-directory ~/username.github.io/
with the same
directory structure.
The setting :body-only t
makes sure that only the body of the HTML
document will be exported (the rest of the webpage will be
constructed by jekyll according to the layout given in the config
section).
The second project ("org-static-blog") just copies anything else
from the _org/
directory to the main jekyll directory. They can be
images, css/js files, etc.
The project "blog" calls both of these publishing projects.
Now we can export the project with C-c C-e P x blog
.
Note: If you don't have org-mode version 8.0 or later (check it
with M-x org-version
), you should update it. You can do it like
this:
M-x package-list-packages
C-s org
<Enter>
[Install]
Let us look closer at the export settings on the org file:
#+OPTIONS: H:3 num:t toc:t \n:nil @:t ::t |:t ^:nil -:t f:t *:t <:t #+OPTIONS: TeX:nil LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc
We notice the option toc:t
which tells the export function to generate
a Table Of Content on the HTML file (to disable it use toc:nil
).
However there is a problem because the TOC is generated before anything else, even before the YAML config section. We don't want this because jekyll can process the config section of an HTML file only if it is at the top of the file.
But we can fix it with commands like this:
(sed -n -e '/^---$/,/^---$/p' file.html; sed -e '/^---$/,/^---$/d' file.html) > file.new mv file.new file.html
(More on it later.)
Now the TOC is displayed at the top of the article. However it is
possible to make it look more nice and professional. Add these lines
at the file _sass/_custom.scss
and you will see that they create
the splendid efect of a dynamic TOC:
Images in the org-mode file are included like this:
#+CAPTION: Tables and their relations. #+NAME: fig:db_diagram #+ATTR_LaTeX: width=13cm [[file:images/btranslator-project/db_diagram.png][file:images/btranslator-project/db_diagram.png] ]
On the final document it looks like this:
The image itself is initially placed on the directory
/_org/images/btranslator_project/
. When the export comand is
executed, besides converting to HTML it will also copy everything on
the directory /_org/
to the main directory of jekyll, so it will go
to /images/btranslator_project/
. However the source of the image on
the generated HTML file will look like this:
It is missing a slash (/) in front. It can be corrected after exporting with a command like this:
sed -e 's|="images/|="/images/|g' -i file.html
I have created a bash script for fixing the problems in the two
sections above (making sure that the YAML section is at the very
top of the generated HTML file, and correcting the URL of images).
It is in the file _org/post-process.sh
and looks like this:
To make sure that all the HTML posts are OK, I call it like this:
_org/post_process.sh _posts/*.html
It is also idempotent (runing it multiple times on the same HTML file will not break it).
My articles often include code examples, so I am interested in
displaying them prettily. I use the tags #+BEGIN_EXAMPLE
and
#+END_EXAMPLE
to display terminal commands and output, and
sometimes even for code, but it does not do syntax highlighting.

For other tips about code examples see also this: http://mmistakes.github.io/hpstr-jekyll-theme/code-highlighting-post/


## Mako can find this base file because it's in the settings.py -> MAKO_TEMPLATES_DIRS list <%inherit file="base_template.htm" /> <%block name="left_side">


We are building something that should make it easier for you to write better blog posts, and write more often.
Details on the other side
We will notify you as soon as we open our beta testing slots.


Bayesian Workflow Example (Strack RRR Analysis Replication)¶
[1]:
import arviz as az
import bambi as bmb
from glob import glob
import pandas as pd
from os.path import basename
[2]:
az.style.use('arviz-darkgrid')
In this Jupyter notebook, we do a Bayesian reanalysis of the data reported in the recent registered replication report (RRR) of a famous study by Strack, Martin & Stepper (1988). The original Strack et al. study tested a facial feedback hypothesis arguing that emotional responses are, in part, driven by facial expressions (rather than expressions always following from emotions). Strack and colleagues reported that participants rated cartoons as more funny when the participants held a pen in their teeth (unknowingly inducing a smile) than when they held a pen between their lips (unknowingly inducing a pout). The article has been cited over 1,400 times, and has been enormously influential in popularizing the view that affective experiences and outward expressions of affective experiences can both influence each other (instead of the relationship being a one-way street from experience to expression). In 2016, a Registered Replication Report led by Wagenmakers and colleagues attempted to replicate Study 1 from Strack, Martin, & Stepper (1988) in 17 independent experiments comprising over 2,500 participants. The RRR reported no evidence in support of the effect.
Here we use the Bambi model-building interface (itself just a wrapper for PyMC3 and Stan) to quickly re-analyze the RRR data using a Bayesian approach. Because the emphasis here is on fitting models in Bambi, we spend very little time on quality control and data exploration; our goal is simply to show how one can replicate and extend the primary analysis reported in the RRR in a few lines of Bambi code.
We begin by importing all of the modules we’ll need, and enabling inline plotting with matplotlib.
Reading in the data¶
The data for the RRR of Strack, Martin, & Stepper (henceforth SMS) is available as a set of CSV files from the project’s repository on the Open Science Framework. For the sake of completeness, we’ll show how to go from the raw CSV to the “long” data format that Bambi can use (and that’s required by most other mixed-effects modeling packages).
One slightly annoying thing about these 17 CSV files–each of which represents a different replication site–is that they don’t all contain exactly the same columns. Some labs added a column or two at the end (mostly for notes). To keep things simple, we’ll just truncate each dataset to only the first 22 columns. Because the variable names are structured in a bit of a confusing way, we’ll also just drop the first two rows in each file, and manually set the column names for all 22 variables. Once we’ve done that, we can simply concatenate all of the 17 datasets along the row axis to create one big dataset.
[3]:
DL_PATH = 'data/facial_feedback/*csv'
dfs = []
columns = ['subject', 'cond_id', 'condition', 'correct_c1', 'correct_c2', 'correct_c3', 'correct_c4',
'correct_total', 'rating_t1', 'rating_t2', 'rating_c1', 'rating_c2', 'rating_c3',
'rating_c4', 'self_perf', 'comprehension', 'awareness', 'transcript', 'age', 'gender',
'student', 'occupation']
count = 0
for idx, study in enumerate(glob(DL_PATH)):
data = pd.read_csv(study, encoding='latin1', skiprows=2, header=None, index_col=False).iloc[:, :22]
data.columns = columns
# Add study name
data['study'] = idx
# Some sites used the same subject id numbering schemes, so prepend with study to create unique ids.
# Note that if we don't do this, Bambi would have no way of distinguishing two subjects who share
# the same id, which would hose our results.
data['uid'] = data['subject'].astype(float) + count
dfs.append(data)
data = pd.concat(dfs, axis=0).apply(pd.to_numeric, errors='coerce', axis=1)
INFO:numexpr.utils:NumExpr defaulting to 4 threads.
Let’s see what the first few rows look like…
[4]:
data.head()
[4]:
| subject | cond_id | condition | correct_c1 | correct_c2 | correct_c3 | correct_c4 | correct_total | rating_t1 | rating_t2 | ... | self_perf | comprehension | awareness | transcript | age | gender | student | occupation | study | uid | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1.0 | 39814.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 4.0 | 6.0 | 3.0 | ... | NaN | 1.0 | 0.0 | NaN | 19.0 | 0.0 | 1.0 | NaN | 0.0 | 1.0 |
| 1 | 2.0 | 39829.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 4.0 | 8.0 | 6.0 | ... | NaN | 1.0 | 0.0 | NaN | 19.0 | 1.0 | 1.0 | NaN | 0.0 | 2.0 |
| 2 | 3.0 | 39817.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 3.0 | 7.0 | 5.0 | ... | NaN | 1.0 | 0.0 | NaN | 20.0 | 0.0 | 1.0 | NaN | 0.0 | 3.0 |
| 3 | 4.0 | 39994.0 | 0.0 | 1.0 | 0.0 | 1.0 | 1.0 | 3.0 | 5.0 | 3.0 | ... | NaN | 1.0 | 0.0 | NaN | 20.0 | 0.0 | 1.0 | NaN | 0.0 | 4.0 |
| 4 | 5.0 | 39541.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 4.0 | 7.0 | 4.0 | ... | NaN | 1.0 | 0.0 | NaN | 18.0 | 0.0 | 1.0 | NaN | 0.0 | 5.0 |
5 rows × 24 columns
Reshaping the data¶
At this point we have our data in a pandas DataFrame with shape of (2612, 24). Unfortunately, we can’t use the data in this form. We’ll need to (a) conduct some basic quality control, and (b) “melt” the dataset–currently in so-called “wide” format, with each subject in a separate row–into long format, where each row is a single trial. Fortunately, we can do this easily in pandas:
[5]:
# Keep only subjects who (i) respond appropriately on all trials,
# (ii) understand the cartoons, and (iii) don't report any awareness
# of the hypothesis or underlying theory.
valid = data.query('correct_total==4 and comprehension==1 and awareness==0')
long = pd.melt(valid, ['uid', 'condition', 'gender', 'age', 'study', 'self_perf'],
['rating_c1', 'rating_c2', 'rating_c3', 'rating_c4'], var_name='stimulus')
Notice that in the melt()
call above, we’re treating not only the unique subject ID (uid) as an identifying variable, but also gender, experimental condition, age, and study name. Since these are all between-subject variables, these columns are all completely redundant with uid
, and adding them does nothing to change the structure of our data. The point of explicitly listing them is just to keep them around in the dataset, so that we can easily add them to our models.
Fitting the model¶
Now that we’re all done with our (minimal) preprocessing, it’s time to fit the model! This turns out to be a snap in Bambi. We’ll begin with a very naive (and, as we’ll see later, incorrect) model that includes only the following terms:
An overall (fixed) intercept.
The fixed effect of experimental condition (“smiling” by holding a pen in one’s teeth vs. “pouting” by holding a pen in one’s lips). This is the primary variable of interest in the study.
A random intercept for each of the 1,728 subjects in the ‘long’ dataset. (There were 2,576 subjects in the original dataset, but about 25% were excluded for various reasons, and we’re further excluding all subjects who lack complete data. As an exercise, you can try relaxing some of these criteria and re-fitting the models, though you’ll probably find that it makes no meaningful difference to the results.)
We’ll create a Bambi model, fit it, and store the results in a new object–which we can then interrogate in various ways.
[6]:
# Initialize the model, passing in the dataset we want to use.
model = bmb.Model(long, dropna=True)
model.add('value ~ condition', random=['1|uid'])
# Set a custom prior on random factor variances—just for illustration
random_sd = bmb.Prior('HalfNormal', sd=10)
random_prior = bmb.Prior('Normal', mu=0, sd=random_sd)
model.set_priors(random=random_prior)
# Fit the model, drawing 1,000 MCMC samples per chain
results = model.fit(samples=1000)
/home/osvaldo/proyectos/00_BM/Bambi/bambi/bambi/models.py:160: UserWarning: Automatically removing 9/6940 rows from the dataset.
warnings.warn(msg)
Auto-assigning NUTS sampler...
INFO:pymc3:Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
INFO:pymc3:Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (2 chains in 2 jobs)
INFO:pymc3:Multiprocess sampling (2 chains in 2 jobs)
NUTS: [value_sd, 1|uid_offset, 1|uid_sd, condition, Intercept]
INFO:pymc3:NUTS: [value_sd, 1|uid_offset, 1|uid_sd, condition, Intercept]
Notice that, in Bambi, the fixed and random effects are specified separately (for technical reasons we won’t get into here). The fixed effects can be specified using R-style formulas. The underlying parsing of the formula is done by patsy, and you can find out much more about the operators patsy supports by reading its excellent documentation. The random effects are specified as a list, where each term is a separate element. In this case, we only have one random factor (unique subject id) to include.
When we run the above code, nothing much will happen for a while. The process of compiling the model in Theano (the numerical library that PyMC3–and hence, Bambi–relies on) can take a few minutes (on newer versions of Bambi, you can also opt to use a Stan back-end that is often faster to compile and sample). Once sampling starts, we get a nice progress bar (courtesy of PyMC3) that keeps us updated on the sampler’s progress. Our model is fairly large, in that it includes nearly 1800 parameters (most of which are the random subject intercepts); even so, it should take only a few minutes to run on newer machines.
Inspecting results¶
We can plot the posterior distributions and traces for all parameters with the plot()
method.
[7]:
model.plot();
[8]:
az.plot_trace(results, var_names=['Intercept', 'condition', 'value_sd', '1|uid_sd']);
If we want a numerical summary of the results, we can call the results object’s summary()
method (again simply a wrapper for PyMC3’s method of the same name). By default, the summary display shows us the mean, standard deviation, and 95% highest posterior density interval.
[9]:
# Show summary statistics for all fixed effects and variances;
az.summary(results)
[9]:
| mean | sd | hpd_3% | hpd_97% | mcse_mean | mcse_sd | ess_mean | ess_sd | ess_bulk | ess_tail | r_hat | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Intercept[0] | 4.564 | 0.048 | 4.479 | 4.656 | 0.002 | 0.001 | 886.0 | 886.0 | 881.0 | 1298.0 | 1.0 |
| condition[0] | -0.030 | 0.059 | -0.137 | 0.084 | 0.002 | 0.001 | 1106.0 | 1003.0 | 1118.0 | 1317.0 | 1.0 |
| 1|uid_offset[0] | -0.489 | 0.743 | -1.914 | 0.878 | 0.018 | 0.015 | 1717.0 | 1264.0 | 1719.0 | 1302.0 | 1.0 |
| 1|uid_offset[1] | -0.293 | 0.792 | -1.897 | 1.150 | 0.018 | 0.019 | 1844.0 | 883.0 | 1867.0 | 1073.0 | 1.0 |
| 1|uid_offset[2] | -0.539 | 0.754 | -1.926 | 0.870 | 0.016 | 0.015 | 2201.0 | 1277.0 | 2189.0 | 1504.0 | 1.0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1|uid[185] | 0.070 | 0.284 | -0.446 | 0.609 | 0.007 | 0.006 | 1868.0 | 997.0 | 1879.0 | 1172.0 | 1.0 |
| 1|uid[186] | -0.084 | 0.306 | -0.688 | 0.480 | 0.007 | 0.008 | 1686.0 | 808.0 | 1696.0 | 1314.0 | 1.0 |
| 1|uid[187] | -0.127 | 0.299 | -0.690 | 0.415 | 0.007 | 0.007 | 1690.0 | 1002.0 | 1693.0 | 1213.0 | 1.0 |
| 1|uid[188] | -0.118 | 0.295 | -0.706 | 0.423 | 0.007 | 0.007 | 1708.0 | 950.0 | 1730.0 | 1140.0 | 1.0 |
| value_sd | 2.401 | 0.020 | 2.363 | 2.438 | 0.000 | 0.000 | 1858.0 | 1856.0 | 1840.0 | 1336.0 | 1.0 |
382 rows × 11 columns
Expanding the model¶
Looking at the parameter estimates produced by our model, it seems pretty clear that there’s no meaningful effect of condition. The posterior distribution is centered almost exactly on 0, with most of the probability mass on very small values. The 95% HPD spans from -0.16 to -0.16–in other words, the plausible effect of the experimental manipulation is, at best, to produce a change of < 0.2 on cartoon ratings made on a 10-point scale. For perspective, the variation between subjects is enormous
in comparison–the standard deviation u_uid
is around 1.2. We can also see that the model is behaving well, and the sampler seems to have converged nicely (the traces for all parameters look stationary).
Unfortunately, our first model has at least two pretty serious problems. First, it gives no consideration to between-study variation–we’re simply lumping all 1,728 subjects together, as if they came from the same study. A better model would properly account for study-level variation. We could model study as either a fixed or a random factor in this case–both choices are defensible, depending on whether we want to think of the 17 studies in this dataset as the only sites of interest, or as if they’re just 17 random sites drawn from some much larger population. For present purposes, we’ll adopt the latter strategy (as an exercise, you can modify the the code below and re-run the model with study as a fixed factor). We’ll “keep it maximal” by adding both random study intercepts and random study slopes to the model. That is, we’ll assume that the subjects at each research site have a different baseline appreciation of the cartoons (some find the cartoons funnier than others), and that the effect of condition also varies across sites.
Second, our model also fails to explicitly model variation in cartoon ratings that should properly be attributed to the 4 stimuli. In principle, our estimate of the fixed effect of condition could change somewhat once we correctly account for stimulus variability (though in practice, the net effect is almost always to reduce effects, not increase them–so in this case, it’s very unlikely that adding random stimulus effects will produce a meaningful effect of condition). So we’ll deal with this by adding random intercepts for the 4 stimuli. We’ll model the stimuli as random rather than fixed, because it wouldn’t make sense to think of these particular cartoons as exhausting the universe of stimuli we care about (i.e., we wouldn’t really care about the facial-feedback effect if we knew that it only applied to 4 specific Far Side cartoons, and no other stimuli).
Lastly, just for fun, we can throw in some additional covariates, since they’re readily available in the dataset, and may be of interest even if they don’t directly inform the core hypothesis. Specifically, we’ll add fixed effects of gender and age to the model, which will let us estimate the degree to which participants’ ratings of the cartoons varies as a function of these background variables.
Once we’ve done all that, we end up with a model that’s in a good position to answer the question we care about–namely, whether the smiling/pouting manipulation has an effect on cartoon ratings that generalizes across the subjects, studies, and stimuli found in the RRR dataset.
[10]:
model = bmb.Model(long, dropna=True)
model.fit('value ~ 1 + condition + age + gender',
random=['condition|study', '1|uid', 'condition|stimulus'],
run=False)
random_sd = bmb.Prior('HalfNormal', sd=10)
random_prior = bmb.Prior('Normal', mu=0, sd=random_sd)
model.set_priors(random=random_prior)
results = model.fit(samples=1000, init='adapt_diag')
/home/osvaldo/proyectos/00_BM/Bambi/bambi/bambi/models.py:160: UserWarning: Automatically removing 33/6940 rows from the dataset.
warnings.warn(msg)
Auto-assigning NUTS sampler...
INFO:pymc3:Auto-assigning NUTS sampler...
Initializing NUTS using adapt_diag...
INFO:pymc3:Initializing NUTS using adapt_diag...
Multiprocess sampling (2 chains in 2 jobs)
INFO:pymc3:Multiprocess sampling (2 chains in 2 jobs)
NUTS: [value_sd, condition|stimulus_offset, condition|stimulus_sd, 1|stimulus_offset, 1|stimulus_sd, 1|uid_offset, 1|uid_sd, condition|study_offset, condition|study_sd, 1|study_offset, 1|study_sd, gender, age, condition, Intercept]
INFO:pymc3:NUTS: [value_sd, condition|stimulus_offset, condition|stimulus_sd, 1|stimulus_offset, 1|stimulus_sd, 1|uid_offset, 1|uid_sd, condition|study_offset, condition|study_sd, 1|study_offset, 1|study_sd, gender, age, condition, Intercept]
There were 25 divergences after tuning. Increase `target_accept` or reparameterize.
ERROR:pymc3:There were 25 divergences after tuning. Increase `target_accept` or reparameterize.
There were 77 divergences after tuning. Increase `target_accept` or reparameterize.
ERROR:pymc3:There were 77 divergences after tuning. Increase `target_accept` or reparameterize.
The estimated number of effective samples is smaller than 200 for some parameters.
ERROR:pymc3:The estimated number of effective samples is smaller than 200 for some parameters.
[12]:
az.plot_trace(results, var_names=['age', '1|stimulus_sd', 'condition|stimulus_sd', 'gender', 'condition|study',
'1|stimulus', 'value_sd', 'Intercept', 'condition|study_sd',
'1|study', 'condition', 'condition|stimulus'], compact=True);
And the answer is…¶
No. There’s still no discernible effect. Modeling the data using a mixed-effects model does highlight a number of other interesting features, however: * The stimulus-level variance u_stimulus_sd
is quite large compared to the other factors. This is potentially problematic, because it suggests that a more conventional analysis that left individual stimulus effects out of the model could potentially run a high false positive rate. Note that this is a problem that affects both the RRR and the
original Strack study equally; the moral of the story is to deliberately sample large numbers of stimuli and explicitly model their influence. * Older people seem to rate cartoons as being (a little bit) funnier. * The variation across sites is surprisingly small–in terms of both the random intercepts (u_study_id
) and the random slopes (u_condition|study_id
). In other words, the constitution of the sample, the gender of
the experimenter, or any of the hundreds of others of between-site differences that one might conceivably have expected to matter, don’t really seem to make much of a difference to participants’ ratings of the cartoons.


Type Safety:
Introduction
Type Safety for JavaScript
When you create a new project with create-composi-app
it comes set up to provide you with the ability to have type safe JavaScript. It does this by telling Visual Studio Code to use its TypeScript Language Service to type check you JavaScript in real time, as you write it.
JavaScript is Loosely Typed
JavaScript is not strongly typed. It was designed to run in the browser, so it has no need to be built or compiled. Unlike every other language out there, JavaScript is the only language that has to deal with the browser's DOM. The DOM has its own type system and JavaScript has to deal with that. To do so, JavaScript uses type coercion. For example, if you use document.getElementById
you get an object of type Element
. But document.querySelector
returns HTMLElement
. These are not the same thing. If the element you were getting was an input, its type would be HTMLInputElement
, which is different from the previous two types. In a strongly typed language these would not be interchangeable because their properties do not overlap. Through type coercion JavaScript lets you use access the DOM without having to thing about the underlying types.
Suppose you use getElementById
to get an input element. Then you want to get the value of that input like this:
const formInput = document.getElementById('form-input')
const value = formInput.value
This seems straightforward. It's something we do all the time. But what's really happening is the query is returning a type Element
. This is a very generic type. It has no value property. So when we try to access the value property on formInput
JavaScript sees that you're trying to access the value property and coerces formInput
to type HTMLInputElement
.
Type coercion makes it easy to work with the DOM. However the when and how type coercion happens is not always clear to developers, resulting in unexpected results. On top of that, because JavaScript is an interpreted language that runs in the browser, there is no way to know if code is correct until it loads in the browser. Basically the browser is your compiler and test environment. Strongly typed languages require a build step where the code is quecked for type correctness. If it is not correct, the build fails with relavant information about the type errors so you can address them.
For simple JavaScript, running it in the browser to catch errors is not a big deal. But if the code base is large and complex, this becomes couterproductive. The number and types of errors in a code base increase exponentially with the number of people working on it.
Prevention is the Best Cure
There are several trends to avoid the many types of JavaScript errors that can cause a crash. Linting with JSHint, ESLint, etc., warns you about dubious patterns and usages in your code. Prettier makes the code formatting consistent between team members. And there are now dozens of typed languages that compile to JavaScript. These provide static type safety at build time. This verifies that the types are correct before the JavaScript loads i the browser. There are many languages that provide static type checking for JavaScript: GWT, ClosureScript, Elm, PureScript, Reason, TypeScript, etc.
Regardless of what language you use to provide type safety for your code, once it is converted to JavaScript and run in the browser or in Node, there is no type safety. If all the code is from you or your team, you can have a high degree of certainty that your code will be type safe. However, if you are using third party libraries, data or APIs you cannot be sure. In such a case the only way to be sure that your code is type safe during runtime is to use type guards.
JSDoc
Composi provides type safety through a different approach. It uses TypeScript for type safety, but directly on JavaScript. TypeScript was designed to be able to understand JavaScript type usage through inference. This is nice, but not that useful because in most cases it treats so many things as type any
. TypeScript understands JSDoc comments, including the type information that closely corresponds to those of TypeScript. This means you can provide your JavaScript code with type information with JSDoc comments that TypeScript can understand, allowing it to verify the correctness of types. This means as you type JavaScript you will get both warning about incorrect type usage and rich Intellisense about the code you are using and code completion with hints.
All the default Composi code already comes typed for you. But any code you write will need type definitions to make it type safe. You can learn how to provide JSDoc type definition in the docs for typing.


--- layout: archive title: Zulip Chat Archive permalink: /stream/113488-general/topic/refl.html ---
why does lean automatically tag certain theorems with @[_refl_lemma]
?
I want to know too!
That attribute means that it's a refl lemma (it was proven by rfl
), and it is basically "@[simp]
for dsimp
"
someday i'd like to write a boilerplate macro that goes into definitions and creates lemmas for each branch
although even better is if we could suggest names while constructing a def
using the equation compiler
That attribute means that it's a refl lemma (it was proven by
rfl
), and it is basically "@[simp]
fordsimp
"
Is this necessary because definitional equality is not transitive?
No, it's just a way to guide the operation of dsimp
. There are loads of things that are defeq, so how should it know you want a + succ b
to "unfold" to succ (a + b)
and not something else? Equation lemmas are used for this purpose, and the ones that are known to be definitional are legal for use by dsimp
Does dsimp
use anything else?
It uses the bracket list of course, but most of its stuff comes from equation lemmas
someday i'd like to write a boilerplate macro that goes into definitions and creates lemmas for each branch
def
does this currently...
Last updated: Apr 24 2020 at 19:02 UTC


NEWS.md
New rscript()
function and rscript_process
class to execute R scripts via Rscript
(#40, #81).
Library paths are now correctly set up for system()
(and similar) calls from the callr subprocesses (#83, #84).
Pass options("repos")
to the child process as is, without checking. Closes #82.
r_session$run_with_output()
now returns an S3 object with class callr_session_result
.
r_session$run*()
handle interrupts properly. It tries to interrupt the background process fist, kills it if it is not interruptable, and then re-throws the interrupt condition, going back to the top level prompt if the re-thrown condition is un-caught.
pkgdown web site at https://callr.r-lib.org (#52, #53).
callr users .Renviron
files now (and R_ENVIRON_USER
as well), but overrides the library path, as requested in r()
, etc. (#30).
callr now handles the case when the subprocess calls quit()
.
callr now uses the processx package, instead of embedded code, to create and control processes.
Fix a race condition on Windows, when creating named pipes for stdout or stderr. The client sometimes didn’t wait for the server, and callr failed with ERROR_PIPE_BUSY (231, All pipe instances are busy).
Fix compilation issues on CRAN’s Solaris machine
Fix a test failure on CRAN’s macOS machine
Run R or R CMD * in the background, see r_bg()
, rcmd_bg()
, and also r_process
and rcmd_process
The defaults for r()
are safer now, the match the defaults of r_safe()
. r_safe()
is kept for compatibility. r_copycat()
has the old r()
defaults.
The defaults for rcmd()
are safer now, the match the defaults of rcmd_safe()
. rcmd_safe()
is kept for compatibility. rcmd_copycat()
has the old rcmd()
defaults.
Support block callbacks, in addition to line callbacks. Block callbacks are called for arbitrary chunks of output, even without a newline
Support timeouts, via the timeout
argument
Fix bug when stdout and stderr are redirected to the same file
rcmd_safe_env()
to allow extending the environment variables set in safe mode
rcmd()
gets a fail_on_status
argument
rcmd()
gets an echo
argument to potentially show the command to be run on the screen (#15)
rcmd()
gets a wd
argument to set the working directory


Hello, The NHS administrator for {{SCHOOL_NAME}} has demerited you using AtYourService. Below is the reason: {{reason}} If you believe this to be in error, you can contact your administrator via email. Thanks, {{SCHOOL_SHORT_NAME}} AtYourService


Ravi Shastri cautious about pink ball's behaviour
by PTIThe historical significance of India's maiden Day/Night Test isn't lost on him but national coach Ravi Shastri on Friday also struck a note of caution by saying that "plenty of questions" on the pink ball's behaviour are still to be answered.
Seven years after the International Cricket Council (ICC) gave its go ahead, India finally embraced the revamp which is aimed at bringing back the crowds to the Test format.
"(It's a) historic occasion, boys will look forward to it. There will be a wait-and-watch game as well," Shastri said while speaking to the official broadcaster here before the start of opening day's play in the game.
"Plenty of questions to be answered and time will tell," he added.
The game is just 12th Day/Night game since the format was rolled out in 2015 during an Australia-New Zealand series.
"Adapting to pink ball will be a challenge, we've got to see how the ball will behave, how close will it be to how Test cricket should be. It seems a little harder and heavier than the red ball," he said.
"So when you're catching, you have to watch it that much closely into the hand. Only time will tell how it behaves, how it will handle dew, how much of swing there is, how much of reverse, how much will it help spinners," he explained.
India captain Virat Kohli has also been guarded in his views and in the pre-match press conference stated that Day/Night games can only be an exception not the rule.
Shastri said he would observe how things work out in this game before giving a definitive view.
"We're making a start, it'll be interesting to see how it goes," he said.


--- layout: page title: About header-img: "img/about-bg.png" ---
Carving my path through the data science jungle