Tutorial 1: “What” models¶
Content creators: Matt Laporte, Byron Galbraith, Konrad Kording
Content reviewers: Dalin Guo, Aishwarya Balwani, Madineh Sarvestani, Maryam Vaziri-Pashkam, Michael Waskom
We would like to acknowledge Steinmetz et al. (2019) for sharing their data, a subset of which is used here.
Tutorial Objectives¶
This is tutorial 1 of a 3-part series on different flavors of models used to understand neural data. In this tutorial we will explore ‘What’ models, used to describe the data. To understand what our data looks like, we will visualize it in different ways. Then we will compare it to simple mathematical models. Specifically, we will:
Load a dataset with spiking activity from hundreds of neurons and understand how it is organized
Make plots to visualize characteristics of the spiking activity across the population
Compute the distribution of “inter-spike intervals” (ISIs) for a single neuron
Consider several formal models of this distribution’s shape and fit them to the data “by hand”
#@title Video 1: "What" Models
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='KgqR_jbjMQg', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
Setup¶
Python requires you to explictly “import” libraries before their functions are available to use. We will always specify our imports at the beginning of each notebook or script.
import numpy as np
import matplotlib.pyplot as plt
Tutorial notebooks typically begin with several set-up steps that are hidden from view by default.
Important: Even though the code is hidden, you still need to run it so that the rest of the notebook can work properly. Step through each cell, either by pressing the play button in the upper-left-hand corner or with a keyboard shortcut (Cmd-Return
on a Mac, Ctrl-Enter
otherwise). A number will appear inside the brackets (e.g. [3]
) to tell you that the cell was executed and what order that happened in.
If you are curious to see what is going on inside each cell, you can double click to expand. Once expanded, double-click the white space to the right of the editor to collapse again.
#@title Figure Settings
import ipywidgets as widgets #interactive display
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
#@title Helper functions
#@markdown Most of the tutorials make use of helper functions
#@markdown to simplify the code that you need to write. They are defined here.
# Please don't edit these, or worry about understanding them now!
def restrict_spike_times(spike_times, interval):
"""Given a spike_time dataset, restrict to spikes within given interval.
Args:
spike_times (sequence of np.ndarray): List or array of arrays,
each inner array has spike times for a single neuron.
interval (tuple): Min, max time values; keep min <= t < max.
Returns:
np.ndarray: like `spike_times`, but only within `interval`
"""
interval_spike_times = []
for spikes in spike_times:
interval_mask = (spikes >= interval[0]) & (spikes < interval[1])
interval_spike_times.append(spikes[interval_mask])
return np.array(interval_spike_times, object)
#@title Data retrieval
#@markdown This cell downloads the example dataset that we will use in this tutorial.
import io
import requests
r = requests.get('https://osf.io/sy5xt/download')
if r.status_code != 200:
print('Failed to download data')
else:
spike_times = np.load(io.BytesIO(r.content), allow_pickle=True)['spike_times']
Section 1: Exploring the Steinmetz dataset¶
In this tutorial we will explore the structure of a neuroscience dataset.
We consider a subset of data from a study of Steinmetz et al. (2019). In this study, Neuropixels probes were implanted in the brains of mice. Electrical potentials were measured by hundreds of electrodes along the length of each probe. Each electrode’s measurements captured local variations in the electric field due to nearby spiking neurons. A spike sorting algorithm was used to infer spike times and cluster spikes according to common origin: a single cluster of sorted spikes is causally attributed to a single neuron.
In particular, a single recording session of spike times and neuron assignments was loaded and assigned to spike_times
in the preceding setup.
Typically a dataset comes with some information about its structure. However, this information may be incomplete. You might also apply some transformations or “pre-processing” to create a working representation of the data of interest, which might go partly undocumented depending on the circumstances. In any case it is important to be able to use the available tools to investigate unfamiliar aspects of a data structure.
Let’s see what our data looks like…
Section 1.1: Warming up with spike_times
¶
What is the Python type of our variable?
type(spike_times)
You should see numpy.ndarray
, which means that it’s a normal NumPy array.
If you see an error message, it probably means that you did not execute the set-up cells at the top of the notebook. So go ahead and make sure to do that.
Once everything is running properly, we can ask the next question about the dataset: what’s its shape?
spike_times.shape
There are 734 entries in one dimension, and no other dimensions. What is the Python type of the first entry, and what is its shape?
idx = 0
print(
type(spike_times[idx]),
spike_times[idx].shape,
sep="\n",
)
It’s also a NumPy array with a 1D shape! Why didn’t this show up as a second dimension in the shape of spike_times
? That is, why not spike_times.shape == (734, 826)
?
To investigate, let’s check another entry.
idx = 321
print(
type(spike_times[idx]),
spike_times[idx].shape,
sep="\n",
)
It’s also a 1D NumPy array, but it has a different shape. Checking the NumPy types of the values in these arrays, and their first few elements, we see they are composed of floating point numbers (not another level of np.ndarray
):
i_neurons = [0, 321]
i_print = slice(0, 5)
for i in i_neurons:
print(
"Neuron {}:".format(i),
spike_times[i].dtype,
spike_times[i][i_print],
"\n",
sep="\n"
)
Note that this time we’ve checked the NumPy dtype
rather than the Python variable type. These two arrays contain floating point numbers (“floats”) with 32 bits of precision.
The basic picture is coming together:
spike_times
is 1D, its entries are NumPy arrays, and its length is the number of neurons (734): by indexing it, we select a subset of neurons.An array in
spike_times
is also 1D and corresponds to a single neuron; its entries are floating point numbers, and its length is the number of spikes attributed to that neuron. By indexing it, we select a subset of spike times for that neuron.
Visually, you can think of the data structure as looking something like this:
| . . . . . |
| . . . . . . . . |
| . . . |
| . . . . . . . |
Before moving on, we’ll calculate and store the number of neurons in the dataset and the number of spikes per neuron:
n_neurons = len(spike_times)
total_spikes_per_neuron = [len(spike_times_i) for spike_times_i in spike_times]
print(f"Number of neurons: {n_neurons}")
print(f"Number of spikes for first five neurons: {total_spikes_per_neuron[:5]}")
#@title Video 2: Exploring the dataset
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='oHwYWUI_o1U', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
Section 1.2: Getting warmer: counting and plotting total spike counts¶
As we’ve seen, the number of spikes over the entire recording is variable between neurons. More generally, some neurons tend to spike more than others in a given period. Lets explore what the distribution of spiking looks like across all the neurons in the dataset.
Are most neurons “loud” or “quiet”, compared to the average? To see, we’ll define bins of constant width in terms of total spikes and count the neurons that fall in each bin. This is known as a “histogram”.
You can plot a histogram with the matplotlib function plt.hist
. If you just need to compute it, you can use the numpy function np.histogram
instead.
plt.hist(total_spikes_per_neuron, bins=50, histtype="stepfilled")
plt.xlabel("Total spikes per neuron")
plt.ylabel("Number of neurons");
Let’s see what percentage of neurons have a below-average spike count:
mean_spike_count = np.mean(total_spikes_per_neuron)
frac_below_mean = (total_spikes_per_neuron < mean_spike_count).mean()
print(f"{frac_below_mean:2.1%} of neurons are below the mean")
We can also see this by adding the average spike count to the histogram plot:
plt.hist(total_spikes_per_neuron, bins=50, histtype="stepfilled")
plt.xlabel("Total spikes per neuron")
plt.ylabel("Number of neurons")
plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
plt.legend();
This shows that the majority of neurons are relatively “quiet” compared to the mean, while a small number of neurons are exceptionally “loud”: they must have spiked more often to reach a large count.
Exercise 1: Comparing mean and median neurons¶
If the mean neuron is more active than 68% of the population, what does that imply about the relationship between the mean neuron and the median neuron?
Exercise objective: Reproduce the plot above, but add the median neuron.
# To complete the exercise, fill in the missing parts (...) and uncomment the code
median_spike_count = ... # Hint: Try the function np.median
# plt.hist(..., bins=50, histtype="stepfilled")
# plt.axvline(..., color="limegreen", label="Median neuron")
# plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
# plt.xlabel("Total spikes per neuron")
# plt.ylabel("Number of neurons")
# plt.legend()
Example output:
Bonus: The median is the 50th percentile. What about other percentiles? Can you show the interquartile range on the histogram?
Section 2: Visualizing neuronal spiking activity¶
Section 2.1: Getting a subset of the data¶
Now we’ll visualize trains of spikes. Because the recordings are long, we will first define a short time interval and restrict the visualization to only the spikes in this interval. We defined a utility function, restrict_spike_times
, to do this for you. If you call help()
on the function, it will tell you a little bit about itself:
help(restrict_spike_times)
t_interval = (5, 15) # units are seconds after start of recording
interval_spike_times = restrict_spike_times(spike_times, t_interval)
Is this a representative interval? What fraction of the total spikes fall in this interval?
original_counts = sum([len(spikes) for spikes in spike_times])
interval_counts = sum([len(spikes) for spikes in interval_spike_times])
frac_interval_spikes = interval_counts / original_counts
print(f"{frac_interval_spikes:.2%} of the total spikes are in the interval")
How does this compare to the ratio between the interval duration and the experiment duration? (What fraction of the total time is in this interval?)
We can approximate the experiment duration by taking the minimum and maximum spike time in the whole dataset. To do that, we “concatenate” all of the neurons into one array and then use np.ptp
(“peak-to-peak”) to get the difference between the maximum and minimum value:
spike_times_flat = np.concatenate(spike_times)
experiment_duration = np.ptp(spike_times_flat)
interval_duration = t_interval[1] - t_interval[0]
frac_interval_time = interval_duration / experiment_duration
print(f"{frac_interval_time:.2%} of the total time is in the interval")
These two values—the fraction of total spikes and the fraction of total time—are similar. This suggests the average spike rate of the neuronal population is not very different in this interval compared to the entire recording.
Section 2.2: Plotting spike trains and rasters¶
Now that we have a representative subset, we’re ready to plot the spikes, using the matplotlib plt.eventplot
function. Let’s look at a single neuron first:
neuron_idx = 1
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);
We can also plot multiple neurons. Here are three:
neuron_idx = [1, 11, 51]
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);
This makes a “raster” plot, where the spikes from each neuron appear in a different row.
Plotting a large number of neurons can give you a sense for the characteristics in the population. Let’s show every 5th neuron that was recorded:
neuron_idx = np.arange(0, len(spike_times), 5)
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);
Question: How does the information in this plot relate to the histogram of total spike counts that you saw above?
#@title Video 3: Visualizing activity
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='QGA5FCW7kkA', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
Section 3: Inter-spike intervals and their distributions¶
Given the ordered arrays of spike times for each neuron in spike_times
, which we’ve just visualized, what can we ask next?
Scientific questions are informed by existing models. So, what knowledge do we already have that can inform questions about this data?
We know that there are physical constraints on neuron spiking. Spiking costs energy, which the neuron’s cellular machinery can only obtain at a finite rate. Therefore neurons should have a refractory period: they can only fire as quickly as their metabolic processes can support, and there is a minimum delay between consecutive spikes of the same neuron.
More generally, we can ask “how long does a neuron wait to spike again?” or “what is the longest a neuron will wait?” Can we transform spike times into something else, to address questions like these more directly?
We can consider the inter-spike times (or interspike intervals: ISIs). These are simply the time differences between consecutive spikes of the same neuron.
Exercise 2: Plot the distribution of ISIs for a single neuron¶
Exercise objective: make a histogram, like we did for spike counts, to show the distribution of ISIs for one of the neurons in the dataset.
Do this in three steps:
Extract the spike times for one of the neurons
Compute the ISIs (the amount of time between spikes, or equivalently, the difference between adjacent spike times)
Plot a histogram with the array of individual ISIs
def compute_single_neuron_isis(spike_times, neuron_idx):
"""Compute a vector of ISIs for a single neuron given spike times.
Args:
spike_times (list of 1D arrays): Spike time dataset, with the first
dimension corresponding to different neurons.
neuron_idx (int): Index of the unit to compute ISIs for.
Returns:
isis (1D array): Duration of time between each spike from one neuron.
"""
#############################################################################
# Students: Fill in missing code (...) and comment or remove the next line
raise NotImplementedError("Exercise: compute single neuron ISIs")
#############################################################################
# Extract the spike times for the specified neuron
single_neuron_spikes = ...
# Compute the ISIs for this set of spikes
# Hint: the function np.diff computes discrete differences along an array
isis = ...
return isis
# Uncomment the following lines when you are ready to test your function
# single_neuron_isis = compute_single_neuron_isis(spike_times, neuron_idx=283)
# plt.hist(single_neuron_isis, bins=50, histtype="stepfilled")
# plt.axvline(single_neuron_isis.mean(), color="orange", label="Mean ISI")
# plt.xlabel("ISI duration (s)")
# plt.ylabel("Number of spikes")
# plt.legend()
Example output:
In general, the shorter ISIs are predominant, with counts decreasing rapidly (and smoothly, more or less) with increasing ISI. However, counts also rapidly decrease to zero with decreasing ISI, below the maximum of the distribution (8-11 ms). The absence of these very low ISIs agrees with the refractory period hypothesis: the neuron cannot fire quickly enough to populate this region of the ISI distribution.
Check the distributions of some other neurons. To resolve various features of the distributions, you might need to play with the value of n_bins
. Using too few bins might smooth over interesting details, but if you use too many bins, the random variability will start to dominate.
You might also want to restrict the range to see the shape of the distribution when focusing on relatively short or long ISIs. Hint: plt.hist
takes a range
argument
Section 4: What is the functional form of an ISI distribution?¶
#@title Video 4: ISI distribution
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='DHhM80MOTe8', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
The ISI histograms seem to follow continuous, monotonically decreasing functions above their maxima. The function is clearly non-linear. Could it belong to a single family of functions?
To motivate the idea of using a mathematical function to explain physiological phenomena, let’s define a few different function forms that we might expect the relationship to follow: exponential, inverse, and linear.
def exponential(xs, scale, rate, x0):
"""A simple parametrized exponential function, applied element-wise.
Args:
xs (np.ndarray or float): Input(s) to the function.
scale (float): Linear scaling factor.
rate (float): Exponential growth (positive) or decay (negative) rate.
x0 (float): Horizontal offset.
"""
ys = scale * np.exp(rate * (xs - x0))
return ys
def inverse(xs, scale, x0):
"""A simple parametrized inverse function (`1/x`), applied element-wise.
Args:
xs (np.ndarray or float): Input(s) to the function.
scale (float): Linear scaling factor.
x0 (float): Horizontal offset.
"""
ys = scale / (xs - x0)
return ys
def linear(xs, slope, y0):
"""A simple linear function, applied element-wise.
Args:
xs (np.ndarray or float): Input(s) to the function.
slope (float): Slope of the line.
y0 (float): y-intercept of the line.
"""
ys = slope * xs + y0
return ys
Interactive Demo: ISI functions explorer¶
Here is an interactive demo where you can vary the parameters of these functions and see how well the resulting outputs correspond to the data. Adjust the parameters by moving the sliders and see how close you can get the lines to follow the falling curve of the histogram. This will give you a taste of what you’re trying to do when you fit a model to data.
“Interactive demo” cells have hidden code that defines an interface where you can play with the parameters of some function using sliders. You don’t need to worry about how the code works – but you do need to run the cell to enable the sliders.
#@title
#@markdown Be sure to run this cell to enable the demo
# Don't worry about understanding this code! It's to setup an interactive plot.
single_neuron_idx = 283
single_neuron_spikes = spike_times[single_neuron_idx]
single_neuron_isis = np.diff(single_neuron_spikes)
counts, edges = np.histogram(
single_neuron_isis,
bins=50,
range=(0, single_neuron_isis.max())
)
functions = dict(
exponential=exponential,
inverse=inverse,
linear=linear,
)
colors = dict(
exponential="C1",
inverse="C2",
linear="C4",
)
@widgets.interact(
exp_scale=widgets.FloatSlider(1000, min=0, max=20000, step=250),
exp_rate=widgets.FloatSlider(-10, min=-200, max=50, step=1),
exp_x0=widgets.FloatSlider(0.1, min=-0.5, max=0.5, step=0.005),
inv_scale=widgets.FloatSlider(1000, min=0, max=3e2, step=10),
inv_x0=widgets.FloatSlider(0, min=-0.2, max=0.2, step=0.01),
lin_slope=widgets.FloatSlider(-1e5, min=-6e5, max=1e5, step=10000),
lin_y0=widgets.FloatSlider(10000, min=0, max=4e4, step=1000),
)
def fit_plot(
exp_scale=1000, exp_rate=-10, exp_x0=0.1,
inv_scale=1000, inv_x0=0,
lin_slope=-1e5, lin_y0=2000,
):
"""Helper function for plotting function fits with interactive sliders."""
func_params = dict(
exponential=(exp_scale, exp_rate, exp_x0),
inverse=(inv_scale, inv_x0),
linear=(lin_slope, lin_y0),
)
f, ax = plt.subplots()
ax.fill_between(edges[:-1], counts, step="post", alpha=.5)
xs = np.linspace(1e-10, edges.max())
for name, function in functions.items():
ys = function(xs, *func_params[name])
ax.plot(xs, ys, lw=3, color=colors[name], label=name);
ax.set(
xlim=(edges.min(), edges.max()),
ylim=(0, counts.max() * 1.1),
xlabel="ISI (s)",
ylabel="Number of spikes",
)
ax.legend()
#@title Video 5: Fitting models by hand
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='uW2HDk_4-wk', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
Summary¶
In this tutorial, we loaded some neural data and poked at it to understand how the dataset is organized. Then we made some basic plots to visualize (1) the average level of activity across the population and (2) the distribution of ISIs for an individual neuron. In the very last bit, we started to think about using mathematical formalisms to understand or explain some physiological phenomenon. All of this only allowed us to understand “What” the data looks like.
This is the first step towards developing models that can tell us something about the brain. That’s what we’ll focus on in the next two tutorials.


Red9_Tools¶
Red9 Tools is the module for general tools within Red9
Main Classes
SceneReviewerUI () |
this is the reporter Dialogue that the scriptNode calls to display and edit the |
SceneReviewer () |
This is a really simple proc that will stamp data onto the time node and retrieve it so that leads can review and enter info into the scene itself. |
RecordAttrs () |
Simple class to use the Mouse as a MoCap input device #BUG : Maya can’t now record fucking rotate channels as it pushes a unitConvert Node between the rotate and the record plugs and nothing gets captured. |
-
class
SceneReviewerUI
¶ Bases:
object
this is the reporter Dialogue that the scriptNode calls to display and edit the sceneReview data, held on the time node
-
classmethod
show
()¶
-
getSceneName
()¶
-
updateInternalDict
(*args)¶
-
addNewComment
(*args)¶
-
resizeTextScrollers
()¶
-
classmethod
-
class
SceneReviewer
¶ Bases:
object
This is a really simple proc that will stamp data onto the time node and retrieve it so that leads can review and enter info into the scene itself. Why the time1 node??? this saves any issues with merging scenes as the time node is one of the only nodes in Maya that can only exist once, and is managed in that way internally
-
addScriptNode
(*args)¶
-
getReportData
()¶
-
storeReportData
()¶
-
pushCommentToHistory
()¶
-
exists
()¶
-
selectScriptNode
()¶
-
deleteScriptNode
(*args)¶
-
scriptNodeFunc
()¶
-
-
class
RecordAttrs
¶ Bases:
object
Simple class to use the Mouse as a MoCap input device #BUG : Maya can’t now record fucking rotate channels as it pushes a unitConvert Node between the rotate and the record plugs and nothing gets captured. Trying to figure a way round this
-
classmethod
show
()¶
-
addAttrsToRecord
(attrs=None, *args)¶
-
removeAttrsToRecord
(attrs=None, *args)¶
-
recordStart
()¶
-
recordStop
()¶
-
classmethod


{% load staticfiles %}
{% csrf_token %}
Floppydisc.in
Word Cloud
(current)
Contact
Upload A Text File To View Word Cloud Built Using D3
Scrape Data from A URL And Build A Word Cloud
Submit
Cloud
Info
info


In Parameter you can set the SH and the ICG period. After editing please hit return.
The SH adjusts the sensitivity of the device. From experience the ICG should be a whole multiply of the SH-period. For example SH 280 | ICG 56000 or SH 1120 | ICG 56000.
For more informations copy and visit https://tcd1304.wordpress.com/timers/
If you observe periodic noise visit https://erossel.wordpress.com/2017/11/10/momodulo
This is a new feature the TCD1304FW (UART) averages 2-25 integrations. The program needs the version from sep. 10th 2017
My approach in handling the signal form the TCD is to subtract the dark spectra und divide it by the maximum signal.
The dark spectra is the spectrometers background noise when there is no light. It can be added to the system by pressing the darkline-button
In order to determine the High Level take a strong signal and/or high SH period. Select the maximum area with the mouse und read the value and edit High level.
The absorbance or transmittance spectra shows the light that is absorbt by a sample e.g. a dye. The baseline is the spectra of the light source, küvette und solvent.
Take a darkline first!


chrome.app.window
module
to create windows.
Windows can either be shells or panels.
Shell windows have no browser chrome.
Panel windows are the same as shell windows,
except they have different size and position restrictions,
for example, a chat panel.


General Details
Invoice No :
Invoice Date :
Bill To Party
Party’s GSTIN:
Vehicle No:
P&D Logistics:
Products
DESCRIPTION
Price
QUANTITY
HSN Code
UOM
Rate/Unit
AMOUNT


Class: Aws::EC2::Types::CreateFlowLogsRequest
| Relationships & Source Files | |
| Super Chains via Extension / Inclusion / Inheritance | |
|
Class Chain:
self,
Struct
|
|
|
Instance Chain:
self,
::Aws::Structure ,
Struct
|
|
| Inherits: |
Struct
|
| Defined in: | gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb |
Overview
Constant Summary
Instance Attribute Summary
-
#client_token ⇒ String
rw
Unique, case-sensitive identifier that you provide to ensure the idempotency of the request.
-
#deliver_cross_account_role ⇒ String
rw
The ARN of the
::Aws::IAM
role that allows Amazon EC2 to publish flow logs across accounts. -
#deliver_logs_permission_arn ⇒ String
rw
The ARN of the
::Aws::IAM
role that allows Amazon EC2 to publish flow logs to a::Aws::CloudWatch
Logs log group in your account. -
#destination_options ⇒ Types::DestinationOptionsRequest
rw
The destination options.
-
#dry_run ⇒ Boolean
rw
Checks whether you have the required permissions for the action, without actually making the request, and provides an error response.
-
#log_destination ⇒ String
rw
The destination for the flow log data.
-
#log_destination_type ⇒ String
rw
The type of destination for the flow log data.
-
#log_format ⇒ String
rw
The fields to include in the flow log record.
-
#log_group_name ⇒ String
rw
The name of a new or existing
::Aws::CloudWatch
Logs log group where Amazon EC2 publishes your flow logs. -
#max_aggregation_interval ⇒ Integer
rw
The maximum interval of time during which a flow of packets is captured and aggregated into a flow log record.
-
#resource_ids ⇒ Array<String>
rw
The IDs of the resources to monitor.
-
#resource_type ⇒ String
rw
The type of resource to monitor.
-
#tag_specifications ⇒ Array<Types::TagSpecification>
rw
The tags to apply to the flow logs.
-
#traffic_type ⇒ String
rw
The type of traffic to monitor (accepted traffic, rejected traffic, or all traffic).
Instance Method Summary
::Aws::Structure
- Included
| #to_hash | Alias for Structure#to_h. |
Instance Attribute Details
#client_token ⇒ String
(rw)
Unique, case-sensitive identifier that you provide to ensure the idempotency of the request. For more information, see How to ensure idempotency.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#deliver_cross_account_role ⇒ String
(rw)
The ARN of the ::Aws::IAM
role that allows Amazon EC2 to publish flow logs
across accounts.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#deliver_logs_permission_arn ⇒ String
(rw)
The ARN of the ::Aws::IAM
role that allows Amazon EC2 to publish flow logs
to a ::Aws::CloudWatch
Logs log group in your account.
This parameter is required if the destination type is
cloud-watch-logs
and unsupported otherwise.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#destination_options ⇒ Types::DestinationOptionsRequest (rw)
The destination options.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#dry_run ⇒ Boolean
(rw)
Checks whether you have the required permissions for the action,
without actually making the request, and provides an error response.
If you have the required permissions, the error response is
DryRunOperation
. Otherwise, it is UnauthorizedOperation
.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#log_destination ⇒ String
(rw)
The destination for the flow log data. The meaning of this parameter depends on the destination type.
- If the destination type is
cloud-watch-logs
, specify the::Aws::ARN
of a CloudWatch Logs log group. For example:
arn:aws:logs:region:account_id:log-group:my_group
Alternatively, use the LogGroupName
parameter.
- If the destination type is
s3
, specify the::Aws::ARN
of an::Aws::S3
bucket. For example:
arn:aws:s3:::my_bucket/my_subfolder/
The subfolder is optional. Note that you can't use AWSLogs
as a
subfolder name.
- If the destination type is
kinesis-data-firehose
, specify the ARN of a Kinesis Data Firehose delivery stream. For example:
arn:aws:firehose:region:account_id:deliverystream:my_stream
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#log_destination_type ⇒ String
(rw)
The type of destination for the flow log data.
Default: cloud-watch-logs
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#log_format ⇒ String
(rw)
The fields to include in the flow log record. List the fields in the
order in which they should appear. If you omit this parameter, the
flow log is created using the default format. If you specify this
parameter, you must include at least one field. For more information
about the available fields, see Flow log records in the Amazon
VPC User Guide or Transit Gateway Flow ::Aws::Log
records in the
Amazon Web Services Transit Gateway Guide.
Specify the fields using the $\{field-id\}
format, separated by
spaces.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#log_group_name ⇒ String
(rw)
The name of a new or existing ::Aws::CloudWatch
Logs log group where Amazon
::Aws::EC2
publishes your flow logs.
This parameter is valid only if the destination type is
cloud-watch-logs
.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#max_aggregation_interval ⇒ Integer
(rw)
The maximum interval of time during which a flow of packets is captured and aggregated into a flow log record. The possible values are 60 seconds (1 minute) or 600 seconds (10 minutes). This parameter must be 60 seconds for transit gateway resource types.
When a network interface is attached to a Nitro-based instance, the aggregation interval is always 60 seconds or less, regardless of the value that you specify.
Default: 600
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#resource_ids ⇒ Array
<String
> (rw)
The IDs of the resources to monitor. For example, if the resource
type is VPC
, specify the IDs of the VPCs.
Constraints: Maximum of 25 for transit gateway resource types. Maximum of 1000 for the other resource types.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#resource_type ⇒ String
(rw)
The type of resource to monitor.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#tag_specifications ⇒ Array
<Types::TagSpecification> (rw)
The tags to apply to the flow logs.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end
#traffic_type ⇒ String
(rw)
The type of traffic to monitor (accepted traffic, rejected traffic, or all traffic). This parameter is not supported for transit gateway resource types. It is required for the other resource types.
# File 'gems/aws-sdk-ec2/lib/aws-sdk-ec2/types.rb', line 7611
class CreateFlowLogsRequest < Struct.new( :dry_run, :client_token, :, :deliver_cross_account_role, :log_group_name, :resource_ids, :resource_type, :traffic_type, :log_destination_type, :log_destination, :log_format, :tag_specifications, :max_aggregation_interval, : ) SENSITIVE = [] include Aws::Structure end


Release 2.0.0-b06:
This release is a rewrite of version 1 focusing on removing features,
clearifying useful concepts and abstracting the storage engines cleanly.
The latter was done to allow the web implementation of dataset to
achieve an appropriate performance and be able to scale to a larger
number of collections and size of collections.
The dataset collection’s structure has changed.
- a dataset is a directory containing a collection.json and
codemeta.json file
- the collection.json no longer contains general metadata or maps to
the keys and pairtree, it focuses on operational settings (e.g. storage
type and access information)
- when running dataset as a web service or on a shared user machine
you can setup the database connection through the environment. I.e. set
DATASET_DSN_URI value (DSN URI is formed with a protocol named for the
SQL driver, a “://” and the DSN for that driver,
e.g. “mysql://DB_USER:DB_PASSWD@/DB_NAME”)
- a codemeta.json file is now used for holding general collection
level metadata. codemeta has
been adopted by the data science community for describing data and
software
- additional JSON configuration files may be used to manage the
collection dependent on storage engine
- Experimental features in dataset v2
- Using SQL JSON columns to store our JSON documents in a relational
database
- SQLite 3
- MySQL 8
- Posgres 14.5 (very experimental, still learning Postgres)
- Object level versioning
- Attachment versioning
- dataset package to support Async, multi-process access via JSON
stored in SQL databases
Golang package changes:
- Minimum Go version is now 1.19
- The dataset v2 package has been substantially reorganized and
simplified, most things have changed
- Collection.Init() now takes two parameters, collection name, an an
optional DSN URI, if a DSN URI is provided it’ll define the storage
engine, e.g. a dataset.SQLSTORE)
- Collection.DocPath() removed, doesn’t make sense anymore since JSON
may be stored in a SQL table
- Collection.Read() only takes two parameters, not three
- Collection.Keys() returns a list of keys and an error value
- Collection.KeyExists() was renamed Collection.HasKey() to be more
idiomatic in Go
- Collection.FrameExists() was renamed Collection.HasFrame() to be
more idiomatic in Go
- Collection.Length() returns an int64 rather than an int
- Collection.MetadataJSON() renamed Collection.Metadata() returns the
codemeta JSON for the collection
- Collection.UpdateMetatada() has been added, takes the name of a
codemeta.json to replace the existing codemeta content for the
collection.
libdataset:
- The C shared library implementation has been dropped for now do to
the challenges of easily cross compiling releases
CLI changes:
- options have be restructured so that most come after the verb
- help has been restructured to better support focusing the help text
on the task needed
- the command line version is single user, single process and be
default assumes pairtree storage.
- Experimental features
- JSON stored in SQL database via JSON column types
- SQLite3
- MySQL 8
- Postgres 14.5
Web Service changes:
- the web service is generally RESTful so the end points no longer map
directly to the client syntax
- it is recommended to use SQL storage for your dataset collections
explosed using the web service (e.q. SQLite3 or MySQL). This is because
a pairtree collection doesn’t provide docuemnt locking
- relies on SQL JSON columns for storage
- Access to the SQL storage engine is through either the environment
or a URI expressing a storage type as protocol and a data source name to
making the connection
Release 1.1.0:
Added attachment support for datasetd.
Updated the metadata fields to include richer PersonOrOrg data
structures for author, contributor, funder as well as added an
annotation map field for custom metadata.
Added “MetadataJSON()” function for Collection to quickly copy out
the metadata values from a collection.
c, err := dataset.Open("MyData.ds")
...
defer c.Close()
fmt.Printf("%s", c.MetadataJSON())
Added “MetadataUpdate()” function to update a collection’s
metadata.
c, err := dataset.Open("MyData.ds")
...
defer c.Close()
meta := new(Collection)
meta.Description = "A test dataset"
meta.Version = "1.0.0"
meta.Author = []*PersonOrOrg{
&Person{
Type: "Person",
GivenName: "Jane",
FamilyName: "Doe",
},
}
err = c.MetadataUpdate(meta)
...
Depreciated dependency on namaste package and Namaste support in
command line tools. Removed “collections.go and collections_test.go”
from repository (redundant code). Updated libdataset/libdataset.go to
hold functions that were needed for the C-Shared library from
collections.go. The Namaste fields in the collection’s metadata are now
depreciated.
The dataset.Init() now places a lock file in the collection directory
and leaves the collection in an “Open” state, it should be explicitly
closed after Init is called.
E.g.
c, err := dataset.Init("MyData.ds")
...
defer c.Close()
Removed “set_*” for collection metadata fields from libdataset.go.
These should be set using the dataset command line tool only.
The dataset.Analyzer() and dataset.Repair() commands expect the
dataset collections to be closed before being called. E.g..
c, err := dataset.Open("MyData.ds")
...
c.Close()
err := dataset.Analyzer("MyData.ds", true)
if err == nil {
c, err = dataset.Open("MyData.ds")
...
}
Release 1.0.2:
Added support for datasetd, a localhost web service
for dataset collections. The web service supports a subset of the
command line tool.
Both datasetd and dataset command
line program now include a “lock.pid” file in the collection root. This
is to prevent multiple processes from clashing when maintaining the
“collections.json” file in the collection root.
Migrated cli package into dataset repository sub-package
“github.com/caltechlibrary/dataset/cli”. Eventually this package will be
replaced by “datasetCli.go” in the root folder.
In the dataset command line program the verb “detach” has been
renamed “retrieve” better describe the action. “detach” is depreciated
and will be removed in upcoming releases.
Release 1.0.1:
- Keys are stored lowercase
- Removed filtering and sorting options from dataset and
libdataset
- Use pairtree 1.0.2 configurable separator
- Added check and repair for migrating to case insensitive keys and
path
- Updated required packages to latest releases
- Added notes about Windows cmd prompt issues when providing JSON
objects on command line
- Added M1 support for libdataset
Release 1.0.0:
Release 0.1.11:
- Requires go1.16 compilation
- Add macOS M1 compiled binaries
Release 0.1.10:
- Improved memory handling when handling for large attachments
Release 0.1.8:
This release focuses on minor bug fixes in libdataset.
- Removing duplicate functions:
delete_frame()
has been superseded by
frame_delete()
- Renamed functions:
make_objects()
has been renamed
create_objects()
to be more consistent with naming
scheme.
- Build Notes:
- Golang v1.14
- Caltech library go packages
- storage v0.1.0
- namaste v0.0.5
- pairtree v0.0.4
- OS used to compiled and test
- macOS Catalina
- Windows 10
- Ubuntu 18.04 LTS
- Python 3.8 (from Miniconda 3)
- zip has replaced tar in the releases of libdataset
- Some tests fail on Windows 10 for libdataset. These will be
addressed in future releases.
Release 0.1.6:
This release focuses on minor bug fixes in libdataset. All functions
which returned an error string only now return True for success and
False otherwise. The error string can be retrieved with
dataset.error_message()
.
- Build Notes:
- Golang v1.14
- Caltech library go packages
- storage v0.1.0
- namaste v0.0.5
- pairtree v0.0.4
- OS used to compiled and test
- macOS Catalina
- Windows 10
- Ubuntu 18.04 LTS
- Python 3.8 (from Miniconda 3)
- zip has replaced tar in the releases of libdataset
- Some tests fail on Windows 10 for libdataset. These will be
addressed in future releases.
Release 0.1.5:
This release focuses on refine function names, simplification and
easy of testing for Windows 10 deployments.
- Build Notes:
- Golang v1.14
- Caltech library go packages
- storage v0.1.0
- namaste v0.0.5
- pairtree v0.0.4
- OS used to compiled and test
- macOS Catalina
- Windows 10
- Ubuntu 18.04 LTS
- Python 3.8 (from Miniconda 3)
- zip has replaced tar in the releases of libdataset
- Renamed functions:
- collection_status() is now collection_exists()
- Depreciated functions and features:
- S3, Google Cloud Storage support dropped.
- grid(), if you need this create a frame first and use
frame_grid().
- Some tests fail on Windows 10 for libdataset. These will be
addressed in future releases.
Release 0.1.4:
This release has breaking changes with release v0.1.3 and early. Many
functions in libdataset have been renamed to prevent collisions in the
environments using libdataset C-shared library. Most function names now
have two parts separated by a underscore (e.g. status has become
collection_status, repair has become collection_repair).
Google Sheet integration has been dropped. It was just more trouble
then it was worth to maintain.
Tests from py_dataset now have been ported to the test library for
libdataset.
Redundant functions have been removed (we had accumulated multiple
definitions for the same thing in libdataset). Where possible code has
been simplified.
Most libdataset functions will cause an “open” on a dataset
collection automatically. Some additional functions around collections
have been added primarily to make testing easier
(e.g. open_collection(), is_open(), close_collection(),
close_all()).
Functions that were overloaded via optional parameters have been
simplified. E.g. keys() now returns all keys in collection, use
key_filter() and key_sort() accordingly.
- Dropped support for GSheet integration
- Only support pairtree layout of collection
- cleaned up libdataset API focusing on removing overloaded
functions
Release 0.1.3:
Release 0.1.2:
- Persisting _Attachments metadata when updating with clean objects
using the same technique as _Key
Release 0.1.1:
- Fixed problem where keys_exist called before an open command.
Release 0.1.0:
- Updated libdataset API, simplified func names and normalized many of
the calls (breaking change)
- libdataset now manages opening dataset collections, inspired by
Oberon System file riders (breaking change)
- Added Python test code for libdataset to make sure libdataset
works
- Added support for check and repair when working on S3 deployed
collections
- Refactored and simplified frame behavior (breaking change)


I applied in Luminus Technical university college for a software engineering diploma, to pursue my passion for web developing. Being in a fast-paced environment where I was able to gain an in-depth knowledge on the programming field through an improvement process, which really solidified for me that software engineering was the right career path for me. I love companies that focus on leaving a small environmental footprint.
Welcome to my website
Get ready to know more about me!!
Education and Experience


Max David Metelus
Full Stack Developer At Lambda School
Scrum Master
Connect on Github
Davina Taylor
Full Stack Developer At Lambda School
Backend Engineer
Connect on Github
Jonathan Bernal
Full Stack Developer At Lambda School
Backend Engineer
Connect on Github
Benjamin Chandler
Full Stack Developer At Lambda School
Frontend Engineer
Connect on Github
Firealem Erko
Full Stack Developer At Lambda School
User Interface Developer
Connect on Github
Tim Rooney
Full Stack Developer At Lambda School
User Interface Developer
Connect on Github


Slime mold mechanisms
Slime mold is a broad term describing organisms that grow selectively based on the available food. If there is no food, slime mould grows exploratively. If there is food, the explorative branches die and the branches toward the food increase their capacity. The CS slime mold metaphor can be be used to handle intractable optimisation problems. In CS there are two slime-mold metaphors, viz. pipeline-based (Japan school 1995-now) and particle based (UK school, 2005-now). The particle-based approach is conceptually much simpler and gives results that are as rewarding the pipeline-based approach. Recently (2014), the particle-model has been used to tackle the travelling salesman problem, with some success.
CS models of slime mold are instances of so-called massive multi-agent learning or, more in particular, stigmercy-based multi-agent learning. Thousants of particles (agents) influence each other by dropping pheromone and perceiving pheromone on a grid.
Presented by Lecturer.
|
Slides T.B.P.
|
Screencast Not available.
|
Assignment Not this year.
|
Literature
Key
- “Influences on the formation and evolution of Physarum Polycephalum inspired emergent transport networks” J. Jones (2011) [scholar, lib]
Support
Physarum:
- “Road planning with slime mould if Physarum built motorways” Adamatzky & Jones (2009) [scholar, lib]
- “Comparison of Bio-Inspired and Graph-Theoretic Algorithms for Design of Fault-Tolerant Networks” Becker (2010) [scholar, lib]
- “Characteristics of pattern formation and evolution in approximations of Physarum transport networks” Jones (2010) [scholar, lib]
- “Design of fault tolerant networks with agent-based simulation of Physarum Polycephalum” M. Becker (2011) [scholar, lib]
- “Computation of the travelling salesman problem by a shrinking blob” Jones & Adamatzky (2014) [scholar, lib]
- “Permeability of terrain in particle-based models of Physarum Polycephalum” Van Dorsten (2015). BSc Thesis [scholar, lib].
- Some Dutch links.
Dicty:
- “A Simulation Study of Mechanisms of Group Selection of the Slime Mold Dictyostelium Discoideum” M. Becker (2010) [scholar, lib]
- “A Simulation Model of Dictyostelium Discoideum for the Study of Evolutionary Selection Mechanisms” M. Becker (2011) [scholar, lib]
- “A Simulation Model for the whole life cycle of the slime mold Dictyostelium Discoideum” M. Becker (2013) [scholar, lib]
Demos
Removed March-April 2018.


Contact information
College of Information and Control Engineering
China University of Mining and Technology
No1, Daxue Road, Xuzhou, Jiangsu, 221116, P . R .China
Tel: +86516-88888888
Fax: +86516-88888888
Email: szwfast at 163.com
Email: szwen1818 at gmail.com
Group Website: IRlab and Computer Vision


Rebecca Prows is a Python developer.
Learn more about her professional experience here.
See her some of her latest projects here.
Suzanne Richman is a fullstack JavaScript developer.
Learn more about her professional experience here.
See her some of her latest projects here.


wavemulcor: Wavelet Routines for Global and Local Multiple Correlation
Wavelet routines that calculate single sets of
wavelet multiple correlations (WMC) and cross-correlations (WMCC)
out of n variables.
They can later be plotted in single graphs, as an alternative to trying
to make sense out of several sets of wavelet correlations or
wavelet cross-correlations.
The code is based on the calculation, at each wavelet scale, of the
square root of the coefficient of determination in a linear combination
of variables for which such coefficient of determination is a maximum.
The code provided here is based on the wave.correlation routine in
Brandon Whitcher's waveslim R package Version: 1.6.4, which in turn is
based on wavelet methodology developed in Percival and Walden (2000) <doi:10.1017/CBO9780511841040>;
Gençay, Selçuk and Whitcher (2002) <doi:10.1016/B978-012279670-8.50013-6> and others.
Version 2 incorporates wavelet local multiple correlations (WLMC).
These are like the previous global WMC but consisting in one
single set of multiscale correlations along time. That is, at each time
t, they are calculated by letting a window of weighted wavelet
coefficients around t move along time. Six weight functions are provided.
Namely, the uniform window, Cleveland's tricube window, Epanechnikov's
parabolic window, Bartlett's triangular window and Wendland's truncated
power window and the Gaussian window.
Version 2.2 incorporates an auxiliary function that calculates local
multiple correlations (LMC). They are calculated by letting move along time
a window of weighted time series values around t. Any of the six weight
functions mentioned above can be used.
| Version: |
2.2.1 |
| Depends: |
R (≥ 3.4.0), waveslim (≥ 1.7.5) |
| Suggests: |
plot3D |
| Published: |
2018-06-08 |
| Author: |
Javier Fernandez-Macho [aut, cre] |
| Maintainer: |
Javier Fernandez-Macho <javier.fernandezmacho at ehu.es> |
| License: |
GPL-2 | GPL-3 [expanded from: GPL (≥ 2)] |
| NeedsCompilation: |
no |
| Materials: |
NEWS |
| CRAN checks: |
wavemulcor results |
Downloads:
Reverse dependencies:
Linking:
Please use the canonical form
https://CRAN.R-project.org/package=wavemulcor
to link to this page.


To start, install virtualenv and create a virtual environment for our experiments.
$ easy_install virtualenv $ virtualenv quickstart
Install zc.async in the virtual environment.
$ cd quickstart/ $ ./bin/easy_install zc.async
This installed several packages.
All of these, and zc.async, are distributed under BSD-like licenses such as LGPL and ZPL.
zc.async relies on a distributed ZODB technology called ZEO (“Zope Enterprise Objects”) to distribute work. ZEO has a central database server to which client processes connect.
Let’s start the ZEO Server:
$ ./bin/runzeo -a 9999 -f test.fs &
That starts a database server, accessible on port 9999 of your local machine, saving the data in the test.fs file.
Now let’s start a Python with a client connection to the database server.
Start up bin/python (not your system python, but the one in virtualenv’s quickstart/bin):
$ ./bin/python
This will be our single client process.
You might have many, each connecting to the main database server, and each able to perform and/or request zc.async jobs.
Connect to the database.
import ZEO.ClientStorage
import ZODB
storage = ZEO.ClientStorage.ClientStorage(
('<IP_ADDRESS>', 9999))
db = ZODB.DB(storage)
Now we do some basic configuration. This first bit installs some default adapters. You might not ever have to worry too much about them.
>>> import zc.async.configure
>>> zc.async.configure.base()
This second part is policy, and if you ever put zc.async in production, you’ll want to understand what’s going on here.
>>> zc.async.configure.start(db, poll_interval=1)
Now the system has a dispatcher polling for jobs every second. As we’ll see below, a dispatcher polls for jobs in one or more queues, to assign them to worker threads.
The start function also installed a queue. To get zc.async to do work, you put a job in a queue, and commit the transaction.
First, let’s get the queue that we have installed. We need to open a connection to the database. Then we get the queue.
>>> conn = db.open()
>>> import zc.async.interfaces
>>> q = zc.async.interfaces.IQueue(conn)
Let’s put a job in our queue. This silly example will return the current time.
>>> import time
>>> j = q.put(time.time)
It’s not done yet.
>>> j.result
>>> j.status
u'pending-status'
We have to commit the transaction for the dispatcher to see the job.
>>> import transaction
>>> transaction.commit()
Now wait a second and then try this. “transaction.begin” will sync up our database with database changes made elsewhere.
>>> _ = transaction.begin()
>>> j.result
1216179006.856108
>>> j.status
u'completed-status'
You can also make closures. The Job class accepts arguments similarly to the Python 2.5 functools.partial(): Job(func, \*args, \*\*keywords). This instantiates a new callable (a Job instance) with partial application of the given arguments and keywords. You can then pass the job instance to the put() method.
Generating RSA keys is actually a reasonable real-world use case for something like this.
import subprocess
j = q.put(zc.async.job.Job(
subprocess.call,
['openssl', 'genrsa', '-out',
'key.pem', '1024']))
transaction.commit()
We need to begin the transaction to see the result...
j.result
_ = transaction.begin()
j.result
...which in this case is simply 0, indicating a successful UNIX process.
0
We can open the file to show the result.
subprocess.call(['cat', 'key.pem'])
This will show you the key, which should look something like this:
-----BEGIN RSA PRIVATE KEY----- MIICXgIBAAKBgQCYAZW+HjDGJhRHnUlZZWqhrGOxU2K/RhssmcMs0JLnWI2cWmZ+ ... CEcz6ZbO8zm4AEGI/dqLicZh3bhunhflAovW6WxbNKLENQ== -----END RSA PRIVATE KEY----- 0
We’ve now seen some simple examples from the standard library. But how do you get your own work done?
Let’s say we want to implement an old chestnut of a problem: use a Monte Carlo simulation (that is, “throwing darts and analyzing the results”) to calculate pi. This will use zc.async much like the map-reduce approach of Hadoop: we will want to distribute the work of running the simulation to multiple machines so the result can be done faster.
You want a job to have a reference to your own callable, so the job will get the work you define performed.
This reference, of the job to your callable, will need to be persisted in the database.
Because zc.async uses the ZODB for its persistence mechanism, the ZODB’s persistence rules are in effect.
Luckily, these are fairly simple.
For now, we’ll stay as simple as it gets: if you use module global functions and immutables, and share software across instances, you’ll be fine.
ZODB allows a lot more, if you’re willing to follow a few more rules, but that one rule will get us moving for this quick-start.
Exit the Python interpreter (control-D). Look around in the directory (ls): you should see a few database files, the key.pem you created, and a new file: uuid.txt. It should look something like this:
$ cat uuid.txt afd1e0d0-52e1-11dd-879b-0017f2c49bdd ------------------------------------------------------------------------ The value above (and this file) is created and used by the zc.async package. It is intended to uniquely identify this software instance when it is used to start a zc.async dispatcher. This allows multiple dispatchers, each in its own software instance, to connect to a single database to do work. In order to decide where to look for this file (or to create it, if necessary), the module looks in ``os.environ['ZC_ASYNC_UUID']`` for a file name. If you are using zdaemon (http://pypi.python.org/pypi/zdaemon) to daemonize your process, you can set this in a zdaemon environment section of your zdaemon.conf. Supervisor (http://supervisord.org/) also provides this functionality. Other similar tools probably do as well. If the ``ZC_ASYNC_UUID`` is not found in the environment, it will use ``os.path.join(os.getgwd(), 'uuid.txt')`` as the file name. To get a new identifier for this software instance, delete this file, restart Python, and import zc.async.instanceuuid. This file will be recreated with a new value.
That text is intended to be self-explanatory, so hopefully it made sense to you. We’ll handle these UUIDs more explicitly in a moment.
Make a new Python file. Let’s call it pi.py. Save this file in lib/python2.5/site-packages/.
Use the following for the file content.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | import random
import math
import ZEO.ClientStorage
import ZODB
import twisted.internet.reactor
import zc.async.configure
def generate_sample(size=100000):
count = 0
for i in range(size):
if math.hypot(random.random(), random.random()) < 1:
count += 1
return count, size
def process_samples(*sample_jobs):
count = 0
size = 0
for j in sample_jobs:
count += j.result[0]
size += j.result[1]
return 4.0 * count / size
if __name__ == '__main__':
storage = ZEO.ClientStorage.ClientStorage(
('<IP_ADDRESS>', 9999))
db = ZODB.DB(storage)
zc.async.configure.base()
zc.async.configure.start(
db, poll_interval=0.1, twisted=True)
twisted.internet.reactor.run()
|
We’ll need the ZEO server running. If you’ve gone through this file from the start, it should still be running. If not, use this:
$ ./bin/runzeo -a 9999 -f test.fs &
Now, for our first experiment with the Monte Carlo simulation, we’ll start a single worker process.
Enter this in the terminal:
$ ./bin/python lib/python2.5/site-packages/pi.py &
That will start our worker process. Now let’s start an interpreter.
$ ./bin/python
Now get a database and a connection, as we’ve seen before. We’ll also set up the base zc.async configuration.
import ZEO.ClientStorage
import ZODB
storage = ZEO.ClientStorage.ClientStorage(
('<IP_ADDRESS>', 9999))
db = ZODB.DB(storage)
conn = db.open()
import zc.async.configure
zc.async.configure.base()
We don’t have any adapters installed, so zc.async.interfaces.IQueue(conn) won’t work. This will though, and still looks pretty good:
>>> import zc.async.queue
>>> q = zc.async.queue.getDefaultQueue(conn)
Now we can start some jobs in parallel().
>>> import pi
>>> import zc.async.job
>>> j = q.put(zc.async.job.parallel(
... pi.generate_sample, pi.generate_sample, pi.generate_sample,
... postprocess=pi.process_samples))
>>> import transaction
>>> transaction.commit()
Wait a few seconds. If the result is empty (None), begin the transaction again and check the result again. Eventually, these next two lines should give you a result: an approximation of pi.
_ = transaction.begin()
j.result
For one run, I got 3.1386666666666665. Cool.
We’ve already seen Jobs used as closures in the openssl example. You can also use them to pass a different size argument to generate_sample.
Let’s try it.
>>> j = q.put(zc.async.job.parallel(
... zc.async.job.Job(pi.generate_sample, 1000000),
... zc.async.job.Job(pi.generate_sample, size=1000000),
... postprocess=pi.process_samples))
>>> transaction.commit()
Wait a bit, again. Retry these two lines until you get a result. It should be well under a minute on most machines.
_ = transaction.begin()
j.result
My run got 3.1434359999999999. Cool.
Unfortunately, the parallel runs we’ve done so far would actually make the calculation go slower than if we did it in a single function call! That’s because we ran it in a single Python worker process, and the overhead of threads just makes the jobs a bit less efficient. Threads help for some uses of zc.async, but not this one.
Let’s assume you are running these experiments on a machine with two processor cores. We should actually start two worker processes then, one for each core. We’ll need to make sure each worker process has its own OID.
Moreover, we need to configure each process to only take one generate_sample job at a time. Let’s adjust our code to do that.
A worker process regularly polls the database for new jobs. The software component that polls is called a dispatcher. Dispatchers look in the database to ask their personal agent (or agents) to determine what they should get their threads to do. Think of the agent as a “talent agent” or a “booking agent” for the process.
By default, using the zc.async.configure helpers, each dispatcher is given a single agent that will choose the first job in the queue, and that wants to run no more than three jobs at a time.
For our Monte Carlo job, we’ll give each process an additional agent. The new agent will only accept up to one instance of a generate_sample job at a time.
We will also reconfigure the existing agent to accept up to three of anything except generate_sample.
We’ll do that in our file. Here’s the revised version. The only changes are the imports, the three new functions, and setting up install_agent in the if __name__ == '__main__': block.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | import random
import math
import ZEO.ClientStorage
import ZODB
import transaction
import twisted.internet.reactor
import zc.async.configure
import zc.async.queue
import zc.async.instanceuuid
import zc.async.agent
def generate_sample(size=100000):
count = 0
for i in range(size):
if math.hypot(random.random(), random.random()) < 1:
count += 1
return count, size
def process_samples(*sample_jobs):
count = 0
size = 0
for j in sample_jobs:
count += j.result[0]
size += j.result[1]
return 4.0 * count / size
def choose_generate_sample(agent):
return agent.queue.claim(
lambda j: j.callable.__name__ == 'generate_sample')
def choose_another(agent):
return agent.queue.claim(
lambda j: j.callable.__name__ != 'generate_sample')
def install_agent(db):
conn = db.open()
try:
q = zc.async.queue.getDefaultQueue(conn)
try:
dispatcher = q.dispatchers[zc.async.instanceuuid.UUID]
except KeyError:
twisted.internet.reactor.callLater(0.05, install_agent, db)
else:
if 'generate_sample' not in dispatcher:
agent = dispatcher['main']
agent.chooser = choose_another
dispatcher['generate_sample'] = zc.async.agent.Agent(
choose_generate_sample, 1)
transaction.commit()
finally:
transaction.abort()
conn.close()
if __name__ == '__main__':
storage = ZEO.ClientStorage.ClientStorage(
('<IP_ADDRESS>', 9999))
db = ZODB.DB(storage)
zc.async.configure.base()
zc.async.configure.start(
db, poll_interval=0.1, twisted=True)
twisted.internet.reactor.callWhenRunning(install_agent, db)
twisted.internet.reactor.run()
|
Now let’s start up our workers, and see how they work. We’re going to have two workers now, and they will each need separate UUIDs. A really simple approach will be to make two separate working directories for the two worker processes. (We also could use the environmental variable, ZC_ASYNC_UUID, described in the Process UUIDs section above.)
$ mkdir worker1 $ mv uuid.txt worker1 $ cd worker1 $ ../bin/python ../lib/python2.5/site-packages/pi.py & $ cd .. $ mkdir worker2 $ cd worker2 $ ../bin/python ../lib/python2.5/site-packages/pi.py &
Now we’ll start the Python process in which we will test our code. We’ll move to the main directory, but as long as we don’t start another worker, it doesn’t really matter.
$ cd .. $ ./bin/python
And now, our test.
import ZEO.ClientStorage
import ZODB
storage = ZEO.ClientStorage.ClientStorage(
('<IP_ADDRESS>', 9999))
db = ZODB.DB(storage)
conn = db.open()
import zc.async.configure
zc.async.configure.base()
import pi
import zc.async.job
import zc.async.queue
q = zc.async.queue.getDefaultQueue(conn)
j = q.put(zc.async.job.parallel(
zc.async.job.Job(pi.generate_sample, 5000000),
zc.async.job.Job(pi.generate_sample, size=5000000),
postprocess=pi.process_samples))
import transaction
transaction.commit()
Wait a few seconds and then try these lines.
_ = transaction.begin()
j.result
If the result is empty (None), repeat those two lines again (thatæ is, begin the transaction again and check the result again). Eventually, these lines should give you a result: an approximation of pi.
Just to prove to ourselves that we saved some time, let’s do a comparison test: the same number of samples, but not in parallel.
j2 = q.put(zc.async.job.parallel(
zc.async.job.Job(pi.generate_sample, 10000000),
postprocess=pi.process_samples))
transaction.commit()
_ = transaction.begin()
j2.result
Once both jobs are complete, compare their run-time.
j.active_end - j.active_start
j2.active_end - j2.active_start
On my machine, even in this simple, short example, running in parallel with a job per processor/core took 7.8 seconds, while running all in one process took 13.4 seconds.
Soon, you may want to be able to monitor or introspect what’s going on in your zc.async work. The package provides several tools to do that. We’ll take a look at a few here.
We will be turning on a monitor port. This port should be protected behind a firewall, like your ZEO ports.
If you like the functionality that we describe here but would prefer to expose it in a different manner, note that most of the Python functions in monitor.py and monitordb.py power the zc.async commands in the monitor port, and can be used without the monitor itself.
To enable the monitoring port, we need to install some extra dependencies for zc.async: “[monitor]”. Exit the Python interpreter an make sure you are in the top “quickstart” directory, and then enter this command:
$ ./bin/easy_install zc.async[monitor]
If you take a glance at the output, you’ll see we’ve only added a few dependencies: simplejson, zc.ngi, and zc.monitor.
Now let’s turn on the port and the zc.async commands.
At the top of the pi.py file, add some imports:
import os
import zope.component
import zc.monitor
import zc.monitor.interfaces
import zc.async.monitor
import zc.async.monitordb
Then down in the if __name__ == '__main__': block, add these lines at the top.
monitor_port = os.environ.get('MONITOR_PORT')
if monitor_port:
for f in (zc.monitor.interactive, zc.monitor.quit, zc.monitor.help,
zc.async.monitor.async, zc.async.monitordb.asyncdb):
zope.component.provideUtility(
f, zc.monitor.interfaces.IMonitorPlugin, f.__name__)
zc.monitor.start(int(monitor_port))
The file should look like this now.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | import os
import random
import math
import ZEO.ClientStorage
import ZODB
import transaction
import twisted.internet.reactor
import zc.monitor
import zc.monitor.interfaces
import zope.component
import zc.async.configure
import zc.async.queue
import zc.async.instanceuuid
import zc.async.agent
import zc.async.monitor
import zc.async.monitordb
def generate_sample(size=100000):
count = 0
for i in range(size):
if math.hypot(random.random(), random.random()) < 1:
count += 1
return count, size
def process_samples(*sample_jobs):
count = 0
size = 0
for j in sample_jobs:
count += j.result[0]
size += j.result[1]
return 4.0 * count / size
def choose_generate_sample(agent):
return agent.queue.claim(
lambda j: j.callable.__name__ == 'generate_sample')
def choose_another(agent):
return agent.queue.claim(
lambda j: j.callable.__name__ != 'generate_sample')
def install_agent(db):
conn = db.open()
try:
q = zc.async.queue.getDefaultQueue(conn)
try:
dispatcher = q.dispatchers[zc.async.instanceuuid.UUID]
except KeyError:
twisted.internet.reactor.callLater(0.05, install_agent, db)
else:
if 'generate_sample' not in dispatcher:
agent = dispatcher['main']
agent.chooser = choose_another
dispatcher['generate_sample'] = zc.async.agent.Agent(
choose_generate_sample, 1)
transaction.commit()
finally:
transaction.abort()
conn.close()
if __name__ == '__main__':
monitor_port = os.environ.get('MONITOR_PORT')
if monitor_port:
for f in (zc.monitor.interactive, zc.monitor.quit, zc.monitor.help,
zc.async.monitor.async, zc.async.monitordb.asyncdb):
zope.component.provideUtility(
f, zc.monitor.interfaces.IMonitorPlugin, f.__name__)
zc.monitor.start(int(monitor_port))
storage = ZEO.ClientStorage.ClientStorage(
('<IP_ADDRESS>', 9999))
db = ZODB.DB(storage)
zc.async.configure.base()
zc.async.configure.start(
db, poll_interval=0.1, twisted=True)
twisted.internet.reactor.callWhenRunning(install_agent, db)
twisted.internet.reactor.run()
|
Now stop and restart your two worker instances, this time providing two different ports in the environment for each worker. Here’s one way to do it. First we’ll shut down the previous instances. If you used the lines above to start them before, type fg and RETURN, and then CTRL-C to stop worker 2; and then do the same thing (fg and RETURN, and then CTRL-C) to stop worker 1.
$ cd worker1 $ MONITOR_PORT=9991 ../bin/python ../lib/python2.5/site-packages/pi.py & $ cd ../worker2 $ MONITOR_PORT=9992 ../bin/python ../lib/python2.5/site-packages/pi.py &
Now you can open connections to these two ports, 9991 and 9992, and query the worker (using the async command, primarily) and the state of zc.async in the database itself (the asyncdb command).
The easiest way to experiment with this is using telnet. Try this.
$ telnet <IP_ADDRESS> 9991
You should see something like this:
Trying <IP_ADDRESS>... Connected to localhost. Escape character is '^]'.
Now you can experiment with commands. Try this (followed by a RETURN):
help
You should see something like this:
Supported commands: async -- Monitor zc.async activity in this process. asyncdb -- Monitor and introspect zc.async activity in the database. help -- Get help about server commands interactive -- Turn on monitor's interactive mode quit -- Quit the monitor Connection closed by foreign host. $
Hm, it dumped us straight back to the shell! zc.monitor behaves that way tp be friendly to automated monitoring processes using the port. We can use the interactive command to make things a bit more pleasant for ourselves.
Reuse the telnet command shown above, or maybe connect to 9992 (telnet <IP_ADDRESS> 9992) to see that you can. This time, type the interactive command first. You should see this reply:
Interactive mode on. Use "quit" To exit.
Now experiment! Try some of these commands to see what you get.
help async async help async help status async status async help poll async poll async UUID async utcnow help asyncdb asyncdb help asyncdb help UUIDs asyncdb UUIDs asyncdb lastcompleted asyncdb count completed asyncdb count completed uuid:THIS
That last command shows how many jobs this worker has completed for as long as the database has records, which by default means between seven and eight days. The help for asyncdb count, which is available from asyncdb help count, is not short, but tells you the many options available.
When you are done, use the quit command to exit the telnet connection to the monitor port.
Another important tool is logging. The zc.async.event logger gets important events–pay special attention to critical events! zc.async.trace lets you follow along with every detail, if so desired.
These monitoring and introspection tools, combined with logging, provide powerful tools to keep track of your zc.async work.
We’re at the end of this quickstart. To close, here’s a quick survey of some other configuration opportunities available that we haven’t seen here.
Note
Unlike Twisted callbacks, all callbacks for the same job get the same result; if you would like to chain results, the callbacks themselves are Jobs, so attach a callback to your callback.
Note
There is no built-in way to stop a running job, short of stopping the process. This can be approximated by use in your job of getLiveAnnotation() to poll for stop requests; or the brave can write some C to use PyThreadState_SetAsyncExc.
There are many other topics to discuss–testing, debugging, and Zope 3 integration, for instance–but this is a quick start, so we’ll end here.


It can be tricky to calculate the absolute path of a link -
especially if you know the page will be loaded from somewhere else.
Just use the page-href
custom binding instead of specifying
a href
directly!
The page-href
link will be relative to the page where the
element is. So if an anchor tag is in the page "user" a value pelle
will be resolved to user/pelle
. The binding also understands ../
if you need to track back up the page hierarchy!
pager.Href.hash
is by default #
but can be changed to #!/
(or any other string). All page-href
bindings will use the prefix from pager.Href.hash
.

 Fry

 Interesting

 Close this tab


--- subject: The Wisemans Surf Lodge Property Trust is now open for investment --- {{>header-preview-text}} {{>header-logo copy="Welcome to a new era for surfing"}} {{>image-header image-desktop-url="https://f.hubspotusercontent30.net/hubfs/8837988/Wiseman%20eDM%20images/wiseman-image-body-3.png" image-mobile-url="https://f.hubspotusercontent30.net/hubfs/8837988/Wiseman%20eDM%20images/wiseman-image-body-3-mob.png"}} {{>body-copy-start}}
Thank you for registering your interest in Wisemans Surf Lodge.
We’re guessing that you, like us, dream of a surf trip where every wave is perfect and uncrowded. Well that dream is soon to become a reality.
The land, the hotel, the wave pool, the restaurant, the golf course - everything will be wrapped inside a property fund that you can invest in, which like any property investment, can earn income and potentially benefit from capital growth over time.
Learn all about how it works and secure your spot today at wisemans.surf
{{>body-copy-end}} {{>callout-two-buttons cta-1='Find out more' url-1='https://www.wisemans.surf/home' cta-2='Let’s chat' url-2='https://preview.wisemans.surf/invest'}} {{>spacer size='30'}} {{>footer}}


$template{name="sample-readme-html-header" title="Custom Load Balancer demo"}$
What does the sample do?
This sample illustrates the implementation of a custom load-balancer that uses information about the nodes and the submitted jobs, to dispatch JPPF tasks to the nodes.
Description of the problem to solve
We are in a situation where we need to submit two types of jobs for execution in the JPPF grid:
- "heavy jobs", made of a small number of long-lived tasks with a large memory footprint
- "light jobs", made of a large number of short-lived tasks with a small memory footprint
We want the JPPF server to be able to dispatch the tasks to the apppropriate node(s), based on what it knows of each node's capabilities and on the jobs characteristics.
Description of the solution
To resolve this problem, we will use a combination of a custom load balancer that is aware of the node's capabilities, execution policies for the jobs, and job metadata the load-balancer will be able to use.
a) Information provided by each job (job metadata):
- "task.memory": the maximum size in bytes neded by each task in the job
- "task.time": the maximum duration of each task in the job
- "allowed.time": the maximum allowed time for execution of a single set of tasks on a node
- "id": the job id (for debugging and logging purposes only)
- for full details, please take a look at the fully commented job runner source code
b) Information provided by each node:
- "maxMemory": the maximum heap size available to the node
- "processing.threads": the number of processing threads available to the node
c) Our load-balancer will compute the maximum number of tasks in the job that can be sent to a node, based on:
- the estimated total memory footprint of the tasks, which must fit in the node's available memory.
- the estimated total execution time, that must be less than the allowed time
- for full details, please take a look at the fully commented load-balancer source code
d) Furthermore, additional node filtering will be applied before the load-balancing comes into play, by the means of an execution policy for each job:
- heavy jobs can only be executed on nodes such that the amount of available memory per processing thread is at least the memory footprint of each task;
this will avoid under-utilization of nodes with many processing threads, while avoiding out of memory issues
- light jobs will only be executed on nodes that have at east 2 processing threads, to minimize the network transport overhead, which can be significant for short-lived tasks
Here is a set of links to the source of each Java class in this sample:
How do I run it?
Before running this sample, you need to install a JPPF server and at least two nodes.
For information on how to set up a node and server, please refer to the JPPF documentation.
Once you have installed a server and node, perform the following steps:
- To configure the two nodes, there are 2 predefined node configurations we will use:
- copy CustomLoadBalancer/config/node1/jppf-node.properties to the config folder of your first node installation (it will replace the existing file).
This will setup the first node with 64 MB of heap and 1 processing thread; this node will be used for "heavy" jobs
- copy CustomLoadBalancer/config/node2/jppf-node.properties to the config folder of your second node installation (it will replace the existing file).
This will setup the second node with 64 MB of heap and 4 processing thread; this node will be used for "light" jobs
- open a command prompt in JPPF-x.y-samples-pack/CustomLoadBalancer
- build the sample: type "ant jar"; this will create a file named CustomLoadBalancer.jar
- copy CustomLoadBalancer.jar in the "lib" folder of the JPPF driver installation, to add it to the driver's classpath. This will effectively install the new load-balancer.
- in the server's installation config/jppf-driver.properties file, replace the property "jppf.load.balancing.algorithm = xxxxx" with "jppf.load.balancing.algorithm = customLoadBalancer",
to let the server know it must use the new load-balancer
- start the JPPF server and each of the 2 nodes
- the demo application should already be configured with a connection pool size of 2; to confirm it you can open the file jppf-client.properties in CustomerLoadbalancer/config, you should see a line "jppf.pool.size = 2"
- start the demo, by opening a console in JPPF-x.y-samples-pack/CustomLoadBalancer, and typing: "ant run"
- in the first node's console output you should see messages of this type:
[java] Starting execution of task Heavy Job - task 1 (as many as there are tasks in the "heavy" job)
- in the second node's console output you should see messages of this type:
[java] Starting execution of task Light Job - task 2 (as many as there are tasks in the "light" job)
- the demo console output should show something like this:
[java] ********** Results for job : Heavy Job **********
[java]
[java] Result for task Heavy Job - task 1 : the execution was performed successfully
[java] Result for task Heavy Job - task 2 : the execution was performed successfully
[java] Result for task Heavy Job - task 3 : the execution was performed successfully
[java] Result for task Heavy Job - task 4 : the execution was performed successfully
[java]
[java] ********** Results for job : Light Job **********
[java]
[java] Result for task Light Job - task 1 : the execution was performed successfully
[java] Result for task Light Job - task 2 : the execution was performed successfully
[java] Result for task Light Job - task 3 : the execution was performed successfully
[java] Result for task Light Job - task 4 : the execution was performed successfully
.....
What features of JPPF are demonstrated?
I have additional questions and comments, where can I go?
If you need more insight into the code of this demo, you can consult the Java source files located in the CustomLoadBalancer/src folder.
In addition, There are 2 privileged places you can go to:
$template{name="sample-readme-html-footer"}$


Setting up Filelug is simple. Register or sign in with your phone number and connect to the computer from your cell phone. No extra devices required. You don't need to know the IP address of your computer, or worry about configuring any network settings.
Filelug transfers files at any time between your various mobile devices and computers. We support Windows/Mac/Linux computers, iPhone, iPad, Android phones and tablets.
Filelug securely transfers your files and protects them from unauthorized access. You need to sign in to use it. All data sent and received is securely encrypted during transmission, and we do not keep nor see any of your uploaded and downloaded files.


All sounds are paused when the window loses focus and resumed when it gets back focus. This behaviour can be prevented using the stopSound() method.
Play sound Pause sound Stop sound


--- layout: post status: publish published: true title: Promotions Galore - Podcast Episode 6.19 author: display_name: Corey Fineran login: Corey email<EMAIL_ADDRESS>url: '' author_login: Corey author_email<EMAIL_ADDRESS>wordpress_id: 10145 wordpress_url: http://ivyenvy.com/?p=10145 date: '2014-06-22 10:51:10 +0000' date_gmt: '2014-06-22 15:51:10 +0000' categories: - Minor Leagues - Starting Pitching - Podcast - Prospects tags: - Jeff Samardzija - Arodys Vizcaino - Armando Rivero - Kris Bryant - Kyle Schwarber comments: - id: 11677 author: Clint Ward author_email: '' author_url: '' date: '2014-06-22 11:19:03 +0000' date_gmt: '2014-06-22 16:19:03 +0000' content: Good I can listen on my way home from work today. - id: 11678 author: Brock Wilson author_email: '' author_url: '' date: '2014-06-22 12:52:28 +0000' date_gmt: '2014-06-22 17:52:28 +0000' content: Going to listen while watching the Cubbies! Good Sunday. - id: 11679 author: Michael Cotton author_email: '' author_url: '' date: '2014-06-22 16:27:44 +0000' date_gmt: '2014-06-22 21:27:44 +0000' content: Looking forward to it but will probably wait til tomorrow anyway. - id: 11687 author: Jason Mollett author_email: '' author_url: '' date: '2014-06-22 22:52:11 +0000' date_gmt: '2014-06-23 03:52:11 +0000' content: Looks like Bryant's slump is over - id: 11688 author: Jason Mollett author_email: '' author_url: '' date: '2014-06-22 22:52:58 +0000' date_gmt: '2014-06-23 03:52:58 +0000' content: I'm going to a Boise Hawks game in Spokane. Who are the players to scout? - id: 11707 author: Shawn O'Hare author_email: '' author_url: '' date: '2014-06-23 14:10:58 +0000' date_gmt: '2014-06-23 19:10:58 +0000' content: 'Clarification: http://www.rollingstone.com/music/news/tom-morello-how-the-chicago-cubs-can-win-the-world-series-20140621 and the rebuttal I agree with: http://www.chicagonow.com/cubs-insider/2014/06/tom-morello-should-stick-to-shredding-and-leave-cubs-strategy-to-the-pros/' ---
As they hit the middle of the minor league season, the Cubs promoted some of their top prospects, including the "will never struggle in AA" and Southern League Home Run Derby Champion, Kris Bryant.
On this week's episode, Andy joins Corey in studio to discuss all of the promotions. Kyle Schwarber was promoted to Kane County after a handful of games in Short-Season Boise. Arodys Vizcaino and Armando Rivero were also promoted in the last week.
Jeff Samardzija recently rejected the most recent (and probably final) contract offer from the Cubs. We talk a little about this, and the seemingly inevitable trade that will send Samardzija to another team.
We read some listener questions and a few Internet Muscle Flexes.


All Classes
-
Class Summary Class Description Calibrator This is an implementation of the camera calibration method described in Z.Calibrator.Parameters Inner class representing a set of parameters for instantiating new objects of type ofCalibrator
.Camera A camera model with parameters as specified in Zhang's paper.CameraIntrinsicsEstimator This class defines methods for estimating the intrinsic camera parameters from multiple homographies.Demo_Draw_3D_Axes This plugin projects and draws the projected X/Y/Z coordinate axes for each of the given camera views.Demo_Rectification This plugin projects opens an image stack containing the 5 Zhang test images and removes the lens distortion based on the calibrated camera parameters.Demo_Replace_Camera This plugin projects opens an image stack containing the 5 Zhang test images (assumed to be taken with camera A) and re-renders the images by mapping them to a new camera B.Demo_View_Interpolation This plugin performs interpolation of views, given a sequence of key views.Demo_Zhang_Calibration This plugin performs Zhang's camera calibration on the pre-calculated point data for the N given target views.Demo_Zhang_Projection This plugin projects opens an image stack containing the 5 Zhang test images and projects the model points into each view, using the (known) camera and view parameters.Demo_Zhang_Projection_Overlay This plugin projects opens an image stack containing the 5 Zhang test images, then outlines the positions of the observed image points and finally projects the points of the calibration model using the calculated intrinsic camera parameters (same for all views) and the extrinsic parameters calculated for each view.ExtrinsicViewEstimator This class defines methods for estimating the extrinsic camera parameters from multiple homographies.GridPainter Helper class for drawing projections of the calibration target.HomographyEstimator This class defines methods for estimating the homography (projective) transformation between pairs of 2D point sets.HomographyEstimator2 Deprecated. IjUtil This class defines ImageJ-related utility methods used for camera calibration.InterCameraMapping This class represents the 2D geometric transformation for an image taken with some camera A to an image taken with another camera B.MathUtil Utility math methods used for camera calibration.NonlinearOptimizer Abstract super-class for non-linear optimizers used for final, overall optimization of calibration parameters.NonlinearOptimizerAnalytic Nonlinear optimizer based on the Levenberg-Marquart method, where the Jacobian matrix is calculated from analytic expressions (obtained with MatLab).NonlinearOptimizerNumeric Nonlinear optimizer based on the Levenberg-Marquart method, where the Jacobian matrix is calculated numerically (i.e., by estimating the first partial derivatives from finite differences).Open_Test_Images Opens Zhang's standard calibration images as a stack of RGB images.RadialDistortionEstimator This class defines methods for estimating the radial lens distortion parametersRectificationMapping This class represents a special geometric mapping for rectifying (i.e., removing the lens distortion from) an image, given the associated camera parameters.RotationTests ViewTransform Instances of this class represent extrinsic camera (view) parameters.ZhangData Supplies all data for Zhang's demo calibration test suite.


S&P 500 Earnings: Maybe The S&P 500's 'Forward Estimate' Has Finally Bottomed
by Brian Gilmartin, CFASummary
- If the trend continues higher, it bodes positively for the S&P 500.
- It's easy to get lost in this wonky world, and if the data is stared at long enough, it can mean whatever we want it too.
- Let's focus on that forward estimate rate of change the next few weeks and see if it has bottomed.
Originally published November 17, 2019
With yesterday's post, the S&P 500 earnings data was not posted.
S&P 500 earnings data (by the numbers):
Using forward earnings data:
- Fwd 4-qtr est: $172.51 vs. last week's $172.90
- Fwd P/E: 18x
- Fwd PEG: 43x (will explain this further down)
- Fwd S&P 500 earnings yield: 5.53% vs. last week's 5.59%
- Year-over-year growth of fwd est: +0.42% vs. last week's +0.42%
- (It's important that readers understand that the "y/y growth of the forward estimate (last bullet point) measures the "forward estimate" today vs. 52 weeks prior. The low number of 0.42% means that there has been little change in the forward estimate the last 52 weeks (pretty amazing, no?), hence the very high PEG ratio, which is calculated as "18x P/E divided by 0.42%" or 43x.
Using trailing earnings data:
- TTM EPS est: $163.92 vs. $163.84 (technically this is still an estimate because it includes the Q3 '19 earnings data still being reported)
- P/E ratio: 19x
- PEG ratio: 3.45x
- TTM S&P 500 earnings yield: 5.25% vs. last week's 5.30%
- TTM y/y growth: 5.24% vs. 5.53% (calculated by dividing the forward estimate of $172.51 by the 4-tr trailing est of $163.92)
This trend while still early is worth watching:
Note the 1-year change % - it hasn't made a new low in the last three weeks.
The stat doesn't sound like much but since the week of September 21, 2018, where the "1-yr chg %" peaked at 22.92% (influenced quite heavily by tax reform), the trend has been straight down for 61 weeks, with the exception of 4 weeks in May '19 when it was flat-to-higher. (Guess when the S&P 500 peaked at 2,940.91 before the 20% Q4 '18 correction? Yep, September 21, 2018.)
This is a wonky stat and to my knowledge, this is the only blog that tracks it (or has tracked it for 20 years) and the statistic isn't well understood, but it really measures the "rate of change" of the forward estimate.
If the trend continues higher, it bodes positively for the S&P 500.
Leon Cooperman was on CNBC earlier this year and thought calendar 2019 S&P 500 earnings would be $171-172 per share. The current estimate today is $162.77 and we still have Q4 '18 earnings to get through. Tax reform heavily distorted 2018 S&P 500 earnings which then distorted 2019 earnings. Calendar 2019 earnings will likely come in around $161-162 per share.
Summary/Conclusion: Looking at the data, I hadn't realized the "forward estimate" y/y change peaked the same week as the S&P 500 in late September 2018. It's easy to get lost in this wonky world, and if the data is stared at long enough, it can mean whatever we want it too.
The goal of this morning's blog was not this, but the data was missing from yesterday.
Let's focus on that forward estimate rate of change the next few weeks and see if it has bottomed.
Thanks for reading.
Editor's Note: The summary bullets for this article were chosen by Seeking Alpha editors.


{% extends "d3b_generic_form.html" %} {% block serviceform %}
|
Defined filters:
none
{% for cfilter in pfkeys %}
{{ cfilter }}
{% endfor %}
|
Selected filter: List of keywords (taxonomic groups): Select only groups in the list Select all except groups in the list |


The procedure to enter Israel is as follows:-
At least 48 hours, preferably more, before arrival, email the Israeli Navy<EMAIL_ADDRESS>stating your routing, timing and crew list and ask them to e-mail you back the form ‘YACHT-IMOT’.
Fill it in (it is a standard Word document) and send it back. This completed IMOT will contain information about your previous ports of call, crew list, yacht description and call sign. Your IMOT will be circulated to Israeli Navy on shore and at sea and all the answers you give tested against those on the IMOT, so it will be worth keeping a copy of what you sent them.
A question not on the IMOT but a pre-emptive answer worth adding is ‘There are no weapons on board.’ If you have a referee in Israel it is worth adding the details, but warn your contact first as they could be contacted at any time, day or night!
Be sure also to e-mail the marina you intend to visit in advance as well.
When about 20 miles from the coast, you are likely to be contacted by the Israeli Navy. If no contact is made, the Navy will make further attempts at contact and visual identification is also common so do no be alarmed if approached by a navy vessel.
All marinas monitor Channel 16, but use different working channels: 11 at Herzliya, Ashkelon and Eilat, 10 at Tel Aviv, and 9 at Ashdod (The Blue Marina). Channel 16 must be monitored permanently.
On approaching the coast yachts may be met by an armed patrol boat and asked for details of the yacht and crew. Sometimes the yacht is boarded for inspection while still offshore. On arrival yachts are met by Customs and Immigration. The yacht may be searched for arms or explosives.
No yacht may proceed from Israel to Lebanon.
It is advisable to reach the Israeli coasts and marinas during daylight hours. Herzlyia and Tel-Aviv Border Control only operates during weekday working hours (09:00-16:00). Be careful also not to arrive during the Shabat (Friday afternoon through Saturday night) as there are substantial charges for clearing-in on these days.
Last updated September 2016.
Passports must be valid for a minimum of six months from the date of expected departure.
Nationals of the following countries do not require visas, or will have them issued free of charge on arrival: The European Union, Argentina, Australia, Bahamas, Barbados, Bolivia, Brazil, Canada, Central African Republic, Chile, Colombia, Costa Rica, Dominican Republic, Ecuador, El Salvador, Fiji, Guatemala, Haiti, Hong Kong, Iceland, Jamaica, Japan, Korea South, Lesotho, Liechtenstein, Malawi, Mauritius, Mexico, Micronesia, Monaco, Mongolia, New Zealand, Norway, Panama, Paraguay, Philippines, San Marino, South Africa, Spain, St Kitts & Nevis Surinam, Swaziland, Switzerland, Trinidad & Tobago, United States of America, Uruguay, Vanuatu.
Other nationalities must apply for visas in advance, and pay a fee.
Normally a three month stay is granted on arrival. After three months, one can apply for a visa or extension of a visa at a Ministry of the Interior office. This extension is usually for a further three months.
Entry stamps are not longer added to foreign passports. Instead, visitors are issued with a an Entry card which should be kept with your passport at all times.
However, if applying for an extension after the initial three months, this will automatically be stamped in the passport.
Last updated August 2015.
Firearms must be declared.
The import of fresh meat, some fruits and vegetables is prohibited.
Approval by Customs is required to take the following items into Israel: flowers, plants and seeds, prescription drugs, all animals, knives or penknives not intended for professional or household use.
Last updated August 2015.
Typhoid and polio vaccinations are recommended.
This must be applied for within 30 days of arrival.
Checking in and out is 236 ILS ($69) each way.
Overtime is payable during the Shabat (Friday afternoon through Saturday night) and on public holidays at 900 ILS ($263).
Harbour fees are calculated on length.
There are light fees.
Last updated July 2014.
There are several areas prohibited to yachts. Information on these areas is obtainable from chart suppliers and is drawn on Navy charts, which are available in every marina.
In areas where there is a concentration of orthodox jews, it is considered a violation of Shabbat (Sabbath, on Saturday) to smoke on that day. There is usually a sign to remind the visitor of this, and to disregard the warning would be regarded as discourteous. In most areas of the country, there is no restriction other than the usual non-smoking in public buildings, and that applies every day.
Dogs, cats and birds need a veterinary health certificate issued in the country of origin. Dogs over three months old need a rabies vaccination certificate issued between one year and one month before arrival.
If the yacht carries more than two of any of these animals and other species of animal, an import permit issued by the Director of Veterinary Services is required. A written application must be received not less than 10 days before the planned arrival of such animals.
Details of every import of animals must be notified to Ramla Quarantine Station at least 48 hours prior to arrival to Israel (Fax 972-8-9229906) Details to include owner, animal species, age, flight number and approximate time of arrival. Animals arriving in good health and correctly certified will generally not be quarantined.
Dogs and cats younger than 3 months will not be admitted.
Unaccompanied animals, or such which require an Import Permit, will require Quarantined in an official quarantine station for a period of 8 days or as decided by the Director.


Lets give the deserted talent back to our country (rodrigo).
They are the present pun intended.
Saint Augustine proposed that the present is a knife edge between the past and the future and could not contain any extended period of time.
Also, cats believes in present as serious stuff.
Rodrigo dont't have a future... who cares? no body has one, lets give rodrigo a present.
Rodrigo represents a country, Colombian people, Colombian youth. Rodrigo IS the present.
Well, in present we have a bunch of talented people who kind of deserted but believe in a future.
They come back in christmas enjoy and discusses the future. Rodri.co is a channel to make that future a present, think less, do more and do it now.
It's more than a think tank. Our deliverables aren't concepts or ideas, in present Rodrigo needs solutions.
Lets call it a BUILD TANK !
As an experiment rodri is a partial ordered set of projects with clearly defined i/o.
As usual, nothing comes before the aleph. Projects details will be disclosed soon.
Let your email to be in touch.
hecho con amor en medallo