comp neuro
Contents
7.4. comp neuro#
7.4.1. introduction#
<IP_ADDRESS>. overview#
does biology have a cutoff level (likecutoffs in computers below which fluctuations don’t matter)
core principles underlying these two questions
how do brains work?
how do you build an intelligent machine?
lacking: insight from neuro that can help build machine
scales: cortex, column, neuron, synapses
physics: theory and practice are much closer
are there principles?
“god is a hacker” - francis crick
theorists are lazy - ramon y cajal
things seemed like mush but became more clear - horace barlow
principles of neural design book
felleman & van essen 1991
ascending layers (e.g. v1-> v2): goes from superficial to deep layers
descending layers (e.g. v2 -> v1): deep layers to superficial
solari & stoner 2011 “cognitive consilience” - layers thicknesses change in different parts of the brain
motor cortex has much smaller input (layer 4), since it is mostly output
<IP_ADDRESS>. historical ai#
people: turing, von neumman, marvin minsky, mccarthy…
ai: birth at 1956 conference
vision: marvin minsky thought it would be a summer project
lighthill debate 1973 - was ai worth funding?
intelligence tends to be developed by young children…
cortex grew very rapidly
<IP_ADDRESS>. historical cybernetics/nns#
people: norbert weiner, mcculloch & pitts, rosenblatt
neuro
hubel & weisel (1962, 1965) simple, complex, hypercomplex cells
neocognitron fukushima 1980
david marr: theory, representation, implementation
<IP_ADDRESS>. types of models#
three types
descriptive brain model - encode / decode external stimuli
mechanistic brian cell / network model - simulate the behavior of a single neuron / network
interpretive (or normative) brain model - why do brain circuits operate how they do
receptive field - the things that make a neuron fire
retina has on-center / off-surround cells - stimulated by points
then, V1 has differently shaped receptive fields
efficient coding hypothesis - learns different combinations (e.g. lines) that can efficiently represent images
sparse coding (Olshausen and Field 1996)
ICA (Bell and Sejnowski 1997)
Predictive Coding (Rao and Ballard 1999)
brain is trying to learn faithful and efficient representations of an animal’s natural environment
same goes for auditory cortex
7.4.2. biophysical models#
<IP_ADDRESS>. modeling neurons#
nernst battery
osmosis (for each ion)
electrostatic forces (for each ion)
together these yield Nernst potential \(E = \frac{k_B T}{zq} ln \frac{[in]}{[out]}\)
T is temp
q is ionic charge
z is num charges
part of voltage is accounted for by nernst battery \(V_{rest}\)
yields \(\tau \frac{dV}{dt} = -V + V_\infty\) where \(\tau=R_mC_m=r_mc_m\)
equivalently, \(\tau_m \frac{dV}{dt} = -((V-E_L) - g_s(t)(V-E_s) r_m) + I_e R_m \)
<IP_ADDRESS>. simplified model neurons#
integrate-and-fire neuron
passive membrane (neuron charges)
when V = V\(_{thresh}\), a spike is fired
then V = V\(_{reset}\)
doesn’t have good modeling near threshold
can include threshold by saying
when V = V\(_{max}\), a spike is fired
then V = V\(_{reset}\)
modeling multiple variables
also model a K current
can capture things like resonance
theta neuron (Ermentrout and Kopell)
often used for periodically firing neurons (it fires spontaneously)
<IP_ADDRESS>. a forest of dendrites#
cable theory - Kelvin
voltage V is a function of both x and t
separate into sections that don’t depend on x
coupling conductances link the sections (based on area of compartments / branching)
Rall model for dendrites
if branches obey a certain branching ratio, can replace each pair of branches with a single cable segment with equivalent surface area and electrotonic length
\(d_1^{3/2} = d_{11}^{3/2} + d_{12}^{3/2}\)
dendritic computation (London and Hausser 2005)
hippocampus - when inputs arrive at soma, similiar shape no matter where they come in = synaptic scaling
where inputs enter influences how they sum
dendrites can generate spikes (usually calcium) / backpropagating spikes
ex. Jeffress model - sound localized based on timing difference between ears
ex. direction selectivity in retinal ganglion cells - if events arive at dendrite far -> close, all get to soma at same time and add
<IP_ADDRESS>. circuit-modeling basics#
membrane has capacitance \(C_m\)
force for diffusion, force for drift
can write down diffeq for this, which yields an equilibrium
\(\tau = RC\)
bigger \(\tau\) is slower
to increase capacitance
could have larger diameter
\(C_m \propto D\)
axial resistance \(R_A \propto 1/D^2\) (not same as membrane lerk), thus bigger axons actually charge faster
<IP_ADDRESS>. action potentials#
channel/receptor types
ionotropic: \(G_{ion}\) = f(molecules outside)
something binds and opens channel
metabotropic: \(G_{ion}\) = f(molecules inside)
doesn’t directly open a channel: indirect
others
photoreceptor
hair cell
voltage-gated (active - provide gain; might not require active ATP, other channels are all passive)
<IP_ADDRESS>. physics of computation#
based on carver mead: drift and diffusion are at the heart of everything
different things realted by the Boltzmann distr. (ex. distr of air molecules vs elevation. Subject to gravity and diffusion upwards since they’re colliding)
nernst potential
current-voltage relation of voltage-gated channels
current-voltage relation of MOS transistor
these things are all like transistor: energy barrier that must be overcome
neuromorphic examples
differential pair sigmoid yields sigmoid-like function
can compute tanh function really simply to simulate
silicon retina
lateral inhibition exists (gap junctions in horizontal cells)
mead & mahowald 1989 - analog VLSI retina (center-surround receptive field is very low energy)
computation requires energy (otherwise signals would dissipate)
von neumann architecture: CPU - bus (data / address) - Memory
moore’s law ending (in terms of cost, clock speed, etc.)
ex. errors increase as device size decreases (and can’t tolerate any errors)
neuromorphic computing
brain ~ 20 Watts
exploit intrinsic transistor physics (need extremely small amounts of current)
exploit electronics laws kirchoff’s law, ohm’s law
new materials (ex. memristor - 3d crossbar array)
can’t just do biological mimicry - need to understand the principles
<IP_ADDRESS>. spiking neurons#
passive membrane model was leaky integrator
voltage-gaed channels were more complicated
can be though of as leaky integrate-and-fire neuron (LIF)
this charges up and then fires a spike, has refractory period, then starts charging up again
rate coding hypothesis - signal conveyed is the rate of spiking (bruno thinks this is usually too simple)
spiking irregulariy is largely due to noise and doesn’t convey information
some neurons (e.g. neurons in LIP) might actually just convey a rate
linear-nonlinear-poisson model (LNP) - sometimes called GLM (generalized linear model)
based on observation that variance in firing rate \(\propto\) mean firing rate
plotting mean vs variance = 1 \(\implies\) Poisson output
these led people to model firing rates as Poisson \(\frac {\lambda^n e^{-\lambda}} {n!}\)
bruno doesn’t really believe the firing is random (just an effect of other things we can’t measure)
ex. fly H1 neuron 1997
constant stimulus looks very Poisson
moving stimulus looks very Bernoulli
spike timing hypothesis
spiece timing can be very precise in response to time-varying signals (mainen & sejnowski 1995; bair & koch 1996)
often see precise timing
encoding: stimulus \(\to\) spikes
decoding: spikes \(\to\) representation
encoding + decoding are related through the joint distr. over simulus and repsonse (see Bialek spikes book)
nonlinear encoding function can yield linear decoding
able to directly decode spikes using a kernel to reproduce signal (seems to say you need spikes - rates would not be good enough)
some reactions happen too fast to average spikes (e.g. 30 ms)
estimating information rate: bits (usually better than snr - can calculate between them) - usually 2-3 bits/spike
7.4.3. neural coding#
<IP_ADDRESS>. neural encoding#
defining neural code
extracellular
fMRI
averaged over space
slow, requires seconds
EEG
noisy
averaged, but faster
multielectrode array
record from several individual neurons at once
calcium imaging
cells have calcium indicator that fluoresce when calcium enters a cell
intracellular - can use patch electrodes
raster plot
replay a movie many times and record from retinal ganglion cells during movie
encoding: P(response | stimulus)
tuning curve - neuron’s response (ex. firing rate) as a function of stimulus
orientation / color selective cells are distributed in organized fashion
some neurons fire to a concept, like “Pamela Anderson”
retina (simple) -> V1 (orientations) -> V4 (combinations) -> ?
also massive feedback
decoding: P(stimulus | response)
simple encoding
want P(response | stimulus)
response := firing rate r(t)
stimulus := s
simple linear model
r(t) = c * s(t)
weighted linear model - takes into account previous states weighted by f
temporal filtering
r(t) = \(f_0 \cdot s_0 + ... + f_t \cdot s_t = \sum s_{t-k} f_k\) where f weights stimulus over time
could also make this an integral, yielding a convolution:
r(t) = \(\int_{-\infty}^t d\tau \: s(t-\tau) f(\tau)\)
a linear system can be thought of as a system that searches for portions of the signal that resemble its filter f
leaky integrator - sums its inputs with f decaying exponentially into the past
flaws
no negative firing rates
no extremely high firing rates
can add a nonlinear function g of the linear sum can fix this
r(t) = \(g(\int_{-\infty}^t d\tau \: s(t-\tau) f(\tau))\)
spatial filtering
r(x,y) = \(\sum_{x',y'} s_{x-x',y-y'} f_{x',y'}\) where f again is spatial weights that represent the spatial field
could also write this as a convolution
for a retinal center surround cell, f is positive for small \(\Delta x\) and then negative for large \(\Delta x\)
can be calculated as a narrow, large positive Gaussian + spread out negative Gaussian
can combine above to make spatiotemporal filtering
filtering = convolution = projection
feature selection
P(response|stimulus) is very hard to get
stimulus can be high-dimensional (e.g. video)
stimulus can take on many values
need to keep track of stimulus over time
solution: sample P(response|s) to many stimuli to characterize what in input triggers responses
find vector f that captures features that lead to spike
dimensionality reduction - ex. discretize
value at each time \(t_i\) is new dimension
commonly use Gaussian white noise
time step sets cutoff of highest frequency present
prior distribution - distribution of stimulus
multivariate Gaussian - Gaussian in any dimension, or any linear combination of dimensions
look at where spike-triggering points are and calculate spike-triggered average f of features that led to spike
use this f as filter
determining the nonlinear input/output function g
replace stimulus in P(spike|stimulus) with P(spike|\(s_1\)), where s1 is our filtered stimulus
use bayes rule \(g=P(spike\|s_1)=\frac{P(s_1\|spike)P(spike)}{P(s_1)}\)
if \(P(s_1\|spike) \approx P(s_1)\) then response doesn’t seem to have to do with stimulus
incorporating many features \(f_1,...,f_n\)
here, each \(f_i\) is a vector of weights
\(r(t) = g(f_1\cdot s,f_2 \cdot s,...,f_n \cdot s)\)
could use PCA - discovers low-dimensional structure in high-dimensional data
each f represents a feature (maybe a curve over time) that fires the neuron
variability
hidden assumptions about time-varying firing rate and single spikes
smooth function RFT can miss some stimuli
statistics of stimulus can effect P(spike|stimulus)
Gaussian white noise is nice because no way to filter it to get structure
identifying good filter
want \(P(s_f\|spike)\) to differ from \(P(s_f)\) where \(s_f\) is calculated via the filter
instead of PCA, could look for f that directly maximizes this difference (Sharpee & Bialek, 2004)
Kullback-Leibler divergence - calculates difference between 2 distributions
\(D_{KL}(P(s),Q(s)) = \int ds P(s) log_2 P(s) / Q(s)\)
maximizing KL divergence is equivalent to maximizing mutual info between spike and stimulus
this is because we are looking for most informative feature
this technique doesn’t require that our stimulus is white noise, so can use natural stimuli
maximization isn’t guaranteed to uniquely converge
modeling the noise
need to go from r(t) -> spike times
divide time T into n bins with p = probability of firing per bin
over some chunk T, number of spikes follows binomial distribution (n, p)
mean = np
var = np(1-p)
if n gets very large, binomial approximates Poisson
\(\lambda\) = spikes in some set time
mean = \(\lambda\)
var = \(\lambda\)
can test if distr is Poisson with Fano factor=mean/var=1
interspike intervals have exponential distribution - if fires a lot, this can be bad assumption (due to refractory period)
generalized linear model adds explicit spike-generation / post-spike filter (Pillow et al. 2008)
\(P(\text{spike at }t)\sim\exp((f_1*s + h_1*r)) \)
post-spike filter models refractory period
Paninski showed that using exponential nonlinearity allows this to be optimized
could add in firing of other neurons
time-rescaling theorem - tests how well we have captured influences on spiking (Brown et al 2001)
scaled ISIs (\(t_{i-1}-t_i\)) r(t) should be exponential
<IP_ADDRESS>. neural decoding#
neural decoding and signal detection
decoding: P(stimulus | response) - ex. you hear noise and want to tell what it is
here \(r\) = response = firing rate
monkey is trained to move eyes in same direction as dot pattern (Britten et al. 92)
when dots all move in same direction (100% coherence), easy
neuron recorded in MT - tracks dots
count firing rate when monkey tracks in right direction
count firing rate when monkey tracks in wrong direction
as coherence decreases, these firing rates blur
need to get P(+ or - | r)
can set a threshold on r by maximizing likelihood
P(r|+) and P(r|-) are likelihoods
Neyman-Pearson lemma - likelihood ratio test is the most efficient statistic, in that is has the most power for a given size
\(\frac{p(r\|+)}{p(r\|-)} > 1?\)
accumulated evidence - we can accumulate evidence over time by multiplying these probabilities
instead we take sum the logs, and compare to 0
\(\sum_i ln \frac{p(r_i\|+)}{p(r_i\|-)} > 0?\)
once we hit some threshold for this sum, we can make a decision + or -
experimental evidence (Kiani, Hanks, & Shadlen, Nat. Neurosci 2006)
monkey is making decision about whether dots are moving left/right
neuron firing rates increase over time, representing integrated evidence
neuron always seems to stop at same firing rate
priors - ex. tiger is much less likely then breeze
scale P(+|r) by prior P(+)
neuroscience ex. photoreceptor cells P(noise|r) is much larger than P(signal|r)
therefore threshold on r is high to minimize total mistakes
cost of acting/not acting
loss for predicting + when it is -: \(L_- \cdot P[+\|r]\)
loss for predicting - when it is +: \(L_+ \cdot P[-\|r]\)
cut your losses: answer + when average Loss\(_+\) < Loss\(_-\)
i.e. \(L_+ \cdot P[-\|r]\) < \(L_- \cdot P[+\|r]\)
rewriting with Baye’s rule yields new test:
\(\frac{p(r\|+)}{p(r\|-)}> L_+ \cdot P[-] / L_- \cdot P[+]\)
here the loss term replaces the 1 in the Neyman-Pearson lemma
population coding and bayesian estimation
population vector - sums vectors for cells that point in different directions weighted by their firing rates
ex. cricket cercal cells sense wind in different directions
since neuron can’t have negative firing rate, need overcomplete basis so that can record wind in both directions along an axis
can do the same thing for direction of arm movement in a neural prosthesis
not general - some neurons aren’t tuned, are noisier
not optimal - making use of all information in the stimulus/response distributions
bayesian inference
\(p(s\|r) = \frac{p(r\|s)p(s)}{p( r)}\)
maximum likelihood: s* which maximizes p(r|s)
MAP = maximum a posteriori: s* which mazimizes p(s|r)
simple continuous stimulus example
setup
s - orientation of an edge
each neuron’s average firing rate=tuning curve \(f_a(s)\) is Gaussian (in s)
let \(r_a\) be number of spikes for neuron a
assume receptive fields of neurons span s: \(\sum r_a (s)\) is const
solving
maximizing log-likelihood with respect to s -
take derivative and set to 0
soln \(s^* = \frac{\sum r_a s_a / \sigma_a^2}{\sum r_a / \sigma_a^2}\)
if all the \(\sigma\) are same, \(s^* = \frac{\sum r_a s_a}{\sum r_a}\)
this is the population vector
maximum a posteriori
\(ln \: p(s\|r) = ln \: P(r\|s) + ln \: p(s) = ln \: P(r )\)
\(s^* = \frac{T \sum r_a s_a / \sigma^2_a + s_{prior} / \sigma^2_{prior}}{T \sum r_a / \sigma^2_a + 1/\sigma^2_{prior}}\)
this takes into account the prior
narrow prior makes it matter more
doesn’t incorporate correlations in the population
stimulus reconstruction
decoding s -> \(s^*\)
want an estimator \(s_{Bayes}=s_B\) given some response r
error function \(L(s,s_{B})=(s-s_{B})^2\)
minimize \(\int ds \: L(s,s_{B}) \: p(s\|r)\) by taking derivative with respect to \(s_B\)
\(s_B = \int ds \: p(s\|r) \: s\) - the conditional mean (spike-triggered average)
add in spike-triggered average at each spike
if spike-triggered average looks exponential, can never have smooth downwards stimulus
could use 2 neurons (like in H1) and replay the second with negative sign
LGN neurons can reconstruct a video, but with noise
recreated 1 sec long movies - (Jack Gallant - Nishimoto et al. 2011, Current Biology)
voxel-based encoding model samples ton of prior clips and predicts signal
get p(r|s)
pick best p(r|s) by comparing predicted signal to actual signal
input is filtered to extract certain features
filtered again to account for slow timescale of BOLD signal
decoding
maximize p(s|r) by maximizing p(r|s) p(s), and assume p(s) uniform
30 signals that have highest match to predicted signal are averaged
yields pretty good pictures
<IP_ADDRESS>. information theory#
information and entropy
surprise for seeing a spike h(p) = \(-log_2 (p)\)
entropy = average information
code might not align spikes with what we are encoding
how much of the variability in r is encoding s
define q as en error
\(P(r_+\|s=+)=1-q\)
\(P(r_-\|s=+)=q\)
similar for when s=-
total entropy: \(H(R ) = - P(r_+) log P(r_+) - P(r_-)log P(r_-)\)
noise entropy: \(H(R\|S=+) = -q log q - (1-q) log (1-q)\)
mutual info I(S;R) = \(H(R ) - H(R\|S) \) = total entropy - average noise entropy
= \(D_{KL} (P(R,S), P(R )P(S))\)
grandma’s famous mutual info recipe
for each s
P(R|s) - take one stimulus and repeat many times (or run for a long time)
H(R|s) - noise entropy
\(H(R\|S)=\sum_s P(s) H(R\|s)\)
\(H(R ) \) calculated using \(P(R ) = \sum_s P(s) P(R\|s)\)
info in spike trains
information in spike patterns
divide pattern into time bins of 0 (no spike) and 1 (spike)
binary words w with letter size \(\Delta t\), length T (Reinagel & Reid 2000)
can create histogram of each word
can calculate entropy of word
look at distribution of words for just one stimulus
distribution should be narrower
calculate \(H_{noise}\) - average over time with random stimuli and calculate entropy
varied parameters of word: length of bin (dt) and length of word (T)
there’s some limit to dt at which information stops increasing
this represents temporal resolution at which jitter doesn’t stop response from identifying info about the stimulus
corrections for finite sample size (Panzeri, Nemenman,…)
information in single spikes - how much info does single spike tell us about stimulus
don’t have to know encoding, mutual info doesn’t care
calculate entropy for random stimulus - \(p=\bar{r} \Delta t\) where \(\bar{r}\) is the mean firing rate
calculate entropy for specific stimulus
let \(P(r=1\|s) = r(t) \Delta t\)
let \(P(r=0\|s) = 1 - r(t) \Delta t\)
get r(t) by having simulus on for long time
ergodicity - a time average is equivalent to averging over the s ensemble
info per spike \(I(r,s) = \frac{1}{T} \int_0^T dt \frac{r(t)}{\bar{r}} log \frac{r(t)}{\bar{r}}\)
timing precision reduces r(t)
low mean spike rate -> high info per spike
ex. rat runs through place field and only fires when it’s in place field
spikes can be sharper, more / less frequent
coding principles
natural stimuli
huge dynamic range - variations over many orders of magnitude (ex. brightness)
power law scaling - structure at many scales (ex. far away things)
efficient coding - in order to have maximum entropy output, a good encoder should match its outputs to the distribution of its inputs
want to use each of our “symbols” (ex. different firing rates) equally often
should assign equal areas of input stimulus PDF to each symbol
adaptation to stimulus statistics
feature adaptation (Atick and Redlich)
spatial filtering properties in retina / LGN change with varying light levels
at low light levels surround becomes weaker
coding sechemes
redundancy reduction
population code \(P(R_1,R_2)\)
entropy \(H(R_1,R_2) \leq H(R_1) + H(R_2)\) - being independent would maximize entropy
correlations can be good
error correction and robust coding
correlations can help discrimination
retina neurons are redundant (Berry, Chichilnisky)
more recently, sparse coding
penalize weights of basis functions
instead, we get localized features
we ignored the behavioral feedback loop
7.4.4. computing with networks#
<IP_ADDRESS>. modeling connections between neurons#
model effects of synapse by using synaptic conductance \(g_s\) with reversal potential \(E_s\)
\(g_s = g_{s,max} \cdot P_{rel} \cdot P_s\)
\(P_{rel}\) - probability of release given an input spike
\(P_s\) - probability of postsynaptic channel opening = fraction of channels opened
basic synapse model
assume \(P_{rel}=1\)
model \(P_s\) with kinetic model
open based on \(\alpha_s\)
close based on \(\beta_s\)
yields \(\frac{dP_s}{dt} = \alpha_s (1-P_s) - \beta_s P_s\)
3 synapse types
AMPA - well-fit by exponential
GAMA - fit by “alpha” function - has some delay
NMDA - fit by “alpha” function - has some delay
linear filter model of a synapse
pick filter (ex. K(t) ~ exponential)
\(g_s = g_{s,max} \sum K(t-t_i)\)
network of integrate-and-fire neurons
if 2 neurons inhibit each other, get synchrony (fire at the same time
<IP_ADDRESS>. intro to network models#
comparing spiking models to firing-rate models
advantages
spike timing
spike correlations / synchrony between neurons
disadvantages
computationally expensive
uses linear filter model of a synapse
developing a firing-rate model
replace spike train \(\rho_1(t) \to u_1(t)\)
can’t make this replacement when there are correlations / synchrony?
input current \(I_s\): \(\tau_s \frac{dI_s}{dt}=-I_s + \mathbf{w} \cdot \mathbf{u}\)
works only if we let K be exponential
output firing rate: \(\tau_r \frac{d\nu}{dt} = -\nu + F(I_s(t))\)
if synapses are fast (\(\tau_s << \tau_r\))
\(\tau_r \frac{d\nu}{dt} = -\nu + F(\mathbf{w} \cdot \mathbf{u}))\)
if synapses are slow (\(\tau_r << \tau_s\))
\(\nu = F(I_s(t))\)
if static inputs (input doesn’t change) - this is like artificial neural network, where F is sigmoid
\(\nu_{\infty} = F(\mathbf{w} \cdot \mathbf{u})\)
could make these all vectors to extend to multiple output neurons
recurrent networks
\(\tau \frac{d\mathbf{v}}{dt} = -\mathbf{v} + F(W\mathbf{u} + M \mathbf{v})\)
\(-\mathbf{v}\) is decay
\(W\mathbf{u}\) is input
\(M \mathbf{v}\) is feedback
with constant input, \(v_{\infty} = W \mathbf{u}\)
ex. edge detectors
V1 neurons are basically computing derivatives
<IP_ADDRESS>. recurrent networks#
linear recurrent network: \(\tau \frac{d\mathbf{v}}{dt} = -\mathbf{v} + W\mathbf{u} + M \mathbf{v}\)
let \(\mathbf{h} = W\mathbf{u}\)
want to investiage different M
can solve eq for \(\mathbf{v}\) using eigenvectors
suppose M (NxN) is symmetric (connections are equal in both directions)
\(\to\) M has N orthogonal eigenvectors / eigenvalues
let \(e_i\) be the orthonormal eigenvectors
output vector \(\mathbf{v}(t) = \sum c_i (t) \mathbf{e_i}\)
allows us to get a closed-form solution for \(c_i(t)\)
eigenvalues determine network stability
if any \(\lambda_i > 1, \mathbf{v}(t)\) explodes \(\implies\) network is unstable
otherwise stable and converges to steady-state value
\(\mathbf{v}_\infty = \sum \frac{h\cdot e_i}{1-\lambda_i} e_i\)
amplification of input projection by a factor of \(\frac{1}{1-\lambda_i}\)
ex. each output neuron codes for an angle between -180 to 180
define M as cosine function of relative angle
excitation nearby, inhibition further away
memory in linear recurrent networks
suppose \(\lambda_1=1\) and all other \(\lambda_i < 1\)
then \(\tau \frac{dc_1}{dt} = h \cdot e_1\) - keeps memory of input
ex. memory of eye position in medial vestibular nucleus (Seung et al. 2000)
integrator neuron maintains persistent activity
nonlinear recurrent networks: \(\tau \frac{d\mathbf{v}}{dt} = -\mathbf{v} + F(\mathbf{h}+ M \mathbf{v})\)
ex. rectification linearity F(x) = max(0,x)
ensures that firing rates never go below
can have eigenvalues > 1 but stable due to rectification
can perform selective “attention”
network performs “winner-takes-all” input selection
gain modulation - adding constant amount to input h multiplies the output
also maintains memory
non-symmetric recurrent networks
ex. excitatory and inhibitory neurons
linear stability analysis - find fixed points and take partial derivatives
use eigenvalues to determine dynamics of the nonlinear network near a fixed point
hopfield nets
hopfield nets can store / retrieve memories
fully connected (no input/output) - activations are what matter
can memorize patterns - starting with noisy patterns can converge to these patterns
marr-pogio stereo algorithm
hopfield three-way connections
\(E = - \sum_{i, j, k} T_{i, j, k} V_i V_j V_k\) (self connections set to 0)
update to \(V_i\) is now bilinear
hopfield nets are all you need
keys: each input has a key vector which “represents info about this input” (e.g. this is a noun)
queries: each input has a query vector which “asks for other inputs that would be useful context” (e.g. what adjectives describe this word)
in self-attention these queries also come from the input whereas in just regular attention they come from somewhere else (e.g. the output of a translation task)
transformer finds similarity between each key with each query then takes softmax - this provides weights for each of the inputs, as context for the original input
in transformer, these weights are used to weight the values but in hopfield nets we would take a weighted sum of the keys and feed it back as the input
as we update becomes more skewed towards the things that match the most
7.4.5. learning#
<IP_ADDRESS>. supervised learning#
net talk was major breakthrough (words -> audio) Sejnowski & Rosenberg 1987
people looked for world-centric receptive fields (so neurons responded to things not relative to retina but relative to body) but didn’t find them
however, they did find gain fields: (Zipser & Anderson, 1987)
gain changes based on what retina is pointing at
trained nn to go from pixels to head-centered coordinate frame
yielded gain fields
pouget et al. were able to find that this helped having 2 pop vectors: one for retina, one for eye, then add to account for it
support vector networks (vapnik et al.) - svms early inspired from nns
dendritic nonlinearities (hausser & mel 03)
example to think about neurons due this: \(u = w_1 x_1 + w_2x_2 + w_{12}x_1x_2\)
\(y=\sigma(u)\)
somestimes called sigma-pi unit since it’s a sum of products
exponential number of params…could be fixed w/ kernel trick?
could also incorporate geometry constraint…
<IP_ADDRESS>. unsupervised learning#
born w/ extremely strong priors on weights in different areas
barlow 1961, attneave 1954: efficient coding hypothesis = redundancy reduction hypothesis
representation: compression / usefulness
easier to store prior probabilities (because inputs are independent)
relich 93: redundancy reduction for unsupervised learning (text ex. learns words from text w/out spaces)
<IP_ADDRESS>.1. hebbian learning and pca#
pca can also be thought of as a tool for decorrelation (in pc dimension, tends to be less correlated)
hebbian learning = fire together, wire together: \(\Delta w_{ab} \propto <a, b>\) note: \(<a, b>\) is correlation of a and b (average over time)
linear hebbian learning (perceptron with linear output)
\(\dot{w}_i \propto <y, x_i> \propto \sum_j w_j <x_j, x_i>\) since weights change relatively slowly
synapse couldn’t do this, would grow too large
oja’s rule (hebbian learning w/ weight decay so ws don’t get too big)
points to correct direction
sanger’s rule: for multiple neurons, fit residuals of other neurons
competitive learning rule: winner take all
population nonlinearity is a max
gets stuck in local minima (basically k-means)
pca only really good when data is gaussian
interesting problems are non-gaussian, non-linear, non-convex
pca: yields checkerboards that get increasingly complex (because images are smooth, can describe with smaller checkerboards)
this is what jpeg does
very similar to discrete cosine transform (DCT)
very hard for neurons to get receptive fields that look like this
retina: does whitening (yields center-surround receptive fields)
easier to build
gets more even outputs
only has ~1.5 million fibers
<IP_ADDRESS>. synaptic plasticity, hebb’s rule, and statistical learning#
if 2 spikes keep firing at same time, get LTP - long-term potentiation
if input fires, but not B then could get LTD - long-term depression
Hebb rule \(\tau_w \frac{d\mathbf{w}}{dt} = \mathbf{x}v\)
\(\mathbf{x}\) - input
\(v\) - output
translates to \(\mathbf{w}_{i+1}=\mathbf{w}_i + \epsilon \cdot \mathbf{x}v\)
average effect of the rule is to change based on correlation matrix \(\mathbf{x}^T\mathbf{x}\)
covariance rule: \(\tau_w \frac{d\mathbf{w}}{dt} = \mathbf{x}(v-E[v])\)
includes LTD as well as LTP
Oja’s rule: \(\tau_w \frac{d\mathbf{w}}{dt} = \mathbf{x}v- \alpha v^2 \mathbf{w}\) where \(\alpha>0\)
stability
Hebb rule - derivative of w is always positive \(\implies\) w grows without bound
covariance rule - derivative of w is still always positive \(\implies\) w grows without bound
could add constraint that \(\|\|w\|\|=1\) and normalize w after every step
Oja’s rule - \(\|\|w\|\| = 1/\sqrt{alpha}\), so stable
solving Hebb rule \(\tau_w \frac{d\mathbf{w}}{dt} = Q w\) where Q represents correlation matrix
write w(t) in terms of eigenvectors of Q
lets us solve for \(\mathbf{w}(t)=\sum_i c_i(0)exp(\lambda_i t / \tau_w) \mathbf{e}_i\)
when t is large, largest eigenvalue dominates
hebbian learning implements PCA
hebbian learning learns w aligned with principal eigenvector of input correlation matrix
this is same as PCA
<IP_ADDRESS>. intro to unsupervised learning#
-
most active neuron is the one whose w is closest to x
competitive learning
updating weights given a new input
pick a cluster (corresponds to most active neuron)
set weight vector for that cluster to running average of all inputs in that cluster
\(\Delta w = \epsilon \cdot (\mathbf{x} - \mathbf{w})\)
related to self-organizing maps = kohonen maps
in self-organizing maps also update other neurons in the neighborhood of the winner
update winner closer
update neighbors to also be closer
ex. V1 has orientation preference maps that do this
<IP_ADDRESS>. sparse coding and predictive coding#
eigenface - Turk and Pentland 1991
eigenvectors of the input covariance matrix are good features
can represent images using sum of eigenvectors (orthonormal basis)
suppose you use only first M principal eigenvectors
then there is some noise
can use this for compression
not good for local components of an image (e.g. parts of face, local edges)
if you assume Gausian noise, maximizing likelihood = minimizing squared error
generative model
images X
causes
likelihood P(X=x|C=c)
Gaussian
proportional to \(\exp(x-Gc)\)
want posterior P(C|X)
prior p(C )
assume priors causes are independent
want sparse distribution
has heavy tail (super-Gaussian distribution)
then P(C ) = \(k\prod \exp(g(C_i))\)
can implement sparse coding in a recurrent neural network
Olshausen & Field, 1996 - learns receptive fields in V1
sparse coding is a special case of predicive coding
there is usually a feedback connection for every feedforward connection (Rao & Ballard, 1999)
recurrent sparse reconstruction (shi…joshi, darrel, wang, 2022) - sparse reconstruction (of a single image) learns a layer that does better than self-attention
<IP_ADDRESS>. sparse, distributed coding#
- \[\underset {\mathbf{D}} \min \underset t \sum \underset {\mathbf{h^{(t)}}} \min ||\mathbf{x^{(t)}} - \mathbf{Dh^{(t)}}||_2^2 + \lambda ||\mathbf{h^{(t)}}||_1\]
D is like autoencoder output weight matrix
h is more complicated - requires solving inner minimization problem
outer loop is not quite lasso - weights are not what is penalized
barlow 1972: want to represent stimulus with minimum active neurons
neurons farther in cortex are more silent
v1 is highly overcomplete (dimensionality expansion)
codes: dense -> sparse, distributed \(n \choose k\) -> local (grandmother cells)
energy argument - bruno doesn’t think it’s a big deal (could just not have a brain)
PCA: autoencoder when you enforce weights to be orthonormal
retina must output encoded inputs as spikes, lower dimension -> uses whitening
cortex
sparse coding different kind of autencoder bottleneck (imposes sparsity)
using bottlenecks in autoencoders forces you to find structure in data
v1 simple-cell receptive fields are localized, oriented, and bandpass
higher-order image statistics
phase alignment
orientation (requires at least 3 points stats (like orientation)
motion
how to learn sparse repr?
foldiak 1990 forming sparse reprs by local anti-hebbian learning
driven by inputs and gets lateral inhibition and sum threshold
neurons drift towards some firing rate naturally (adjust threshold naturally)
use higher-order statistics
projection pursuit (field 1994) - maximize non-gaussianity of projections
CLT says random projections should look gaussian
gabor-filter response histogram over natural images look non-Gaussian (sparse) - peaked at 0
doesn’t work for graded signals
sparse coding for graded signals: olshausen & field, 1996
\(\underset{Image}{I(x, y)} = \sum_i a_i \phi_i (x, y) + \epsilon (x,y)\)
loss function \(\frac{1}{2} |I - \phi a|^2 + \lambda \sum_i C(a_i)\)
can think about difference between \(L_1\) and \(L_2\) as having preferred directions (for the same length of vector) - prefer directions which some zeros
in terms of optimization, smooth near zero
there is a network implementation
\(a_i\)are calculated by solvin optimization for each image, \(\phi\) is learned more slowly
can you get \(a_i\) closed form soln?
wavelets invented in 1980s/1990s for sparsity + compression
these tuning curves match those of real v1 neurons
applications
for time, have spatiotemporal basis where local wavelet moves
sparse coding of natural sounds
audition like a movie with two pixels (each ear sounds independent)
converges to gamma tone functions, which is what auditory fibers look like
sparse coding to neural recordings - finds spikes in neurons
learns that different layers activate together, different frequencies come out
found place cell bases for LFP in hippocampus
nonnegative matrix factorization - like sparse coding but enforces nonnegative
can explicitly enforce nonnegativity
LCA algorithm lets us implement sparse coding in biologically plausible local manner
explaining away - neural responses at the population should be decodable (shouldn’t be ambiguous)
good project: understanding properties of sparse coding bases
SNR = \(VAR(I) / VAR(|I- \phi A|)\)
can run on data after whitening
graph is of power vs frequency (images go down as \(1/f\)), need to weighten with f
don’t whiten highest frequencies (because really just noise)
need to do this softly - roughly what the retina does
as a result higher spatial frequency activations have less variance
whitening effect on sparse coding
if you don’t whiten, have some directions that have much more variance
projects
applying to different types of data (ex. auditory)
adding more bases as time goes on
combining convolution w/ sparse coding?
people didn’t see sparsity for a while because they were using very specific stimuli and specific neurons
now people with less biased sampling are finding more sparsity
in cortex anasthesia tends to lower firing rates, but opposite in hippocampus
<IP_ADDRESS>. self-organizing maps#
homunculus - 3d map corresponds to map in cortex (sensory + motor)
visual cortex
visual cortex mostly devoted to center
different neurons in same regions sensitive to different orientations (changing smoothly)
orientation constant along column
orientation maps not found in mice (but in cats, monkeys)
direction selective cells as well
maps are plastic - cortex devoted to particular tasks expands (not passive, needs to be active)
kids therapy with tone-tracking video games at higher and higher frequencies
7.4.6. probabilistic models + inference#
Wiener filter
has Gaussian prior + likelihoodfor gaussian function, \(d/dx f(x) = -x f(x)\)
<IP_ADDRESS>. boltzmann machines#
hinton & sejnowski 1983
starts with a hopfield net (states \(s_i\) weights \(\lambda_{ij}\)) where states are \(\pm 1\)
define energy function \(E(\mathbf{s}) = - \sum_{ij} \lambda_{ij} s_i s_j\)
assume Boltzmann distr \(P(s) = \frac{1}{z} \exp (- \beta \phi(s))\)
learning rule is basically expectation over data - expectation over model
could use wake-sleep algorithm
during day, calculate expectation over data via Hebbian learning (in Hopfield net this would store minima)
during night, would run anti-hebbian by doing random walk over network (in Hopfield net this would remove spurious local minima)
learn via gibbs sampling (prob for one node conditioned on others is sigmoid)
can add hidden units to allow for learning higher-order interactions (not just pairwise)
restricted boltzmann machine: no connections between “visible” units and no connections between “hidden units”
computationally easier (sampling is independent) but less rich
stacked rbm: hinton & salakhutdinov (hinton argues this is first paper to launch deep learning)
don’t train layers jointly
learn weights with rbms as encoder
then decoder is just transpose of weights
finally, run fine-tuning on autoencoder
able to separate units in hidden layer
cool - didn’t actually need decoder
in rbm
when measuring true distr, don’t see hidden vals
instead observe visible units and conditionally sample over hidden units
\(P(h|v) = \prod_i P(h_i | v)\) ~ easy to sample from
when measuring sampled distr., just sample \(P(h|v)\) then sample \(P(v|h)\)
ising model - only visible units
basically just replicates pairwise statistics (kind of like pca)
pairwise statistics basically say “when I’m on, are my neighbors on?”
need 3-point statistics to learn a line
generating textures
learn the distribution of pixels in 3x3 patches
then maximize this distribution - can yield textures
reducing the dimensionality of data with neural networks
7.4.7. data-driven neuroscience#
<IP_ADDRESS>. data types#
EEG |
ECoG |
Local Field potential (LFP) -> microelectrode array |
single-unit |
calcium imaging |
fMRI |
|
|---|---|---|---|---|---|---|
scale |
high |
high |
low |
tiny |
low |
high |
spatial res |
very low |
low |
mid-low |
x |
low |
mid-low |
temporal res |
mid-high |
high |
high |
super high |
high |
very low |
invasiveness |
non |
yes (under skull) |
very |
very |
non |
non |
pro big-data
Artificial neural networks can compute in several different ways. There is some evidence in the visual system that neurons in higher layers of visual areas can, to some extent, be predicted linearly by higher layers of deep networks (yamins2014performance)
when comparing energy-efficiency, must normalize network performance by energy / number of computations / parameters
anti big-data
could neuroscientist understand microprocessor
no canonical microcircuit
cellular
extracellular microeelectrodes
intracellular microelectrode
neuropixels
optical
calcium imaging / fluorescence imaging
whole-brain light sheet imaging
voltage-sensitive dyes / voltage imaging
adaptive optics
fNRIS - like fMRI but cheaper, allows more immobility, slightly worse spatial res
oct - noninvasive - can look at retina (maybe find biomarkers of alzheimer’s)
fiber photometry - optical fiber implanted delivers excitation light
alteration
optogenetic stimulation
tms
genetically-targeted tms: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4846560/
local microstimulation with invasive electrodes
high-level
EEG/ECoG
MEG
fMRI/PET
MRI with millisecond temporal precision
molecular fmri (bartelle)
MRS
event-related optical signal = near-infrared spectroscopy
implantable
neural dust
<IP_ADDRESS>. datasets#
-
MRNet: knee MRI diagnosis
springer 10k calcium imaging recording: https://figshare.com/articles/Recordings_of_ten_thousand_neurons_in_visual_cortex_during_spontaneous_behaviors/6163622
springer 2: 10k neurons with 2800 images
stringer et al. data
10000 neurons from visual cortex
neuropixels probes
10k neurons visual coding from allen institute
this probe has also been used in macaques
allen institute calcium imaging
An experiment is the unique combination of one mouse, one imaging depth (e.g. 175 um from surface of cortex), and one visual area (e.g. “Anterolateral visual area” or “VISal”)
predicting running, facial cues
dimensionality reduction
enforcing bottleneck in the deep model
how else to do dim reduction?
responses to 2800 images
overview: http://www.scholarpedia.org/article/Encyclopedia_of_computational_neuroscience
keeping up to date: https://sanjayankur31.github.io/planet-neuroscience/
lots of good data: http://home.earthlink.net/~perlewitz/index.html
connectome
fly brain: http://temca2data.org/
models
senseLab: https://senselab.med.yale.edu/
modelDB - has NEURON code
model databases: http://www.cnsorg.org/model-database
comp neuro databases: http://home.earthlink.net/~perlewitz/database.html
raw misc data
crcns data: http://crcns.org/
visual cortex data (gallant)
hippocampus spike trains
allen brain atlas: http://www.brain-map.org/
includes calcium-imaging dataset: http://help.brain-map.org/display/observatory/Data+-+Visual+Coding
wikipedia page: https://en.wikipedia.org/wiki/List_of_neuroscience_databases
human fMRI datasets: https://docs.google.com/document/d/1bRqfcJOV7U4f-aa3h8yPBjYQoLXYLLgeY6_af_N2CTM/edit
Kay et al 2008 has data on responses to images
calcium imaging for spike sorting: http://spikefinder.codeneuro.org/
fmri responses to language++
A natural language fMRI dataset for voxelwise encoding models (lebel, … huth, 2022)
8 participants listening to ~6 hours each of the moth radio hour
Narratives Dataset (Nastase et al. 2019) - more subjects, less data per subject
345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words) and total collection time is ~6.4 days
Huth et al. 2016 released data from one subject
Visual and linguistic semantic representations are aligned at the border of human visual cortex (popham, huth et al. 2021) - compared semantic maps obtained from two functional magnetic resonance imaging experiments in the same participants: one that used silent movies as stimuli and another that used narrative stories (data link)
More datasets available at openneuro and visual cortex data on crcns
misc ideas
could a neuroscientist understand a deep neural network? - use neural tracing to build up wiring diagram / function
prediction-driven dimensionality reduction
deep heuristic for model-building
joint prediction of different input/output relationships
joint prediction of neurons from other areas
7.4.8. advanced topics#
<IP_ADDRESS>. high-dimensional (hyperdimensional) computing#
computing with random high-dim vectors (also known as vector-symbolic architectures)
Good overview website: https://www.hd-computing.com
ovw talk (kanerva, 2022)
has slide with related references
A comparison of vector symbolic architectures (schlegel et al. 2021)
motivation
high-level overview
draw inspiration from circuits not single neurons
the brain’s circuits are high-dimensional
elements are stochastic not deterministic
no 2 brains are alike yet they exhibit the same behavior
basic question of comp neuro: what kind of computing can explain behavior produced by spike trains?
recognizing ppl by how they look, sound, or behave
learning from examples
remembering things going back to childhood
communicating with language
operations
ex. vectors \(A\), \(B\) both \(\in \{ +1, -1\}^{10,000}\) (also extends to real / complex vectors)
3 operations
addition: A + B = (0, 0, 2, 0, 2,-2, 0, ….)
alternatively, could take mean
multiplication: A * B = (-1, -1, -1, 1, 1, -1, 1, …) - this is XOR
want this to be invertible, dsitribute over addition, preserve distance, and be dissimilar to the vectors being multiplied
number of ones after multiplication is the distance between the two original vectors
can represent a dissimilar set vector by using multiplication
permutation: shuffles values (like bit-shift)
ex. rotate (bit shift with wrapping around)
multiply by rotation matrix (where each row and col contain exactly one 1)
can think of permutation as a list of numbers 1, 2, …, n in permuted order
many properties similar to multiplication
random permutation randomizes
secondary operations
weighting by a scalar
similarity = dot product (sometimes normalized)
A \(\cdot\) A = 10k
A \(\cdot\) A = 0 (orthogonal)
in high-dim spaces, almost all pairs of vectors are dissimilar A \(\cdot\) B = 0
goal: similar meanings should have large similarity
normalization
for binary vectors, just take the sign
for non-binary vectors, scalar weight
fractional binding - can bind different amounts rather than binary similar / dissimilar
data structures
the operations above allow for encoding many normal data structures into a single vector
set - can be represented with a sum (since the sum is similar to all the vectors)
can find a stored set using any element
if we don’t store the sum, can probe with the sum and keep subtracting the vectors we find
multiset = bag (stores set with frequency counts) - can store things with order by adding them multiple times, but hard to actually retrieve frequencies
sequence - could have each element be an address pointing to the next element
problem - hard to represent sequences that share a subsequence (could have pointers which skip over the subsquence)
soln: index elements based on permuted sums
can look up an element based on previous element or previous string of elements
could do some kind of weighting also
pairs - could just multiply (XOR), but then get some weird things, e.g. A * A = 0
instead, permute then multiply
can use these to index (address, value) pairs and make more complex data structures
named tuples - have smth like (name: x, date: m, age: y) and store as holistic vector \(H = N*X + D * M + A * Y\)
individual attribute value can be retrieved using vector for individual key
representation substituting is a little trickier….
we blur what is a value and what is a variable
can do this for a pair or for a named tuple with new values
this doesn’t always work
examples
ex. semantic word vectors
goal: get good semantic vectors for words
baseline (e.g. latent-semantic analysis LSA): make matrix of word counts, where each row is a word, and each column is a document
add counts to each column – row vector becomes semantic vector
HD computing alternative: each row is a word, but each document is assigned a few ~10 columns at random
thus, the number of columns doesn’t scale with the number of documents
can also do this randomness for the rows (so the number of rows < the number of words)
can still get semantic vector for a row/column by adding together the rows/columns which are activated by that row/column
this examples still only uses bag-of-words (but can be extended to more)
ex. semantic word vectors 2 (like word2vec)
each word in vocab is given 2 vectors
random-indexing vector - fixed random from the beginning
semantic vector - starts at 0
as we traverse sequence, for each word, add random-indexing vector from words right before/after it to its semantic vector
can also permute them before adding to preserve word order (e.g. “Permutations as a means to encode order in word space” (2008))
can instead use placeholder vector to help bring in word order (e.g. BEAGLE - Jones & Mewhort, 2007)
ex. learning rules by example
particular instance of a rule is a rule (e.g mother-son-baby \(\to\) grandmother)
as we get more examples and average them, the rule gets better
doesn’t always work (especially when things collapse to identity rule)
ex. what is the dollar of mexico? (kanerva, 2010)
initialize US = (NAME * USA) + (MONEY * DOLLAR)
initialize MEXICO = (NAME * MEXICO) + (MONEY * PESO)
query: “Dollar of Mexico”? = DOLLAR * US * MEXICO = PESO
ex. text classification (najafabadi et al. 2016)
ex. language classification - “Language Recognition using Random Indexing” (joshi et al. 2015)
scalable, easily us any-order ngrams
data
train: given million bytes of text per language (in the same alphabet)
test: new sentences for each language
training: compute a 10k profile vector for each language and for each test sentence
could encode each letter with a seed vector which is 10k
instead encode trigrams with rotate and multiply
1st letter vec rotated by 2 * 2nd letter vec rotated by 1 * 3rd letter vec
ex. THE = r(r(T)) * r(H) * r(E)
approximately orthogonal to all the letter vectors and all the other possible trigram vectors…
profile = sum of all trigram vectors (taken sliding)
ex. banana = ban + ana + nan + ana
profile is like a histogram of trigrams
testing
compare each test sentence to profiles via dot product
clusters similar languages - cool!
can query the letter most likely to follow “TH”
form query vector \(Q = r(r(T)) * r(H)\)
query by using multiply \(X = Q\) * english-profile-vec
find closest letter vecs to \(X\): yields “e”
details
frequent “stopwords” should be ignored
mathematical background
randomly chosen vecs are dissimilar
sum vector is similar to its argument vectors
product vector and permuted vector are dissimilar to their argument vectors
multiplication distibutes over addition
permutation distributes over both additions and multiplication
multiplication and permutations are invertible
addition is approximately invertible
comparison to DNNs
both do statistical learning from data
data can be noisy
both use high-dim vecs although DNNs get bad with him dims (e.g. 100k)
new codewords are made from existing ones
HD memory is a separate func
HD algos are transparent, incremental (on-line), scalable
somewhat closer to the brain…cerebellum anatomy seems to be match HD
HD: holistic (distributed repr.) is robust
HD papers
HDComputing Github Repos (see torchhd)
HD computing overview paper (Kanerva, 2009)
in these high dimensions, most points are close to equidistant from one another (L1 distance), and are approximately orthogonal (dot product is 0)
memory
heteroassociative - can return stored X based on its address A
autoassociative - can return stored X based on a noisy version of X (since it is a point attractor), maybe with some iteration
this adds robustness to the memory
this also removes the need for addresses altogether
-
note: for sparse vectors, might need some threshold before computing mean (otherwise will have too many zeros)
Neural Statistician (Edwards & Storkey, 2016) summarises a dataset by averaging over their embeddings
kanerva machine (yu…lillicrap, 2018)
like a VAE where the prior is derived from an adaptive memory store
theory of sequence indexing and working memory in RNNs
trying to make key-value pairs
VSA as a structured approach for understanding neural networks
reservoir computing = state-dependent network = echos-state network = liquid state machine - try to represen sequential temporal data - builds representations on the fly
different names
Tony plate: holographic reduced representation (1995)
related to TPR by paul smolensky
ross gayler: multiply-add-permute arch
gayler & levi: vector-symbolic arch
gallant & okaywe: matrix binding with additive termps
fourier holographic reduced reprsentations (FHRR; Plate)
…many more names
connecting to DNNs
Attention Approximates Sparse Distributed Memory https://arxiv.org/pdf/2111.05498.pdf
<IP_ADDRESS>. visual sampling#
Emergence of foveal image sampling from learning to attend in visual scenes (cheung, weiss, & olshausen, 2017) - using neural attention model, learn a retinal sampling lattice
can figure out what parts of the input the model focuses on
<IP_ADDRESS>. dynamic routing between capsules#
hinton 1981 - reference frames require structured representations
mapping units vote for different orientations, sizes, positions based on basic units
mapping units gate the activity from other types of units - weight is dependent on if mapping is activated
top-down activations give info back to mapping units
this is a hopfield net with three-way connections (between input units, output units, mapping units)
reference frame is a key part of how we see - need to vote for transformations
olshausen, anderson, & van essen 1993 - dynamic routing circuits
ran simulations of such things (hinton said it was hard to get simulations to work)
learn things in object-based reference frames
inputs -> outputs has weight matrix gated by control
zeiler & fergus 2013 - visualizing things at intermediate layers - deconv (by dynamic routing)
save indexes of max pooling (these would be the control neurons)
when you do deconv, assign max value to these indexes
arathom 02 - map-seeking circuits
tenenbaum & freeman 2000 - bilinear models
trying to separate content + style
hinton et al 2011 - transforming autoencoders - trained neural net to learn to shift imge
sabour et al 2017 - dynamic routing between capsules
units output a vector (represents info about reference frame)
matrix transforms reference frames between units
recurrent control units settle on some transformation to identify reference frame
notes from this blog post
problems with cnns
pooling loses info
don’t account for spatial relations between image parts
can’t transfer info to new viewpoints
capsule - vector specifying the features of an object (e.g. position, size, orientation, hue texture) and its likelihood
ex. an “eye” capsule could specify the probability it exists, its position, and its size
magnitude (i.e. length) of vector represents probability it exists (e.g. there is an eye)
direction of vector represents the instantiation parameters (e.g. position, size)
hierarchy
capsules in later layers are functions of the capsules in lower layers, and since capsule has extra properties can ask questions like “are both eyes similarly sized?”
equivariance = we can ensure our net is invariant to viewpoints by checking for all similar rotations/transformations in the same amount/direction
active capsules at one level make predictions for the instantiation parameters of higher-level capsules
when multiple predictions agree, a higher-level capsule is activated
steps in a capsule (e.g. one that recognizes faces)
receives an input vector (e.g. representing eye)
apply affine transformation - encodes spatial relationships (e.g. between eye and where the face should be)
applying weighted sum by the C weights, learned by the routing algorithm
these weights are learned to group similar outputs to make higher-level capsules
vectors are squashed so their magnitudes are between 0 and 1
outputs a vector
<IP_ADDRESS>. hierarchical temporal memory (htm)#
binary synapses and learns by modeling the growth of new synapses and the decay of unused synapses
separate aspects of brains and neurons that are essential for intelligence from those that depend on brain implementation
<IP_ADDRESS>.1. necortical structure#
evolution leads to physical/logical hierarchy of brain regions
neocortex is like a flat sheet
neocortex regions are similar and do similar computation
Mountcastle 1978: vision regions are vision becase they receive visual input
number of regions / connectivity seems to be genetic
before necortex, brain regions were homogenous: spinal cord, brain stem, basal ganglia, …
<IP_ADDRESS>.2. principles#
common algorithims accross neocortex
hierarchy
sparse distributed representations (SDR) - vectors with thousands of bits, mostly 0s
bits of representation encode semantic properties
inputs
data from the sense
copy of the motor commands
“sensory-motor” integration - perception is stable while the eyes move
patterns are constantly changing
necortex tries to control old brain regions which control muscles
learning: region accepts stream of sensory data + motor commands
learns of changes in inputs
ouputs motor commands
only knows how its output changes its input
must learn how to control behavior via associative linking
sensory encoders - takes input and turnes it into an SDR
engineered systems can use non-human senses
behavior needs to be incorporated fully
temporal memory - is a memory of sequences
everything the neocortex does is based on memory and recall of sequences of patterns
on-line learning
prediction is compared to what actually happens and forms the basis of learning
minimize the error of predictions
<IP_ADDRESS>.3. papers#
“A Theory of How Columns in the Neocortex Enable Learning the Structure of the World”
network model that learns the structure of objects through movement
object recognition
over time individual columns integrate changing inputs to recognize complete objects
through existing lateral connections
within each column, neocortex is calculating a location representation
locations relative to each other = allocentric
much more motion involved
multiple columns - integrate spatial inputs - make things fast
single column - integrate touches over time - represent objects properly
“Why Neurons Have Thousands of Synapses, A Theory of Sequence Memory in Neocortex”
learning and recalling sequences of patterns
neuron with lots of synapses can learn transitions of patterns
network of these can form robust memory
<IP_ADDRESS>. forgetting#
Continual Lifelong Learning with Neural Networks: A Review
main issues is catastrophic forgetting / stability-plasticity dilemma
2 types of plasticity
Hebbian plasticity (Hebb 1949) for positive feedback instability
compensatory homeostatic plasticity which stabilizes neural activity
approaches: regularization, dynamic architectures (e.g. add more nodes after each task), memory replay
<IP_ADDRESS>. population coding#
saxena_19_pop_cunningham: “Towards the neural population doctrine”
correlated trial-to-trial variability
Ni et al. showed that the correlated variability in V4 neurons during attention and learning — processes that have inherently different timescales — robustly decreases
‘choice’ decoder built on neural activity in the first PC performs as well as one built on the full dataset, suggesting that the relationship of neural variability to behavior lies in a relatively small subspace of the state space.
decoding
more neurons only helps if neuron doesn’t lie in span of previous neurons
encoding
can train dnn goal-driven or train dnn on the neural responses directly
testing
important to be able to test population structure directly
population vector coding - ex. neurons coded for direction sum to get final direction
reduces uncertainty
correlation coding - correlations betweeen spikes carries extra info
independent-spike coding - each spike is independent of other spikes within the spike train
position coding - want to represent a position
for grid cells, very efficient
sparse coding
hard when noise between neurons is correlated
measures of information
eda
plot neuron responses
calc neuron covariances
<IP_ADDRESS>. interesting misc papers#
berardino 17 eigendistortions
Fisher info matrix under certain assumptions = \(Jacob^TJacob\) (pixels x pixels) where Jacob is the Jacobian matrix for the function f action on the pixels x
most and least noticeable distortion directions corresponding to the eigenvectors of the Fisher info matrix
gao_19_v1_repr
don’t learn from images - v1 repr should come from motion like it does in the real world
repr
vector of local content
matrix of local displacement
why is this repr nice?
separate reps of static image content and change due to motion
disentangled rotations
learning
predict next image given current image + displacement field
predict next image vector given current frame vectors + displacement
friston_10_free_energy
7.4.9. neuro-inspired ai (niAI)#
<IP_ADDRESS>. neuro-dl reviews#
https://xcorr.net/2023/01/01/2022-in-review-neuroai-comes-of-age/
Neuroscience-Inspired Artificial Intelligence (hassabis et al. 2017)
Toward next-generation artificial intelligence: catalyzing the NeuroAI revolution (zador, …bengio, dicarlo, lecun, …sejnowski, tsao, 2022)
Computational language modeling and the promise of in silico experimentation (jain, vo, wehbe, & huth, 2023) - 4 experimental design exampmles
compare concrete & abstract words (binder et al. 2005)
contrast-based study of composition in 2-word phrase (Bemis & Pylkkanen, 2011)
checks for effects between group and individual (lerner et al. 2011)
forgetting behavior using controlled manipulations (chien & honey, 2020)
Dissociating language and thought in large language models (mahowald, …, tenebaum, fedorenko, 2023)
2 competences
formal linguistic competence - knowledge of rules and patterns of a given language
functional linguistic competence - cognitive abilities required for language understanding and use in the real world, e.g. formal reasoning, world knowledge, situation modeling, communicative intent
much of world knowledge is implied: people are much more likely to communicate new or unusual information rather than commonly known facts
language and thought are robustly dissociable in the brain
aphasia studies: despite the nearly complete loss of linguistic abilities, some individuals with severe aphasia have intact non-linguistic cognitive abilities: they can play chess, compose music, solve arithmetic problems and logic puzzles, …
fMRI studies: the language network is extremely selective for language processing: it responds robustly and reliably when people listen to, read, or generate sentences , but not when they perform arithmetic tasks, engage in logical reasoning, understand computer programs, listen to music, …
Merrill et al. [2022] - semantic information is in-principle learnable from language data
Piantadosi and Hill [2022] - an argument that models can genuinely learn meaning
Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks (collins…tenebaum, 2022)
as situation gets more OOD, LLM gets worse compared to human, e.g.
Get your sofa onto the roof of your house, without using a pulley, a ladder, a crane...
recommendations
modularity, curated data / diverse objectives, new benchmarks
Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems (smolensky, …, gao, 2022)
Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks (2023)
Designing Ecosystems of Intelligence from First Principles (friston et al. 2022)
NeuroAI - A strategic opportunity for Norway and Europe (2022)
Perceptual Inference, Learning, and Attention in a Multisensory World (nopponey, 2021)
Engineering a Less Artificial Intelligence (sinz…tolias, 2019) - overview of ideas to make DNNs more brain-like
<IP_ADDRESS>. biological constraints for DNNs#
Aligning DNN with brain responses
haven’t found anything like this for NLP
Aligning Model and Macaque Inferior Temporal Cortex Representations Improves Model-to-Human Behavioral Alignment and Adversarial Robustness (dapello, kar, shrimpf…cox, dicarlo, 2022) - finetune CNN embedding to match monkey brain (IT electrode recordings) before making classifications
Towards robust vision by multi-task learning on monkey visual cortex (safarani…sinz, 2021) - simultaneously predict monkey v1 (electrode data) and imagenet
Improved object recognition using neural networks trained to mimic the brain’s statistical properties - ScienceDirect (federer et al. 2020) - simultaneously train CNN to classify objects + have similar reprs to monkey electrode data
Learning from brains how to regularize machines (li …, tolias 2019) - regularize intermediate representations using mouse v1 data (optical imaging) for image classification
A Neurobiological Evaluation Metric for Neural Network Model Search (blanchard, …, bashivan, scheirer, 2019) - compare fMRI kernel matrix to DNN kernel matrix - find that the closer it is, the better a network is (and use this metric to perform early stopping)
aligning with experimental/psychological data
How Well Do Unsupervised Learning Algorithms Model Human Real-time and Life-long Learning? | OpenReview (zhuang…dicarlo, yamins, 2022)
Biologically-inspired DNNs (not data-driven)
Simulating a Primary Visual Cortex at the Front of CNNs Improves Robustness to Image Perturbations (dapello…cox, dicarlo, 2020) - biologically inspired early neural-network layers (gabors etc.) improve robustness of CNNs
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs (kubilius, schrimpt, kar, …, yamins, dicarlo, 2019)
Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs (baidya, dapello, dicarlo, & marques, 2021)
Surround Modulation: A Bio-inspired Connectivity Structure for Convolutional Neural Networks (hasani, …, aghajan, 2019) - add inhibitory lateral connections in CNNs
Biologically inspired protection of deep networks from adversarial attacks (nayebi & ganguli, 2017) - change training to get highly nonlinear, saturated neural nets
Biological constraints on neural network models of cognitive function (pulvermuller, …, wennekers, 2021) - review on biological constraints
Disentangling with Biological Constraints: A Theory of Functional Cell Types
<IP_ADDRESS>. nlp / fMRI#
interpreting brain encoding models
Brains and algorithms partially converge in natural language processing (caucheteux & king, 2022)
best brain-mapping are obtained from the middle layers of DL models
whether an algorithm maps onto the brain primarily depends on its ability to predict words context
average ROIs across many subjects
test “compositionality” of features
Tracking the online construction of linguistic meaning through negation (zuanazzi, …, remi-king, poeppel, 2022)
Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains (abnar, … zuidema, emnlp workshop, 2019) - use RSA to compare representations from language models with fMRI data from Wehbe et al. 2014
encoding models
Seminal language-semantics fMRI study (Huth…Gallant, 2016) - build mapping of semantic concepts across cortex using word vecs
(caucheteux, gramfort, & king, facebook, 2022) - predicts fMRI with gpt-2 on the narratives dataset
GPT‐2 representations predict fMRI response + extent to which subjects understand corresponding narratives
compared different encoding features: phoneme, word, gpt-2 layers, gpt-2 attention sizes
brain mapping finding: auditory cortices integrate information over short time windows, and the fronto-parietal areas combine supra-lexical information over long time windows
gpt2 models predict brain responses well (caucheteux & king, 2021)
Disentangling syntax and semantics in the brain with deep networks (caucheteux, gramfort, & king, 2021) - identify which brain networks are involved in syntax, semantics, compositionality
Incorporating Context into Language Encoding Models for fMRI (jain & huth, 2018) - LSTMs improve encoding model
The neural architecture of language: Integrative modeling converges on predictive processing (schrimpf, .., tenenbaum, fedorenko, 2021) - transformers better predict brain responses to natural language (and larger transformers predict better)
Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data | Neurobiology of Language (antonello & huth, 2022)
LLM brain encoding performance correlates not only with their perplexity, but also generality (skill at many different tasks) and translation performance
Prediction with RNN beats ngram models on individual-sentence fMRI prediction (anderson…lalor, 2021)
Interpret transformer-based models and find top predictions in specific regions, like left middle temporal gyrus (LMTG) and left occipital complex (LOC) (sun et al. 2021)
Semantic representations during language comprehension are affected by context (i.e. how langauge is presented) (deniz…gallant, 2021) - stimuli with more context (stories, sentences) evoke better responses than stimuli with little context (Semantic Blocks, Single Words)
Combining computational controls with natural text reveals new aspects of meaning composition (toneva, mitchell, & wehbe, 2022) - study word interactions by using encoding vector emb(phrase) - emb(word1) - emb(word2)…
decoding models
Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding (chen et al. 2022)
Semantic reconstruction of continuous language from non-invasive brain recordings (lebel, jain, & huth, 2022) - reconstruct continuous natural language from fMRI
Decoding speech from non-invasive brain recordings (defossez, caucheteux, …, remi-king, 2022)
Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies (nishimoto, …, gallant, 2011)
<IP_ADDRESS>. overview#
explaining concepts from neuroscience to inform deep learning
The brain currently outperforms deep learning in a number of different ways: efficiency, parallel computation, not forgetting, robustness. Thus, in these areas and others, the brain can offer high-level inspiration as well as more detailed algorithmic ideas on how to solve complex problems.
We begin with some history and perspective before further exploring these concepts at 3 levels: (1) the neuron level, (2) the network level, and (3) high-level concepts.
brief history
The history of deep learning is intimately linked with neuroscience. In vision, the idea of hierarchical processing dates back to Hubel and Weisel
Ranges from neurally-inspired -> biologically plausible
Computational neuroscientists often discuss understanding computation at Marr’s 3 levels of understanding: (1) computational, (2) algorithmic, and (3) mechanistic
cautionary notes
There are dangers in deep learning researchers constraining themselves to biologically plausible algorithms. First, the underlying hardware of the brain and modern von Neumman-based architectures is drastically different and one should not assume that the same algorithms will work on both systems. Several examples, such as backpropagation, were derived by deviating from the mindset of mimicking biology.
Second, the brain does not solve probleDangers for going too far…. One wouldn’t want to draw inspiration from the retina to put a hole in the camera.
Gallery of brain failures. Example, inside-out retina, V1 at back…
neuron-level
The fundamental unit of the brain is the neuron, which takes inputs from other neurons and then provides an output.
Individual neurons perform varying computations. Some neurons have been show to linearly sum their inputs
neurons are complicated (perceptron -> … -> detailed comparmental model)
For more information, see a very good review on modeling individual neurons
converting to spikes introduces noise
- perhaps just price of long-distance communication
network-level
Artificial neural networks can compute in several different ways. There is some evidence in the visual system that neurons in higher layers of visual areas can, to some extent, be predicted linearly by higher layers of deep networks
For the simplest intuition, here we provide an example of a canonical circuit for computing the maximum of a number of elements: the winner-take-all circuit.
Other network structures, such as that of the hippocampus are surely useful as well.
Questions at this level bear on population coding, or how groups of neurons jointly represent information.
engram - unit of cognitive information imprinted in a physical substance, theorized to be the means by which memories are stored
high-level concepts
Key concepts differentiate the learning process. Online,
learning
high-level
attention
memory
robustness
recurrence
topology
glial cells
inspirations
canonical cortical microcircuits
nested loop architectures
avoiding catostrophic forgetting through synaptic complexity
learning asymmetric recurrent generative models
spiking networks (bindsnet)
Computational Theory of Mind
Classical associationism
Connectionism
Situated cognition
Memory-prediction framework
Fractal Theory: https://www.youtube.com/watch?v=axaH4HFzA24
Brain sheets are made of cortical columns (about .3mm diameter, 1000 neurons / column)
Have ~6 layers
Brain as a Computer – Analog VLSI and Neural Systems by Mead (VLSI – very large scale integration)
Process info
Signals represented by potential
Signals are amplified = gain
Power supply
Knowledge is not stored in knowledge of the parts, but in their connections
Based on electrically charged entities interacting with energy barriers
http://scienceblogs.com/developingintelligence/2007/03/27/why-the-brain-is-not-like-a-co/
Brain’ storage capacity is about 2.5 petabytes (Scientific American, 2005)
Electronics
Voltage can be thought of as water in a reservoir at a height
It can flow down, but the water will never reach above the initial voltage
A capacitor is like a tank that collects the water under the reservoir
The capacitance is the cross-sectional area of the tank
Capacitance – electrical charge required to raise the potential by 1 volt
Conductance = 1/ resistance = mho, siemens
We could also say the word is a computer with individuals being the processors – with all the wasted thoughts we have – the solution is probably to identify global problems and channel people’s focus towards working on them
Brain chip: http://www.research.ibm.com/articles/brain-chip.shtml
Brains are not digital
Brains don’t have a CPU
Memories are not separable from processing
Asynchronous and continuous
Details of brain substrate matter
Feedback and Circular Causality
Asking questions
Brains has lots of sensors
Lots of cellular diversity
NI uses lots of parallelism
Delays are part of the computation
-
problems with brain simulations
Not possible to test specific scientific hypotheses (compare this to the large hadron collider project with its perfectly defined hypotheses)
Does not simulate real brain processing (no firing connections, no biological interactions)
Does not give any insight into the functionality of brain processing (the meaning of the simulated activity is not assessed)
Neuron information processing parts
Dendritic spikes are like first layer of conv net
Neurons will typically have a genome that is different from the original genome that you were assigned to at birth. Neurons may have additional or fewer chromosomes and have sequences of information removed or added from certain chromosomes.
http://timdettmers.com/2015/03/26/convolution-deep-learning/
The adult brain has 86 billion neurons, about 10 trillion synapse, and about 300 billion dendrites (tree-like structures with synapses on them


Click or drag file to this area to upload
Support for a single upload. Supports only .json, and .csv files.


Compilation Guide¶
Compilation of the package is quite simple (says an expert …)
Compilation on Windows¶
Windows only supports compilation for python3.5 and higher. This is for the reason that python2 needs an ancient MSCV compiler, which does not support the new coding standard C++14 (also the old compiler created 3x slower code).
For compilation of python version 3.5 and higher on Windows download Microsoft Visual Studio . Then clone the repository into a folder:
git clone --depth 1 --branch master --single-branch https://github.com/qd-cae/qd-cae-python.git
cd qd-cae-python
The argument --depth
only makes a shallow copy, which saves a lot time and memory. Thereafter run the setup script, the code will automatically compile and install.
python setup.py install
Compilation on Linux¶
Note that in contrast to windows, on Linux the library can be compiled for both, python2 and python3. Check first whether your compiler is new enough.
g++ --version
You will need g++ version 4.8 or higher. If your compiler is fine, just install as follows:
git clone https://github.com/qd-cae/qd-cae-python.git
cd qd-cae-python
sudo python setup.py install
As a little hint, if you want force python to take a modern compiler version (it may happen that you have multiple g++), simply overwrite the CC attribute before the running the setup script (bash shell):
sudo CC=g++-5 python setup.py install
Tests¶
It is highly recommended to run the test suite after compilation to check the installation!
python setup.py test
Hints on OpenMP¶
By default the library uses and supports OpenMP parallelization. Sometimes compatability issues arise though if the path to OpenMP (on your Linux) is not configured correctly (it can confuse the Anaconda OpenMP with your system one). To check if OpenMP is really an issue during compilation, you can optionally disable OpenMP in the top of the setup.py
with use_openmp=False
and the library will be compiled and run without it.
Compilation with FEMZIP support¶
Important note: The installation skips femzip if the libraries are missing (checks if folder with link libraries exists). If it can not find the folder, then the compiler definition QD_USE_FEMZIP
is not used. This cancels out all code in the source files regarding FEMZIP.
If one wants to compile with FEMZIP support, download the libraries from the Sidact Homepage. There are two things to download, the actual femunzip library:
Windows: FEMZIP_[version]_dyna_NO_OMP_Windows_VS2012_MD_x64.tar.gz
Linux: FEMZIP_[version]_dyna_NO_OMP_Linux_64Bit.tar.gz
as well as the link libraries (FEMUNZIPLIB-LPR_[version]_link_libraries.tar.gz). Put all content into a folder and register the path in the header of the setup.py.
femzip_path_windows = "path/to/femzip/windows/libs"
femzip_path_linux = "path/to/femzip/linux/libs"


8.8 Questions and Answers
Q: In the
implementation of chained hash tables presented in this
chapter, the actual hash code used for accessing the table is
the hash code modulo the table size. Why is this?
A: This
transformation ensures that the hash coding does not position
us past the end of the table. Although the hash function
should ensure this itself, it is worthwhile for the hash table
implementation to provide the guarantee as well, especially
since the hash function is provided by the caller. However,
this is not the same reason that the modulo is performed when
double hashing a key in an open-addressed hash table. In this
case, the process of double hashing may produce a hash coding
that falls outside of the bounds of the table, even for two
auxiliary hash functions each producing hash codings within
the table. This is because the two hash codings are added
together.
Q: Why are
hash tables good for random access but not sequential access?
For example, in a database system in which records are to be
accessed in a sequential fashion, what is the problem with
hashing?
A: Hash tables
are excellent for random access because each key hashes us
precisely to where we need to be in the table to access the
data, or at least within a few steps when a collision occurs.
However, hash tables do not support sequential access. After
hashing to some position, we have no way to determine where
the next smallest or largest key resides. Compare this with a
linked list containing elements that are sorted. Assuming some
initial position in the list, the next key is easy to
determine: we simply look at the next element in the list.
Q: What is the
worst-case performance of searching for an element in a
chained hash table? How do we ensure that this case will not
occur?
A: A chained
hash table performs the worst when all elements hash into a
single bucket. In this case, searching for an element is O (n),
where n is the number of
elements in the table. A ridiculous hash function that would
result in this performance is h
(k) = c, where c is some constant within the bounds
of the hash table. Selecting a good hash function ensures that
this case will not occur. If the hash function approximates
uniform hashing well, we can expect to locate an element in
constant time.
Q: What is the
worst-case performance of searching for an element in an
open-addressed hash table? How do we ensure that this case
will not occur?
A: The
worst-case performance of searching for an element in an
open-addressed hash table occurs once the hash table is
completely full and the element we are searching for is not in
the table. In this case, searching for an element is an O (m)
operation, where m is the
number of positions in the table. This case can occur with any
hash function. To ensure reasonable performance in an
open-addressed hash table, we should not let the table become
more than 80% full. If we choose a hash function that
approximates uniform hashing well, we can expect performance
consistent with what is presented in Table
8.1. |


Google & Intel have been focusing on the lack of diversity within their companies for the last 2 years. Their public analysis of the diversity of their employees hasn't changed much over the last 2 years - 71% men and 60% white. Two years, no changes - so what are they trying to do now?
All factors are being looked at. One may surprise you that attrition is a problem for these low-ranking diversity companies. Studies found that companies with low diversity had high turnover amoungst women and minorities. Attrition is a focus and has changed the way some are paying employees and benefits are structured.
Some are trying more outside the box approaches. Google and NASA have pressured Disney to start adding female characters that find joy in programming as a hobby. The makers of "Miles From Tomorrowland" have added characters that change the science stereotype into a more positive one for children. The goal being to get young girls to think about the possibilities of a career in these fields.
Maybe even this is a bigger picture thing that companies are only part of the equation. Who do we blame when young adults don't turn out the way they should? Parents! Yes parents can be a scapegoat for why women don't get in tech jobs. Did you know in 1918 blue was considered a feminine shade? Girls are pressured to wear pink for the first years of childhood and only play with dolls. Many parents and studies have came out stating that these push young girls to certain roles and away from fields that boys identify with.


ORC ID 0000-0002-0560-6023
Contact<EMAIL_ADDRESS>(firstname.name = clovis.galiez)
My research focuses on building computational tools for environmental data analysis. My main application field is biology and in particular the microbial world (metagenomics data), but I'm also interested in application to ecology and geosciences. More details on research interest or publications webpages.
I give several lectures related to data science at Ensimag (Grenoble INP group) in Université Grenoble Alpes.


Résumé
Summary
UX designer skilled in interface design, user experience research, and HTML prototyping.
Experience
- Senior Interaction Designer
- NPR, Washington, DC, November 2014 - Present
- Design, moderate, analyze, and present user experience research
- Create designs and collaborate with multi-displinary Scrum team to impliment and iterate
- Collaborate with newsroom staff for various initiatives, including the Morning Edition book club
- Co-chair: Community Engagement
- IA Summit, Washington, DC, March 2014 - April 2015
- Curate the professional and social programs of a 600-person, national conference
- Led first ever research initiative with attendees, speakers, and stakeholders
- Recruit and enable volunteers to complete their respective duties
- Senior Interaction Designer
- EightShapes, Washington, DC, June 2011 - January 2014
- Design, moderate, analyze, and present user experience research
- Create designs and accompanying design documentation
- Prototype designs using HTML, CSS, and JavaScript within the Blocks prototype toolkit
- User Experience Designer
- AED (now FHI 360), Washington, DC, December 2008 - May 2011
- Research, design, and prototype user experiences while collaborating with teammates
- Introduced and grew UX design and research into a prominent component of team’s process
- Moderated usability testing and interviews with teachers in Rwandan schools
- Led redesign of Global Learning Portal logo, including research activity with stakeholders
- Class Administrator for Information Architecture and the User Experience
- GS Graduate School, Washington, DC, September 2010 - December 2011
- Assist instructor Thom Haller in teaching and discussing IA and UX principles
- Provide connection for students to DC IA/UX community
- Gallery Assistant
- Healing Arts Gallery at Smith Farm Center, Washington, DC, August 2008 - December 2008
- Monitored gallery and assisted Gallery Director with special event coordination
- Designed bimonthly postcards and regular Constant Contact emails
- Chair of the Student Educational Policy Committee
- Grinnell College Department of Art, Grinnell, IA, August 2007 - May 2008
- Organized the annual Student Art Salon held in Faulconer Gallery
- Recruited and coordinated guest juror of the Student Art Salon
- Technology Consultant
- Grinnell College ITS, January 2005 - May 2008
- Troubleshot PC and Apple computers remotely and on-site from the IT Helpdesk
- Mentored newly hired staff, explaining technical concepts in plain language
Presentations, Publications, & Community Involvement
- Guest lecturer at Intermediate Web Design at the Art Institute of Washington, with instructor Patricia Kruep, 17 November 2014: Fluent UX Sketching
- Guest lecturer at User Experience Design at the General Assembly, with instructors Jennifer Romano Bergstrom and Sree Anirudh Bhandaram, 10 November 2014: Fluent UX Sketching
- Presentation at Midwest UX 2014, 24 October 2014: The Self-Aware Researcher (Description, Slide deck, Video)
- Presentation at DCWW Code(Her) Conference 2014, 13 September 2014: A Nicer Kind of Interrogation (Conference Website, Slide deck)
- Presentation at 2014 IA Summit, 29 March 2014: The Self-Aware Researcher (Description, Slide deck)
- Workshop for DC Web Women Code(Her) Workshops, 14 January 2014: Intro to Syntactically Awesome Style Sheets (Sass) (Slide deck, Example code)
- Discussion at Mobile UXCamp DC 2013, 14 September 2013: Native App Reviews — Bring your App! with James Melzer
- Workshops at Sketchcamp Grand Rapids 2013, 19 July 2013: Sketchnoting in the Deep End (Event website)
- Presentation at Sketchcamp Grand Rapids 2013, 19 July 2013: Learning to Sketch through Sketchnoting (Event website)
- Guest Lecturer at INFM 702 Users and Use Context at the University of Maryland, with instructor Vera Rhoads, 18 April 2013: Sketching and the User Experience (Program website)
- Guest Post for the Sketchnote Army, 9 May 2013: How to Start Sketchnoting (Blog post)
- Sketchnote Army Field Marshall for the IA Summit 2013, April 2012: How to Start Sketchnoting (Announcement and Flickr pool)
- Article for the Bulletin of the Association for Information Science and Technology, October/November 2012: How to Start Sketchnoting (Article)
- Podcast on Radio Johnny with Jeff Parks, 24 September 2012: Sketching Out a Greater Understanding (Podcast)
- Presentation at IxDA DC, 28 August 2012: How and Why to Start Sketchnoting (Event website)
- Presentation at UX Week 2012, 24 March 2012: How and Why to Start Sketchnoting (Video)(Description)
- Workshop at Midwest UX 2012, 31 May 2012: Let's Sketchnote! with Binaebi Akah and Charlene McBride (Summary and slides)(Sketchnote Army post)
- Guest Lecturer at Information Architecture & the User Experience at the Graduate School, with instructor Thom Haller, 2 May 2012: Sketching and User Experience
- Podcast for LDNIA Podcast, 2 April 2012: Special Edition from IA Summit 12 (Podcast & Summary)
- Presentation at IA Summit 2012, 24 March 2012: Why and How to Start Sketchnoting (Slide deck)
- Presentation at The Big Redux DC, 30 April 2011: UX Research in the Real World (Slideshare)
- Presentation at Midwest UX Conference, 9 April 2011: UX Research in the Real World (Slideshare) (PDF) (Talk description)
- Poster at IA Summit 2011, 1 April 2011: UX Research in the Real World: Stories from Rwanda (PDF)
- Presentation at UXCamp DC 2011, 8 January 2011: My Rwandan User Experience
Skills
- Skilled in interaction design, information architecture, and user experience research
- Skilled at prototyping with HTML, CSS, jQuery
- Proficient with Adobe products: especially InDesign, Illustrator
- Resourceful and quick-learning
- Skilled at rapport building
Education
- Bachelor of Arts in Art, with honors; GPA: 3.7/4.0
- Grinnell College, Grinnell, IA, May 2008
- Research Seminar in the Humanities
- Newberry Library, Chicago, IL, Fall 2006


{{ page.title }}
{{ content }}Sagefy
An open-content adaptive learning platform.
- Home
- Ideas
- Data Structure
- User Stories
- Technology Stack
- Contributing
- Planning
- How to Build a Course in Sagefy
- Reference
- Web API
© Copyright Sagefy 2017. All contents are licensed under Apache 2.0


Summer weather is generally settled, but during the rest of the year it can change very quickly. The winds during summer alternate between NW and NE. At other times, SW winds are more frequent and this is also the direction of most gales. More than strong winds, the main danger in the islands is the extremely strong tidal streams, the notorious Alderney Race reaching 8 knots on occasions. Other hazards are concealed rocks and poor visibility.
Jersey Coastguard - VHF Ch 25 and Ch 82 - Shipping Forecast (and any gale or navigational warnings) at 06:45, 07:45, 08:45, 12:45, 18:45, 22:45. Gale Warnings, on receipt and at 03:07, 09:07, 15:07,21:07. Wind information (speed and gusts at St Helier Port Control) transmitted on VHF 18 every two minutes.
Channel Islands Weather Forecast
For links to free global weather information, forecast services and extreme weather information see the Noonsite Weather Page
Alderney: Braye *
Guernsey: Beaucette * , St Peter Port * , St Sampson
Jersey: Gorey * , St Aubin's Harbour , St Helier *
* indicates port of entry


| Home | Trees | Indices | Help |
|
|---|
|
|
object --+ | Arbiter
| Instance Methods | |||
|
|||
|
Inherited from |
| Properties | |
|
contacts Information on the contact points between the objects. |
|
|
a The first shape involved in the collision |
|
|
b The second shape involved in the collision |
|
|
elasticity Elasticity |
|
|
friction Friction |
|
|
surface_velocity Surface velocity |
|
|
stamp Time stamp of the arbiter. |
|
|
Inherited from |
| Method Details |
|
| Property Details |
contactsInformation on the contact points between the objects.
|
aThe first shape involved in the collision
|
bThe second shape involved in the collision
|
elasticityElasticity
|
frictionFriction
|
surface_velocitySurface velocity
|
stampTime stamp of the arbiter. (from the space)
|
| Home | Trees | Indices | Help |
|
|---|
| Generated by Epydoc 3.0.1 on Sat Nov 01 23:36:27 2008 | http://epydoc.sourceforge.net |


Table
General
Tables are lists in which the individual items have common elements with each other. The rows of a table
represent each item of the list and its different elements. Item elements are divided into columns with
descriptive headings that describe each column of information.
Purpose
Visually organize related items of information within a single component.
Examples
Appearance
Table Specifics
Implementation
- Column headings describe the information found in the column.
- Columns can be made sortable by adding .ias-sortable which sorts information in ascending or descending
order when clicked.
- .ias-align-end right aligns text within the table cell.
- Action icon-buttons can be added to rows to allow for more options to edit row data.
Common Patterns
- The most recognizable element of a row is placed in the first column where most readers will start scanning.
- Icons can be added the first column to help identify important rows of information.
- Numerical values will often use .ias-align-end to be displayed as right aligned in a table cell.
- Checkboxes allow users to select multiple rows of data at once.
- A checkbox in the header signifies a "select all" checkbox.
Nesting
Nestable Components
- Icons .ias-icon
- Icon Buttons .ias-icon-button
- Inputs (Checkboxes)
- Links <a>
Nesting Tables in Other Components
Tables are a component used to organize information in an visually recognizable format and can be implemented in
many places throughout an application. A component with a moderately sized amount of information could make use
of a table to organize that information.
Responsive Displays
As the display width decreases a table will also reduce its width. If the text exceeds the width of the table
cell containing it, it will be wrapped to a new line and the cell will increase its height.


Stafford, several others out for Lions vs. Washington
DETROIT (AP) - The Detroit Lions have ruled out quarterback Matthew Stafford for a third straight game and will also be without several other players this weekend at Washington.
Stafford is out with back and hip problems, putting Jeff Driskel in the lineup at quarterback. Detroit has also ruled out cornerback Jamal Agnew (ankle), defensive end Trey Flowers (concussion), defensive lineman Da’Shawn Hand (ankle), offensive lineman Frank Ragnow (concussion) and defensive back Tracy Walker (knee).
The Lions have lost three in a row and six of their last seven. Washington has lost four in a row.
More AP NFL: https://apnews.com/NFL and https://twitter.com/AP_NFL
Sign up for Daily Newsletters
Copyright © 2019 The Washington Times, LLC.
Please read our comment policy before commenting.


Enter following command into your terminal and the script will do everything for you:
wget -qO- "{{this.apiServer}}api/device/new/{{this.http.username}}/{{selectedDevice.id}}/auto" | sudo python3 -
You can link your device with this account by using simple-guardian-client:
sudo simple-guardian-client login "{{this.apiServer}}api/device/new/{{this.http.username}}/{{selectedDevice.id}}"
sudo service simple-guardian restart
By adding deb repository you can take advantage of using automatic updates.
Paste following command to your terminal to install simple-guardian on your device:
# first make sure that you have root rights
sudo ls
# then import the repository key
wget -O - https://deb.adamhlavacek.com/pub.gpg | sudo apt-key add -
# then add the repository to your system
echo "deb https://deb.adamhlavacek.com ./" | sudo tee -a /etc/apt/sources.list
# update
sudo apt update
# install
sudo apt install simple-guardian
# login with the server
simple-guardian-client login "{{this.apiServer}}api/device/new/{{this.http.username}}/{{selectedDevice.id}}"


Every GEOS application has an object called the Process object. This object is run by the application's primary thread and is an instance of a subclass of
GenProcessClass
. Because the Process is an event-driven object, there is no
main()
routine that is executed when the program is launched. Instead, the object will wait until it receives messages (events), at which time it will execute the proper methods.
Applications can be of two basic models: The procedural model puts the entire functionality of the application within the Process object, not using any other objects in the application's thread (UI objects are run by the UI thread). All messages sent to the application are handled by the Process object, and most OOP issues can be avoided. The object-oriented model allows for other objects to be run by the application's primary thread; each of these objects will have its own instance data and be located in an object block (a resource) associated with its own message queue.
There is no command or directive that determines which model is used--the distinction is inherent within the format of message handlers and therefore can be virtually ignored. Hello World, for example, is simple enough to use the procedural model, handling all functionality with the Process object.
The Process object of Hello World is shown in Hello World's Process Object .
Code Display 2-3 Hello World's Process Object
This code display is part of hello3.goc and follows the previous display directly. */
/*********************************************************************** * Class & Method Definitions * This section contains the definition of the application's Process * class and its methods. Other classes can also be defined here, * along with the message each handles. ***********************************************************************/
/* * Here we define "HelloProcessClass" as a subclass of the system-provided * "GenProcessClass". As this application is launched, an instance of this class * will be created automatically to handle application-related events (messages). * The application thread will be responsible for running this object, * meaning that whenever this object handles a message, it will be executing * in the application's thread. */
/* You will find no object in this file declared to be of this class. Instead, * this class is bound to the application thread in hello3.gp.
@class HelloProcessClass, GenProcessClass;
/* The messages HelloProcessClass objects can handle that are not * system-defined are enumerated here. Each of these messages is sent * by one of the triggers in the dialog box. This is where class- * specific messages for this application (not system-defined messages) * are defined. */
@message void MSG_HELLO_CHANGE_TO_BLUE(); /* sent by Blue trigger */ @message void MSG_HELLO_CHANGE_TO_GOLD(); /* sent by Gold trigger */ @message void MSG_HELLO_REDRAW_DOCUMENT(); /* sent by Process to itself */
@endc /* signifies end of class definition */
/* Because this class definition must be stored in memory at runtime, * we must declare it here along with a flag indicating how it should * be loaded. The "neverSaved" flag is used because Process classes * are never saved to state files and therefore no relocation tables * need be built. */ @classdecl HelloProcessClass, neverSaved;