Table of Contents
Optimizing Gradient Descent
Blog: SGD, Momentum and Adaptive Learning Rate
EE227C Notes: Convex Optimization and Approximation
Convex Functions
Strong Convexity
Lipschitz continuous gradient (condition)
Conjugate Gradient
NOTES ON FIRST-ORDER METHODS FOR MINIMIZING SMOOTH FUNCTIONS
Gradient Descent (paperspace)
An Intuitive Introduction to the Hessian for Deep Learning
NIPS Optimization Lecture!!
10 Gradient Descent Optimisation Algorithms + Cheat Sheet (Blog!)
Gradient-Based Optimization
- Gradient-Based Optimization:
Gradient Methods are algorithms to solve problems of the form:$$\min_{x \in \mathbb{R}^{n}} f(x)$$
with the search directions defined by the gradient of the function at the current point.
- Gradient-Based Algorithms:
Examples, include:- Gradient Descent: minimizes arbitrary differentiable functions.
- Conjugate Gradient: minimizes sparse linear systems w/ symmetric & positive-definite matrices.
- Coordinate Descent: minimizes functions of two variables.
Gradient Descent
-
Gradient Descent:
Gradient Descent is a first-order, iterative algorithm to minimize an objective function \(J(\theta)\) parameterized by a model’s parameters \(\theta \in \mathbb{R}^{d}\) by updating the parameters in the opposite direction of the gradient of the objective function \(\nabla_{\theta} J(\theta)\) w.r.t. to the parameters.
Intuition (for derivation):
- Local Search from a starting location on a hill
- Feel around how a small movement/step around your location would change the height of the surrounding hill (is the ground higher or lower)
- Make the movement/step consistent as a small fixed step along some direction
- Measure the steepness of the hill at the new location in the chosen direction
- Do so by Approximating the steepness with some local information
- Find the direction that decreases the steepness the most
\(\iff\) .
- Local Search from an initial point \(x_0\) on a function
- Explore the value of the function at different small nudges around \(x_0\)
- Make the nudges consistent as a small fixed step \(\delta\) along a normalized direction \(\hat{\boldsymbol{u}}\)
- Evaluate the function at the new location \(x_0 + \delta \hat{\boldsymbol{u}}\)
- Do so by Approximating the function w/ first-order information (Taylor expansion)
- Find the direction \(\hat{\boldsymbol{u}}\) that minimizes the function the most
- Derivation:
A small change in \(\boldsymbol{x}\):
We would like to know how would a small change in \(\boldsymbol{x}\), namely \(\Delta \boldsymbol{x}\) would affect the value of the function \(f(x)\). This will allow us to evaluate the function:$$f(\mathbf{x}+\Delta \mathbf{x})$$
to find the direction that makes \(f\) decrease the fastest.
Let’s set up \(\Delta \boldsymbol{x}\), the change in \(\boldsymbol{x}\), as a fixed step \(\delta\) along some normalized direction \(\hat{\boldsymbol{u}}\):
$$\Delta \boldsymbol{x} = \delta \hat{\boldsymbol{u}}$$
The Gradient:
The gradient tells us how that small change in \(f(\mathbf{x}+\Delta \mathbf{x})\) affects \(f\) through the first-order approximation:$$f(\mathbf{x}+\Delta \mathbf{x}) \approx f(\mathbf{x})+\Delta \mathbf{x}^{T} \nabla_{\mathbf{x}} f(\mathbf{x})$$
In the single variable case, \(f(x+\delta) \approx f(x)+\delta f'(x)\), we know that \(f\left(x-\delta \operatorname{sign}\left(f^{\prime}(x)\right)\right)\) is less than \(f(x)\) for small enough \(\delta\).
We can thus reduce \(f(x)\) by moving \(x\) in small steps with the opposite sign of the derivative.The Change in \(f\):
The change in the objective function is:$$\begin{aligned} \Delta f &= f(\boldsymbol{x}_ 0 + \Delta \boldsymbol{x}) - f(\boldsymbol{x}_ 0) \\ &= f(\boldsymbol{x}_ 0 + \delta \hat{\boldsymbol{u}}) - f(\boldsymbol{x}_ 0)\\ &= \delta \nabla_x f(\boldsymbol{x}_ 0)^T\hat{\boldsymbol{u}} + \mathcal{O}(\delta^2) \\ &= \delta \nabla_x f(\boldsymbol{x}_ 0)^T\hat{\boldsymbol{u}} \\ &\geq -\delta\|\nabla f(\boldsymbol{x}_ 0)\|_ 2 \end{aligned} $$
using the first order approximation above.
Notice:$$\nabla_x f(\boldsymbol{x}_ 0)^T\hat{\boldsymbol{u}} \in \left[-\|\nabla f(\boldsymbol{x}_ 0)\|_ 2, \|\nabla f(\boldsymbol{x}_ 0)\|_ 2\right]$$
since \(\hat{\boldsymbol{u}}\) is a unit vector; either aligned with \(\nabla_x f(\boldsymbol{x}_ 0)\) or in the opposite direction; it contributes nothing to the magnitude of the dot product.
So, the \(\hat{\boldsymbol{u}}\) that changes the above inequality to equality, achieves the largest negative value (moves the most downhill). That vector \(\hat{\boldsymbol{u}}\) is, then, the one in the negative direction of \(\nabla_x f(\boldsymbol{x}_ 0)\); the opposite direction of the gradient.
The Directional Derivative:
The directional derivative in direction \(\boldsymbol{u}\) (a unit vector) is the slope of the function \(f\) in direction \(\boldsymbol{u}\). In other words, the directional derivative is the derivative of the function \(f(\boldsymbol{x}+\alpha \boldsymbol{u})\) with respect to \(\delta\), evaluated at \(\delta= 0\).
Using the chain rule, we can see that \(\frac{\partial}{\partial \delta} f(\boldsymbol{x}+\delta \boldsymbol{u})\) evaluates to \(\boldsymbol{u}^{\top} \nabla_{\boldsymbol{x}} f(\boldsymbol{x})\) when \(\delta=0\).Minimizing \(f\):
To minimize \(f\), we would like to find the direction in which \(f\) decreases the fastest. We do so by using the directional derivative:$$\begin{aligned} & \min _{\boldsymbol{u}, \boldsymbol{u}^{\top} \boldsymbol{u}=1} \boldsymbol{u}^{\top} \nabla_{\boldsymbol{x}} f(\boldsymbol{x}) \\ =& \min_{\boldsymbol{u}, \boldsymbol{u}^{\top} \boldsymbol{u}=1}\|\boldsymbol{u}\|_{2}\left\|\nabla_{\boldsymbol{x}} f(\boldsymbol{x})\right\|_{2} \cos \theta \\ =& \min_{\boldsymbol{u}} \cos \theta \\ \implies& \boldsymbol{u} = -\nabla_x f(x)\end{aligned}$$
by substituting \(\|\boldsymbol{u}\|_2 = 1\) and ignoring factors that do not depend on \(\boldsymbol{u}\), we get \(\min_{\boldsymbol{u}} \cos \theta\); this is minimized when \(\boldsymbol{u}\) points in the opposite direction as the gradient.
Or rather, because \(\hat{\boldsymbol{u}}\) is a unit vector, we need:$$\hat{\boldsymbol{u}} = - \dfrac{\nabla_x f(x)}{\|\nabla_x f(x)\|_ 2}$$
In other words, the gradient points directly uphill, and the negative gradient points directly downhill. We can decrease \(f\) by moving in the direction of the negative gradient.
The method of steepest/gradient descent:
Proposes a new point to decrease the value of \(f\):$$\boldsymbol{x}^{\prime}=\boldsymbol{x}-\epsilon \nabla_{\boldsymbol{x}} f(\boldsymbol{x})$$
where \(\epsilon\) is the learning rate, defined as:
$$\epsilon = \dfrac{\delta}{\left\|\nabla_{x} f(x)\right\|_ {2}}$$
-
The Learning Rate:
The learning rate is a hyper-parameter that controls how much we are adjusting the weights of our network with respect the loss gradient.The learning rate comes from a modification of the step-size in the GD derivation.
We get the learning rate by employing a simple idea:
We have a fixed step-size \(\delta\) that dictated how much we should be moving in the direction of steepest descent. However, we would like to keep the step-size from being too small or overshooting. The idea is to make the step-size proportional to the magnitude of the gradient (i.e. some constant multiplied by the magnitude of the gradient):$$\delta = \epsilon \left\|\nabla_{x} f(x)\right\|_ {2}$$
If we do so, we get a nice cancellation as follows:
$$\begin{aligned}\Delta \boldsymbol{x} &= \delta \hat{\boldsymbol{u}} \\ &= -\delta \dfrac{\nabla_x f(x)}{\|\nabla_x f(x)\|_ 2} \\ &= - \epsilon \left\|\nabla_{x} f(x)\right\|_ {2} \dfrac{\nabla_x f(x)}{\|\nabla_x f(x)\|_ 2} \\ &= - \dfrac{\epsilon \left\|\nabla_{x} f(x)\right\|_ {2}}{\|\nabla_x f(x)\|_ 2} \nabla_x f(x) \\ &= - \epsilon \nabla_x f(x) \end{aligned}$$
where now we have a fixed learning rate instead of a fixed step-size.
Choosing the Learning Rate:
- Set it to a small constant
- Line Search: evaluate \(f\left(\boldsymbol{x}-\epsilon \nabla_{\boldsymbol{x}} f(\boldsymbol{x})\right)\) for several values of \(\epsilon\) and choose the one that results in the smallest objective value.
Finds a local minimum along a search direction by solving an optimization problem in 1-D.e.g. for smooth \(f\): Secant Method, Newton-Raphson Method (may need Hessian, hard for large dims)
for non-smooth \(f\): use direct line search e.g. golden section search
Note: usually NOT used in DL - Trust Region Method
- Grid Search: is simply an exhaustive searching through a manually specified subset of the hyperparameter space of a learning algorithm. It is guided by some performance metric, typically measured by cross-validation on the training set or evaluation on a held-out validation set.
- Population-Based Training (PBT): is an elegant implementation of using a genetic algorithm for hyper-parameter choice.
In PBT, a population of models are created. They are all continuously trained in parallel. When any member of the population has had sufficiently long to train to show improvement, its validation accuracy is compared to the rest of the population. If its performance is in the lowest \(20\%\), then it copies and mutates the hyper-parameters and variables of one of the top \(20\%\) performers.
In this way, the most successful hyper-parameters spawn many slightly mutated variants of themselves and the best hyper-parameters are likely discovered. - Bayesian Optimization: is a global optimization method for noisy black-box functions. Applied to hp optimization, it builds a probabilistic model of the function mapping from hp values to the objective evaluated on a validation set. By iteratively evaluating a promising hp configuration based on the current model, and then updating it, it, aims to gather observations revealing as much information as possible about this function and, in particular, the location of the optimum. It tries to balance exploration (hps for which the outcome is most uncertain) and exploitation (hps expected close to the optimum).
In practice, it has been shown to obtain better results in fewer evaluations compared to grid search and random search, due to the ability to reason about the quality of experiments before they are run.
Line Search VS Trust Region:
Trust-region methods are in some sense dual to line-search methods: trust-region methods first choose a step size (the size of the trust region) and then a step direction, while line-search methods first choose a step direction and then a step size.Learning Rate Schedule:
A learning rate schedule changes the learning rate during learning and is most often changed between epochs/iterations. This is mainly done with two parameters: decay and momentum.- Decay: serves to settle the learning in a nice place and avoid oscillations, a situation that may arise when a too high constant learning rate makes the learning jump back and forth over a minima, and is controlled by a hyperparameter.
- Momentum: is analogous to a ball rolling down a hill; we want the ball to settle at the lowest point of the hill (corresponding to the lowest error). Momentum both speeds up the learning (increasing the learning rate) when the error cost gradient is heading in the same direction for a long time and also avoids local minima by ‘rolling over’ small bumps.
Types of learning rate schedules for Decay:
- Time-based learning schedules alter the learning rate depending on the learning rate of the previous time iteration. Factoring in the decay the mathematical formula for the learning rate is:
$${\displaystyle \eta_{n+1}={\frac {\eta_{n}}{1+dn}}}$$
where \(\eta\) is the learning rate, \(d\) is a decay parameter and \(n\) is the iteration step.
- Step-based learning schedules changes the learning rate according to some pre defined steps:
$${\displaystyle \eta_{n}=\eta_{0}d^{floor({\frac {1+n}{r}})}}$$
where \({\displaystyle \eta_{n}}\) is the learning rate at iteration \(n\), \(\eta_{0}\) is the initial learning rate, \(d\) is how much the learning rate should change at each drop (0.5 corresponds to a halving) and \(r\) corresponds to the droprate, or how often the rate should be dropped (\(10\) corresponds to a drop every \(10\) iterations). The floor function here drops the value of its input to \(0\) for all values smaller than \(1\).
- Exponential learning schedules are similar to step-based but instead of steps a decreasing exponential function is used. The mathematical formula for factoring in the decay is:
$$ {\displaystyle \eta_{n}=\eta_{0}e^{-dn}}$$
where \(d\) is a decay parameter.
-
Convergence:
Gradient Descent converges when every element of the gradient is zero, or very close to zero within some threshold.With certain assumptions on \(f\) (convex, \(\nabla f\) lipschitz) and particular choices of \(\epsilon\) (chosen via line-search etc.), convergence to a local minimum can be guaranteed.
Moreover, if \(f\) is convex, all local minima are global minimia, so convergence is to the global minimum.
-
Choosing (tuning) the hyperparameters:
We can set/tune most hyperparameters by reasoning about their effect on model capacity.
Important HPs:
- Learning Rate
- # Hidden Units
- Mini-batch Size
- Momentum Coefficient
Hyperparameter search:
Sample at random in a grid (hypercube) of different parameters, then zoom in to a tighter range of “good” values.
Search (sample) on a logarithmic scale to get uniform sizes between values:- Select value \(r \in [a, b]\) (e.g. \(\in [-4, 0]\), and set your hp as \(10^r\) (e.g. \(\epsilon = 10^{r}\)). You’ll be effectively sampling \(\in [10^{-4}, 10^0] \iff [0.0001, 1]\).
- Notes:
- Gradient descent can be viewed as applying Euler’s method for solving ordinary differential equations \({\displaystyle x'(t)=-\nabla f(x(t))}\) to a gradient flow.
- Neural nets are unconstrained optimization problems with many, many local minima. They sometimes benefit from line search or second-order optimization algorithms, but when the input data set is very large, researchers often favor the dumb, blind, stochastic versions of gradient descent.
- Grid search suffers from the curse of dimensionality, but is often embarrassingly parallel because typically the hyperparameter settings it evaluates are independent of each other.
Gradient Descent Variants
There are three variants of gradient descent, which differ in the amount of data used to compute the gradient. The amount of data imposes a trade-off between the accuracy of the parameter updates and the time it takes to perform the update.
- Batch Gradient Descent:
Batch Gradient Descent AKA Vanilla Gradient Descent, computes the gradient of the objective wrt. the parameters \(\theta\) for the entire dataset:$$\theta=\theta-\epsilon \cdot \nabla_{\theta} J(\theta)$$
Since we need to compute the gradient for the entire dataset for each update, this approach can be very slow and is intractable for datasets that can’t fit in memory.
Moreover, batch-GD doesn’t allow for an online learning approach.
- Stochastic Gradient Descent:
SGD performs a parameter update for each data-point:$$\theta=\theta-\epsilon \cdot \nabla_{\theta} J\left(\theta ; x^{(i)} ; y^{(i)}\right)$$
SGD exhibits a lot of fluctuation and has a lot of variance in the parameter updates. However, although, SGD can potentially move in the wrong direction due to limited information; in-practice, if we slowly decrease the learning-rate, it shows the same convergence behavior as batch gradient descent, almost certainly converging to a local or the global minimum for non-convex and convex optimization respectively.
Moreover, the fluctuations it exhibits enables it to jump to new and potentially better local minima.Notes:
- Why reduce the learning rate after every epoch?
This is due to the fact that the random sampling of batches acts as a source of noise which might make SGD keep oscillating around the minima without actually reaching it.
It is necessary to guarantee convergence. - The following conditions guarantee convergence under convexity conditions for SGD:
$$\begin{array}{l}{\sum_{k=1}^{\infty} \epsilon_{k}=\infty, \quad \text { and }} \\ {\sum_{k=1}^{\infty} \epsilon_{k}^{2}<\infty}\end{array}$$
- Stochastic Gradient Descent Escapes Saddle Points Efficiently (M.J.)
- Why reduce the learning rate after every epoch?
- Mini-batch Gradient Descent:
A hybrid approach that perform updates for a, pre-specified, mini-batch of \(n\) training examples:$$\theta=\theta-\epsilon \cdot \nabla_{\theta} J\left(\theta ; x^{(i : i+n)} ; y^{(i : i+n)}\right)$$
This allows it to:
- Reduce the variance of the parameter updates \(\rightarrow\) more stable convergence
- Makes use of matrix-vector highly optimized libraries
Gradient Descent “Optimization”
\(\:\) Evolutionary Map Of Optimizers
- Challenges in vanilla approaches to gradient descent:
All the variants described above, however, do not guarantee “good” convergence due to some challenges:- Choosing a proper learning rate is usually difficult:
A learning rate that is too small leads to painfully slow convergence, while a learning rate that is too large can hinder convergence and cause the loss function to fluctuate around the minimum or even to diverge. - Learning rate schedules1 try to adjust the learning rate during training by e.g. annealing, i.e. reducing the learning rate according to a pre-defined schedule or when the change in objective between epochs falls below a threshold. These schedules and thresholds, however, have to be defined in advance and are thus unable to adapt to a dataset’s characteristics2.
- The learning rate is fixed for all parameter updates:
If our data is sparse and our features have very different frequencies, we might not want to update all of them to the same extent, but perform a larger update for rarely occurring features. - Another key challenge of minimizing highly non-convex error functions common for neural networks is avoiding getting trapped in their numerous suboptimal local minima. Dauphin et al.3 argue that the difficulty arises in fact not from local minima but from saddle points, i.e. points where one dimension slopes up and another slopes down. These saddle points are usually surrounded by a plateau of the same error, which makes it notoriously hard for SGD to escape, as the gradient is close to zero in all dimensions.
- Choosing a proper learning rate is usually difficult:
- Preliminaries - Important Concepts:
Exponentially Weighted Averages:
EWAs (NG)$$v_{t}=\beta v_{t-1}+ (1-\beta) \theta_{t}$$
You can think of \(v_t\) as approximately averaging over \(\approx \dfrac{1}{1-\beta}\) previous values \(\theta_i\).
The larger the \(\beta\) the slower \(v_t\) adapts to changes in (new) \(\theta\) and the less noisy the value of \(v_t\).
Intuition:
It is a recursive equation. Thus,$$v_{100} = (1-\beta) \theta_{100} + (1-\beta)\beta \theta_{99} + (1-\beta)\beta^{2} \theta_{98} + (1-\beta)\beta^{3} \theta_{97} + (1-\beta)\beta^{4} \theta_{96} + \ldots + (1-\beta)\beta^{100} \theta_{1}$$
- It is an element-wise product between the values of \(\theta_i\) and an exponentially decaying function \(v(i)\).
For \(T=100, \beta=0.9\):
- The sum of the coefficients of \(\theta_i\) is equal to \(\approx 1\)
But not exactly \(1\) which is why bias correction is needed.
- It takes about \((1-\beta)^{\dfrac{1}{\beta}}\) time-steps for \(v\) to decay to about a third of its peak value. So, after \((1-\beta)^{\dfrac{1}{\beta}}\) steps, the weight decays to about a third of the weight of the current time-step \(\theta\) value.
In general:$$(1-\epsilon)^{\dfrac{1}{\epsilon}} \approx \dfrac{1}{e} \approx 0.35 \approx \dfrac{1}{3}$$
Exponentially Weighted Averages Bias Correction:
The Problem:
The estimate of the first value \(\theta_1\) will not be a good estimate of because it will be multiplied by \((1-\beta) << 1\). This will be a much lower estimate especially during the initial phase of the estimate. It will produce the purple curve instead of the green curve:
Bias Correction:
Replace \(v_t\) with:
\(\:\:\:\:\:\: \dfrac{v_{t}}{1-\beta^{t}}\)- Small \(t\): \(\implies \beta^t\) is large \(\implies \dfrac{1}{1-\beta^t}\) is large
- Large \(t\): \(\implies \beta^t\) is small \(\implies \dfrac{1}{1-\beta^t} \approx 1\)
- It is an element-wise product between the values of \(\theta_i\) and an exponentially decaying function \(v(i)\).
-
Momentum:
Motivation:
SGD has trouble navigating ravines (i.e. areas where the surface curves much more steeply in one dimension than in another4) which are common around local optima.
In these scenarios, SGD oscillates across the slopes of the ravine while only making hesitant progress along the bottom towards the local optimum.Momentum:
Momentum5 is a method that helps accelerate SGD in the relevant direction and dampens oscillations (image^). It does this by adding a fraction \(\gamma\) of the update vector of the past time step to the current update vector:$$\begin{aligned} v_{t} &=\gamma v_{t-1}+\eta \nabla_{\theta} J(\theta) \\ \theta &=\theta-v_{t} \end{aligned}$$
Note: Some implementations exchange the signs in the equations. The momentum term \(\gamma\) is usually set to \(0.9\) or a similar value, and \(v_0 = 0\).
Essentially, when using momentum, we push a ball down a hill. The ball accumulates momentum as it rolls downhill, becoming faster and faster on the way (until it reaches its terminal velocity if there is air resistance, i.e. \(\gamma < 1\)). The same thing happens to our parameter updates: The momentum term increases for dimensions whose gradients point in the same directions and reduces updates for dimensions whose gradients change directions. As a result, we gain faster convergence and reduced oscillation.
In this case we think of the equation as:$$v_{t} =\underbrace{\gamma}_{\text{friction }} \: \underbrace{v_{t-1}}_{\text{velocity}}+\eta \underbrace{\nabla_{\theta} J(\theta)}_ {\text{acceleration}}$$
Instead of using the gradient to change the position of the weight “particle,” use it to change the velocity. - Hinton
- Momentum NG
Momentum Calculation (EWAs):
$$\begin{align} v_{dw} &= \beta\:v_{dw}+(1-\beta) dw \\ v_{db} &=\beta\:v_{db}+(1-\beta) db \end{align}$$
Parameter Updates:
$$\begin{align} w &= w - \epsilon\:v_{dw} \\ b &= b - \epsilon\:v_{d_b} \end{align}$$
- Why Momentum Really Works (distill)
Notes:
- Bias Correction is NOT used in practice; only 10 iterations needed to catch up.
- The \((1-\beta)\) coefficient usually gets dropped in the literature. The effect is that that lr needs to be rescaled which is not a problem.
- Learning representations by back-propagating errors (Rumelhart, Hinton)
- visualizing Momentum (video)
- Momentum NG
-
Nesterov Accelerated Gradient:
Motivation:
Momentum is good, however, a ball that rolls down a hill, blindly following the slope, is highly unsatisfactory. We’d like to have a smarter ball, a ball that has a notion of where it is going so that it knows to slow down before the hill slopes up again.Nesterov Accelerated Gradient (NAG):
NAG6 is a way to five our momentum term this kind of prescience. Since we know that we will use the momentum term \(\gamma\:v_{t-1}\) to move the parameters \(\theta\), we can compute a rough approximation of the next position of the parameters with \(\theta - \gamma v_{t-1}\) (w/o the gradient). This allows us to, effectively, look ahead by calculating the gradient not wrt. our current parameters \(\theta\) but wrt. the approximate future position of our parameters:$$\begin{aligned} v_{t} &=\gamma v_{t-1}+\eta \nabla_{\theta} J\left(\theta-\gamma v_{t-1}\right) \\ \theta &=\theta-v_{t} \end{aligned}$$
\(\gamma = 0.9\),
While Momentum first computes the current gradient (small blue vector) and then takes a big jump in the direction of the updated accumulated gradient (big blue vector), NAG first makes a big jump in the direction of the previous accumulated gradient (brown vector), measures the gradient and then makes a correction (red vector), which results in the complete NAG update (green vector). This anticipatory update prevents us from going too fast and results in increased responsiveness, which has significantly increased the performance of RNNs on a number of tasks7.
Notes:
- This really helps the optimization of recurrent neural networks8
- Momentum allows us to adapt our updates to the slope of our error function and speed up SGD
- A Dynamical Systems Perspective on Nesterov Acceleration (M.Jordan)
- Adagrad:
Motivation:
Now that we are able to adapt our updates to the slope of our error function and speed up SGD in turn, we would also like to adapt our updates to each individual parameter to perform larger or smaller updates depending on their importance.The magnitude of the gradient can be very different for different weights and can change during learning: This makes it hard to choose single global learning rate. - Hinton
Adagrad:
Adagrad9 is an algorithm for gradient-based optimization that does just this: It adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.Adagrad per-parameter update:
Adagrad uses a different learning rate for every parameter \(\theta_i\) at every time step \(t\), so, we first show Adagrad’s per-parameter update.The SGD update for every parameter \(\theta_i\) at each time step \(t\) is:
$$\theta_{t+1, i}=\theta_{t, i}-\eta \cdot g_{t, i}$$
where \(g_{t, i}=\nabla_{\theta} J\left(\theta_{t, i}\right)\), is the partial derivative of the objective function w.r.t. to the parameter \(\theta_i\) at time step \(t\), and \(g_{t}\) is the gradient at time-step \(t\).
In its update rule, Adagrad modifies the general learning rate \(\eta\) at each time step \(t\) for every parameter \(\theta_i\) based on the past gradients that have been computed for \(\theta_i\):
$$\theta_{t+1, i}=\theta_{t, i}-\frac{\eta}{\sqrt{G_{t, i i}+\epsilon}} \cdot g_{t, i}$$
\(G_t \in \mathbb{R}^{d \times d}\) here is a diagonal matrix where each diagonal element \(i, i\) is the sum of the squares of the gradients wrt \(\theta_i\) up to time step \(t\)10, while \(\epsilon\) is a smoothing term that avoids division by zero (\(\approx 1e - 8\)).
Without the sqrt, the algorithm performs much worse
As \(G_t\) contains the sum of the squares of the past gradients w.r.t. to all parameters \(\theta\) along its diagonal, we can now vectorize our implementation by performing a matrix-vector product \(\odot\) between \(G_t\) and \(g_t\):
$$\theta_{t+1}=\theta_{t}-\frac{\eta}{\sqrt{G_{t}+\epsilon}} \odot g_{t}$$
Properties:
- Well-suited for dealing with sparse data (because it adapts the lr of each parameter wrt feature frequency)
- Pennington et al.11 used Adagrad to train GloVe word embeddings, as infrequent words require much larger updates than frequent ones.
- Eliminates need for manual tuning of lr:
One of Adagrad’s main benefits is that it eliminates the need to manually tune the learning rate. Most implementations use a default value of \(0.01\) and leave it at that. - Weakness -> Accumulation of the squared gradients in the denominator:
Since every added term is positive, the accumulated sum keeps growing during training. This in turn causes the learning rate to shrink and eventually become infinitesimally small, at which point the algorithm is no longer able to acquire additional knowledge. - Visualization - How adaptive gradient methods speedup convergence
- Well-suited for dealing with sparse data (because it adapts the lr of each parameter wrt feature frequency)
-
Adadelta:
Motivation:
Adagrad has a weakness where it suffers from aggressive, monotonically decreasing lr by accumulation of the squared gradients in the denominator. The following algorithm aims to resolve this flow.Adadelta:
Adadelta12 is an extension of Adagrad that seeks to reduce its aggressive, monotonically decreasing learning rate. Instead of accumulating all past squared gradients, Adadelta restricts the window of accumulated past gradients to some fixed size \(w\).Instead of inefficiently storing \(w\) previous squared gradients, the sum of gradients is recursively defined as a decaying average of all past squared gradients. The running average \(E\left[g^{2}\right]_ {t}\) at time step \(t\) then depends (as a fraction \(\gamma\) similarly to the Momentum term) only on the previous average and the current gradient:
$$E\left[g^{2}\right]_{t}=\gamma E\left[g^{2}\right]_{t-1}+(1-\gamma) g_{t}^{2}$$
We set \(\gamma\) to a similar value as the momentum term, around \(0.9\).
For clarity, we now rewrite our vanilla SGD update in terms of the parameter update vector \(\Delta \theta_{t}\):$$\begin{aligned} \Delta \theta_{t} &=-\eta \cdot g_{t, i} \\ \theta_{t+1} &=\theta_{t}+\Delta \theta_{t} \end{aligned}$$
In the parameter update vector of Adagrad, we replace the diagonal matrix \(G_t\) with the decaying average over past squared gradients \(E[g^2]_ t\):
$$-\frac{\eta}{\sqrt{E[g^2]_ t+\epsilon}} \odot g_{t}$$
Since the denominator is just the root mean squared (RMS) error criterion of the gradient, we can replace it with the criterion short-hand:
$$-\frac{\eta}{RMS[g]_{t}} \odot g_{t}$$
This modified parameter update vector does NOT have the same hypothetical units as the parameter.
We accomplish this by first defining another exponentially decaying average, this time not of squared gradients but of squared parameter updates:$$E[\Delta \theta^2]_t = \gamma E[\Delta \theta^2]_{t-1} + (1 - \gamma) \Delta \theta^2_t$$
The RMS Error of parameter updates is thus:
$$RMS[\Delta \theta]_ {t} = \sqrt{E[\Delta \theta^2]_ t + \epsilon}$$
Since \(RMS[\Delta \theta]_ {t}\) is unknown, we approximate it with the \(RMS\) of parameter updates up to (until) the previous time step. Replacing the learning rate \(\eta\) in the previous update rule with \(RMS[\Delta \theta]_ {t-1}\) finally yields the Adadelta update rule:
$$\begin{align} \begin{split} \Delta \theta_t &= - \dfrac{RMS[\Delta \theta]_{t-1}}{RMS[g]_{t}} g_{t} \\ \theta_{t+1} &= \theta_t + \Delta \theta_t \end{split} \end{align}$$
Properties:
- Eliminates need for lr completely:
With Adadelta, we do not even need to set a default learning rate, as it has been eliminated from the update rule.
- Eliminates need for lr completely:
-
RMSprop:
Motivation:
RMSprop and Adadelta have both been developed independently around the same time stemming from the need to resolve Adagrad’s radically diminishing learning rates.RMSprop:
RMSprop is an unpublished, adaptive learning rate method proposed by Geoff Hinton in Lecture 6e of his Coursera Class.RMSprop in fact is identical to the first update vector of Adadelta that we derived above:
$$\begin{align} \begin{split} E[g^2]_t &= 0.9 E[g^2]_{t-1} + 0.1 g^2_t \\ \theta_{t+1} &= \theta_{t} - \dfrac{\eta}{\sqrt{E[g^2]_t + \epsilon}} g_{t} \end{split} \end{align}$$
RMSprop as well divides the learning rate by an exponentially decaying average of squared gradients.
Hinton suggests \(\gamma\) to be set to \(0.9\), while a good default value for the learning rate \(\eta\) is \(0.001\).RMSprop as an extension of Rprop:
Hinton, actually, thought of RMSprop as a way of extending Rprop to work with mini-batches.
Rprop: is equivalent to using the gradient but also dividing by the magnitude of the gradient.
The problem with mini-batch rprop is that we divide by a different number for each mini-batch.
So why not force the number we divide by to be very similar for adjacent mini-batches?
That is the idea behind RMSprop.
Notes:
- It is of note that Hinton has tried to add momentum to RMSprop and found that “it does not help as much as it normally does - needs more investigation”.
- Visualizing Rprop - How adaptive gradient methods speedup convergence
-
Adam:
Motivation:
Adding momentum to Adadelta/RMSprop.Adam:
Adaptive Moment Estimation (Adam)13 is another method that computes adaptive learning rates for each parameter. In addition to storing an exponentially decaying average of past squared gradients \(v_t\) like Adadelta and RMSprop, Adam also keeps an exponentially decaying average of past gradients \(m_t\), similar to momentum.Adam VS Momentum:
Whereas momentum can be seen as a ball running down a slope, Adam behaves like a heavy ball with friction, which thus prefers flat minima in the error surface[^15].We compute the decaying averages of past and past squared gradients \(m_t\) and \(v_t\) respectively as follows:
$$\begin{align} \begin{split} m_t &= \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ v_t &= \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 \end{split} \end{align}$$
\(m_t\) and \(v_t\) are estimates of the first moment (the mean) and the second moment (the uncentered variance) of the gradients respectively, hence the name of the method.
As \(m_t\) and \(v_t\) are initialized as vectors of \(0\)’s, the authors of Adam observe that they are biased towards zero, especially during the initial time steps, and especially when the decay rates are small (i.e. \(\beta_1\) and \(\beta_2\) are close to \(1\)).They counteract these biases by computing bias-corrected first and second moment estimates:
$$\begin{align} \begin{split} \hat{m}_ t &= \dfrac{m_t}{1 - \beta^t_1} \\ \hat{v}_ t &= \dfrac{v_t}{1 - \beta^t_2} \end{split} \end{align}$$
They then use these to update the parameters just as we have seen in Adadelta and RMSprop, which yields the Adam update rule:
$$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{\hat{v}_ t} + \epsilon} \hat{m}_ t$$
The authors propose default values of \(0.9\) for \(\beta_1\), \(0.999\) for \(\beta_2\), and \(10^{ −8}\) for \(\epsilon\). They show empirically that Adam works well in practice and compares favorably to other adaptive learning-method algorithms.
-
AMSGrad:
Motivation:
Adaptive LR methods fail to converge to an optimal solution in some cases, e.g. for object recognition [17] or machine translation [18].
Reddi et al. (2018) [19] formalize this issue and pinpoint the exponential moving average of past squared gradients as a reason for the poor generalization behaviour of adaptive learning rate methods.In settings where Adam converges to a suboptimal solution, it has been observed that some minibatches provide large and informative gradients, but as these minibatches only occur rarely, exponential averaging diminishes their influence, which leads to poor convergence.
To fix this behaviour, the authors propose a new algorithm, AMSGrad that uses the maximum of past squared gradients \(v_t\) rather than the exponential average to update the parameters.Instead of using \(v_{t}\) (or its bias-corrected version \(\hat{v}_{t}\)) directly, we now employ the previous \(v_{t-1}\) if it is larger than the current one:
\(\hat{v}_{t}=\max \left(\hat{v}_{t-1}, v_{t}\right)\)This way, AMSGrad results in a non-increasing step size, which avoids the problems suffered by Adam. For simplicity, the authors also remove the debiasing step that we have seen in Adam.
Parameter Updates:
$$ \begin{aligned} m_{t} &=\beta_{1} m_{t-1}+\left(1-\beta_{1}\right) g_{t} \\ v_{t} &=\beta_{2} v_{t-1}+\left(1-\beta_{2}\right) g_{t}^{2} \\ \hat{v}_{t} &=\max \left(\hat{v}_{t-1}, v_{t}\right) \\ \theta_{t+1} &=\theta_{t}-\frac{\eta}{\sqrt{\hat{v}_{t}}+\epsilon} m_{t} \end{aligned} $$
TLDR: AMSGrad = Adam but always divide by max \(\hat{v}_ i\) for \(i \in [1, t]\).
-
Visualization of the Algorithms
SGD optimization on loss surface contours:
We see their behavior on the contours of a loss surface (the Beale function) over time. Note that Adagrad, Adadelta, and RMSprop almost immediately head off in the right direction and converge similarly fast, while Momentum and NAG are led off-track, evoking the image of a ball rolling down the hill. NAG, however, is quickly able to correct its course due to its increased responsiveness by looking ahead and heads to the minimum.
SGD optimization on saddle point:
Image shows the behavior of the algorithms at a saddle point, i.e. a point where one dimension has a positive slope, while the other dimension has a negative slope, which pose a difficulty for SGD as we mentioned before. Notice here that SGD, Momentum, and NAG find it difficulty to break symmetry, although the two latter eventually manage to escape the saddle point, while Adagrad, RMSprop, and Adadelta quickly head down the negative slope.
Analysis:
As we can see, the adaptive learning-rate methods, i.e. Adagrad, Adadelta, RMSprop, and Adam are most suitable and provide the best convergence for these scenarios. - Analysis - Choosing an Optimizer:
- Sparse Input Data:
If your input data is sparse, then you likely achieve the best results using one of the adaptive learning-rate methods. An additional benefit is that you won’t need to tune the learning rate but likely achieve the best results with the default value.
In summary, RMSprop is an extension of Adagrad that deals with its radically diminishing learning rates. It is identical to Adadelta, except that Adadelta uses the RMS of parameter updates in the numerator update rule. Adam, finally, adds bias-correction and momentum to RMSprop. Insofar, RMSprop, Adadelta, and Adam are very similar algorithms that do well in similar circumstances. Kingma et al.13 show that its bias-correction helps Adam slightly outperform RMSprop towards the end of optimization as gradients become sparser. Insofar, Adam might be the best overall choice.
Interestingly, many recent papers use vanilla SGD without momentum and a simple learning rate annealing schedule. As has been shown, SGD usually achieves to find a minimum, but it might take significantly longer than with some of the optimizers, is much more reliant on a robust initialization and annealing schedule, and may get stuck in saddle points rather than local minima. Consequently, if you care about fast convergence and train a deep or complex neural network, you should choose one of the adaptive learning rate methods.
- Sparse Input Data:
Parallelizing and distributing SGD
Additional strategies for optimizing SGD
-
Shuffling and Curriculum Learning:
Generally, we want to avoid providing the training examples in a meaningful order to our model as this may bias the optimization algorithm. Consequently, it is often a good idea to shuffle the training data after every epoch.On the other hand, for some cases where we aim to solve progressively harder problems, supplying the training examples in a meaningful order may actually lead to improved performance and better convergence. The method for establishing this meaningful order is called Curriculum Learning15.
Zaremba and Sutskever16 were only able to train LSTMs to evaluate simple programs using Curriculum Learning and show that a combined or mixed strategy is better than the naive one, which sorts examples by increasing difficulty.
Note: “Generally, we want to avoid providing the training examples in a meaningful order to our model as this may bias the optimization algorithm.”
Yes, ok, but this is just the kind of bias that we want to introduce to the network. -
Batch Normalization:
To facilitate learning, we typically normalize the initial values of our parameters by initializing them with zero mean and unit variance. As training progresses and we update parameters to different extents, we lose this normalization, which slows down training and amplifies changes as the network becomes deeper.Batch normalization17 reestablishes these normalizations for every mini-batch and changes are back-propagated through the operation as well. By making normalization part of the model architecture, we are able to use higher learning rates and pay less attention to the initialization parameters. Batch normalization additionally acts as a regularizer, reducing (and sometimes even eliminating) the need for Dropout.
-
Early Stopping:
According to Geoff Hinton: “Early stopping (is) beautiful free lunch” (NIPS 2015 Tutorial slides, slide 63). You should thus always monitor error on a validation set during training and stop (with some patience) if your validation error does not improve enough.
-
Gradient Noise:
Neelakantan et al.18 add noise that follows a Gaussian distribution \(\mathcal{N}(0, \sigma^2_t)\) to each gradient update:$$g_{t, i} = g_{t, i} + \mathcal{N}(0, \sigma^2_t)$$
They anneal the variance according to the following schedule:
$$\sigma^2_t = \dfrac{\eta}{(1 + t)^\gamma}$$
They show that adding this noise makes networks more robust to poor initialization and helps training particularly deep and complex networks. They suspect that the added noise gives the model more chances to escape and find new local minima, which are more frequent for deeper models.
Further Advances in DL Optimization
-
H. Robinds and S. Monro, “A stochastic approximation method,” Annals of Mathematical Statistics, vol. 22, pp. 400–407, 1951. ↩
-
Darken, C., Chang, J., & Moody, J. (1992). Learning rate schedules for faster stochastic gradient search. Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop, (September), 1–11. http://doi.org/10.1109/NNSP.1992.253713 ↩
-
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. arXiv, 1–14. Retrieved from http://arxiv.org/abs/1406.2572 ↩
-
Sutton, R. S. (1986). Two problems with backpropagation and other steepest-descent learning procedures for networks. Proc. 8th Annual Conf. Cognitive Science Society. ↩
-
Qian, N. (1999). On the momentum term in gradient descent learning algorithms. Neural Networks : The Official Journal of the International Neural Network Society, 12(1), 145–151. http://doi.org/10.1016/S0893-6080(98)00116-6 ↩
-
Nesterov, Y. (1983). A method for unconstrained convex minimization problem with the rate of convergence o(1/k2). Doklady ANSSSR (translated as Soviet.Math.Docl.), vol. 269, pp. 543– 547. ↩
-
Bengio, Y., Boulanger-Lewandowski, N., & Pascanu, R. (2012). Advances in Optimizing Recurrent Networks. Retrieved from http://arxiv.org/abs/1212.0901 ↩
-
Sutskever, I. (2013). Training Recurrent neural Networks. PhD Thesis. ↩
-
Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12, 2121–2159. Retrieved from http://jmlr.org/papers/v12/duchi11a.html ↩
-
Duchi et al. [3] give this matrix as an alternative to the full matrix containing the outer products of all previous gradients, as the computation of the matrix square root is infeasible even for a moderate number of parameters \(d\). ↩
-
Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 1532–1543. http://doi.org/10.3115/v1/D14-1162 ↩
-
Zeiler, M. D. (2012). ADADELTA: An Adaptive Learning Rate Method. Retrieved from http://arxiv.org/abs/1212.5701 ↩
-
Kingma, D. P., & Ba, J. L. (2015). Adam: a Method for Stochastic Optimization. International Conference on Learning Representations, 1–13. ↩ ↩2
-
LeCun, Y., Bottou, L., Orr, G. B., & Muller, K. R. (1998). Efficient BackProp. Neural Networks: Tricks of the Trade, 1524, 9–50. http://doi.org/10.1007/3-540-49430-8_2 ↩
-
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. Proceedings of the 26th Annual International Conference on Machine Learning, 41–48. http://doi.org/10.1145/1553374.1553380 ↩
-
Zaremba, W., & Sutskever, I. (2014). Learning to Execute, 1–25. Retrieved from http://arxiv.org/abs/1410.4615 ↩
-
Ioffe, S., & Szegedy, C. (2015). Batch Normalization : Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv Preprint arXiv:1502.03167v3. ↩
-
Neelakantan, A., Vilnis, L., Le, Q. V., Sutskever, I., Kaiser, L., Kurach, K., & Martens, J. (2015). Adding Gradient Noise Improves Learning for Very Deep Networks, 1–11. Retrieved from http://arxiv.org/abs/1511.06807 ↩


My name is Captain Jack Sparrow. I have been a pirate forever.  When I call myself captain, the real captain makes me walk the plank. I also help the navigator by pointing the way to go, but they make me walk the plank for "pointing the wrong direction". I also have lots of gold that I will not share.


--- date: 2015-01-29 14:00:03 author: Jen Gray title: "PhoneGap Day US 2014: Intel XDK - An IDE for Cordova" format: html tags: - PhoneGap Day - Event - Community ---
A big thanks to everyone who was able to make it to PhoneGap Day US. We'll be posting more videos from the conference so check back on our blog or Youtube channel.
Our next session comes from Paul Fischer, Technical Consultant, Intel Corporation.
The Intel® XDK HTML5 is a free mobile app development tool utilizing a collection of open-source technologies to simplify the hybrid mobile app development cycle. In this quick survey of the tool he will demonstrate:
- Utilizing the Crosswalk webview for Android Cordova apps
- Build services for Crosswalk, Android, iOS and Win 8 Tablet
- App testing and debug with on-device debug and live preview
- Linking drag-and-drop UI widgets directly to RESTful web service APIs


LEVENSHTEIN_DISTANCE(value1, value2) → levenshteinDistance
Return the calculated Levenshtein distance between the input strings value1 and value2.
LEVENSHTEIN_DISTANCE(value1, value2) → levenshteinDistance
Return the calculated Levenshtein distance between the input strings value1 and value2.
LEVENSHTEIN_DISTANCE("foobar", "bar") // 3
LEVENSHTEIN_DISTANCE(" ", "") // 1
LEVENSHTEIN_DISTANCE("The quick brown fox jumps over the lazy dog", "The quick black dog jumps over the brown fox") // 13
LEVENSHTEIN_DISTANCE("der mötör trötet", "der trötet") // 6
LENGTH() can also determine the number of elements in an array, the number of attribute keys of an object / document and the amount of documents in a collection.


Blackbaud’s next-generation user-experience framework brings a consistent, cohesive experience to Blackbaud products, and you can use it to bring that same consistent experience to your customizations and applications.
{{ stache.config.product_name_short }} provides an HTML, CSS, and JavaScript framework to implement Blackbaud’s design patterns, along with the guidance to handle visual design and user interactions.
{{ description }}
Blackbaud’s products are already taking advantage of {{ stache.config.product_name_short }}’s design patterns.
{{ description }}


Final project for Introduction to Rapid Prototyping in Fall 2016. I used Focal Camera's plans and modified the the lens appearance and the ergonomics of the shutter button. I also replaced wooden parts with plastic pieces to create a color motif. I also created a case out of lasercut plastic to keep the wooden panels together and to prevent light leaks.


Noel Recchia
Software Engineer. Web Developer. Passionate Learner.
Please feel free to check out
my projects
and
my resume
.
My Email
My Instagram
My LinkedIn
My GitHub
My Spotify


Microsoft Botbuilder telemetry package for analytics.
Allows bot session, incoming and outgoing messages, and LUIS recognizer to be captured and consumed. This payload can be sent to any endpoint desired.
Purpose
This middleware allows bot developers to collect telemetry on incoming user messages, outgoing bot responses, dialog, stack, response latency and cognitive services and more. No additional REST calls are made to gather these data points. On every user input, the package will create a new payload with the most recent information.
A host of data is by default already collected from the user (below), but the developer can add, manipulate and restructure the payload however they require. Once the payload contains all the telemetry required, it can be sent to any endpoint specified by the developer. This can include API calls, database connections, etc.
Analytics and telemetry
botName // "Test-Bot"
userName // "Kevin L."
userMessage // "hello"
userMessageLength // 5
userMessageTimestamp // 2017-10-26T20:21:54.977Z
botResponse // string[] ["Hello! I am your friendly bot."]
botResponseLength // 10
botResponseTimestamp // 2017-10-26T20:21:57.424Z
botResponseLatency // 2447
currentDialog // Object {dialog: "*:/", step: 0}
dialogStack // string[] ["*:/", "*:/greeting"]
luisIntent // "greeting"
Consume
// from Microsoft botbuilder SDK
var bot = new builder.UniversalBot(connector)
// Use botbuilder-telemetry package
bot = ApplyTelemetryMiddleware(bot, configObject, dataHandleFunction, dataMutationFuncOrPromise)
// ... Rest of your bot definition
configObject
Any variables can be passed in using the required configObject. Such as bot version, developer, any endpoint requirements, user data, etc..
dataHandleFunction
Define and implement the endpoint call you wish to make. Where would you like to send the payload?
dataMutationFuncOrPromise
Define and implement any function or promise that manipulates or adds onto the payload that is created by default. This can be left undefined if no futher processing is required, but this parameter allows the developer to customize the package before sending.
NPM Package
Check out the source code and NPM package, give it a try and let me know what you think! Github repo /examples/app.js has a full example of how to setup a bot using the Microsoft Bot Framework which uses this middleware package.
NPM package: botbuilder-telemetry NPM package
Source code in Github here: botbuilder-telemetry Github Repo


Tourism Database
Home
Customer
Bookings
Assignments
Tour guides
Locations
Ratings
Payment
Add New Tour Guide:
First Name
Last Name
Add
Return to Manage Tour Guides


Tarzan and the Lost City (PG) no stars
One review of 1994's Sylvester Stallone-Sharon Stone actioner The Specialist called that superstar team-up "the acting equivalent of the Hindenburg meeting the Titanic." If that's the case, then the pairing of hack-tors Casper Van Dien (Teen Beat-ready block of meat from Starship Troopers) and Jane March (last seen delivering what should have been a career-ending performance as three characters in the atrocious Color of Night) is the acting equivalent of armageddon meeting the end of the universe. In this horribly conceived and executed update of Edgar Rice Burroughs's classic character, the King of the Jungle (Van Dien) returns to his African homeland to prevent some treasure hunters from finding the sacred lost city of Opar; following him to the jungle is his loyal fiancee Jane (March, thankfully keeping her clothes on for a change).
Tarzan and the Lost City, directed by Carl Schenkel, is the latest in what's becoming a most unwelcome annual cinematic tradition--the horrid jungle adventure (the most recent entrants being last year's Anaconda and 1995's unspeakable Congo). There are lousy, sub-Lost in Space effects; cheap gorilla suits; flat-out atrocious performances (Van Dien's barely-there, why-the-hell-even-bother attempt at a British accent makes Kevin Costner look like Meryl Streep); and, worst of all, a lack of genuine thrills. Tarzan swings on vines and everything else, but there's nothing exciting about anything, certainly not the boring, anti-climactic "climax" in a pyramid. The audience of ten with whom I saw the film got the most amusement from Van Dien's hilariously delivered animal noises and sloppily dubbed Tarzan call. If this is truly, as the Warner Bros. publicity machine stated, "A new Tarzan for a new generation," then everyone falling into the Generation X category should be incredibly offended. I know I am.
Michael Dequina Chat Forum Host, The Official Michael Jordan Web Site http://jordan.sportsline.com
<EMAIL_ADDRESS><EMAIL_ADDRESS>|<EMAIL_ADDRESS>>My personal WWW sites< Mr. Brown's Movie Site: http://welcome.to/mrbrown Michael Jordan Beyond the Court: http://fly.to/michaeljordan A Michael Jordan Fan's Heartbreak: http://fly.to/mj23 Personal Page: http://welcome.to/w3md >Other WWW sites I work on< CompuServe Hollywood Hotline: http://www.HollywoodHotline.com Albany Online: http://www.AlbanyOnline.com Eyepiece Network: http://www.eyepiece.com
"I didn't know what to expect. It's like something you chase for so long, but then you don't know how to react when you get it. I still don't know how to react." --Michael Jordan, on winning his first NBA championship in 1991 ...or, my thoughts after meeting him on November 21, 1997
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Triol Corporation covers a wide range of industry sectors that requires a diversity of solutions
We are a part of each industry by designing superior solutions that make your life easier, more comfortable and safe, solutions that you can rely on in everyday life. Triol team is dedicated to bringing the best results that exceed Customers’ expectations with the help of our extraordinary vision and approach.
Our products find their place in
- Heavy industry
- Energy
- Public Utilities
Products
Triol Variable Frequency Drives are the perfect match for your industrial sector to increase your efficiency level, extend the lifetime of equipment by the means of Triol special algorithms, choose the right power range and minimize the costs implementing our high-technological solutions.
Triol AT27
Medium-Voltage Variable Frequency Drive
Medium-voltage Variable Frequency Drives Triol AT27 is designed for automatic control of a wide range of industrial systems with induction, synchronous (with an external exciting system) and Permanent Magnet Motors.
- Minimized maintenance
- Investment savings
- Reduced installation and operating costs
- Transient graphs
Triol AT24
Low-Voltage Variable Frequency Drive
Variable Frequency Drive (VFD) AT24 series is designed to automatically control of a wide range of production equipment, systems and complexes with induction and permanent magnet synchronous motors in various processes.
- Minimized maintenance
- Investment savings
- Commissioning is a snap
- Precision control
- Transient graphs
- No long downtimes


{% extends "dash/data.html" %} {% block content %} {% if metadata is defined %} {% include "data/metadata/report_metadata_short.html" %} {% if is_allowed('read') %} Back to report {% endif %}
Below is a list of outage periods for the report. Any forecast submissions that fall on or between the start and end of an outage will not be included in report analysis.
System outages are defined by Solar Forecast Arbiter administrators and cannot be edited. If you do not wish to exclude these periods, recreate the report and uncheck the exclude system outages option.
| Start | End | {% if system_outages |length > 0 %} {% for outage in system_outages %}
|---|---|
| {{ outage['start'] | format_datetime }} | {{ outage['end'] | format_datetime }} |
| No system outages. | {% endif %}
This report is not configured to include arbiter system outages. To include system outages, you may clone the report parameters and check the exclude system outages option.
{% endif %}Report outages may be be added and deleted freely, but this report will need to be recomputed to reflect any changes.
| Start | End | {% if report_outages |length > 0 %} {% for outage in report_outages %} |
|---|---|---|
| {{ outage['start'] | format_datetime }} | {{ outage['end'] | format_datetime }} | Delete |
| No outages for this report. | {% endif %}


xmake is a lightweight cross-platform build tool based on Lua. It uses xmake.lua to maintain project builds. Compared with makefile/CMakeLists.txt, the configuration syntax is more Concise and intuitive, it is very friendly to novices, and you can get started quickly in a short time, allowing users to focus more on actual project development.
In this version, we have added a lot of new features. Not only did we increase the compilation support for Vala and Metal languages, we also improved the package dependency management, which can support the locking and updating of dependent packages like npm/package.lock, so The user’s project will not be affected by the update and change of the upstream package repository.
In addition, we also provide some more practical rules. For example, utils.bin2c
allows users to easily and quickly embed some binary resource files into the code, and obtain relevant data in the form of header files.
New feature introduction
Added Vala language support
In this version, we can already initially support the construction of Vala programs, just apply the add_rules("vala")
rule.
At the same time, we need to add some dependency packages, among which the glib package is necessary because Vala itself will also use it.
add_values("vala.packages")
is used to tell valac which packages the project needs, it will introduce the vala api of the relevant package, but the dependency integration of the package still needs to be downloaded and integrated through add_requires("lua")
.
E.g:
add_rules("mode.release", "mode.debug")
add_requires("lua", "glib")
target("test")
set_kind("binary")
add_rules("vala")
add_files("src/*.vala")
add_packages("lua", "glib")
add_values("vala.packages", "lua")
More examples: Vala examples
Added package dependency lock support
This feature is similar to npm’s package.lock and cargo’s cargo.lock.
For example, if we quote some packages, by default, if the version is not specified, xmake will automatically pull the latest version of the package for integrated use each time, for example:
add_requires("zlib")
However, if the upstream package repository is updated and changed, for example, zlib adds a new version 1.2.11, or the installation script is changed, the user’s dependent packages will change.
This can easily lead to some projects that were originally compiled and passed, and there may be some unstable factors due to changes in dependent packages, and the compilation may fail, etc.
In order to ensure that the package used by the user’s project is fixed each time, we can enable the package dependency lock through the following configuration.
set_policy("package.requires_lock", true)
This is a global setting and must be set to the global root scope. If enabled, xmake will automatically generate a xmake-requires.lock
configuration file after executing package pull.
It contains all the packages that the project depends on, as well as the current package version and other information.
{
__meta__ = {
version = "1.0"
},
["macosx|x86_64"] = {
["cmake#31fecfc4"] = {
repo = {
branch = "master",
commit = "4498f11267de5112199152ab030ed139c985ad5a",
url = "https://github.com/xmake-io/xmake-repo.git"
},
version = "3.21.0"
},
["glfw#31fecfc4"] = {
repo = {
branch = "master",
commit = "eda7adee81bac151f87c507030cc0dd8ab299462",
url = "https://github.com/xmake-io/xmake-repo.git"
},
version = "3.3.4"
},
["opengl#31fecfc4"] = {
repo = {
branch = "master",
commit = "94d2eee1f466092e04c5cf1e4ecc8c8883c1d0eb",
url = "https://github.com/xmake-io/xmake-repo.git"
}
}
}
}
Of course, we can also execute the following command to force the upgrade package to the latest version.
$ xmake require --upgrade
upgrading packages ..
zlib: 1.2.10 -> 1.2.11
1 package is upgraded!
option supports runtime detection of code snippets
Option itself provides two interfaces add_csnippets/add_cxxsnippets
, which are used to quickly detect whether a specific piece of c/c++ code has been compiled, and if the compilation passes, the corresponding option option will be enabled.
But the previous version can only provide compile-time detection, and in the new version, we also added runtime detection support.
We can set the two parameters {tryrun = true}
and {output = true}
to try to run detection and capture output.
Try to run the test
Set tryrun to try to run to detect
option("test")
add_cxxsnippets("HAS_INT_4", "return (sizeof(int) == 4)? 0: -1;", {tryrun = true})
If the compile and run pass, the test option will be enabled.
Detect and capture output at runtime
Setting output will also try to detect and additionally capture the output content of the run.
option("test")
add_cxxsnippets("INT_SIZE",'printf("%d", sizeof(int)); return 0;', {output = true, number = true})
If the compile and run pass, the test option will be enabled, and the corresponding output content can be obtained as the value of option.
Note: Set to capture output, the current option cannot set other snippets
We can also get the output bound to the option through is_config
.
if is_config("test", "8") tben
- xxx
end
In addition, we have also improved the auxiliary detection interface of includes("check_csnippets")
to support runtime detection.
includes("check_csnippets.lua")
target("test")
set_kind("binary")
add_files("*.c")
add_configfiles("config.h.in")
check_csnippets("HAS_INT_4", "return (sizeof(int) == 4)? 0: -1;", {tryrun = true})
check_csnippets("INT_SIZE",'printf("%d", sizeof(int)); return 0;', {output = true, number = true})
configvar_check_csnippets("HAS_LONG_8", "return (sizeof(long) == 8)? 0: -1;", {tryrun = true})
configvar_check_csnippets("PTR_SIZE",'printf("%d", sizeof(void*)); return 0;', {output = true, number = true})
If capture output is enabled, ${define PTR_SIZE}
in config.h.in
will automatically generate #define PTR_SIZE 4
.
Among them, the number = true
setting can be forced as a number instead of a string value, otherwise it will be defined as #define PTR_SIZE "4"
by default
Quickly embed binary resource files into code
We can use the utils.bin2c
rule to introduce some binary files into the project, and see them as c/c++ header files for developers to use to obtain the data of these files.
For example, we can embed some png/jpg resource files into the code in the project.
target("console")
set_kind("binart")
add_rules("utils.bin2c", {extensions = {".png", ".jpg"}})
add_files("src/*.c")
add_files("res/*.png", "res/*.jpg")
Note: The setting of extensions is optional, the default suffix is .bin
Then, we can import and use it through #include "filename.png.h"
, xmake will automatically generate the corresponding header file for you, and add the corresponding search directory.
static unsigned char g_png_data[] = {
#include "image.png.h"
};
int main(int argc, char** argv)
{
printf("image.png: %s, size: %d\n", g_png_data, sizeof(g_png_data));
return 0;
}
The content of the generated header file is similar:
cat build/.gens/test/macosx/x86_64/release/rules/c++/bin2c/image.png.h
0x68, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0x78, 0x6D, 0x61, 0x6B, 0x65, 0x21, 0x0A, 0x00
Added iOS/macOS application Metal compilation support
We know that the xcode.application
rule can compile iOS/macOS applications, generate .app/.ipa packages, and complete the signature operation at the same time.
However, it did not support the compilation of code with .metal before. In the new version, we have added the xcode.metal
rule, which is associated with the xcode.application
rule by default to support metal compilation by default.
xmake will automatically compile .metal and then package to generate the default.metallib file, and it will be built into .app/.ipa automatically.
If the user’s metal is accessed through [_device newDefaultLibrary]
, it can be automatically supported, just like compiling with xcode.
Here is a complete one we provide: Project Example.
add_rules("mode.debug", "mode.release")
target("HelloTriangle")
add_rules("xcode.application")
add_includedirs("Renderer")
add_frameworks("MetalKit")
add_mflags("-fmodules")
add_files("Renderer/*.m", "Renderer/*.metal") ------- add metal files
if is_plat("macosx") then
add_files("Application/main.m")
add_files("Application/AAPLViewController.m")
add_files("Application/macOS/Info.plist")
add_files("Application/macOS/Base.lproj/*.storyboard")
add_defines("TARGET_MACOS")
add_frameworks("AppKit")
elseif is_plat("iphoneos") then
add_files("Application/*.m")
add_files("Application/iOS/Info.plist")
add_files("Application/iOS/Base.lproj/*.storyboard")
add_frameworks("UIKit")
add_defines("TARGET_IOS")
For example, on macOS, after compiling and running, the desired effect will be rendered through metal.
If our project does not use the default metal library, we can also use the above-mentioned utils.bin2c
rule as a source file and embed it into the code library, for example:
add_rules("utils.bin2c", {extensions = ".metal"})
add_files("Renderer/*.metal")
Then in the code, we can access:
static unsigned char g_metal_data[] = {
#include "xxx.metal.h"
};
id<MTLLibrary> library = [_device newLibraryWithSource:[[NSString stringWithUTF8String:g_metal_data]] options:nil error:&error];
Improve add_repositories
If we use the local repository built into the project, we used to introduce it through add_repositories("myrepo repodir")
.
However, it is not based on the relative directory of the current xmake.lua file directory like add_files()
, and there is no automatic path conversion, so it is easy to encounter the problem of not being able to find the repo.
Therefore, we have improved it, and you can specify the corresponding root directory location through an additional rootdir parameter, such as the script directory relative to the current xmake.lua.
add_repositories("myrepo repodir", {rootdir = os.scriptdir()})
os.cp supports symbolic links
In the previous version, the os.cp
interface could not handle the copying of symbolic links very well. It would automatically expand the link and copy the actual file content, which would only cause the symbolic link to be lost after copying.
If you want to keep the symbolic link as it is after copying, you only need to set the following parameter: {symlink = true}
os.cp("/xxx/symlink", "/xxx/dstlink", {symlink = true})
Compile automatically generated code more easily
Sometimes, we have such a requirement to automatically generate some source files to participate in the later code compilation before compilation. But because the files added by add_files
are already determined when the compilation is executed, they cannot be added dynamically during the compilation process (because parallel compilation is required).
Therefore, to achieve this requirement, we usually need to customize a rule, and then actively call the compiler module to deal with a series of issues such as compilation of generated code, injection of object files, and dependency updates.
This is not a big problem for the xmake developers themselves, but for users, it is still more cumbersome and difficult to get started.
In the new version, we have improved the support for add_files
and added the {always_added = true}
configuration to tell xmake that we always need to add the specified source file, even if it does not exist yet.
In this way, we can rely on the default compilation process of xmake to compile the automatically generated code, like this:
add_rules("mode.debug", "mode.release")
target("autogen_code")
set_kind("binary")
add_files("$(buildir)/autogen.cpp", {always_added = true})
before_build(function (target)
io.writefile("$(buildir)/autogen.cpp", [[
#include <iostream>
using namespace std;
int main(int argc, char** argv)
{
cout << "hello world!" << endl;
return 0;
}
]])
end)
There is no need for additional rule definitions, only the compilation order needs to be guaranteed, and the code files are generated at the correct stage.
However, we also need to pay attention that since the currently automatically generated source files may not yet exist, we cannot use pattern matching in add_files
, and can only explicitly add each source file path.
Changelog
New features
- #1534: Support to compile Vala lanuage project
- #1544: Add utils.bin2c rule to generate header from binary file
- #1547: Support to run and get output of c/c++ snippets in option
- #1567: Package “lock file” support to freeze dependencies
- #1597: Support to compile *.metal files to generate *.metalib and improve xcode.application rule
Change
- #1540: Better support for compilation of automatically generated code
- #1578: Improve add_repositories to support relative path better
- #1582: Improve installation and os.cp to reserve symlink
Bugs fixed
- #1531: Fix error info when loading targets failed


SIS 0.7 Release notes
Summary
The main new features of Apache SIS 0.7 are:
- Creation of Coordinate Reference System (CRS) objects from an EPSG code.
The list of EPSG codes supported by latest SIS release can be viewed here.
- Find coordinate operation paths from a pair of CRS, using the EPSG geodetic dataset if possible.
- Completion of Geographic Markup Language (GML) version 3.2 (ISO 19136) for
DerivedCRS
, ConcatenatedOperation
and PassThroughOperation
objects.
New features
- [SIS-131] - Database backend for CRS definitions
- [SIS-224] - Oblique and Equatorial Stereographic (EPSG:9809)
- [SIS-257] - Geographic/Geocentric conversions (EPSG:9602)
- [SIS-260] - Geographic 3D to 2D conversions (EPSG:9659)
- [SIS-270] - Vertical Offset (EPSG:9616)
- [SIS-274] - Position Vector transformation geocentric domain (EPSG:1033)
- [SIS-275] - Coordinate Frame Rotation geocentric domain (EPSG:1032)
- [SIS-278] - 3-parameter geocentric translations (EPSG:1031)
- [SIS-279] - France geocentric interpolation (EPSG:9655)
- [SIS-280] - Abridged Molodensky transformation (EPSG:9605)
- [SIS-281] - Geographic Offsets (EPSG:9660)
- [SIS-282] - Geographic Offset by Interpolation of Gridded Data (EPSG:9613, 9615, 9633)
- [SIS-211] - Support parsing and formatting of ParametricCRS
Improvements
- [SIS-292] - Complete JAXB annotations for ConcatenatedOperation
- [SIS-293] - Complete JAXB annotations for PassThroughOperation
- [SIS-294] - Complete JAXB annotations for DerivedCRS
- [SIS-173] - Support references from xlink:href to gml:id
- [SIS-124] - PT_FreeText should be able to marshall a CodeList instead than the CharacterString
- [SIS-286] - Add @XmlSchemaType annotations where appropriate
- [SIS-295] - Verify validity of unmarshalled CRS
- [SIS-326] - ISO 19111-2 : parametric CRS,CS,Datum and WKT parser implementations
- [SIS-317] - On-the-fly Geographic3D ↔ CompoundCRS conversion when parsing/formatting WKT 1
- [SIS-176] - Provide an optimized MathTransform implementation for diagonal matrices
- [SIS-322] - Non affine 1D transform patch
- [SIS-179] - Extract Database class from Shapefile class to allow reading DBF without shapefile
- [SIS-180] - Place a crude JDBC driver over Dbase files
- [SIS-305] - Shapefile reader should be able to use .SHX (shapefile index)
- [SIS-323] - Quadruple precision float point number to java double method
- [SIS-75] - Replace javadoc annotation by JDK5 annotations in Mojo
Bug fixes
- [SIS-107] - Hole in metadata "unmodifiable" mode
- [SIS-111] - Can not upgrade jetty-maven-plugin past 8.0.1
- [SIS-301] - Wrong elements order in RS_ and MD_Identifier
- [SIS-308] - InputStream provided by StorageConnector not always at the beginning of the stream
- [SIS-309] - URI in the ID element of WKT 2 wrongly taken as ID version number
- [SIS-310] - WKT parser fails to parse UNIT["grade", 0.015707963267948967]
- [SIS-311] - WKT parser ignores AREA and BBOX elements
- [SIS-312] - Axis abbreviation with nested parenthesis confuse the WKT parser
- [SIS-313] - Formatter should tell that "North along 130°W" axis direction requires version 2 of WKT
Tasks
- [SIS-191] - Declare relocation of 0.2-incubating artifacts
- [SIS-212] - Coordinate operation methods to implement
- [SIS-187] - Reduce visibility of Shapefile fields
- [SIS-188] - Hide FieldDescriptor (a DBase3 internal format structure)
- [SIS-189] - InvalidDbaseFileFormatException should extend DataStoreException


We're sorry but buecherbox doesn't work properly without JavaScript enabled. Please enable it to continue.


Canceled. You did not complete the registration process If you would like to register please go to
register


| as.chromatogram | convert dataset to chromatography dataset |
| toChromatogram | Convert chromatogram tick point data to a chromatogram object |
| add | Add a chromatogram data in the chromatogram overlap collection |
| subset | get subset of the chromatogram data by names |
| topInto | |
| labels | set new labels to the chromatogram overlap data |
| scale_time | scale the RT into different time data unit |
| overlapsMatrix | Convert the overlap list to the matrix |
| overlaps | Create chromatogram overlaps from a set of chromatogram objects |
| write.pack | save the chrome overlaps data as the cdf data file |
| read.pack |


Pick a mailing list domain to start viewing lists:
You are currently not logged in
If you are only browsing public archives, that's totally fine! But remember
to log in if you want to reply to emails
through the web interface, or search private archives.


AJAX
XML (hosted remotely)
Goal
As a user, I want to see the news headlines
Technologies
RSS
XML
jQuery AJAX
jQuery
JavaScript
HTML
Live demo


Git post-receive for multiple remote branches and work-trees
A while ago I set up my server to automatically deploy new content from my git repositories when changes where pushed to them. This automatic workflow is great and fairly simple to setup, but I recently needed to add a twist to it: what if I want to deploy separate remote branches to their own individual work-trees? Here’s how…
Below is a walkthrough of my findings along the way of debugging the initial implementation, but if you’re in a hurry, you can go straight to the solution
Initial, single remote branch setup
The initial setup has been explained here before, so you only have to follow these basic instructions to get yourself going on a single master branch auto-deployment.
More branches, more work-trees please!
Right, now let’s say you want not only the master branch of your code to be deployed to a live website, but you also want the development branch to be deployed to it’s own web root directory so you can test a development version of your site (and potentially f** it all up) while the live site still runs the last “gold” version of your code. You will undoubtedly have several local branches, several corresponding tracking remote branches. For this setup, you will also need several corresponding work-trees, all with their automated deployments upon pushing to the main repository.
Unsurprisingly, there is very little to change to our method explained in the previous setup, but the basic instructions I found did not initially work on my Ubuntu server, so you’ll have to be aware of a few things.
Initial findings
Dylan over dylanserver.com posted this handy bit of code earlier this year:
if ! [ -t 0 ]; then read -a ref fi IFS='/' read -ra REF <<< "${ref[2]}" branch="${REF[2]}" if [ "live" == "$branch" ]; then git --work-tree=/var/www/dylanstestserver.com/ --git-dir=.. checkout -f echo 'Changes pushed live.' fi
If you read carefully the post, you can notice that the checkout command is still there, but augmented with the –work-tree and –git-dir arguments.
Great! Thats a great starting point, firing up vi on my remote server, I edited my ‘post-receive’ hook and pasted the following code in it (after changing the branch name to “dev” instead of “live” to test only with my dev site). Note that I had also added the shebang at the top of the file…
#!bin/sh if ! [ -t 0 ]; then read -a ref fi IFS='/' read -ra REF <<< "${ref[2]}" branch="${REF[2]}" if [ "dev" == "$branch" ]; then git --work-tree=/path/under/root/dir/dev-site/ --git-dir=.. checkout -f echo 'Changes pushed dev.' fi
The next git push executed the hook but I got the following error in the git push output:
read: 4: Illegal option -a
The right shebang
The problem was that the shebang I added uses sh, which, according to some stackoverflow answers doesn’t use bash on ubuntu (my server). Incidentally, I found this out after fiddling around with the read command and removing the -a option, only to find out that an “unexpected redirect” has been found.
The solution was simple: change the shebang to:
#!bin/bash
Now, that hook was going to checkout my dev branch into my development site’s root directory… err… Not quite yet.
The right git directory
The next error I encountered was:
fatal: Not a git repository: '..'
That’s from the script itself, but I can’t understand why this was failing, as –git-dir should accept absolute or relative directories. However, the GIT_DIR environment variable should be set to the current directory, which during the push, is the git directory. So I elected to remove the option altogether:
#!bin/sh if ! [ -t 0 ]; then read -a ref fi IFS='/' read -ra REF <<< "${ref[2]}" branch="${REF[2]}" if [ "dev" == "$branch" ]; then git --work-tree=/path/under/root/dir/dev-site/ checkout -f echo 'Changes pushed dev.' fi
Now the push didn’t report any error, and “some” code to the correct development website root directory! Yay! BUT… That code was the MASTER branch’s code. Not the DEV branch.
The right solution
As all we are doing in the hook is performing a git checkout (with some options), we can simply pass the branch we want to checkout as a parameter and all works well! I’m yet to understand what environment variables are set and used by the hook, but for the moment, my final post-receive hook looks like this:
Edit - Thanks to Peter and his comments below, the script below now works while pushing multiple branches at once! Thanks to him!
#!/bin/bash while read oldrev newrev ref do branch=`echo $ref | cut -d/ -f3` if [ "master" == "$branch" ]; then git --work-tree=/path/under/root/dir/live-site/ checkout -f $branch echo 'Changes pushed live.' fi if [ "dev" == "$branch" ]; then git --work-tree=/path/under/root/dir/dev-site/ checkout -f $branch echo 'Changes pushed to dev.' fi done
You will notice I have added the master branch in there too, and you can expand the script with whatever branch you want to checkout, along with the correct work-tree.
comments
Matt Tue, 01 Nov 2011 - 23:46
It seems that this version of the hook doesn't fully work in scenarios where I'm pushing both master and dev (read multi-branches push) to the remote. As the script is executed once, I'm not sure if the references are correctly set. Maybe all this should go in a different hook? Suggestions?
Peter Fri, 02 Dec 2011 - 14:57
To solve Matt's problem I think this whole script should be wrapped in a while loop. I took the post-receive script from http://book.git-scm.com/5_git_hooks.html and combined it with yours to come up with a new version. I haven't tested it, but it should would work for multi-branch commits:
Matt Fri, 02 Dec 2011 - 15:03
Hi Peter, I'll give that one a go, that looks alright to me. I did not have time yet to investigate myself but that's what I had in mind as well. I'll try that asap and report. Thanks! (And I'll have to find a way to format code in comments...)
Peter Fri, 02 Dec 2011 - 18:38
The last one I posted won't work unless you change the shabang. Here's a better version that should be a little easier to read. I have tested this one successfully.
Peter Fri, 02 Dec 2011 - 18:56
And here is a version you can use if your live site is on another server. Keep in mind that you will have to setup a user who can ssh to the other server using keys instead of passwords to authenticate. I’m sure you can find those instructions online.
Matt Sun, 04 Dec 2011 - 18:33
Hi Peter, I've edited your comments so they look alright. They won't show indenting as I don't apply the pretty formatter to those. That's for another time! Regarding your code, I can confirm that the second one you posted (for a solution on the same server) does work properly and I've updated my original post as well to reflect those changes, which should help a few out there! Thanks a lot!
Marty Sun, 01 Jul 2012 - 14:01
Many thanks for your solution and explanation. I tested it on Windows machine and had to do a little edit. I changed this row:
to this:
And everything work fine. Thanks again.
Matt Tue, 03 Jul 2012 - 16:04
Happy to see these little tips still help!
Gustavo Mon, 16 Jul 2012 - 03:13
I've a little problem here... Here is step by step that i'm doing:
For now, "tutto bene" (is fine). I want to use the master branch as development branch, so, any push to master branch, deploy to dev/test server, and i want to add another branch for production enviroment, "production" branch. So I edit my config and post-receive files in server:
Now I change some code and push it to test/dev (master):
Fine, works for test/dev deploy, now test with production branch:
I push production branch to the origin because, that way, other developers can merge and push to production your works... now I merge master into production, for deploy a copy of test/dev into production enviroment: git merge master git push Fine!!!, deployed into production server, awesome!! OK, here come the bad think, now i change to master branch to fix some bug, push to dev/test, and.... nothing, don't deploy to test/dev server...  :( Any suggestion?
Matt Tue, 17 Jul 2012 - 09:12
Hey Gustavo! Other than some permission issues, I don't see what could be wrong here, as your method looks ok to me. To debug what is happening, instead of doing a checkout in the "master branch block", you could stick in there a bunch of logging lines of code that would print the git history being pushed and on which $ref you are etc.
Jake Wed, 10 Apr 2013 - 18:38
Sorry to post in an old page. I have been trying to set up a deployment similar to this, but I have been having an issue with speed. With both the script I wrote and the script here, it will sometimes take 15 minutes for a push to complete. It seems to me that the problem is when switching between the two working directories. When I push the same branch as I had previously, it only takes a few seconds to make a small change. However, if I had last pushed the master branch and now push the dev branch, it will take 15 minutes to complete, even just to make one change on a 2kB file. Additionally, it this case, windows shows all the files in the directory as having been modified, rather than just the 1 file that had been changed and committed.
Jake Wed, 10 Apr 2013 - 18:42
EDIT: After the 15 minutes, the checkout is correct.
Matt Thu, 11 Apr 2013 - 08:49
Hey Jake, I'm not sure what the problem could be here. Are you sure your machine has the correct permissions and that your filesystem is ok?
David Tue, 13 Aug 2013 - 16:45
Wonderful. Just a few questions... I'd like to incorporate this method without running or managing a git server like gitolite,... preferring to rely on GitHub and Bitbucket for things like collaboration, user management, rwx permissions, etc. So... a) Does a server running git and ssh qualify for using this, or is something like gitolite a must? b) Imagining a scenario in which you have some code that is part of a site that needs local => stage and => live branches, as well as a version that lives on Bitbucket or GH,...erhm...what might that look like? In other words, is there a sensible way of managing both sharing/collaboration AND the convenience of "push publishing", or is that asking too much of one repo?
Matt Tue, 13 Aug 2013 - 17:02
Hi David, I'll try to answer your questions below: a) Gitolite is only a permissions management system for repositories, not git itself. This is a git hook, so it should work on any git install providing that you modify the script to use the correct file system. You can also write the same script in pretty much any language the shell of said server can interpret. This is bash, but you could use ruby and others. You can find a more advanced script doign something else here: http://git-scm.com/book/en/Customizing-Git-An-Example-Git-Enforced-Policy b) There is nothing that would prevent you from building a more complex script, checking what branch you are pushing to and deploying the right code to the right staging, integration, live etc servers. Just bare in mind that this particular hook above is only intended to work on the same machine than the repository is. But because these hooks are essentially scripts, you can do whatever you want in them, even stuff completely unrelated to git itslef, like sending an email to all your team telling them you've pushed code etc. I don't have example of such scripts handy, but that's essentially how heroku or openshift work. The sharing/collaboration is the pure git branch part, the push-publish is in the hooks. I don't think the limitation will be in the repo here, but in how complex your hooks and management system will be.
David Thu, 15 Aug 2013 - 01:02
Thanks, Matt. Cleared up a lot for me. This solution is working great for me. I'd also like to recommend this write-up... http://toroid.org/ams/git-website-howto ...which covers some of the fundamentals you cover in part I. You can use the script Peter came up with in this solution as well. Very clever.
Vincent Thu, 22 Aug 2013 - 19:55
This worked perfectly for me. I've been looking for exactly this sort of solution for months. Not sure how I didn't come across this earlier. I created a Gist with my slightly modified version. http://blog.ekynoxe.com/2011/10/22/git-post-receive-for-multiple-remote-branches-and-work-trees/ Hopefully that'll make it easier to contribute to this block of code in the future. Fork and modify to your heart's content. Thanks again!
Vincent Thu, 22 Aug 2013 - 19:56
Oops--the link above is incorrect, obviously. It should be this: https://gist.github.com/bigsweater/6311240#file-post_receive
Norbert Sun, 29 Sep 2013 - 12:07
I encountered a problem with pushing my master branch to the live site. My workflow is as follows: - make changes locally on develop branch, test, commit, push to develop - test on server (develop url) - merge develop into master on local, push to master The problem is that when I push to master, files that I deleted locally don't get removed on the server. New files are added as expected. This only happens when I push to master, pushing to develop works just fine. Any ideas?
Matt Tue, 01 Oct 2013 - 08:25
Norbert, I am unsure why this happens, but possible issues could include file permissions that could be different for whatever reason on the directories your master branch checks out into, or you may have had some files in there prior to deploying with git. If these files weren't in your repository before, then it's normal they won't get deleted since git doesn't track them. That's only speculation though. Matt
Guillaume Lang Fri, 21 Mar 2014 - 15:38
Hello, i just wont to say big thanks to you for this tip, thank you very much, and sorry for my bad english, I'm french!
Jesse Mon, 15 Sep 2014 - 20:37
Great article. One problem I have, is I want all of my files to be backed up in the repo but I want to prevent certain files/directories from being checked out into the live directory. So gitignore doesn't seem to be the solution since it prevents certain directories from making it into the git backup. For instance, I have a '_db' directory where I store database backups. I want those to be registered in my git repo, but I don't want it to be accessible in my web root. Any ideas how to dot that?
Matt Mon, 15 Sep 2014 - 22:47
Hi Jesse, Thanks for taking the time to comment. You are right, .gitignore will only help in case you don't want to *commit* files to your repo. As far as I'm aware, there isn't a way to tell checkout to ignore some files, that would complicate things a lot! However, since the hook is a bash script, you're free to do whatever you want in it. One solution would be to use the --work-tree option to target a temporary folder, from which you would copy only what you really need to your web root: Instead of
You could have something like:
There are quite a few options to chose from. Rsync / cp / find with xargs / etc etc. Pick the one you prefer! And you're not bound to bash either, you can write your hooks in ruby perl etc as well. Whatever suits you, make those examples your own! /!\/!\ I haven't tried this code, so just make sure you know where you're going with it!


@* * Copyright 2021 HM Revenue & Customs * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. *@ @import config.AppConfig @import models.request.AuthenticatedRequest @import views.html.components.{heading, input_accessible_auto_complete, manager_tools_secondary_navigation} @import views.html.includes.main @import views.html.partials.{back_link, error_summary} @this() @(allKeywords: Seq[Keyword], newKeywordForm: Form[String])(implicit request: AuthenticatedRequest[_], messages: Messages, appConfig: AppConfig) @main(bodyClasses = None, customTitle = Some(messages("management.create-keyword-title"))) { @back_link() @error_summary(newKeywordForm.errors)


Number theory is the branch of mathematics that studies integers, which are all the whole numbers on either side of the number line. Number theory looks at specific properties of integers and seeks patterns in the ways different types of numbers are distributed or related to each other.
-2, -1, 0, 1,2 ..
.., -4,-2,0,2,4 ..
.., -5,-3,-1,1,3 ..
0,1,2,3 ..
1,2,3,4 ..
-1,-2,-3..
\( {\displaystyle \pm }2, {\displaystyle \pm }3, {\displaystyle \pm }5\) ...
1 and -1
\( {\displaystyle \pm }4, {\displaystyle \pm }6, {\displaystyle \pm }8\) ...
\(\sqrt{2}, \sqrt{3}, \pi , e ...\) etc
Integers can be considered either in themselves or as solutions to equations (Diophantine geometry). Questions in number theory are often best understood through the study of analytical objects (for example, the Riemann zeta function) that encode properties of the integers, primes or other number-theoretic objects in some fashion (analytic number theory). One may also study real numbers in relation to rational numbers, for example, as approximated by the latter (Diophantine approximation).
Analytic number theory may be defined as the study of the integers by means of tools from real and complex analysis.In terms of its concerns, as the study within number theory of estimates on size and density, as opposed to identities.
Some of the examples of problems in analytic number theory: the prime number theorem, the Goldbach conjecture (or the twin prime conjecture, or the Hardy–Littlewood conjectures), the Waring problem and the Riemann hypothesis. Some of the most important tools of analytic number theory are the circle method, sieve methods and L-functions (or, rather, the study of their properties).
The central problem of Diophantine geometry is to determine when a Diophantine equation has solutions, and if it does, how many. The approach taken is to think of the solutions of an equation as a geometric object.
An equation in two variables defines a curve in the plane. More generally, an equation, or system of equations, in two or more variables defines a curve, a surface or some other such object in n-dimensional space. In Diophantine geometry, one asks whether there are any rational points (points all of whose coordinates are rationals) or integral points (points all of whose coordinates are integers) on the curve or surface. If there are any such points, the next step is to ask how many there are and how they are distributed. A basic question in this direction is if there are finitely or infinitely many rational points on a given curve (or surface).
In the Pythagorean equation \({\displaystyle x^{2}+y^{2}=1,}\) we would like to study its rational solutions, that is, its solutions \((x,y)\) such that x and y are both rational. This is the same as asking for all integer solutions to \(a^{2}+b^{2}=c^{2} \); any solution to the latter equation gives us a solution \(x = a/c, y=b/c\) to the former. It is also the same as asking for all points with rational coordinates on the curve described by \(x^2 + y^2 = 1\). (This curve happens to be a circle of radius 1 around the origin.)
Much of probabilistic number theory can be seen as an important special case of the study of variables that are almost, but not quite, mutually independent. For example, the event that a random integer between one and a million be divisible by two and the event that it be divisible by three are almost independent, but not quite.
It is sometimes said that probabilistic combinatorics uses the fact that whatever happens with probability greater than {\displaystyle 0}{\displaystyle 0} must happen sometimes; one may say with equal justice that many applications of probabilistic number theory hinge on the fact that whatever is unusual must be rare. If certain algebraic objects (say, rational or integer solutions to certain equations) can be shown to be in the tail of certain sensibly defined distributions, it follows that there must be few of them; this is a very concrete non-probabilistic statement following from a probabilistic one.
If we begin from a fairly "thick" infinite set \(A\), does it contain many elements in arithmetic progression: \(a\), \({\displaystyle a+b,a+2b,a+3b,\ldots ,a+10b}\), say? Should it be possible to write large integers as sums of elements of \(A\).
These questions are characteristic of arithmetic combinatorics. This is a presently coalescing field; it subsumes additive number theory (which concerns itself with certain very specific sets \(A\) of arithmetic significance, such as the primes or the squares) and, arguably, some of the geometry of numbers, together with some rapidly developing new material. Its focus on issues of growth and distribution accounts in part for its developing links with ergodic theory, finite group theory, model theory, and other fields. The term additive combinatorics is also used; however, the sets \(A\) being studied need not be sets of integers, but rather subsets of non-commutative groups, for which the multiplication symbol, not the addition symbol, is traditionally used; they can also be subsets of rings, in which case the growth of \(A+A\) and \(A.A\) may be compared.


T.I. claims he 'exaggerated' about checking daughter's hymen
T.I. has now backtracked on his controversial admission that he takes his daughter Deyjah to the doctor to have her hymen checked, claiming he ‘exaggerated’ the story.
The 39-year-old, real name Clifford Harris Jr, sparked outrage when he said that he took his 18-year-old daughter to the gynaecologist every year to check whether her hymen was still intact.
It was his way of making sure Deyjah hadn’t had sex.
But in his new Red Table Talk interview with Jada Pinkett Smith, T.I. says his words were ‘misconstrued’.
‘I came to clear up any misconceptions there have been surrounding how we parent and what is appropriate and inappropriate,’ T.I. explained to Jada, Willow Smith and Adrienne Banfield-Jones.
‘I think all of this comes from a conversation I was having in a very joking manner when asked how I deal with parenting in this day and age.’
Advertisement Advertisement
The Live Your Life rapper continued: ‘From a place of truth, I began to embellish and exaggerate and I think a lot of people took it extremely literal because if you put any of my reputation, who I am as a father, who I’ve been, I honestly thought people knew me better than that.’
When asked he is understood the sensitivity towards the issue, T.I. replied: ‘No, I did not. I understand it now, absolutely.’
‘However, I think my intentions have been terribly misconstrued and misconceived.’
According to T.I., he was not present in the exam room and has not taken Deyjah to the doctor since she was 18 – his wife Tiny Harris who was also in the interview, stated she was 15 and 16.
‘Let me set this record straight. I never said I was in any exam room, that was an assumption. That is a falsity. I never said it was being done present day as an 18 year old,’ he stated.
‘I never said that her mother wasn’t present. She was present every time. All of this false narrative has just been sensationalised.’
Addressing Deyjah’s feelings about the check-ups, T.I. continued: ‘There was never any objection. She did have a problem with me talking about it, however, and I understand that. And I am incredibly apologetic to her for that.

Advertisement Advertisement
‘She understands my intentions, who I am and who I’ve always been.’
Part two of T.I.’s Red Table Talk interview is expected to air next Monday.
Got a showbiz story?
If you've got a celebrity story, video or pictures get in touch with the Metro.co.uk entertainment team by emailing us<EMAIL_ADDRESS>calling 020 3615 2145 or by visiting our Submit Stuff page - we'd love to hear from you.


Rallies and Events
In order to raise awareness, we focus on organizing events throughout the United States, that range from city cleanups to informational workshops.
Social Media
Our largest platform to increase awareness is social media. Go follow our Instagram!
Community Projects
In order to enact change in communities around the world, we create initiatives to plant gardens and increase eco-friendly products.
Our Mission
Our mission to create a comprehensive framework from which solutions and actions can be taken to prevent climate change from continuing to impact our world.


The name of the application, which will affect strings in titles and the icon.
The url to point the application at.
Width of the packaged application, defaults to 1280px.
Height of the packaged application, defaults to 800px.
The app will be generated for these platforms.
Processor architecture (32/64 bits).
Specifies if the source code within the nativefied app should be packaged into an archive, defaults to false.
Forces the packaged app to ignore web security errors, such as Mixed Content errors.
Forces the packaged app to ignore certificate errors.
Specifies if the menu bar should be shown.
By default, Nativefier uses a preset user agent string for your OS and masquerades as
a regular Google Chrome browser, so that sites like WhatsApp Web
will not say that the current browser is unsupported.
If this flag is passed, it will not override the user agent.
(OSX Only) Specifies to quit the app after closing all windows, defaults to false.
Sets a default zoom factor to be used when the app is opened, defaults to 1.0.